Moving to outcome measures won’t make things perfect in terms of evaluating charities.
There are some downsides to consider, as mentioned by Nonprofit Quarterly – Want Charities to be Evaluated Based on Impact? Be Careful What You Wish For.
The article raises three concerns, all of which we need to think about very carefully. I will mention the three issues and comment on each.
Impact measures are less reliable
The article suggests that outcome measures will be more uncertain and less verifiable than the functional allocation of expenses.
If you think the debate over valuing donated medicine is messy, just wait until we open up discussion of the recovery rate of addiction treatment programs or changed literacy rates in a community created by one NPO.
The article mentions accusations that one cancer hospital allegedly screens out potential patients who are not likely to be treatable. What would be the motivation?
Marketing. Survivor rates are a key metric people look at when selecting a hospital for cancer treatment. If a hospital doesn’t admit really sick people, as the rumors allege, then the survival rate rises.
Outcomes will be complicated, messy, subject to manipulation, hard to explain, and hard to verify. The range of uncertainty will go up. The verifiability will go down.
Some people say that adopting SFAS 157 contributed to the meltdown that led to the great recession. Professionally, I find those arguments to be invalid and bordering on silly, but there are many people who have reached the conclusion that booking assets at what they are worth caused a recession.
Outcome measures will be even softer than fair value. They will be more subject to disagreement that what value to put on purchased 500 mg mebendazole.
The conversation over GIK might be tiny compared to what we could eventually have with outcome measures.
Impact measures are less easily compared
The article suggests outcome measures will be less comparable across sectors and within sectors.
I agree, but that same issue applies today.
How do you compare a literacy organization with X% learning rate with an alcohol recovery charity that sees a Z% short-term recovery rate?
For that matter, how do you compare a literacy organization working with at-risk children who are already at the point of learning to read versus a literacy organization that works with teens who just dropped out of school? Or a literacy organization that works with functionally illiterate adults who have decided they are tired of living on the fringe of the economy?
I am sure those three organizations would have substantially different outcomes. Can you pick which will have the highest and lowest success rates? I can.
By the way, the methodology used by all the rating agencies would go out the window.
As the article points out, this issue already exists with overhead ratios. If a donor gives to the cancer research organization across town only because their overhead ratio is half that of your drug rehab program, then outcome measures showing a huge portion of your program participants get on their feet and are still sober five years later won’t do you any good.
With the current accounting method of calculating overhead ratios, those charities with a mission that can easily incorporate lots of GIK look far ‘better’ than all other charities. Charities with an unpopular cause or who are new must spend more to communicate their message and look far ‘worse’ than all others. I perceive few donors understand that dynamic.
Third concern mentioned in the article:
Impact measures are less controllable
Management controls how much money is spent on fundraising and G&A and how it is spent. Management can’t control variables like what happens in the economy and how that affects program outcomes.
The article suggests that will make it nire difficult to get good outcome measures and make sure they are interpreted correctly. Accountants call this representational faithfulness.
Picture two child sponsorship programs each working in a different country. For whatever reasons, the country where NPO A is located sees a tremendous boom in the economy over the next decade. There will be a magnified impact from the efforts in A’s program compared to NPO B.
Looking at the outcome measures will show A is more effective than B. In that scenario, A get the benefit of things outside their control.
The irony would be that more contributions likely would be flowing to NPO A even though the economy then starts growing in the country where B is located and their outcomes improve dramatically while A’s slow down.
Which organization was more effective? We won’t know for sure because the economic growth skewed the outcomes.
This would be a bigger impact if one of the organizations successfully isolated the economic growth from their outcome measure and the other did not. Another researcher would have to disassemble the data to find out if the stats were properly adjusted for the same outside variables.
I believe the movement toward outcome measures is a good thing.
We will need to carefully consider the downside risks and figure out how to manage them.
The article is superb. I recommend you check it out.
Taken together, these concerns about reliability, comparability, and controllability of impact measures suggest the staying power of accounting measures amidst a barrage of criticism is no accident. … I am not suggesting measuring impact is not worth the trouble. What I am saying is that the notable weaknesses of impact measures are the very things at which accounting excels. For this reason, impact measures are better viewed as a complement to, not substitute for, accounting measures.