A case study and discussion of Charity Navigator’s current efforts to move beyond rating financial accountability to measuring outcomes

1. Charity Navigator
CN 2.0 Case Study
Presentation at
Managing to Outcomes Forum
Paul Brest, President, The William and Flora Hewlett Foundation
With Ken Berger, President & CEO, Charity Navigator,
David Bonbright, CEO, Keystone Accountability,
Xandy Brown, Pilot Project Coordinator, Charity Navigator,
And Professor David Campbell, SUNY Binghamton
June 13, 2011

2. The Data Proves Impact
 Estimated 3.3 million
distinct visitors per
year (~5 million hits)
 92% say evaluations
affected their decision
to support individual
public charities
 CN ratings influence
decisions on billions in
donations annually

4. CN 2.0
Financial Accountability/ Results More
Health Transparency Comprehensive
Rating System

5. Note: Strategic Plan is in draft and not yet finalized.

6. 10,000 CHARITIES RATED BY VOLUNTEER RATERS
• Why 10,000? Because ~70% of all revenues that come
into the NP sector annually go to them.
• How to scale this effort? A new process to move beyond
our staff and involve consumers in the rating of charities
in cause areas they’re passionate about.
• now… 2.0…

7. Reporting Results R&D Process

10. What is the charity’s commitment to
reporting results?
• Clear commitment to reporting results stated.
• Specified time period for which results are
presented.
• Some mission-related results in current period
compared to earlier period.
• Reporting distinguishes between activities,
outputs and outcomes.
• Credible intention to validate results evidence.

11. How does the charity demonstrate
the demand for its services?
• Reports indicating the aggregate numbers
of individuals accessing the charity’s
outputs in a given period.
• Evidence of demand for more than half of
the charity’s outputs is provided.
• A statement of evidenced demand set
against the larger demand that the charity
does not presently have the capacity to
reach.

12. Does the charity report its activities
and outputs?
• A clear description of the majority of the
charity’s mission-related activities and
outputs for a defined reporting period
• A significant portion of mission-related
activities and outputs reported are
compared to an earlier reporting period
• Activities and outputs reported can readily
be related to reported expenditure

13. Does the charity report its outcomes
(medium and longer-term results)?
• A clear description of mission-related
outcomes achieved by the charity in the
reporting period.
• Outcomes are described for more than
half of the charity’s mission-related
activities or those mission-related
outcomes described relate to activities that
consume more than half of total charity’s
expenditure.

14. What is the quality of evidence for
reported results?
• Evidence of most mission-related outputs
includes some element of independent
validation
• Low level of outcome evidence
• Medium level of outcome evidence
• High level of outcome evidence:
beneficiary feedback
• High level of outcome evidence:
independent validation

15. Does the charity adjust and
improve in light of its results?
• Evidence that the organization assesses its mission-
related performance in light of what was planned for
the reporting period
• Admits mistakes and publicizes corrective actions
• Admits mistakes, publicizes corrective actions and
commits itself to validating proposed corrective
measures through dialogue with those affected
• Admits mistakes, publicizes corrective actions, and
provides evidence that it has in fact validated
proposed corrective measures through dialogue with
those affected

16. Framing Questions
• Given the low level of current reporting of
results, what are the best initial rating criteria?
• What is the best way to communicate and
implement rating criteria that will ratchet up over
time?
• What initial criteria will be (a) possible to rate
reliably by volunteers and (b) sufficiently
achievable to have a basic level of compliance
within 12 months, but (c) not be so easy as to be
credible and widely gamed.
• Who else needs to be on the bus?

17. What do student raters learn?
• More questions raised than answered:
– How to differentiate activities, outputs, and
outcomes when they are not explicitly
labeled?
– How does a charity report results at this level
and still have a website that is accessible and
clear to lay people?
– How to decrease subjectivity?

18. How was student feedback
• Materials
incorporated?
– Online platform
– Centralized communication
• Rating
– Illustrative examples of where to find
information
– 4 level confidence scale
• Process
– Thanking volunteers
– Providing background

19. Binghamton University Courses
• Two courses
– Public Administration
• 16 Students
• Issues in Nonprofit Administration
– Social Work
• Advanced Social Work Practice with Communities
• 19 Students
• Unique Features
– Local Organization Assessment
– $3,000 in “Philanthropy Incubator” grants
– Student Blog (“Navigating Southern Tier
Charity”)

20. Course Structure
• Public Administration
– Course Focus: Effectiveness and Accountability
– CN Recommended Readings, Forces for Good
• Social Work
– Forces for Good
• Inter-disciplinary Dimensions
– Charity Navigator Assessment Teams
– Local Organization Assessment Teams
– Class Discussion

21. CN: Students’ Practical Issues
• Assessing Confidence Levels
• Positive Response to 5 Minute Rule
• Most Diligent: Assessments Time
Consuming
• Importance of Preparation, Knowledge
(SW vs. PA)
• Technology Challenges (CN
Responsiveness)

22. CN: Students’ Philosophical Issues
• Standards Incomplete
– Mistrust
– “Stars” system simplistic.
– Good organizations negatively affected.
• Resource Limitations Constrain Performance
• Assessing Transparency, not Performance
– Performance data depend on trust (not verifiable).
• Discomfort with “Admitting Mistakes”
• Donor vs. Beneficiary Focus

23. Standards and Students’ Funding
Decisions
• Real-world Challenge of Assessment
• Charity Navigator Standards OK After All…
– Financial Measures
– Transparency
• Role of Executive Compensation
• Frustrations with Limitations of Secondary Data
– Assessment Incomplete
• Creation of New Standards
– Role of Social Media
– Student Concerns (small vs. large, effect of
contribution)

24. End of Semester CN Rating
Attitudes
• The perfect not the enemy of the
good.
• Value of standards
• Not all performance standards apply.
• Create standards for stakeholder
dialogue
• SW students: Limited utility for
beneficiaries

25. End Semester Performance
Assessment Attitudes
• Effectiveness a social construct
• Negotiate standards with
stakeholders.
• More questions than answers; “it
depends”
• Importance of responsiveness to
performance perceptions.