Pittsburgh Nonprofit Summit - Measuring Change - Moving From Outputs to Outcomes to Impact

Measuring Change
Moving from Outputs to Outcomes to Impact
10:00am – 11:15am
Facilitator: Lisa Kuzma, Richard King Mellon Foundation
Speakers: Mike Bangser, MDRC / Junlei Li, Office of Child Development / Art Maxwell and Pam Meadowcroft, Meadowcroft & Associates / Tim Weidemann, Rondout Consulting

Share your thoughts!
Text “Summit” to 57682 with your
feedback and thoughts on today’s
event!
Share your thoughts on Twitter
by adding #GPNPSummit to your
tweets!

Measuring Change – Moving From Outputs to Outcomes to Impacts

The Context
[We are] trying to put much more emphasis on evidence-based policy decisions here at OMB. Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges — expanding the approaches that work best, fine tuning the ones that get mixed results, and shutting down those that are failing.
-- Peter Orszag, former Director of the Office of Management and Budget
4

Key Assumptions
Agreement on what it means to be “effective”
Reliable evidence on which programs are effective, using the proper measures
Reliable evidence on the core elements that contribute to effectiveness
Documentation and communication of these elements
Understanding of how the elements can or should be adapted to local conditions
Providers have sufficient resources and capacity to replicate evidence-based practices
5

Key Distinctions
Outputs (typically, units of service)
Gross or absolute outcomes(e.g., employment, learning, or behavioral gains for those who are served)
Net impacts(how the outcomes differ from what would have occurred without the intervention)
6

5/11/2011
70
80
40
20
Job Program Group
Control Group
Outcomes vs. Impacts: Which program is more effective?
100
100
90
90
80
80
70
70
60
60
Percent Placed In A Job
Percent Placed In A Job
50
50
40
40
30
30
20
20
10
10
0
0
A
B
7

Positive Outcomes, But for Whom?

Key Challenges in Replication
Rigorous evidence of what works
The right level of model specification
Proper “fit” between the replicated program model and the local population being served
Other factors that may affect results (staffing, funding levels, etc.)
Finding useful “leading indicators” for program staff
Data issues

Implications for Planning and Practice
Need a realistic meeting of the minds on the roles of policymakers, funders, researchers , and providers
The goal is fidelity to the right things, at scale and at reasonable cost
Need appropriate support for replication and organizational infrastructure (dissemination of research findings , tools and training, data systems, etc.)
Beware of potential distortions in interpreting outcomes – especially the influence of participant selection
Recognize that moving from outputs to outcomes to impacts is an evolutionary process

For More Information
Michael Bangser
mike.bbangser@mdrc.org
860-586-8577
www.mdrc.org

Thank you from the Greater Pittsburgh Nonprofit Partnership!

Workshop Evaluation
Text “Impact” to 57682 with your answers to the following questions:
Please rate the overall value of this workshop
E – Excellent G – Good P – Poor F – Fair
Did you learn anything that you will apply at your own organization?
Y – Yes N - No
Please text other comments and feedback.
Submit by hitting “send!” You will receive an auto-reply from the GPNP.
Sample text: “Impact E Y I’ve learned great tips today.”

Pittsburgh Nonprofit Summit - Measuring Change - Moving From Outputs to Outcomes to Impact

by GPNP on May 17, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Pittsburgh Nonprofit Summit - Measuring Change - Moving From Outputs to Outcomes to Impact — Presentation Transcript