Real world nonprofit outcomes measurement - Mass Nonprofit Network Oct 29 2013 10 2013
1. Real World Introduction
to
Outcomes Management
Massachusetts Nonprofit Network Conference
Tuesday, October 29, 2013
Steve Backman, Database Designs
Ellen Bass, Capacity Institute
Prepared by Ellen Bass and Steven Backman. You may use and adapt this
under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
2. Outcomes are Changes in
People’s Lives
Initial outcomes
(direct result of program activities)
new knowledge
increased skills
changed attitudes or values
Intermediate outcomes
(significant milestone or graduation requirement)
modified behavior
Long-term outcomes
improved condition
altered status
10/29/2013
Real World Introduction to Outcomes
Measurement 2
3. Nonprofit Bottom Line
Our Their
Outcomes = Profit
10/29/2013
Real World Introduction to Outcomes
Measurement 3
4. Who’s In the Room?
Ellen, Steve…
and you!
10/29/2013
Real World Introduction to Outcomes
Measurement 4
5. Workshop Agenda
Introductory (10 min)
Define performance management (15 min)
Role of mission statement in PM (10 min)
Role of logic model in PM, criteria (15 min)
Client data tracking systems (30 min)
Your next steps (5 min)
Total 85 min
10/29/2013
Real World Introduction to Outcomes
Measurement 5
6. Workshop Goals
Understand elements of outcome
management system
Understand your organization’s status
and next steps in building outcome
management system
Help all of us contribute to centering
nonprofit work on improving participant
outcomes
10/29/2013
Real World Introduction to Outcomes
Measurement 6
7. Your Take-Aways Today
Assessment of your agency’s PM system
Assessment of your mission statement
Know how to assess a logic model
One piece of mission-critical measurable
outcomes data for your agency
Assessment of your agency’s data system
Your action steps to strengthen PM in your
agency
10/29/2013
Real World Introduction to Outcomes
Measurement 7
8. Four Assessments
Benchmark Assessment Tool (PM)
Mission Statement Assessment
Logic Model Assessment
Data System Tools
10/29/2013
Real World Introduction to Outcomes
Measurement 8
9. Nonprofit Performance
Management System
Definition: What we do to ensure
improved effectiveness, or measurable
intermediate participant outcomes
Four big questions:
What participant outcomes are we aiming for?
How well are we doing?
With whom must we partner?
How can we improve?
10/29/2013
Real World Introduction to Outcomes
Measurement 9
10. What Does an Effective
Nonprofit Agency Look Like?
Improved participant outcomes
Use outcomes data to drive decision-
making about programs, staff,
partnerships and resources
Measure and analyze outcomes
Use learning to guide next cycle of work
10/29/2013
Real World Introduction to Outcomes
Measurement 10
11. What does a useful Mission
Statement look like?
Tells why agency is in business;
definition of success
Clear
Concise
Compelling
Concrete
Should describe target population and
outcome
Example: “Beat Rome”
10/29/2013
Real World Introduction to Outcomes
Measurement 11
12. Role of Mission Statement in
Outcomes Management
Focuses agency resources on priority
activities
Guides staff to make strategic decisions
Tells if we are successful or not
How strong is your agency’s mission
statement?
Take the Assessment for your agency
5 min
10/29/2013
Real World Introduction to Outcomes
Measurement 12
13. Benchmark Assessment Tool
Purpose: help you build your
performance management system from
ground up
17 effectiveness practices
Pre- and post-test
Goal: level 4 practice
General or youth program
10/29/2013
Real World Introduction to Outcomes
Measurement 13
14. Benchmark Assessment Tool
Pair up with a colleague
Take the Assessment for your agency
10 mins.
Questions?
10/29/2013
Real World Introduction to Outcomes
Measurement 14
15. What is a Logic Model?
One page conceptual map showing flow
of influence
Shows how program helps participants
achieve outcomes
Target population, inputs, activities,
outputs, outcomes
Outcome measurement plan (separate)
to track only most important outcome
10/29/2013
Real World Introduction to Outcomes
Measurement 15
16. Role of Logic Model in
Performance Management
Identify specific target participant
Clarify logical expected outcomes
progression
Clarify program strategy necessary to help
participants reach outcomes
Provide some indication that program
provides realistic dosage and duration to lead
to outcomes
Help with program planning and/or
improvement
10/29/2013
Real World Introduction to Outcomes
Measurement 16
17. Logic Model Criteria
1. Logical flow activities through outcomes?
2. Outcomes clearly benefits to participants?
3. Program accountable for initial &
intermediate outcomes?
4. Longest term outcomes meaningful,
influenced by program?
5. Appropriate stakeholders included?
6. Inputs, activities, outputs, outcomes are
clear, comprehensive, and detailed?
10/29/2013
Real World Introduction to Outcomes
Measurement 17
18. Outcome Measurement Plan
Select most important outcome to
measure
Identify indicators to signal outcome
has been achieved
Select/create measurement tool and
data collection process
10/29/2013
Real World Introduction to Outcomes
Measurement 18
20. What does an effective Client
Tracking System look like?
Intake: Know your community
Enroll: Demographics and cohort
Manage: Logic Model activities
Measure: Measure outcomes
Analyze: Up and down the organization
10/29/2013
Real World Introduction to Outcomes
Measurement 20
23. What system do I have?
What do I need?
Basic List Management (Excel, Google)
Custom database (Access, Filemaker)
Customizable off the shelf (Salesforce,
CiviCRM)
General Human Services (ETO)
Specialized Human Services (KidTrax)
10/29/2013
Real World Introduction to Outcomes
Measurement 23
24. Look for the Secret
Spreadsheets
Positives Negatives
Easy, Quick Flattens the data
Low Tech Hinders comparisons in
time
Self Managed Hinders holistic view
Adapt to change easily Lots of hand counting
10/29/2013
Real World Introduction to Outcomes
Measurement 24
25. Client Tracking System
10/29/2013
Real World Introduction to Outcomes
Measurement 25
ParticipantParticipant
Enroll 1Enroll 1
ClassClass CaseCase
Program
A
SY13
Site 3
Stacey
Referral
10/1/13
Marie
Health Cntr
Health Careers
Oct-Nov 13
Peter
Roosevelt HS
26. Participation Differences
10/29/2013
Real World Introduction to Outcomes
Measurement 26
ParticipantParticipant
Enroll 2Enroll 2
CourseCourse CaseCase
Program
A
SY13
Site 3
Stacey
Referral
10/1/13
Marie
Health Cntr
Health Careers
Oct-Nov 13
Peter
Roosevelt HS
ClassesClasses
Enroll 1Enroll 1
CourseCourse CaseCase
StepsStepsActivitiesActivities
27. Where to record outcomes
10/29/2013
Real World Introduction to Outcomes
Measurement 27
ParticipantParticipant
Enroll 2Enroll 2
CourseCourse CaseCase
Program
A
SY13
Site 3
Stacey
Referral
10/1/13
Marie
Health Cntr
Health Careers
Oct-Nov 13
Peter
Roosevelt HS
ClassesClasses
Enroll 1Enroll 1
CourseCourse CaseCase
StepsStepsActivitiesActivities
28. Outcomes And Processes
Find measures that indicate outcomes
“Successful Referral”
Define processes behind the measures
“Steps in making a Referral”
Empower All Staff to monitor
“My referrals over a week old
Empower leadership
“Which case types get stuck most often?”
10/29/2013
Real World Introduction to Outcomes
Measurement 28
29. Steps to a New System
Assess Needs
Select Software
Discovery/Planning
Build/Configure
Data Import
Reporting and Analytics
Train/Administer/Support
10/29/2013
Real World Introduction to Outcomes
Measurement 29
30. Current System Assessment
Pair up with a colleague
Take the Assessment for your agency
10 mins.
Questions?
10/29/2013
Real World Introduction to Outcomes
Measurement 30
31. Software Selection Tool
Using the Worksheet
10/29/2013
Real World Introduction to Outcomes
Measurement 31
32. Ready to start planning?
Imagine project leader
Imagine planning team
Imagine executive sponsorship
Imagine board, development funding
Imagine fitting to annual calendar
Imagine a data manager
Imagine when to go live
10/29/2013
Real World Introduction to Outcomes
Measurement 32
33. Your Next Steps to Advocate
for Outcomes Management
Identify champions
Draft a change statement
Identify costs and benefits
Develop a strategy to win key people to
your vision
Invest in outside assistance, long term
Keep your measurement plan simple
10/29/2013
Real World Introduction to Outcomes
Measurement 33
34. Are You Ready for
Performance Management?
Get ready: Read, learn, assess, plan
Build your system:
Identify funded peer cohort of organizations
(Capacity Institute, Bridgespan)
Build agency system solo (David Hunter, FSG
Social Impact Consultants, Wellspring
Consulting)
Performance contracts (Pay for Success
bonds)
10/29/2013
Real World Introduction to Outcomes
Measurement 34
35. Useful books and reports
Leap of Reason, Mario Morino
http://www.vppartners.org/leapofreason/getit
Working Hard and Working Well, David E. K.
Hunter
http://www.vppartners.org/leapofreason/get-working-
hard-and-working-well
Software for Program Evaluation, Idealware
http://idealware.org/reports/understanding-software-
program-evaluation?key=10035930
10/29/2013
Real World Introduction to Outcomes
Measurement 35
36. About Us
Ellen Bass, ebass@bmaboston.org
Capacity Institute of the Black Ministerial
Alliance. bmaboston.org/node/4
Steve Backman, sbackman@dbdes.com
Database Designs, dbdes.com
10/29/2013
Real World Introduction to Outcomes
Measurement 36
Editor's Notes
OM vs PM??
Outcomes are the profit of the nonprofit sector; results. Outcomes are our reason to exist, to be in business
Just like a for-profit manages to maximize $ profit, a nonprofit can manage to maximize our profit: outcomes
How can you maximize something if you can’t even define it? Or measure it?
Define effective organization: One able to improve its outcomes over time
Performance management is the term used nationally
MNN uses outcomes management to distinguish between HR
PM includes HR
We will not teach/train you how to do any of this.
This is the framework – reflect on it
(If there are challenges, put them in parking lot until end)
MEASURE AND ANALYZE OUTCOMES
Have a culture that values the use of data in making decisions about the constituents
Have a process for examining the data and understanding what it is saying
Have a process for making necessary changes based on the data
We are convinced that any nonprofit that wants to can define its most important outcomes clearly and can learn to manage to improve them.
You can build the individual steps needed to get to these behaviors
Can Small to Medium Sized Agencies Create this Culture?
Normally we work with 15 agencies over 2 years to build 17 practices through a combination of group sessions individualized consultation
Self-assessment or outside consultant
4-point scale with specific indicators
High performing agencies say takes 3-10 years to incorporate these practices into your agency’s DNA
Culture change
What did you learn? Take 2 comments
Who are the OM champions? How inspired are the leaders to adopt outcomes management?
Board
Executive Director
Program staff
Resource development staff
What should characterize your agency related to outcomes management? Why? What should your agency look like?
With which of the champions do you have the best relationship? Who are the early influencers?
Takes 2 years
Measure only the data you need to ensure success
If you want this:
You will need outside help to articulate outcomes, select tools, hold yourselves accountable, envision and manage change
If you want it, you can do it!
Persuade your funders that this is important, get the help you need to build and sustain your system
Replicate or scale effective practice: New Profit, Social Innovation Fund
We hope many agencies in MA can take advantage of Pay for Success