Slides from class taught at Tidewater Community College's Academy for Nonprofit Excellence. Focuses on the basics of data collection, outcome measurement, logic models,and performance management.
NO1 Certified Best vashikaran specialist in UK USA UAE London Dubai Canada Am...
Proving You Make a Difference
1. Can You Really Prove
That You are Making a Difference?
1
Isaac D. Castillo
Director of Data and Evaluation
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomes
Email: Isaac.Castillo@dcpni.org
August 20, 2014
2. Why Should You Care About
Outcomes?
August 20, 2014 Isaac Castillo - @isaac_outcomes 2
• LAYC domestic violence story
– LeapOfReason.org
– First Do No Harm…Then Do More Good
• New domestic violence program
component designed to teach three
things:
– Partner violence is not an OK expression of
love
– Partner violence is not OK in Latino culture
– There are safe ways to get out of violent
relationships
3. Learning Objectives
• Ability to define success for your organization
and its programs/interventions
• Identify ways to measure progress and
outcomes (using multiple approaches)
• Understand how to begin creating an
organizational culture that embraces
information and uses data to make
programmatic improvements
August 20, 2014 Isaac Castillo - @isaac_outcomes 3
4. Who Counts?
• May seem like a
simple thing, but
even defining who
‘counts’ as one of
your participants is
a very important
decision.
August 20, 2014 Isaac Castillo - @isaac_outcomes 4
5. Duplicated vs. Unduplicated Counts
• An unduplicated count is one that counts human
beings
– A person / human being would only be counted ONCE,
regardless of how many activities or sessions they have
attended.
– Example: consider this class – what is the unduplicated
count of this class?
• A duplicated count is one that counts ‘service slots’
– Not a count of human beings – rather it is a count of how
many service slots are occupied.
– Example – if we all came back tomorrow – what would the
duplicated count be? Would the unduplicated count
change?
August 20, 2014 Isaac Castillo - @isaac_outcomes 5
6. Talking and thinking about your
counts
• What do you present publically? A duplicated
count or an unduplicated count?
• Do you ‘count’ someone who only comes to a
single event at your location?
• Do you collect different things for duplicated
or unduplicated populations?
August 20, 2014 Isaac Castillo - @isaac_outcomes 6
7. Same Concept: Different Words
What do you call this sandwich?
August 20, 2014 Isaac Castillo - @isaac_outcomes 7
8. Outputs vs Outcomes
• Output measures assess what you do and who you serve.
Examples include:
• Served 100 youth during summer camp
• Provided 2,250 hours of tutoring during the academic year
• 9 out of 10 youth attended at least 75 % of available art instruction
classes offered
Outcome measures assess changes in your target population.
Examples include:
• 75 % of youth increased their knowledge of local history during the
summer camp
• 50% of youth increased math grades by one grade level during the
academic year
• 25% fewer youth reported being involved in bullying over the last year
August 20, 2014 Isaac Castillo - @isaac_outcomes 8
9. Outputs
• Outputs DO:
– Tell you about whether your program was
implemented well. For example, they indicate
whether a program:
• delivered the intended number of sessions
• reached its intended population
• resulted in adequate participation levels
• Outputs DO NOT:
– Tell you if participants benefited from your program
– Serve as indicators of program success or
effectiveness
August 20, 2014 Isaac Castillo - @isaac_outcomes 9
10. Outcomes
• Outcomes DO:
– Tell you if participants benefited from your program
– Serve as indicators of program success or effectiveness
• Outcomes DO NOT:
– Tell you about whether your program was implemented
well (or provide clues about how your program improved
participant outcomes)
August 20, 2014 Isaac Castillo - @isaac_outcomes 10
11. Do You Need To Measure Outcomes
for Every Program?
• No.
• You should measure outcomes for your
activities that provide a consistent set of
programming or services for participants.
• Measuring just outputs is OK for some
activities or programmatic approaches.
• Activities or services that are only used once
by people likely will NOT produce outcomes.
August 20, 2014 Isaac Castillo - @isaac_outcomes 11
12. Need to Collect Both Outputs and
Outcomes
• Determine if more services lead to
more/better outcomes
• Determine if there is a ‘minimum’ amount of
service needed to improve outcomes
• Determine if too many services actually lead
to decreased or negative outcomes
August 20, 2014 Isaac Castillo - @isaac_outcomes 12
13. Should You Measure Outcomes For
These Activities?
• An organization provides a health fair for community
members
– No – this is a service that a person would access once. Unlikely
this will lead to any lasting change.
– But you should still track outputs – how many attended?
• An organization provides a series of classes lasting 10
weeks that teaches how to lower cholesterol.
– Yes – should measure changes in knowledge, attitudes, and
possibly behavior.
– Outputs should also be tracked (attendance)
• You could potentially also track:
– How many health fair attendees ended up enrolling in and
completing the classes.
August 20, 2014 Isaac Castillo - @isaac_outcomes 13
14. Malleability: Can the Outcome Be
Changed?
August 20, 2014 Isaac Castillo - @isaac_outcomes 14
15. Reality Check
• Is your program likely to have a measurable
effect on an outcome?
– Example: A delinquency prevention program with
100 urban youth is unlikely to reduce a city’s
crime rate
August 20, 2014 Isaac Castillo - @isaac_outcomes 15
16. Confirmatory Outcomes
• Confirmatory outcomes: outcomes you
expect will change as a result of program
participation
– You will hold yourself responsible for confirmatory
outcomes for all (or the majority) of your regular
participants.
– You will devote adequate resources and time to achieving
confirmatory outcomes .
– You will measure your confirmatory outcomes
August 20, 2014 Isaac Castillo - @isaac_outcomes 16
17. Exploratory Outcomes
• Exploratory outcomes : outcomes potentially, but
not definitely, improved by your program
– You think (have a theory) that you may have an effect
on exploratory outcomes.
– But you will not hold yourself responsible for
exploratory outcomes.
– You may or may not measure exploratory outcomes.
– If you do not measure your exploratory outcomes, you
should have a theory (or prior research) that supports
why you think the exploratory outcomes are likely to
happen.
August 20, 2014 Isaac Castillo - @isaac_outcomes 17
18. Confirmatory vs Exploratory
Outcomes Example
• A 24-week program designed to teach teens about the
dangers of tobacco use and decrease (or prevent)
smoking of cigarettes
• Confirmatory outcome: decrease in the self-reported
number of cigarettes smoked during the past 7 days
• Exploratory outcome: decrease in the self-reported
use of alcohol, marijuana, or other drugs during the
past 7 days
The program SHOULD decrease use of cigarettes. And if
the program decreases cigarette use, then participants
MIGHT decrease use of other substances as well.
August 20, 2014 Isaac Castillo - @isaac_outcomes 18
19. Some Examples of Outcomes
August 20, 2014 Isaac Castillo - @isaac_outcomes 19
Healthy eating
habits
Program Outcome(s)
Sex education
for teens
Parent / child
communication
skills
Consumption of
more fruits and
vegetables
Healthier
weight
Decreased
yelling / anger
during conflicts
Increased
knowledge of
dangers of
unprotected /
early sex
Decreased
risky
sexual
activity
Fewer
unplanned
teenage
pregnancies
20. Timing of Outcomes
• Best to think about sequence of outcomes – and
not focus just on when outcomes may/should
occur.
• Early / initial outcomes should lead to more
outcomes, which in turn will lead to more
outcomes.
• Important question: how far down the chain of
outcomes will you measure?
August 20, 2014 Isaac Castillo - @isaac_outcomes 20
21. Short Term Outcomes
• Expect to be affected in a short period of time
• Closely related to/influenced by the program’s
outputs
• Necessary steps
– Ensure program is on track
– Realistic measure of success
August 20, 2014 Isaac Castillo - @isaac_outcomes 21
22. Intermediate and Long-Term
Outcomes
• Intermediate Outcomes
– Link between short and longer-term outcomes
– Most will be confirmatory outcomes
• Long-Term Outcomes
– Change after longer period of time
– Ultimate outcomes
– Mix of confirmatory and exploratory outcomes
August 20, 2014 Isaac Castillo - @isaac_outcomes 22
23. How Long Will Each Type of Outcome
Take?
• Will depend on the length of your
intervention.
August 20, 2014 Isaac Castillo - @isaac_outcomes 23
Short Term Intermediate Long Term
Six MonthsOne Month Twelve Months
6 Years1 Year 12 Years
24. What is a Logic Model?
• Simple representation of a program / service / organization
• Connects program services or activities to anticipated
outputs and outcomes
• Logic models are never ‘finished’ – they should evolve over
time as programs change
• Logic models should be for program/staff use – serve as
guide for direct service work and expected outcomes
• Dozens of frameworks/formats, but all contain at least 3
things:
– Activities/Services
– Outputs
– Outcomes
August 20, 2014 Isaac Castillo - @isaac_outcomes 24
25. Basic Logic Model
August 20, 2014 Isaac Castillo - @isaac_outcomes 25
Activities
Planned services,
offerings, or
instruction
provided to your
participants.
“What you plan to
do or deliver?”
Outputs
What actually
happened when
you offered your
activities.
“Who actually
received services
and in what
amount?”
Outcomes
Changes in
knowledge,
attitudes,
behavior, or
conditions among
your participants.
“What changes
actually happened
among your
participants as a
result of your
activities?”
26. Simplified Logic Model
• Logic models should help you think through the flow of your program:
• If you do Activity 1, you will see evidence of the activity being delivered by
measuring Output 1 and Output 2. Out of these Outputs and Activity, you
would expect Outcome 1 and Outcome 2 to result.
August 20, 2014 Isaac Castillo - @isaac_outcomes 26
Activity / Service Outputs Outcomes
Activity 1
• Output 1
• Output 2
Outcome 1
Outcome 2
Activity 2
• Output 3
• Output 4
Outcome 3
Outcome 4
27. Afterschool Tutoring Logic Model
August 20, 2014 Isaac Castillo - @isaac_outcomes 27
Activity Outputs
Short-term
Outcomes
Intermediate
Outcomes
After-school
tutoring on
mathematics
• # of students
tutored
• # of tutoring
sessions offered
• # of minutes of
tutoring each
student receives
% of students
maintaining or
improving school
attendance
% of students
completing and
submitting math
homework
% of students
increasing math
grade
% of students
with “A”s in math
maintain “A”
Goal: Increase academic performance of students in mathematics
28. The ‘logic’ of the Afterschool Tutoring Logic Model
• IF your program provides after-school tutoring in mathematics, THEN you would
expect three things to happen:
– Students will attend (# of students tutored)
– Each student will attend a certain amount of tutoring sessions (# of tutoring
sessions offered)
– Each student will receive a certain amount of tutoring over the program year
measured in minutes (# of minutes of tutoring each student receives)
• IF you provide tutoring, and students show up and receive a certain amount of
tutoring, THEN you would expect these two outcomes:
– Students maintain or improve school attendances (% of students maintaining
or improving school attendance)
– Students will complete (or continue to complete) and submit math homework
assignments to teachers (% of students completing and submitting math
homework)
• If students attend school, complete their math homework, submit their math
homework, then you would expect these two outcomes:
– Students would increase their math grades (% of students increasing math
grade)
– Students that already have “A”s in math will maintain those “A”s. (% of
students with As in math maintain A)
August 20, 2014 Isaac Castillo - @isaac_outcomes 28
30. Logic Model With Flow
August 20, 2014 Isaac Castillo - @isaac_outcomes 30
Inputs Activities Outputs Outcomes
Activity 3
Activity 2
Outcome 1
Output 1Activity 1
Output 3
Activity 4 Output 2
Input 2
Input1
Other logic models use arrows to show detailed progression or flow of logic.
Boxes or other shapes can be used to separate different concepts within categories.
Outcome 2
31. Logic Models Do Not Need to be
Linear
August 20, 2014 Isaac Castillo - @isaac_outcomes 31
Values
Mission
Vision
32. Nested Logic Models
August 20, 2014 Isaac Castillo - @isaac_outcomes 32
Activities Outputs Outcomes
Nutrition Education Program
Activities Outputs Outcomes
Student Exercise Program
Student Health
Division
Educational Support
Division
Parent Outreach
Division
Organization
Location 1 Location 2 Location 3 Location 4
Enrichment
Division
33. Benefits of Logic Models
• Set up a clear path to success or a roadmap for achieving
program goals
• Provide a visual test of whether inputs, and activities align
with outputs and outcomes
• Help to get everyone “on the same page”
• Facilitate performance management and evaluation
August 20, 2014 Isaac Castillo - @isaac_outcomes 33
34. Who Would You Fund?
• You have $1 million to provide funding to a
tutoring program for “at-risk” youth.
• You need to pick one of two programs to fund
– but you can only pick one!
• Assume everything else is equal aside from
the information provided on the next page.
– Same service population
– Same areas of service
– Same tutoring approach
August 20, 2014 Isaac Castillo - @isaac_outcomes 34
35. Which Program Would You Fund?
August 20, 2014 Isaac Castillo - @isaac_outcomes 35
Program # 1 Program # 2
Served 500 “at-risk” youth Served 50 “at-risk” youth
Provided 2,500 total hours
of tutoring
Provided 2,500 total hours
of tutoring
Each youth received
average of 5 hours of
tutoring
Each youth received
average of 50 hours of
tutoring
5 % of youth showed
improved math grades on
report cards
90 % of youth showed
improved math grades on
report cards
36. Does this Change Things?
August 20, 2014 Isaac Castillo - @isaac_outcomes 36
MathProficiency
Time
Program 1
Comparison Group
Program 1
MathProficiency
Program 2
Comparison Group
Program 2
Time
37. How Do You Create an Outcome
Focused Organizational Culture?
August 20, 2014 Isaac Castillo - @isaac_outcomes 37
38. Five Things to Keep in Mind
• Senior level leadership on outcomes
• Making outcomes someone’s responsibility
• Messaging and changing the conversation
• Making outcomes useful for front-line staff
• Think of it as multi-year culture change
August 20, 2014 Isaac Castillo - @isaac_outcomes 38
39. Leadership Commitment
• Organization’s leadership must buy into
importance of outcomes measurement.
• Leadership must also communicate the
importance of outcomes measurement
throughout the entire organization.
• Work with Development/Fundraising and
Finance/Accounting staff to identify sources of
funding.
August 20, 2014 Isaac Castillo - @isaac_outcomes 39
40. Making Outcomes Someone’s
Responsibility
• A staff person should be given the responsibility
for making the culture change happen.
• Ideally, this staff person would have limited other
responsibilities – most of their work should focus
on outcomes.
• Person should be respected within organization.
• This is a people initiative, not a technology
initiative.
• Technology can help, but it is not the answer.
August 20, 2014 Isaac Castillo - @isaac_outcomes 40
41. Messaging and Changing the
Conversation
• Frame question as: “how can we provide better services to
participants?”
• “Evaluation” as a positive and useful activity, not as a
negative or judgmental process.
• Messaging needs to be consistent – think of this as an
internal marketing activity.
• Staff and organizations will have legitimate concerns, and
they should be addressed honestly.
– “Not enough time to do outcomes measurement”
– “Not enough resources to do outcomes measurement”
– “Don’t have the expertise or skills to do outcomes
measurement”
August 20, 2014 Isaac Castillo - @isaac_outcomes 41
42. Making Outcomes Useful
• Work with staff to develop functional reports.
• Share reports with staff.
• Train staff on how to access and use their own
information.
• Set aside time to talk about data and reports in
meetings.
August 20, 2014 Isaac Castillo - @isaac_outcomes 42
43. Information to Better Serve Clients
• Collect information to modify or improve your
services.
• Human beings and their behaviors are very
complex – so focus on a few things and measure
those as best as you can.
• You should have a clearly articulated connection
between what you do and the outcomes you
hope to achieve.
• In most cases, you will be collecting more than
your funders require.
August 20, 2014 Isaac Castillo - @isaac_outcomes 43
44. Be Honest With Staff
• Focus on the positive – you will collect more
information to better serve clients……
• But be honest.
– Will take some additional work on their part,
especially at the beginning.
– Some people will resist the idea – “not in the job to
collect data, in the job to help people”
• Provide reports back as quickly as possible. Use
these to illustrate the importance of the process.
August 20, 2014 Isaac Castillo - @isaac_outcomes 44
45. Process Will Take Time
• Most organizations will take 3-5 years to achieve the
culture change, depending on size and leadership.
• Avoid temptation to rush or overcommit.
• Set expectations up front – let everyone know the
process will take years (not weeks or months).
August 20, 2014 Isaac Castillo - @isaac_outcomes 45
46. Be Realistic About Process
• Do not try to attempt to measure everything.
– Will only result in poor quality information.
– People will become frustrated or overwhelmed.
• Use what you collect.
– Have reports in place where people can view and discuss
the information collected.
– If you don’t plan on using it – don’t bother collecting it.
• Start small, then expand.
– Better to collect 2-3 things very accurately, and use this
information, than try to collect dozens of things poorly.
– Can always add on additional things to collect in future
years.
August 20, 2014 Isaac Castillo - @isaac_outcomes 46
47. What is Performance Management?
• Performance management is the collection of
information by programs, organizations, or
people for the primary purpose of program /
service improvement.
• Successful performance management relies on
three important features:
1) The regular and accurate collection of information, which is then…...
2) Shared in useable formats, which is then…..
3) Used to make changes in the hope of improving something (services,
programs, conditions, decision making, etc.)
August 20, 2014 Isaac Castillo - @isaac_outcomes 47
48. Performance
Management
Cycle
August 20, 2014 Isaac Castillo - @isaac_outcomes 48
Define Success
Data Collection
Methods
Train Staff
Begin Program
Ongoing
Checks
Interim
Reports
Complete Data
Collection
Examine Data
Prepare
Reports
Share Outcomes
Use Outcomes To
Change Program
49. Performance Management and
Evaluation
August 20, 2014 Isaac Castillo - @isaac_outcomes 49
Performance
Management
Regular measurement,
Improve outcomes
Evaluation
Occasional measurement,
Improve effectiveness
50. PM and Eval: Speed
Performance Management
is rapid and frequent
August 20, 2014 Isaac Castillo - @isaac_outcomes 50
Evaluation is methodical
and delayed
51. PM and Eval: Data Collection and
Analysis
August 20, 2014 Isaac Castillo - @isaac_outcomes 51
Performance Management
is simple and requires little
expertise
Evaluation can be complex
and often requires
specialized training
52. PM and Eval: Completeness
August 20, 2014 Isaac Castillo - @isaac_outcomes 52
Performance Management
gives you enough to guess at
whole puzzle
Evaluation lays out as much
of the puzzle as possible
53. Why Bother With All of This?
August 20, 2014 Isaac Castillo - @isaac_outcomes 53
Ultimately, you should be measuring
outcomes or effectiveness for a
single reason:
To better serve your clients / population.
54. Isaac’s Contact Information
54October 2014 @Isaac_outcomes
Isaac D. Castillo
Director of Data and Evaluation
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomes
Email: Isaac.Castillo@dcpni.org