Downloaded 23 times
Uploaded byIsaac Castillo
Program Evaluation Basics - Center for Nonprofit Success slides
This document summarizes a presentation about program evaluation. The presentation discusses why program evaluation is important, defines what program evaluation is, and provides examples of how to conduct an effective program evaluation. Specifically, it covers evaluating outcomes, using both quantitative and qualitative data, comparing results to baselines or control groups, communicating findings, and determining the appropriate level of rigor and whether to conduct the evaluation internally or externally. The overall goal is to use data to improve programs and demonstrate their impact.
More Related Content
Similar to Program Evaluation Basics - Center for Nonprofit Success slides
![What is program evaluation lecture 100207 [compatibility mode]](https://cdn.slidesharecdn.com/ss_thumbnails/whatisprogramevaluationlecture-100207compatibilitymode-110923090703-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Program Evaluation Basics - Center for Nonprofit Success slides
- 1. Program Evaluation: Usingevaluation data to set direction, expand impact, and maintain accountability October 21, 2014 Presented by: Isaac D. Castillo Director of Data and Evaluation DC Promise Neighborhood Initiative Twitter: @isaac_outcomes
- 2. Today’s Agenda Whyshould you care about program evaluation? What is program evaluation? How can your organization successfully conduct program evaluation work? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 3. Why Should YouCare About Program Evaluation? LAYC domestic violence story LeapOfReason.org First Do No Harm…Then Do More Good New domestic violence program component designed to teach three things: o Partner violence is not an OK expression of love o Partner violence is not OK in Latino culture o There are safe ways to get out of violent relationships Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 4. Why Should YouCare About Program Evaluation? No human being is perfect. Staff will make mistakes Organizations will make mistakes Services will be delivered poorly Despite the best of intentions, some people will be harmed. How do you know you are not harming people with your services? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 5. What is ProgramEvaluation? Process to determine if your program / intervention / approach is effective. Need to define what ‘success’ is for your program. Program evaluation does NOT need to be done by specialists or outsiders – but those people do add credibility and rigor (in most cases) Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 6. The Basics ofProgram Evaluation – An Example The concept of dieting – if you understand dieting, you understand the basics of program evaluation. What is the goal of dieting (how do you define dieting ‘success’)? How do you know if your diet ‘works’? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 7. Data and Dieting Person weighs 200 Person weighs 200 Pounds (90 Kilograms) (90 Kilograms) • Does that data point alone tell us anything? Pounds • Context Matters – what if person is 4 feet tall and 10 years old? • Timing Matters – is this at beginning, end, or middle of diet? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 8. Could Be AboutMore Than Weight • Other things that could be measured: • Body Mass Index (BMI) • Physical fitness • Blood measures (cholesterol levels) • Own perceptions of health / feeling • Appearance / muscle tone Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 9. Five Evaluation Conceptsto be Covered Types and timing of evaluation Who or what will you evaluate, and how will they be selected? Quantitative, Qualitative, and Mixed Methods approaches How detailed or rigorous does it need to be? Who does the work – internal or external? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 10. Types of Evaluation What do I mean by ‘type’? Really it is about timing - When do you collect data? What will you compare your data to? Much of this discussion relies on: o Costs o Availability of potential comparison data o What you are trying to learn Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 11. Traditional (Time Series) Most common type of program evaluation. Looking to see if things have changed over time. What was situation before program, then what was situation after program. BBeefoforree P Prrooggrraamm PPrrooggrraamm D Deelilviveerreedd AAftfeterr P Prrooggrraamm Must measure same things, in same ways, at both points in time. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 12. Comparison Group Atime series study that compares to another group (that does not receive programming). PPrrooggrraamm D Deelilviveerreedd BBeefoforree P Prrooggrraamm No (or minimal) programming AAftfteerr P Prrooggrraamm No (or minimal) programming More rigorous, but more challenging. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 13. Who / WhatWill You Evaluate? Need to define the population that will be evaluated. Need to define ‘success measures’ (outcomes) – what are you trying to achieve? Once these questions are answered, then need to consider which participants will be part of the evaluation (and maybe who gets programming). Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 14. In Time Series,This is Simple Usually just serve and evaluate those that enroll in the program: BBeefoforree P Prrooggrraamm PPrrooggrraamm D Deelilviveerreedd AAftfteerr P Prrooggrraamm SSeelfl-fs-seeleleccttioionn First come, first served is what is frequently used if there are too many potential participants. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 15. Comparison Groups AreMore Complicated • Can select by randomizing participants into groups: PPrrooggrraamm D Deelilviveerreedd BBeefoforree P Prrooggrraamm No (or minimal) programming AAftfteerr P Prrooggrraamm No (or minimal) programming Random Selection Random Selection Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 16. Compare across high/low dosage • Can use self-selection: PPrrooggrraamm D Deelilviveerreedd BBeefoforree P Prrooggrraamm No (or minimal) programming AAftfteerr P Prrooggrraamm No (or minimal) programming High High Attendance Attendance Low Attendance Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014 Self-selection Self-selection Low Attendance
- 17. Quantitative, Qualitative, and Mixed Methods Evaluation Quantitative = more numerical information. Qualitative = less numerical information. One not better than other, just different types of information. Both can be high quality – both can be poor. Most modern program evaluation is Mixed Methods – both quantitative and qualitative to varying degrees. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 18. What does MixedMethods Evaluation Look Like? Quantitative •Survey responses (that can be quantified) •Numerical data (test scores, report cards, medical data, etc.) •External data (collected by others) Qualitative •Interviews with participants •Focus groups with participants •Interviews with staff •Interviews or focus groups with key stakeholders •Process / fidelity study •Open ended survey responses Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 19. Where Does ThisGet Tricky? Need to focus on outcomes – changes in knowledge, attitudes, behavior or conditions. Satisfaction surveys do not equal evaluation. Just because someone liked the program it does not mean the program led to successful outcomes. Some things can be either qualitative or quantitative depending on who you ask. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 20. How Detailed orRigorous Does the Evaluation Need to Be? What do you want to do with the results? Prove to yourself the program works? Use the results to market/fundraise? Publish the results through your own materials? Publish the results in peer-reviewed journals? How ‘certain’ do you want to be about the results? Are you fine with some doubt? Will you be comfortable answering concerns and criticisms? Are you willing to live with negative results? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 21. Using Evaluation Datato Inform Change Changes in program type / delivery (this parenting program isn’t working, time for a different program) Changes in dosage (classes are offered once a month, increase classes to once a week) Changes in measurement tools or approaches (this survey question is flawed, let’s find a better one) Changes in staff training (staff do not seem to know how to deliver this program – time for training) Changes in organizational culture (no one is taking this approach seriously – time for larger conversation) Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 22. Communicating Evaluation Results Be honest about your data and the limitations of your method. What you omit is as telling as what you communicate. When communicating negative / ‘bad’ results, follow this formula: Finding / Result + Theory + New Solution Use different formats to communicate the information. Celebrate the successes, and identify areas for improvement. No need to share everything in detail – but have it ready if someone requests it. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 23. Balancing Cost andRigor Program evaluation does have a resource cost – but so does everything else. Simple internal evaluation / performance management can be done at low cost. However, larger picture requires more rigorous (and more expensive evaluation) Start small and focus on 2-3 outcomes – then expand over time. Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 24. Internal vs. External Evaluation Who does design of evaluation? Who does selection / creation of data collection tools? Who does actual data collection? Who does the analysis of data? Who creates the reports / charts / publications? Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 25. Resources - Books The Nonprofit Outcomes Toolbox By Robert M. Penna Handbook of Practical Program Evaluation By Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer - editors Performance Measurement: Getting Results By Harry P. Hatry Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 26. Resources - Articles First Do No Harm…Then Do More Good By Isaac Castillo http://tinyurl.com/isaacLOR Good Stories Aren’t Enough By Martha A. Miles http://tinyurl.com/milesgoodstories Yes We Can! Performance Management in Nonprofit Human Services By David E.K. Hunter http://tinyurl.com/hunteryeswecan Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 27. Resources - Links PerformWell www.performwell.org Performance Management and Evaluation: Two Sides of the Same Coin By Isaac Castillo and Ann Emery https://www.youtube.com/watch?v=nC7AG8XxrI4 Leap of Reason http://www.leapofreason.org Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014
- 28. Isaac’s Contact Information Isaac D. Castillo Director of Data and Evaluation DC Promise Neighborhood Initiative On Twitter: @Isaac_outcomes Email: Isaac.Castillo@dcpni.org October 21, 2014 Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014