Successfully reported this slideshow.

Going Beyond Evaluation Buzzwords


Published on

Lauren Decker, ACE Conference 2013

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Going Beyond Evaluation Buzzwords

  1. 1. Going Beyond Evaluation Buzzwords Lauren Decker, Ph.D. Edvance Research Presented at Texas ACE Annual Conference, Austin, TX. August 7, 2013
  2. 2. One word or phrase expressing how you feel about evaluation
  3. 3. “If you don’t know where you are going, how are you gonna’ know when you get there?”   Yogi Berra
  4. 4. Where are we going and how will we get there?   What’s a logic model and how does it help?   Defining seemingly inter-changeable terms   Time and tool to begin a logic model (or refine one)   What is actually happening and how does it help to know?   Program and evaluation importance of understanding implementation fidelity   Bringing thoughtful program lessons into evaluation plan and use   What makes a great evaluation?   Planning and asking the right question(s)
  5. 5. Logic Models and Theories of Change
  6. 6. Logic Model vs. Theory of Change   Logic Model = graphically depicts what your program is intended to do   Key program components   Theory of Change = why your program operates as intended and how components and activities are hypothesized to move outcomes   Assumptions underlying expected change
  7. 7. Simple Logic Model Inputs Outputs Outcomes Resources Activities Participation Short Medium Long What we do Who we reach What results we expect What we invest
  8. 8. Logic Model Example
  9. 9. How does having a logic model help my program? Program phase   New program   Existing program   Redesigning existing program Logic model use   Creation & planning   Documentation & communication   Reinvention, improvement, & expansion
  10. 10.   Use the paper provided to begin a logic model for your program   Remember to include resources, intended activities, participants, short-, mid-, and long- term outcomes   If you can also represent the theory of change (arrows) What does our program logic model look like? 10 minutes
  11. 11. What should we include in our program Logic Model?   Identify:   All key components of program   Resources, intended activities, participation, short-, mid-, and long-term outcomes   Components and pathways(mediators) through which program is expected to produce intended outcomes (theory)   Student outcome(s) program is designed to improve   Short-, mid-, long-term outcomes   Other outcomes as well if relevant
  12. 12. I have a Logic Model, now what? – What is fidelity of implementation?
  13. 13. Fidelity of Implementation   The extent to which key components of the program are delivered as originally intended by the developer   Key components = strategies, practices, activities and behaviors that are critical to defining your program
  14. 14. Logic Model Example
  15. 15. Fidelity of Implementation   the extent to which key components of the program are delivered as originally intended by the developer   Key components = strategies, practices, activities and behaviors that are critical to defining your program   When is your program not your program?
  16. 16. How do I measure fidelity of my program? Fidelity Structural key components Interactional key components Structure Process
  17. 17. How does measuring fidelity help my program?   Understanding fidelity helps you know about:   Participation   Activity delivery   Content   Example questions you can answer:   Were activities implemented by program staff according to design?   Did staff receive program content as planned?   What were the adaptations that were made to the program?   What were the factors that may have affected program fidelity?
  18. 18. “I think you should be more explicit here in Step Two.” Evaluations without measuring fidelity…
  19. 19.   Visit someone else’s logic model in process   For the activities listed suggest:   What type of fidelity could be measured   Ways of measuring that type of fidelity for the activity How to measure fidelity in your program? 10 minutes
  20. 20. What do we need to do to understand fidelity of our program?   Periodically measure implementation fidelity   Measured separately for each key component   Thresholds specified for determining whether key components of program were implemented with fidelity   Assess and Report whether each key component was implemented with fidelity
  21. 21. What makes a great evaluation?
  22. 22. High-quality evaluations include:   Evaluators who clearly use sound evaluation principles and practices (transparency, consideration of context, etc.)   Complex programs require thoughtful, directed evaluations   Focus from the start on use of evaluation processes and products to provide information to, and about, the program being evaluated   Alignment to the program’s logic model and theory of change   Include provision for some level of implementation fidelity
  23. 23. High-quality evaluations include:   Preparation and planning for variation within multi- site evaluations   Consideration of clear deadlines for completion and deliverables set by program   Upfront acknowledgement of the limitations of the proposed evaluation   Minimizing jargon to increase clarity of communication of results   Striking a balance between consultancy/ recommendation support and maintaining unbiased distance as an independent evaluator
  24. 24. Before meeting with an evaluator:   Complete your program logic model and theory of change   Or have discussions and ideas of key components for development assistance from evaluator   Consider what you really want to learn   What outcomes do you want an evaluation to focus on for your program?
  25. 25. What do you really want to know? Make sure you ask the right question!
  26. 26. Before meeting with an evaluator:   Complete your program logic model and theory of change   Consider what you really want to learn   What data is already available or will need to be collected to answer your right question(s)?   If you believe data will need to be collected do you want to collect them or have the evaluator conduct the data collection?   Think about whether you have identified SMART outcomes!
  27. 27. SMART Outcomes are:   Specific   Reflect simple information that is communicable and easily understood   Measurable   Can changes by measured in reliable and valid ways?   Achievable   Able to be collected and sensitive to change during the allotted time   Relevant   Reflect information that is important and likely to be used   Time bound   Progress can be tracked at desired frequency within allotted time Source: Adapted from World Bank Group.
  28. 28. Questions for an evaluator   How does your proposed evaluation align with our program logic model and theory of change?   What data do you need from us to conduct the evaluation? What data to you propose to collect, when, and in what format?   How intrusive will the data collection be for our program?   Will your evaluation be able to tell us if the program was in place as intended?   What we really want to know from an evaluation is _________. How will your proposed evaluation answer this question?
  29. 29. Questions to ask yourself about an evaluation proposed to you   Does the evaluation design reflect my logic model and theory of change?   Do the proposed outcomes follow my logic model and theory of change?   Is it feasible to collect outcome data at the time I expect to see changes (short- medium- and long- term outcomes) in those selected?   Are the selected outcomes and measures relevant for my program and my stakeholders?   Does the evaluation include SMART outcomes that will be evaluated?
  30. 30. Logic Model Example
  31. 31. Questions? Thank you. Lauren Decker, Ph.D.