Your SlideShare is downloading. ×
Going Beyond Evaluation Buzzwords
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Going Beyond Evaluation Buzzwords


Published on

Lauren Decker, ACE Conference 2013 …

Lauren Decker, ACE Conference 2013

Published in: Technology

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Going Beyond Evaluation Buzzwords Lauren Decker, Ph.D. Edvance Research Presented at Texas ACE Annual Conference, Austin, TX. August 7, 2013
  • 2. One word or phrase expressing how you feel about evaluation
  • 3. “If you don’t know where you are going, how are you gonna’ know when you get there?”   Yogi Berra
  • 4. Where are we going and how will we get there?   What’s a logic model and how does it help?   Defining seemingly inter-changeable terms   Time and tool to begin a logic model (or refine one)   What is actually happening and how does it help to know?   Program and evaluation importance of understanding implementation fidelity   Bringing thoughtful program lessons into evaluation plan and use   What makes a great evaluation?   Planning and asking the right question(s)
  • 5. Logic Models and Theories of Change
  • 6. Logic Model vs. Theory of Change   Logic Model = graphically depicts what your program is intended to do   Key program components   Theory of Change = why your program operates as intended and how components and activities are hypothesized to move outcomes   Assumptions underlying expected change
  • 7. Simple Logic Model Inputs Outputs Outcomes Resources Activities Participation Short Medium Long What we do Who we reach What results we expect What we invest
  • 8. Logic Model Example
  • 9. How does having a logic model help my program? Program phase   New program   Existing program   Redesigning existing program Logic model use   Creation & planning   Documentation & communication   Reinvention, improvement, & expansion
  • 10.   Use the paper provided to begin a logic model for your program   Remember to include resources, intended activities, participants, short-, mid-, and long- term outcomes   If you can also represent the theory of change (arrows) What does our program logic model look like? 10 minutes
  • 11. What should we include in our program Logic Model?   Identify:   All key components of program   Resources, intended activities, participation, short-, mid-, and long-term outcomes   Components and pathways(mediators) through which program is expected to produce intended outcomes (theory)   Student outcome(s) program is designed to improve   Short-, mid-, long-term outcomes   Other outcomes as well if relevant
  • 12. I have a Logic Model, now what? – What is fidelity of implementation?
  • 13. Fidelity of Implementation   The extent to which key components of the program are delivered as originally intended by the developer   Key components = strategies, practices, activities and behaviors that are critical to defining your program
  • 14. Logic Model Example
  • 15. Fidelity of Implementation   the extent to which key components of the program are delivered as originally intended by the developer   Key components = strategies, practices, activities and behaviors that are critical to defining your program   When is your program not your program?
  • 16. How do I measure fidelity of my program? Fidelity Structural key components Interactional key components Structure Process
  • 17. How does measuring fidelity help my program?   Understanding fidelity helps you know about:   Participation   Activity delivery   Content   Example questions you can answer:   Were activities implemented by program staff according to design?   Did staff receive program content as planned?   What were the adaptations that were made to the program?   What were the factors that may have affected program fidelity?
  • 18. “I think you should be more explicit here in Step Two.” Evaluations without measuring fidelity…
  • 19.   Visit someone else’s logic model in process   For the activities listed suggest:   What type of fidelity could be measured   Ways of measuring that type of fidelity for the activity How to measure fidelity in your program? 10 minutes
  • 20. What do we need to do to understand fidelity of our program?   Periodically measure implementation fidelity   Measured separately for each key component   Thresholds specified for determining whether key components of program were implemented with fidelity   Assess and Report whether each key component was implemented with fidelity
  • 21. What makes a great evaluation?
  • 22. High-quality evaluations include:   Evaluators who clearly use sound evaluation principles and practices (transparency, consideration of context, etc.)   Complex programs require thoughtful, directed evaluations   Focus from the start on use of evaluation processes and products to provide information to, and about, the program being evaluated   Alignment to the program’s logic model and theory of change   Include provision for some level of implementation fidelity
  • 23. High-quality evaluations include:   Preparation and planning for variation within multi- site evaluations   Consideration of clear deadlines for completion and deliverables set by program   Upfront acknowledgement of the limitations of the proposed evaluation   Minimizing jargon to increase clarity of communication of results   Striking a balance between consultancy/ recommendation support and maintaining unbiased distance as an independent evaluator
  • 24. Before meeting with an evaluator:   Complete your program logic model and theory of change   Or have discussions and ideas of key components for development assistance from evaluator   Consider what you really want to learn   What outcomes do you want an evaluation to focus on for your program?
  • 25. What do you really want to know? Make sure you ask the right question!
  • 26. Before meeting with an evaluator:   Complete your program logic model and theory of change   Consider what you really want to learn   What data is already available or will need to be collected to answer your right question(s)?   If you believe data will need to be collected do you want to collect them or have the evaluator conduct the data collection?   Think about whether you have identified SMART outcomes!
  • 27. SMART Outcomes are:   Specific   Reflect simple information that is communicable and easily understood   Measurable   Can changes by measured in reliable and valid ways?   Achievable   Able to be collected and sensitive to change during the allotted time   Relevant   Reflect information that is important and likely to be used   Time bound   Progress can be tracked at desired frequency within allotted time Source: Adapted from World Bank Group.
  • 28. Questions for an evaluator   How does your proposed evaluation align with our program logic model and theory of change?   What data do you need from us to conduct the evaluation? What data to you propose to collect, when, and in what format?   How intrusive will the data collection be for our program?   Will your evaluation be able to tell us if the program was in place as intended?   What we really want to know from an evaluation is _________. How will your proposed evaluation answer this question?
  • 29. Questions to ask yourself about an evaluation proposed to you   Does the evaluation design reflect my logic model and theory of change?   Do the proposed outcomes follow my logic model and theory of change?   Is it feasible to collect outcome data at the time I expect to see changes (short- medium- and long- term outcomes) in those selected?   Are the selected outcomes and measures relevant for my program and my stakeholders?   Does the evaluation include SMART outcomes that will be evaluated?
  • 30. Logic Model Example
  • 31. Questions? Thank you. Lauren Decker, Ph.D.