Four key tasks in  impact assessment of complex interventions Rethinking Impact: Understanding the Complexity of Poverty a...
Some context <ul><li>1997 Causal models for evaluation, Harvard Project on Schooling and Children </li></ul><ul><li>2001 I...
What is impact? <ul><li>… the positive and negative, primary and secondary long-term effects produced by a  development   ...
Different types of impact may need  different methods  <ul><li>Long-term impacts  that will only be evident in future gene...
Different types of intervention may need  different methods  <ul><li>Simple  interventions that can be tightly specified a...
Different types of impact assessment  may need different methods  <ul><li>Intended use and intended users : </li></ul><ul>...
Four key tasks in impact assessment <ul><li>Decide  impacts to be included in assessment (conceptualise valued impacts) </...
A.  Decide impacts to include. <ul><li>Identifying and negotiating impacts to include: </li></ul><ul><ul><li>Not only stat...
B.  Gather evidence of impacts. <ul><li>Balancing accuracy, utility, feasibility and ethics: </li></ul><ul><ul><li>Dealing...
<ul><li>Analyse causal contribution or attribution </li></ul><ul><li>Avoiding false negatives and false positives: </li></...
D. Report synthesis and support use <ul><li>Providing useful information to intended users: </li></ul><ul><ul><li>Balancin...
Following a Recipe   A Rocket to the Moon   Raising a Child   <ul><li>Formulae are critical and necessary </li></ul><ul><l...
What works here and now? What do we mean by ‘works’? What works in what contexts? (implementation environments and partici...
Types of interventions Simple intervention  Complicated or complex intervention Single causal strand.  Intervention is suf...
Simple causal attribution intervention is both necessary and sufficient to produce the impact Impact Intervention
Causality in an INUS world Intervention is an Insufficient but Necessary part of a causal package which is itself Unnecess...
Causal analysis of problems Predisposition Triggering events or conditions Impacts
Causal analysis of interventions Intervention Favourable context Impacts
Eg of developing testable hypotheses not involving a comparison group Change in rate of all road fatalities
Change in rate of road fatalities on Fri and Sat nights
Uptake requires further ongoing adaptation Complicated message  Clear messages Reporting and supporting use Unique, highly...
Upcoming SlideShare
Loading in …5
×

Rogers Revised Presentation For Cali Workshop

712
-1

Published on

Presentation to &quot;Rethinking Impact: Understanding the Complexity of Poverty and Change&quot;, 26 - 28 March 2008 Cali, Colombia. It oulined four key tasks for impact evaluation and sets out some methods that can be used for each of these.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
712
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Rogers Revised Presentation For Cali Workshop

    1. 1. Four key tasks in impact assessment of complex interventions Rethinking Impact: Understanding the Complexity of Poverty and Change 26 - 28 March 2008 Cali, Colombia Professor Patricia Rogers Royal Melbourne Institute of Technology, Australia
    2. 2. Some context <ul><li>1997 Causal models for evaluation, Harvard Project on Schooling and Children </li></ul><ul><li>2001 Impact evaluation research project, Department of Primary Industries, Victoria, Australia </li></ul><ul><li>2006 Qualitative cost benefit analysis of Stronger Families and Communities Strategy, Australia </li></ul><ul><li>2008 NONIE (Network Of Networks on Impact Evaluation) – all multilateral banks, all UN agencies, all aid agencies of OEDC countries </li></ul><ul><li>2008 Edinburgh Summer School, Health Scotland ‘Getting real about impact’ </li></ul><ul><li>2008 NORAD Conference ‘Evaluating the complex’ </li></ul>
    3. 3. What is impact? <ul><li>… the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. These effects can be economic, socio-cultural, institutional, environmental, technological or of other types . </li></ul><ul><li>DAC definition </li></ul>
    4. 4. Different types of impact may need different methods <ul><li>Long-term impacts that will only be evident in future generations OR Short term impacts evident during the life of a project and an evaluation </li></ul><ul><li>Transformational impacts which are likely to remain once achieved OR fragile impacts which can be easily undone </li></ul><ul><li>‘ Smoking gun’ impacts where one intervention is both necessary and sufficient to produce the impact OR … </li></ul>
    5. 5. Different types of intervention may need different methods <ul><li>Simple interventions that can be tightly specified and standardized intervention (e.g. product, technique) </li></ul><ul><li>Complicated interventions that are part of a larger multi-component intervention </li></ul><ul><li>Complex , emergent program or policy (e.g. community development, natural resources management, emergency situation) </li></ul>
    6. 6. Different types of impact assessment may need different methods <ul><li>Intended use and intended users : </li></ul><ul><ul><ul><li>Knowledge building for replication and upscaling (by others?) </li></ul></ul></ul><ul><ul><ul><li>Knowledge building for learning and improvement </li></ul></ul></ul><ul><ul><ul><li>Accountability – to whom, for what, how? </li></ul></ul></ul><ul><li>Timing : </li></ul><ul><ul><ul><li>Ex-ante </li></ul></ul></ul><ul><ul><ul><li>Built into implementation </li></ul></ul></ul><ul><ul><ul><li>Retrospective </li></ul></ul></ul>
    7. 7. Four key tasks in impact assessment <ul><li>Decide impacts to be included in assessment (conceptualise valued impacts) </li></ul><ul><li>B) Gather evidence of impacts </li></ul><ul><li>(describe and/or measure actual impacts) </li></ul><ul><li>C) Analyse causal attribution or contribution </li></ul><ul><li>D) Report synthesis of impact assessment and support use </li></ul>Each of these tasks requires appropriate methods.
    8. 8. A. Decide impacts to include. <ul><li>Identifying and negotiating impacts to include: </li></ul><ul><ul><li>Not only stated objectives – also unintended outcomes (positive and negative) </li></ul></ul><ul><ul><li>Values about good and bad impacts and about the distribution of impacts </li></ul></ul><ul><ul><li>Conceptualising how impact occurs </li></ul></ul><ul><ul><li>Prioritising information needs </li></ul></ul><ul><ul><li>Adequate consultation and legitimisation </li></ul></ul><ul><li>Some approaches: </li></ul><ul><ul><li>Program theory (impact pathway) - possibly developing multiple models of the program, eg Soft Systems, negotiate boundaries (eg Critical Systems Heuristics) </li></ul></ul><ul><ul><li>Participatory approaches to values clarification –eg Most Significant Change </li></ul></ul>
    9. 9. B. Gather evidence of impacts. <ul><li>Balancing accuracy, utility, feasibility and ethics: </li></ul><ul><ul><li>Dealing with time lags before impacts are evident </li></ul></ul><ul><ul><li>Avoiding accidental or systematic distortion of level of impacts </li></ul></ul><ul><ul><li>Evidence that is sufficiently comprehensive </li></ul></ul><ul><ul><li>Making use of existing data as well as additional data </li></ul></ul><ul><ul><li>Valuing the impacts </li></ul></ul><ul><li>Some approaches: </li></ul><ul><ul><li>Program theory (impact pathway) – identify short-term results that can indicate longer-term impacts </li></ul></ul><ul><ul><li>Participatory approaches – engaging community in evidence gathering to increase reach and engagement </li></ul></ul><ul><ul><li>Real world evaluation – mixed methods, triangulation, making maximum use of existing data, strategic sampling, rapid data collection methods </li></ul></ul>
    10. 10. <ul><li>Analyse causal contribution or attribution </li></ul><ul><li>Avoiding false negatives and false positives: </li></ul><ul><ul><li>Systematic search for disconfirming evidence and analysis of exceptions </li></ul></ul><ul><ul><li>Distinguish between theory failure and implementation failure </li></ul></ul><ul><ul><li>Understanding the contribution of context: implementation environment, participant characteristics and other interventions </li></ul></ul><ul><li>Some approaches: </li></ul><ul><ul><li>Addressing through design – </li></ul></ul><ul><ul><ul><li>eg experimental designs (random assignment) and quasi-experimental designs (construction of comparison group eg propensity scores) </li></ul></ul></ul><ul><ul><li>Addressing through data collection – </li></ul></ul><ul><ul><ul><li>eg participatory Beneficiary Assessment, expert judgement </li></ul></ul></ul><ul><ul><li>Addressing through iterative analysis and collection – </li></ul></ul><ul><ul><ul><li>eg Contribution Analysis, Multiple Levels and Lines of Evidence (MLLE), List of Possible Causes (LOPC) and General Elimination Methodology (GEM), systematic qualitative data analysis, realist analysis of testable hypotheses </li></ul></ul></ul>
    11. 11. D. Report synthesis and support use <ul><li>Providing useful information to intended users: </li></ul><ul><ul><li>Balancing overall pattern and detail </li></ul></ul><ul><ul><li>Assisting uptake/translation of evidence </li></ul></ul><ul><li>Some approaches: </li></ul><ul><ul><li>Utilization-focused evaluation - Identification and involvement of intended users from the start </li></ul></ul><ul><ul><li>Layered reports (1 page, 5 pages, 25 pages) </li></ul></ul><ul><ul><li>Scenarios showing different outcomes in different contexts </li></ul></ul><ul><ul><li>Workshopping report to support knowledge translation </li></ul></ul>
    12. 12. Following a Recipe A Rocket to the Moon Raising a Child <ul><li>Formulae are critical and necessary </li></ul><ul><li>Sending one rocket increases assurance that next will be ok </li></ul><ul><li>High level of expertise in many specialized fields + coordination </li></ul><ul><li>Rockets similar in critical ways </li></ul><ul><li>High degree of certainty of outcome </li></ul><ul><li>Formulae have only a limited application </li></ul><ul><li>Raising one child gives no assurance of success with the next </li></ul><ul><li>Expertise can help but is not sufficient; relationships are key </li></ul><ul><li>Every child is unique </li></ul><ul><li>Uncertainty of outcome remains </li></ul>Complicated Complex <ul><li>The recipe is essential </li></ul><ul><li>Recipes are tested to assure replicability of later efforts </li></ul><ul><li>No particular expertise; knowing how to cook increases success </li></ul><ul><li>Recipes produce standard products </li></ul><ul><li>Certainty of same results every time </li></ul>Simple (Diagram from Zimmerman 2003)
    13. 13. What works here and now? What do we mean by ‘works’? What works in what contexts? (implementation environments and participant characteristics) What works? COMPLEX COMPLICATED SIMPLE
    14. 14. Types of interventions Simple intervention Complicated or complex intervention Single causal strand. Intervention is sufficient to produce the impacts Multiple simultaneous causal strands required to produce the impacts Universal mechanism. Intervention is necessary to produce the impacts Different causal mechanisms operating in different contexts Linear causality, proportional impact Recursive, with feedback loops, leading to disproportionate impact at critical levels Agreement and certainty about pre-identified outcomes Disagreement and uncertainty about outcomes and emergent outcomes
    15. 15. Simple causal attribution intervention is both necessary and sufficient to produce the impact Impact Intervention
    16. 16. Causality in an INUS world Intervention is an Insufficient but Necessary part of a causal package which is itself Unnecessary (not the only way) but Sufficient to produce the impact Impact Intervention Other factor – context or contribution of another intervention Alternative intervention Mackie, 1974, Mark, 2001
    17. 17. Causal analysis of problems Predisposition Triggering events or conditions Impacts
    18. 18. Causal analysis of interventions Intervention Favourable context Impacts
    19. 19. Eg of developing testable hypotheses not involving a comparison group Change in rate of all road fatalities
    20. 20. Change in rate of road fatalities on Fri and Sat nights
    21. 21. Uptake requires further ongoing adaptation Complicated message Clear messages Reporting and supporting use Unique, highly contingent causality Causal packages and non-linearity Clear counter-factual likely Analysing cause Harder to plan for given emergence Evidence needed about multiple components More likely to have standardised measures developed Describing impacts May be emergent Likely to differ, reflecting different agendas Likely to be agreed Deciding impacts Complex Complicated Simple

    ×