Successfully reported this slideshow.
Your SlideShare is downloading. ×

Impact Evaluation for Policy Making_Promoting Uptake of Impact Evaluation Findings

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 25 Ad
Advertisement

More Related Content

Slideshows for you (20)

Viewers also liked (16)

Advertisement

Similar to Impact Evaluation for Policy Making_Promoting Uptake of Impact Evaluation Findings (20)

Recently uploaded (20)

Advertisement

Impact Evaluation for Policy Making_Promoting Uptake of Impact Evaluation Findings

  1. 1. Impact Evaluation for policy-making Promoting uptake of impact evaluation findings: the importance of relevance, utility and engagement What 3ie is learning Beryl Leach, deputy director, 3ie 18 February 2015 Istanbul, Turkey
  2. 2. Overview • Who we are and how we promote high-quality, relevant and useful studies that are taken up and used • What we are learning about impact evaluation and policy influence • How think tanks can produce, engage and use impact evaluation in policy advocacy
  3. 3. What is 3ie? • International grant-making NGO founded in 2008 • Filling evidence gaps in what we know works, why, how and the costs in development • We fund particular types of evidence production – high-quality experimental and quasi-experimental, theory-based, mixed-method impact evaluations – systematic reviews – evidence syntheses Why are IEs important for policy-making?
  4. 4. Policy relevance and usefulness • Evidence that can improve the effectiveness of development policies and programmes • Must be policy relevant • We expect policy influence and impact on policy and programming because of how we approach design and implementation
  5. 5. What do we mean by useful?  RELEVANT: Helps answer a specific policy question  CONTEXTUAL: Appropriate political context—it makes sense  CLEAR: message gives options  FEASIBLE: Affordable and possible  TIMELY: interest exists
  6. 6. 3ie funds high-quality policy-relevant studies 3ie encourages researchers to engage with key stakeholders Uptake of study findings and improved policies and practice Applications - Ask policy relevant question(s) - Have potential for policy impact - Study team has experience in policy influence ASSUMPTIONS Researchers -Committed - Understand how policy influence happens - Have the tools and resources to invest in policy engagement - Decision-makers are interested Study -Makes policy- relevant recommendations – Answers main IE questions - Proposes feasible solutions - Had ongoing engagement ASSUMPTIONS ASSUMPTIONS
  7. 7. What we learned from analysing early influencing in our grants • 3ie needed to be directly engaging with researchers and much earlier • Researchers needed to be engaging with implementers much earlier and differently • Paper-driven processes were not effective • 3ie approach needed to be based on evidence about policy change and research uptake
  8. 8. Main ways we are ensuring policy relevance and usefulness • Early and direct engagement with implementing agencies during the development of funding windows • Require country nationals on the research teams in substantive roles; more funding is for country-based teams – role and opportunity for think tanks • Preparation phase • Direct engagement with researchers about PIPs from the start and ongoing dialogue with the team
  9. 9. Communicating impact evaluation evidence: ongoing, integrated, multi-level • Engage from the start • Explain the study and why it will be useful--build interest • Report preliminary findings-get feedback-promotes ownership • Engage a range of key actors: beneficiaries and other local actors, civil society, media from early phases • Translate into plain language and produce in a variety of formats and disseminate through multiple channels: – Meetings, conferences, social media, multiple briefs, papers, reports Think Tanks
  10. 10. Impact evaluation and think tank policy influencing: opportunities and constraints
  11. 11. Bridging researchers & policymakers Translating, relationship building, networking
  12. 12. Using impact evaluation evidence for policy advocacy: continued • Constraints • Two-sided coin – Rigor as the basis for being high quality and for sound decision-making – IEs seek to measure causal relationships, we need to be able to assess validity • Two-part problem – Poor or weak designs mean you can’t really know which cause – Users have limited critical appraisal capacity
  13. 13. Using impact evaluations for advocacy • How much can you generalise based on one IE study? – Not much, even though we stress generalisability a lot – More than IE evidence needed to know about potential for scaling up – Researchers should always bring knowledge of existing evidence to the analysis, so that the new evidence is situated in the existing body of evidence, doesn’t always happen • Evidence for programme change can be more immediately useful than for policy change – Especially the case with theory-based evaluation, which is what we require – Modification to existing programme theory of change and operations
  14. 14. Benefits of increased production and use of impact evaluations by think tanks • IEs particularly need a strong emphasis on policy relevance and utility that think tanks can provide • More think tanks doing quality IEs and being effective intermediaries in policymaking will help increase uptake • Participating in IE teams and commissioning can strengthen relevance and utility • Big potential for improving think tank understanding of IE evidence and critical appraisal to be knowledge translators in advocacy process
  15. 15. Thank you www.3ieimpact.org #3ieNews #TTIX2015
  16. 16. END OF TTIX PRESENTATION
  17. 17. What are we looking at next? • Developing our approaches to working with implementers • Doing case studies on the cumulative impact of IEs done over time: direct and indirect contributions to what types of change • Evaluating evidence uptake from 3ie studies – Finding an appropriate method and framework – Exploring QCA – Why we probably will not be doing IEs of policy influence – Why we will look at synthesising evidence
  18. 18. Preparation phase-why is it so important • Implementing agencies do not necessarily understand impact evaluations • Researchers do not necessarily speak the implementers’ language or understand their evidence needs well enough • Fund working together • Monitor through direct contact • Have inception workshops
  19. 19. Policy influencing using impact evaluations • Evidence does not even play a major role in most decision-making • Research uptake is a political process, not a technical one • Researchers are vital to translating and building trust and credibility • Single study evidence most often does not result in major policy change, nor should it • 3ie does not advocate for wider change based on single studies, nor should you
  20. 20. Limitations • Expensive • Work best with large programmes • Highly specialised methods • Demands of counterfactuals • Need to translate for policy and programming uptake
  21. 21. Prevailing view of policy engagement in the IE community Get evidence Disseminate as reports and papers Change happens Assumption: You need the evidence before you engage and promote uptake
  22. 22. 3ie funds high- quality policy relevant studies Applications - Ask policy relevant question(s) - Have potential for policy impact - Study team has demonstrated experience in policy influence ASSUMPTIONS Increased scoring for policy aspects National researchers on team Preparation phase How we support
  23. 23. 3ie requires stakeholder engagement Researchers -Committed - Understand how policy influence happens - Have the tools and resources to invest in policy engagement - Decision-makers interested ASSUMPTIONS Inception workshop includes implementing agencies Identify useful evaluable questions Develop a policy influence plan Earmark study budget How we support
  24. 24. Uptake of study findings and improved policies and practice Study -Makes policy- relevant recommendations - Answers what works and why - Proposes feasible solutions - Has had ongoing engagement ASSUMPTIONS Ongoing engagement with 3ie as part of grant monitoring– graduating to a more dynamic model of interaction Intensive review and feedback of study reports by internal and external reviewers How we support
  25. 25. Using impact evaluation evidence for policy advocacy: 3ie perspective • Benefits – Policymakers can understand numbers – Talking in effect size and costs can be compelling – Policymakers often are looking for cost effectiveness evidence – Programme managers want the IE and are invested in the questions and want to be able to act on findings – They have credibility in today’s climate

×