Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ODI: Methods for M&E of Policy Advocacy

11,743 views

Published on

Overseas Development Institute presentation to "Assessing the Impact of Policy Engagement: RIMISP / IFAD Learning Event, Rome 23 June 2015"

  • Be the first to comment

  • Be the first to like this

ODI: Methods for M&E of Policy Advocacy

  1. 1. John Young Head of Programme, RAPID, ODI For Assessing the Impact of Policy Engagement RIMISP-IFAD Learning Event IFAD, Rome 23rd June 2015 Methods for M&E of Policy Advocacy
  2. 2. Overview • Me and my background • Why is it important? • What do you want to evaluate? • Some ways of thinking (frameworks) • Some ways of doing it (tools) • Some conclusions
  3. 3. Why do M&E? • To learn about what works • To manage better • To account: – to donors – to recipients
  4. 4. Image source: LQ-CN7499 World Bank What to evaluate? • Do the policies deliver better outcomes? • Do the policies support improved education? • Do your programmes influence education policy? • Does your programme do it well?
  5. 5. Monitoring and Evaluation Agenda Setting Decision Making Policy Implementation Policy Formulation Policy processes are complex Civil Society Donors Cabinet Parliament Ministries Private Sector
  6. 6. The Working Paper • Trends and issues in M&E of policy engagement. • Frameworks for understanding policy engagement. • Methods and tools for M&E of policy engagement. • Case studies.
  7. 7. Image source: LQ-CN7499 World Bank Current issues • What is the purpose of M&E for policy engagement? • How do you measure success? • Attribution vs Contribution? • How can you assess causality?
  8. 8. Tactics: Inputs, activities, outputs Outcomes: Changes in behavior, relationships, policy Impact: Changes in state, wellbeing, environment Sphere of control Sphere of influence Sphere of concern Spheres of influence
  9. 9. The RAPID framework The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc. The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc External Influences Socio-economic and cultural influences, donor policies etc The links between policy and research communities – networks, relationships, power, competing discourses, trust, knowledge etc.
  10. 10. Eight “policy” outcomes 1. Attitudes of stakeholders. 2. Public opinion. 3. Capacity/engagement of other actors. 4. Change in discussions among policy actors. 5. Improvements in policy processes. 6. Change in policy content. 7. Behaviour change for implementation. 8. Systems for supporting delivery of change.
  11. 11. A systematic approach 1. Strategy and direction –are you doing the right thing? 2. Management –are you doing what you planned to do? 3. Outputs – are the outputs appropriate for the audience? 4. Uptake – are people aware of your work? 5. Outcomes and impacts –are you having any impact? 1. Strategy and direction 2. Management 3. Outputs 4. Uptake 5. Outcomes and impacts
  12. 12. Logical frameworks Goal Indicator MOV Purpose Indicator MOV Assumptions/Risks Output 1 Indicator MOV Assumptions/Risks Output 2 Indicator MOV Output 3 Indicator MOV Output 4 Indicator MOV √ √
  13. 13. Scalar models & rubrics 0. No engagement with CDA. 1. Occasional meetings, no participation 2. Some meetings, poor communication 3. Some meetings, limited inputs 4. Regular meetings, opinion considered. 5. …. and can monitor implementation 6. …. and consulted on proposed plans 7. …. and influence plans 8. …. and documented and shared 9. Commitment to action 10. Policy or practice change.
  14. 14. Impact logs
  15. 15. Impact: stories of change 1. Stories of change: • Situation before • Context • Situation after • What changed and why http://www.mande.co.uk/docs/MSCGuide.pdf 2. Most significant change (Davies) • Stories of change from different stakeholders • Systematic analysis of significance. 3. Micro-narratives (Snowden)
  16. 16. Stories of change (DFID 10%) 1. What is the story about? 2. Is there any other important background information? 3. What was the existing behaviour? 4. What did the RPC do that contributed to the change? 5. What was the behaviour change? 6. What factors helped make this change successful? 7. What comes next for the key actors? 8. Where can we find more information or supporting evidence?
  17. 17. Stories of change (TTI Evaluation)
  18. 18. http://www.redstonestrategy.com/reports/global-think-tank-network/ Redstone Strategy
  19. 19. Case studies
  20. 20. RAPID Outcome Assessment Assessment www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html
  21. 21. Conclusions • What exactly are you trying to do? • What do you want to know, why and who for? • What is your theory of change? • Are you seeing straight? • Do the changes have anything to do with you? • Are you changing? • “Perfect is the enemy of good1” 1 Voltaire, La Bégueule: “Dans ses écrits, un sage Italien Dit que le mieux est l'ennemi du bien".

×