Downloaded 36 times
Uploaded byArnaldo Pellini
Measuring policy influence: like measuring thin air?
The document discusses measuring policy influence through various approaches such as outcome mapping and social network analysis, emphasizing the importance of clear objectives and a theory of change. It outlines methods for assessment including case studies and stories of change, and highlights lessons learned regarding the impact of cultural norms and process management on policy influence. Recommendations for improving policy influence through research and monitoring and evaluation (M&E) strategies are also provided.
More Related Content
Similar to Measuring policy influence: like measuring thin air?
Measuring policy influence: like measuring thin air?
- 1. Measuring policy influence: like measuring thin air? John Young: j.young@odi.org.uk Arnaldo Pellini: a.pellini.ra@odi.org.uk
- 2. RAPID Power, Politicsand evidence use Knowledge intermediaries and interactions Evidence production and communication
- 3. Policy processes are... Cabinet Donors Policy Formula/on Agenda Parliament Se3ng Decision Making Civil Society Monitoring and Ministries Evalua/on Policy Implementa/on Private Sector
- 4.
- 5. Policy change • Discursive: Client- focused services • Attitudinal: Farmers have good ideas • Procedural: Participatory approaches to service development • Content: UU20, UU25. New guidelines • Behavioural: Approach being applied in practice 5
- 6. Focus on behaviourchange Project Team Other Actors Inputs Activities Outputs Outcomes Outcome Outcomes Impact Impact Impact Behaviour 6 Change
- 7. RAPID Outcome MappingApproach Define your policy objectives
- 8. RAPID Outcome MappingApproach Media strategy Online communications Develop a network or partnership Academic research communications Policy advocacy coalition More research
- 9. Why do M&E? • To learn about what works • To manage better • To account: – to donors – to recipients 9
- 10. Methods • Classical casestudies (IDRC, IFPRI) • Episode studies (ODI/RAPID) • Stories of Change (Denning) • Micro-Narratives (Snowden) • Impact matrices (Davies) • Peer evaluations (CHSRF) • Systematic reviews? • RCTs? 10
- 11. Outcome Mapping OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo http://www.idrc.ca/en/ev-9330-201-1- DO_TOPIC.html 11
- 12. Social Network Analysis 12 www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysi
- 13. RAPID Outcome Assessment 13 www.odi.org.uk/RAPID/Publications/RAPID_WP_266.htm
- 14. A systematic approach 1. Strategy and direction –are you doing the right thing? 2. Management –are you doing what you planned to do? 3. Outputs – are the outputs appropriate for the audience? 4. Uptake – are people aware of your work? 5. Outcomes and impacts –are you having any impact? 14
- 15. Conclusions Research to influence: Measuring impact • Clear objectives • Clear objectives • Theory of change • Understand the context • 5-levels • Theory of change • Multiple methods • Iterative / learning • Triangulation approach • Expect the unexpected 15
- 16. Recommendations • Strategy: theoryof change, impact pathways, peer review, log frame • Management: process records, appreciative inquiry, AARs, PRINCE2 • Output: logs, peer-review, • Uptake: logs, webstats, surveys • Impact: outcome mapping, stories of change, episode studies, peer review 16
- 17. VDR 2010 VDR 2010:assessing policy influence
- 18. VDR 2010 VDR 2010:Modern Institutions • Research output of a large governance programme (DfID) • 12 months of research and consultations • Presented by WB + 13 donors in 12/2009 • Policy influence assessment in 03/2011
- 19. Confirm the policychange Attitudinal change Procedural change Policy content Behaviour change
- 20. Policy influencing approaches Evidence / science based Advising Advocacy VDR 2010 Inside track / Outside track / cooperation pressure Lobbying Activism Interest / values based Start and Hovland (2004)
- 21. Areas of M&E Strategyand direction Logframes, ToC Project Governance, Management processes Research Management Peer reviews, After Outputs Action Reviews Citation Analysis, Uptake Uptake logs, User Surveys, … Stories of Change, Outcomes and impacts Episode Studies, … Hovland 2007
- 22. Areas of M&E Areas of evaluation Planned Strategy and direction Management processes Outputs Data on uptake of Uptake VDR 2010 Outcomes and impacts Stories of Change
- 23. What we did • Collaborate with a local research/advocacy organization • Citation analysis Google search • Interviews with donors representatives • Interviews with GoV advisers involved in the VDR 2010 process • Interviews with GoV officials
- 24. What we did/learned • Cultural norms influence the process • Citation, Google search, and time • Uptake and www traffic data • Management of the process emerged as an important theme • Perception of the quality of the report • Too early to speak of impact
- 25. What we did/learned Areas of evaluation Planned What we did/learned Strategy and Perception about the direction timing of the VDR 2010 Management Perception of the processes process Perception of the Outputs quality of the VDR 2010 Data on uptake of Data on uptake of the Uptake VDR 2010 VDR 2010 Outcomes and Story of Change / impacts Episode Study
- 26. What we did/learned Assessingthe policy influence of research: a case study of governance research in Viet Nam ODI Background Notes, May 2012 (http://bit.ly/LNLHE0) Assessment of the delivery of the Vietnam development report 2010 (Full report) (http://bit.ly/LNOJZ1)
- 27. M&E in ODI • ODI has 12 core research programs • 145 publications / 74 blogs / 13 issues Development Policy Review and Disasters (2009-2010) • 20 public events / 4500 participants (2009-2010) • Advisory work
- 28. M&E in ODI Strategyand ODI Strategy / Program Strategy / Reviews by direction SMT / ToCs Institute-wide management processes / staff Management surveys / project completion reports / after processes action reviews / internal peer review Peer reviews / quality assurance / communication Outputs team input / user surveys / end of project report + after action reviews
- 29. M&E in ODI Note by communication team / Uptake CommStat / regular updates on uptake / uptake log
- 30. M&E in ODI CommStatODI RAPID July – September 2011
- 31.
- 32. M&E in ODI Outcome/ Stories of Change page / Review by TT Impact heads (2008) / Program evaluations
- 33.
- 34.