Your SlideShare is downloading. ×
  • Like
TTI PEC Nairobi Workshop - Unpacking impact and influence
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

TTI PEC Nairobi Workshop - Unpacking impact and influence

  • 33 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
33
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Unpacking Impact and Influence: Approaches to Monitoring & Evaluating April 28, 2014 TTI PEC Africa workshop - Nairobi Courtney Tolmie (Results for Development Institute)
  • 2. A few notes to start  This session focuses on Organization M&E. That said, most of the lessons can be scaled down to the project level.  You will guide the session – and giving real time feedback: 2 | R4D.org Way too fast. Please slow down. Just right. You can speed up.
  • 3. How do we think about Impact and Influence? 1. What IMPACT does your organization seek to have? 2. What and who do you work to INFLUENCE as an organization? 3. What is the link between IMPACT and INFLUENCE in the work that you do? 3 | R4D.org
  • 4. Monitoring and Evaluation – the link to Impact and Influence In other words, a way of following whether you are doing what you planned to do and whether you are achieving the IMPACT you wanted to, including Policy INFLUENCE. 4 | R4D.org Monitoring: a continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention with early indications of progress, or lack thereof, in the achievement of results. Evaluation: systematic and objective assessment of an on-going or completed project, program, or policy, and its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. Definitions from the World Bank - http://siteresources.worldbank.org/INTBELARUS/Resources/M&E.pdf
  • 5. What Makes an organization-level M&E System Effective? Defining Impact and how we get there (TOC) Defining how to measure impact & outputs (indicators) Carrying out M&E (Methods) Using results to improve work (Feedback Loop) 5 | R4D.org
  • 6. M&E Systems – Defining a Theory of Change Defining Impact and how we get there (TOC) Defining how to measure impact & outputs (indicators) Carrying out M&E (Methods) Using results to improve work (Feedback Loop) 6 | R4D.org Defining a Theory of Change  Clarifying your organization objectives and desired impacts  Mapping out what is required to achieve these impacts  Linking to the activities and actions your organization will take to achieve the impact  Also highlighting assumptions (including context issues)  Similar structures - logframes
  • 7. M&E Systems – Defining Indicators Defining Impact and how we get there (TOC) Defining how to measure impact & outputs (indicators) Carrying out M&E (Methods) Using results to improve work (Feedback Loop) 7 | R4D.org Defining Indicators  Turning what you hope to achieve into what you can actually measure  Making indicators SMART  Value (and tradeoff) in collecting indicators on impact and outputs
  • 8. M&E Systems – Designing Methods Defining Impact and how we get there (TOC) Defining how to measure impact & outputs (indicators) Carrying out M&E (Methods) Using results to improve work (Feedback Loop) 8 | R4D.org Defining and designing Methods  Identifying how to actually collect data on the indicators that you have defined  Timing – monitoring, evaluating or both  Things to consider – capacity, resources, what you are trying to achieve
  • 9. M&E Systems – Ensuring Feedback Loops Defining Impact and how we get there (TOC) Defining how to measure impact & outputs (indicators) Carrying out M&E (Methods) Using results to improve work (Feedback Loop) 9 | R4D.org Ensuring Feedback Loops  The most forgotten step of M&E systems  Results for M&E can help improve programs and ultimately impact  Best to have formal structure to build results back into future work  Moving from culture of M&E to MEL
  • 10. Mapping areas for development  Based on your existing M&E system, assess how you are doing in each element of effective M&E  Scale from “we don’t do this at all” to “we do this very well on all areas of work”  10 minutes to map your organization – then we will come back to review 10 | R4D.org
  • 11. Defining Theories of Change 11 | R4D.org
  • 12. Defining Theories of Change  Examples from other organizations  Tips from the group about how to do it well  Ideas for moving forward 12 | R4D.org
  • 13. 13 | R4D.org Build evidence on better ways to budget for higher education Share recommendations with Minster of Education and Finance New budget passed with recommendation It would be easier if organization TOCs looked like this: But this is not realistic WHY NOT? • TTs are trying to achieve bigger changes than just this • A single TT is going to likely need to take a lot of different actions to get to that end impact • A single TT is not going to individually achieve this impact
  • 14. 14 | R4D.org
  • 15. 15
  • 16. Note about Theories of Change  A good TOC for influencing policy is incredibly complex.  Your organization may not try to affect all parts of the TOC – and thus may not choose to do M&E for all factors that go into policy influence.  But good to be aware. 16
  • 17. Defining Theories of Change  Examples from other organizations  Tips from the group about how to do it well  Ideas for moving forward 17 | R4D.org
  • 18. Defining Theories of Change – TOC Pitfalls to avoid from Stanford Social Science Review 1. Confusing accountability with hope 2. Creating a mirror instead of a target 3. Failing to take external context into account 4. Not confirming the plausibility of your theory 5. Creating a theory that is not measurable 6. Assuming you have figured it all out 18 | R4D.orghttp://www.ssireview.org/blog/entry/six_theory_of_change_pitfalls_to_avoid What lessons can you share from your organizations – things that have worked and what has not worked?
  • 19. Defining Indicators 19 | R4D.org
  • 20. Defining Indicators  What types of indicators work well, especially for measuring policy influence and impact?  Tips from the group about how to do it well  Ideas for moving forward 20 | R4D.org
  • 21. Examples of good indicators – for the full organization  While the focus of this program is on PEC, it is worth noting that there are multiple components of a think tank’s performance worth monitoring: 21 | R4D.org From Raymond Struyk’s forthcoming book – Improving Think Tank Management Public Policy Perspective Funder Perspective Internal Business Perspective Innovation & Learning Perspective Financial Perspective
  • 22. Examples of good indicators – for the full organization  While the focus of this program is on PEC, it is worth noting that there are multiple components of a think tank’s performance worth monitoring: 22 | R4D.org From Raymond Struyk’s forthcoming book – Improving Think Tank Management Public Policy Perspective Funder Perspective Internal Business Perspective Innovation & Learning Perspective Financial Perspective
  • 23. Examples of good indicators for PEC impact and influence  Some helpful indicators from the center for International Governance Innovations: 23 | R4D.org From www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks Indicators of Exposure: • Media mentions • # and type of publications • Scholarly citations • Government citations • Think tank ratings Indicators of Resources: • Quality, diversity, stability of funding • Characteristics of researchers • Quality, # of networks, partnerships Indicators of Demand: • Events • Digital traffic and engagement • Access to officials • Publications sold, downloaded Indicators of Policy Impact & Quality of Work: • Policy recommendations considered or adopted • Testimonials • Quality of think tank work
  • 24. Other indicators we have seen used 24 | R4D.org If the objective is … Specific policy recommendation implemented Change in the debate … Possible indicators are:  Invitation to share findings with specific audiences (government, others)  Attendance of decision makers at dissemination events  (if available) Information on issue being debated in policy discussions  Request for research by policymakers on the new topics  Greater debate on topic in media  (if available) Information on issue being debated in policy discussions Longer term – access to policymakers  Individual meetings with policymakers
  • 25. A note about impact indicators …  In general, these are not going to be attributable.  Attribution is difficult  Almost without exception, it takes a village.  This does not mean impact indicators are not helpful – especially if seeking to improve results. 25 | R4D.org
  • 26. Defining Indicators  What types of indicators work well, especially for measuring policy influence and impact?  Tips from the group about how to do it well  Ideas for moving forward 26 | R4D.org
  • 27. Defining Indicators  What types of indicators work well, especially for measuring policy influence and impact?  Tips from the group about how to do it well  Ideas for moving forward 27 | R4D.org
  • 28. Designing methods for M&E 28 | R4D.org
  • 29. Designing methods for M&E  What common options are others using?  Tips from the group about how to do it well  Ideas for moving forward 29 | R4D.org
  • 30. Options for M&E approaches 30 | R4D.orgTable from ODI Background Note – A guide to monitoring and evaluating policy influence (Harry Jones) Does not include common impact evaluation methods – RCTs, process tracing, etc.
  • 31. Designing methods for M&E  What common options are others using?  Tips from the group about how to do it well  Ideas for moving forward 31 | R4D.org
  • 32. Designing methods for M&E  What common options are others using?  Tips from the group about how to do it well  Ideas for moving forward 32 | R4D.org
  • 33. Ensuring Feedback Loops 33 | R4D.org
  • 34. Ensuring feedback loops  Do results feed back into the think tank automatically?  Tips from the group about how to do it well  Ideas for moving forward 34 | R4D.org
  • 35. Connecting M&E Results to Think Tank Changes  Two potential reasons to undertake M&E:  To achieve the second, M&E can’t stop at data analysis – it needs to feed back into the think tank. 35 | R4D.org ACCOUNTABILITY to funders and others BETTER RESULTS for think tank impact and influence
  • 36. Questions to ask to design a strong Feedback Loop (Knowledge Management) 36 | R4D.org The Question WHO should hear the M&E results? WHEN should they get the information? The Answer  (1) Who is the best position to exploit the information – capacity and interest  (2) More generally, anyone who SHOULD know  (1) At a time when they can act – capacity and opportunity  (2) Generally more than once WHAT should they receive?  (1) Format – useful but not necessarily fancy  (2) Ideally comparisons to baseline data In general, the Feedback Loop is more likely to be effective if designed early – proactive rather than reactive. Built from Raymond Struyk’s forthcoming book – Improving Think Tank Management
  • 37. Ensuring feedback loops  Do results feed back into the think tank automatically?  Tips from the group about how to do it well  Ideas for moving forward 37 | R4D.org
  • 38. Ensuring feedback loops  Do results feed back into the think tank automatically?  Tips from the group about how to do it well  Ideas for moving forward 38 | R4D.org
  • 39. A final word 39 | R4D.org
  • 40. Conclusion  M&E is not easy – and requires capacity. This is especially true for organization-level M&E and for policy influence M&E.  Match M&E plan to capacity  Moving from M&E to MEL (L = “learning”)  Linking the two objectives of M&E – accountability and better results  Some methods and indicators can address both – where possible, don’t double your work  Example – how can you use the IDRC monitoring indicators for PEC to help you achieve better results? 40 | R4D.org
  • 41. Other Great Resources (in addition to those referenced on this powerpoint)  ODI work on RAPID  On Think Tanks page on Monitoring and Evaluating Influence (http://onthinktanks.org/topic-pages/topic-page-monitoring-and- evaluating-influence/)  Research 2 Action PEC page 41 | R4D.org