Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Meaningful and manageable evaluation  Susie Rabin Campaigning Effectiveness, NCVO Hugh Mouser , Global Witness
Why monitor and evaluate?   <ul><li>Strengthen the impact your campaign can make </li></ul><ul><li>Empower people and comm...
M&E can be a valuable part of your campaign <ul><li>In development and planning stages </li></ul><ul><li>During campaign d...
Some principles of M&E <ul><li>Build evidence, not proof </li></ul><ul><li>“ if it quacks and has feathers, it’s probably ...
Some principles of M&E <ul><li>Some information is better than none </li></ul><ul><li>“ It’s not the purity of the process...
Creating an impact Activities: Consumer research to identify clearest labelling, Lobby  government departments, email acti...
Summary of key steps <ul><li>Identify the purpose of m&e </li></ul><ul><li>Decide who should be involved </li></ul><ul><li...
Developing an M&E framework <ul><li>Set clear change objectives </li></ul><ul><ul><li>What steps should you take to achiev...
The simplest approach <ul><li>Creating space for reflection </li></ul><ul><ul><li>At key stages throughout the campaign no...
Evaluation Framework How to create the data Target Indicator (if any) Impacts How to create the data Target Indicator (if ...
Don’t just focus on outputs.. <ul><li>It can be easier to count outputs, but you need to consider your outcomes as well  <...
Outputs vs Outcomes <ul><li>Output Impact </li></ul>Visits to sexual health clinic Better sexual health 15 people attend h...
Creating a credible mass of evidence <ul><li>Surveys commissioned by the organisation </li></ul><ul><li>One-off detailed a...
Building a learning culture in your organisation <ul><li>Monitor and test ‘theories of change’ for your campaign </li></ul...
Aim to… <ul><li>Carry monitoring and evaluation out as well as you can in the circumstances and then learn from your exper...
Communicating impact <ul><li>Keep your audience in mind </li></ul><ul><li>Tailor to your audience  </li></ul><ul><li>Summa...
S u dan Campaign: Evaluation case study Hugh Mouser Global Witness 25 th  January 2011
<ul><li>Aim: To ensure that Sudan’s oil revenues are used to help maintain peace and prevent further conflict between Nort...
<ul><li>What was required for the campaign aim to be met: </li></ul><ul><li>Clear and fair figures on the who, where, when...
<ul><li>4.  Public backing from donors for audit recommendations to be adopted  and new wealth sharing agreement to includ...
E valuation challenges and solutions <ul><li>Challenges : </li></ul><ul><li>Assessing and communicating contribution; </li...
 
So what did we learn?
Objective 1  –  Met <ul><li>“ Backing from key donor governments and institutions for a transparent audit with strong ToR....
Objective 2: Partially met <ul><li>“ Strong ToR drafted by donor-backed team, and accepted by the Sudanese govt ” </li></u...
Objectives 3 & 4  –  Not met <ul><li>“ The audit to be carried out.  ” </li></ul><ul><li>“ Public backing from donors for ...
<ul><li>Any contingency actions or course corrections? </li></ul><ul><li>South Sudan’s first ever petroleum policy; satell...
What evidence did we have for our effectiveness? <ul><li>Private congratulations on our work: </li></ul><ul><li>Off record...
So what did we know after 6 months? <ul><li>Positive impact?  </li></ul><ul><ul><li>Not yet.  Continual monitoring necessa...
How did this contribute towards building a learning culture? <ul><li>Review of findings and process with individual Campai...
What should I take away from this? <ul><li>Gather attributions as you campaign </li></ul><ul><li>Keep evaluation light-tou...
Questions and discussion   Questions to think about: <ul><li>What information do you have which could help with M & E? </l...
What can you do next? <ul><li>If you have 30 minutes: </li></ul><ul><li>Download and read Part 1 of  Is your campaign maki...
Contact us:  <ul><li>Contact NCVO’s Campaigning Effectiveness Programme for bespoke advice and support </li></ul><ul><li>w...
Upcoming SlideShare
Loading in …5
×

Meaningful and Manageable Evaluation

1,453 views

Published on

Susie Rabin's presentation at the "Meaningful and Manageable Evaluation" at the NCVO Campaigns Conference 2011.

More presentations can be found here:
http://www.ncvo-vol.org.uk/campaignsconference

Susie Rabin

Published in: News & Politics
  • Be the first to comment

  • Be the first to like this

Meaningful and Manageable Evaluation

  1. 1. Meaningful and manageable evaluation Susie Rabin Campaigning Effectiveness, NCVO Hugh Mouser , Global Witness
  2. 2. Why monitor and evaluate? <ul><li>Strengthen the impact your campaign can make </li></ul><ul><li>Empower people and communities </li></ul><ul><li>Learn from what you have done </li></ul>
  3. 3. M&E can be a valuable part of your campaign <ul><li>In development and planning stages </li></ul><ul><li>During campaign delivery /implementation </li></ul><ul><li>At the end of the campaign </li></ul>
  4. 4. Some principles of M&E <ul><li>Build evidence, not proof </li></ul><ul><li>“ if it quacks and has feathers, it’s probably a duck” </li></ul><ul><li>Outcomes are more important that outputs </li></ul><ul><li>Measure the important things, not the easy things </li></ul><ul><li>Keep it simple and user-friendly </li></ul>
  5. 5. Some principles of M&E <ul><li>Some information is better than none </li></ul><ul><li>“ It’s not the purity of the process that’s important, it’s the quality of the thinking” </li></ul><ul><li>Establish the link to action </li></ul><ul><li>Create a learning culture </li></ul>
  6. 6. Creating an impact Activities: Consumer research to identify clearest labelling, Lobby government departments, email actions directed at companies Outputs: Outcomes: Government issues clear guidelines on food labelling, major companies adopt clear food labelling Impact: People’s health improved by having access to clear information about food
  7. 7. Summary of key steps <ul><li>Identify the purpose of m&e </li></ul><ul><li>Decide who should be involved </li></ul><ul><li>Deciding your focus </li></ul><ul><li>Deciding what information you want to monitor </li></ul><ul><li>Deciding how you will collect the information you need </li></ul>
  8. 8. Developing an M&E framework <ul><li>Set clear change objectives </li></ul><ul><ul><li>What steps should you take to achieve your overall aim? </li></ul></ul><ul><li>Identify capacity and resources </li></ul><ul><ul><li>Be realistic about what you can achieve </li></ul></ul><ul><li>Identify indicators </li></ul><ul><ul><li>What information will show whether or not objectives are being met? </li></ul></ul><ul><li>Verification </li></ul><ul><ul><li>What sources will you explore and what techniques will you use to gather the evidence? </li></ul></ul>
  9. 9. The simplest approach <ul><li>Creating space for reflection </li></ul><ul><ul><li>At key stages throughout the campaign not, just at the end </li></ul></ul><ul><ul><li>Discuss key questions </li></ul></ul><ul><ul><li>Identify clear and achievable recommendations for future working </li></ul></ul>
  10. 10. Evaluation Framework How to create the data Target Indicator (if any) Impacts How to create the data Target Indicator (if any) Outputs
  11. 11. Don’t just focus on outputs.. <ul><li>It can be easier to count outputs, but you need to consider your outcomes as well </li></ul>
  12. 12. Outputs vs Outcomes <ul><li>Output Impact </li></ul>Visits to sexual health clinic Better sexual health 15 people attend health workshop Improvement in family wellbeing 30% reduction in STIs STI tests completed Increased confidence 100 people access helpline
  13. 13. Creating a credible mass of evidence <ul><li>Surveys commissioned by the organisation </li></ul><ul><li>One-off detailed analysis </li></ul><ul><li>Qualitative research </li></ul><ul><li>Secondary data </li></ul><ul><li>Tracking analysis </li></ul><ul><li>Anecdotal evidence </li></ul><ul><li>External evaluation </li></ul>
  14. 14. Building a learning culture in your organisation <ul><li>Monitor and test ‘theories of change’ for your campaign </li></ul><ul><li>Facilitate learning, inform ongoing decision making, underpin accountability </li></ul><ul><li>Embed as part of planning and reporting, not an add on </li></ul><ul><li>Feedback to stakeholders appropriately </li></ul>
  15. 15. Aim to… <ul><li>Carry monitoring and evaluation out as well as you can in the circumstances and then learn from your experiences </li></ul>
  16. 16. Communicating impact <ul><li>Keep your audience in mind </li></ul><ul><li>Tailor to your audience </li></ul><ul><li>Summarise and focus </li></ul><ul><li>Think about different media </li></ul><ul><li>Use pictures and words </li></ul><ul><li>Quotes and case studies </li></ul>
  17. 17. S u dan Campaign: Evaluation case study Hugh Mouser Global Witness 25 th January 2011
  18. 18. <ul><li>Aim: To ensure that Sudan’s oil revenues are used to help maintain peace and prevent further conflict between North and South </li></ul><ul><li>Tactics: Research, policy & advocacy </li></ul>The campaign at a glance
  19. 19. <ul><li>What was required for the campaign aim to be met: </li></ul><ul><li>Clear and fair figures on the who, where, when and how much of oil extraction and trade </li></ul><ul><li>Equitable revenue sharing </li></ul><ul><li>Peace to be maintained </li></ul>
  20. 20. <ul><li>4. Public backing from donors for audit recommendations to be adopted and new wealth sharing agreement to include transparency requirements </li></ul><ul><li>Activities : </li></ul><ul><li>Research & publish a briefing outlining continued discrepancies in oil figures </li></ul><ul><li>Extensive advocacy targeting US, Norway and UK decision makers, as well as International Finance Institutions </li></ul>Campaign objectives & activities <ul><li>Objectives: </li></ul><ul><li>Backing from key donor governments and institutions </li></ul><ul><li>Strong Terms of Reference for an audit drafted by a donor-backed team, and accepted by the Sudanese govt </li></ul><ul><li>The audit to be carried out , to include strong recommendations, and its results to be made public </li></ul>
  21. 21. E valuation challenges and solutions <ul><li>Challenges : </li></ul><ul><li>Assessing and communicating contribution; </li></ul><ul><li>Making the process light touch; </li></ul><ul><li>Solutions: </li></ul><ul><li>Campaigner self-assessment to be scrutinised by management; </li></ul><ul><li>How long it took - 1-2 hours for campaigner, 1 hour for management; </li></ul><ul><li>What happened next - Campaigns Director reviewed alongside other campaign evaluations; overall summary assessment sent to senior management for review </li></ul>
  22. 23. So what did we learn?
  23. 24. Objective 1 – Met <ul><li>“ Backing from key donor governments and institutions for a transparent audit with strong ToR. ” </li></ul><ul><li>Activities </li></ul><ul><li>Research new evidence of oil discrepancies. </li></ul><ul><li>Lobby for language in US Appropriations legislation to force the US to act. </li></ul><ul><li>Outputs </li></ul><ul><li>Short briefing using research published. </li></ul><ul><li>Story covered by at least three high impact international, and three well known Sudanese media outlets. </li></ul><ul><li>Progress towards objective </li></ul><ul><li>Backing secured </li></ul>
  24. 25. Objective 2: Partially met <ul><li>“ Strong ToR drafted by donor-backed team, and accepted by the Sudanese govt ” </li></ul><ul><li>Activities </li></ul><ul><li>Lobby IFIs, US government, Norway. </li></ul><ul><li>Lobby Sudan govt, South Sudan govt, Chinese / Malay authorities. </li></ul><ul><li>Outputs: </li></ul><ul><li>US private commitment to push Sudan govt to accept audit ToR. </li></ul><ul><li>Norway govt confirms that audit has been drafted. </li></ul><ul><li>Commitments from Sudan govt, South Sudan govt, Chinese / Malay authorities. </li></ul><ul><li>Progress towards objective </li></ul><ul><li>Text drafted, but decision now rests with Sudanese president’s office. Now behind on ambitious schedule. </li></ul>
  25. 26. Objectives 3 & 4 – Not met <ul><li>“ The audit to be carried out. ” </li></ul><ul><li>“ Public backing from donors for audit recommendations to be adopted and new wealth sharing agreement to include transparency requirements ” </li></ul><ul><li>Activities </li></ul><ul><li>Lobby US govt, IFIs, Sudan govt. </li></ul><ul><li>Outputs </li></ul><ul><li>Commitment from donors secured to press Sudanese govt, commitment from Sudanese govt secured to carry out audit / adopt recommendations. </li></ul><ul><li>Progress towards both objectives </li></ul><ul><li>None. </li></ul>
  26. 27. <ul><li>Any contingency actions or course corrections? </li></ul><ul><li>South Sudan’s first ever petroleum policy; satellite photography research; unable to get visas for Sudan </li></ul><ul><li>How was our overall progress? </li></ul><ul><li>Some good progress & reputation improving – helped get the audit drafted and public calls for action. Still disappointing slow rate of change however </li></ul><ul><li>What kind of a difference were we making? </li></ul><ul><li>Positive – in terms of positioning: </li></ul><ul><li>Raising awareness of transparency issues in Sudan </li></ul><ul><li>Working towards getting an audit carried out </li></ul><ul><li>Being poised to influence South Sudan’s first ever petroleum policy </li></ul><ul><li>Securing language in US appropriations act to get the govt to act on corruption issues in South Sudan </li></ul><ul><li>Negative: </li></ul><ul><li>Delays in getting visas! </li></ul>What else did we find out?
  27. 28. What evidence did we have for our effectiveness? <ul><li>Private congratulations on our work: </li></ul><ul><li>Off record: Donor government </li></ul><ul><li>A major Sudan expert said: “Congratulations on your report! It was an amazing piece of work, and has had a larger impact than any other public report that I can think of on Sudan, and that's no small feat!” </li></ul><ul><li>Donor government: our Sudan campaigner was referred to as “a force of nature” on the issue of transparency in Sudan. </li></ul>Public calls following GW lobbying: Senior political figures in South Sudan International Assessment and Evaluation Commission reports UK’s Africa Minister, Glenys Kinnock Several UK Lords <ul><ul><li>Policy tracking: </li></ul></ul><ul><ul><li>UK Associate Parliamentary Group on Sudan report namechecks our work and mirrors many of our policy calls. </li></ul></ul><ul><ul><li>Media presence and response: </li></ul></ul><ul><ul><li>Sudanese United government energy minister invites GW to Khartoum to inspect the oil books, in a live TV debate with our Sudan campaigner on BBC Arabic TV </li></ul></ul>
  28. 29. So what did we know after 6 months? <ul><li>Positive impact? </li></ul><ul><ul><li>Not yet. Continual monitoring necessary. </li></ul></ul><ul><li>Positive outcomes? </li></ul><ul><ul><li>Yes - incremental improvements, but not as much as we had hoped </li></ul></ul><ul><ul><li>No – delays in getting hold of visas! </li></ul></ul><ul><li>Contribution to outcomes? </li></ul><ul><ul><li>Yes. Disappointed with progress but knew we had done a good job – of course our work had raised the heckles of North Sudan and prevented us from getting visas. </li></ul></ul><ul><li>On schedule? </li></ul><ul><ul><li>To the greatest extent, yes, and all activities carried out </li></ul></ul><ul><li>How did this help us? </li></ul><ul><ul><li>Informed tactics: US & Norway were now more responsive to our work </li></ul></ul><ul><ul><li>Helped us show our effectiveness and secure new funding for 3 years </li></ul></ul><ul><li>When would we be able to evaluate impact? </li></ul><ul><ul><li>When improvements are secured in people's lives. </li></ul></ul>
  29. 30. How did this contribute towards building a learning culture? <ul><li>Review of findings and process with individual Campaign Managers individually and together </li></ul><ul><li>Encouraging staff to gather attributions as they go along </li></ul><ul><li>Organisation-wide presentation to follow each round of assessments summarising where the organisation is at </li></ul><ul><li>End of year internal report and external annual review to explain better the timeline of activities leading to outputs, outcomes and chances of impact </li></ul>
  30. 31. What should I take away from this? <ul><li>Gather attributions as you campaign </li></ul><ul><li>Keep evaluation light-touch </li></ul><ul><li>Try to ensure someone neutral checks your evaluation </li></ul><ul><li>Build a picture of the consequences of your work </li></ul>
  31. 32. Questions and discussion Questions to think about: <ul><li>What information do you have which could help with M & E? </li></ul><ul><li>What information does your team or organisation possess? </li></ul><ul><li>What information is available from other sources that you can you tap into to help with your M & E? </li></ul>
  32. 33. What can you do next? <ul><li>If you have 30 minutes: </li></ul><ul><li>Download and read Part 1 of Is your campaign making a difference? from our website </li></ul><ul><li>Apply the basic evaluation framework to your campaign </li></ul><ul><li>If you have longer: </li></ul><ul><li>Purchase the complete publication for how to conduct more in depth m&e </li></ul><ul><li>Read The Good Guide to Campaigning and Influencing </li></ul><ul><li>Carry on the discussion at www.forumforchange.org.uk </li></ul>
  33. 34. Contact us: <ul><li>Contact NCVO’s Campaigning Effectiveness Programme for bespoke advice and support </li></ul><ul><li>www.ncvo-vol.org.uk/campaigningeffectiveness </li></ul><ul><li>[email_address] </li></ul>

×