Outcome Mapping presentation for donor coordination meeting


Published on

Civil society coordination meeting Sarajevo, 6.12.2007

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Outcome Mapping presentation for donor coordination meeting

    1. 1. Outcome Mapping Civil society coordination meeting Sarajevo, 6.12.2007 Steve Powell & Ivona Čelebičić proMENTE social research, Sarajevo an evaluation of six civil society projects in BiH
    2. 2. Contents <ul><li>TOR of our evaluation project </li></ul><ul><li>What is OM? </li></ul><ul><li>Findings </li></ul><ul><li>Conclusions </li></ul><ul><li>Recommendations </li></ul><ul><li>Discussion </li></ul>
    3. 3. Who are we <ul><li>Steve Powell & Ivona Čelebičić proMENTE social research, Sarajevo </li></ul><ul><li>Sources for OM: International Development Research Centre www.idrc.ca </li></ul><ul><li>See: www.promente.org/ om </li></ul>
    4. 4. Evaluation TOR <ul><li>Explorative evaluation of six civil society projects </li></ul><ul><li>Sida-funded civil society programming : lessons on sustainability </li></ul><ul><li>Exploration of OM as a tool </li></ul>
    5. 5. Our project in B&H : 1 donor, 3 framework partners, 6 implementing partners, 6 projects : 2007 &quot;Improvement of access to justice – Raising public awareness on access to rights, legislative changes and legal procedures in BiH Vasa Prava Work with judges and journalists on Press Code BiH Press Council (funded by SHC) Swedish Helsinki Committee for Human Rights &quot;Citizen in action&quot;– Project of community-based advocacy campaigns for solving priority citizens problems from &quot;Civic Platform for 2006 Elections&quot;. &quot;Local Government leadership building activities&quot; Civil Society Promotion Center – GROZD, &quot;Citizen in action&quot; Academy for political leaders Helsinki Citizens Assembly (HCA) – Academy for political leaders) Olof Palme International Center Village activities (including round tables in towns) Most Womens political lobby Zenski Centar Kvinna til Kvinna Evaluated activity Partner organisation Framework organisation
    6. 6. What is Outcome Mapping (OM)? <ul><li>An approach to project planning, monitoring and evaluation </li></ul><ul><li>A new, soft alternative to replace or complement LFA </li></ul><ul><li>A new paradigm/way of thinking. A backlash? </li></ul>
    7. 7. Problems with RBM Strategy maps inspire thinking about different dimensions of planning Mechanistic approach to strategy Quality , not quantity of change Focus too much on impact in areas “where their influence … is low and decreasing relative to that of other actors”* Fits better with what NGOs feel they are doing: stimulating change, not delivering outputs NGOs tend to see it as alien OM Results Based Management (RBM) / Logframe (LFA) Focus on: strategy, learning as well as outcomes Exclusive focus on impact/results Rich , useful feedback Provide only “ clueless feedback ” Contribution (what did they do, what worked?) Attribution (did they really cause the change?) Flexible Inflexible Focus on development/change of key partners; outcomes, not impacts Force implementing orgs to try to demonstrate that they caused numerically large impacts
    8. 8. Logframe vs. OM Vision Implementing agency: strategy Boundary partners. changes in attitudes, behaviour, relationships Other actors? ? Outputs Outcomes Objectives Goal
    9. 9. <ul><li>Boundary partners ≠ beneficiaries! </li></ul><ul><ul><li>(might be, do not have to be) </li></ul></ul><ul><li>There can be more than one group of boundary partners </li></ul><ul><li>Development work is really about leveraging the influence of a limited number of key partners … </li></ul><ul><li>… focus is NOT on the stakeholders with whom the boundary partners interact </li></ul><ul><li>changing consciousness, ways of working … </li></ul><ul><li>… in service of a vision </li></ul>Key outcomes: changes in attitudes, behaviour, relationships in boundary partner groups
    10. 10. Example Implementing agency: Helsinki Citizen’s Assembly Boundary partners: young politicians Outcome challenge: Empowered young politicians influence the change of the everyday politics in their parties. Recognized need for improvement in the areas selected in the party modules, continuous activities on the empowerment of youth engaged in politics. Other actors: political parties, other young politicians … Vision The partners become part of a new, younger, gender-equal generation in politics, who through their influence in their political parties initiate a change in the direction of everyday political issues – away from national/ethnic issues towards themes which respect democracy and human rights including gender rights and which will contribute to social prosperity in the long term. …. … …. …
    11. 11. OM: 12 steps Helping the implementing partner to learn Internal M &E Considering all the dimensions of strategy Focus on outcomes 11. Performance journal 7. Organisational practices Evaluation Outcome & performance monitoring Intentional design 10. Strategy journal 6. …Strategy maps 12. Evaluation plan 8. Monitoring priorities 9. Outcome journals 5. ...Progress markers 4. ...Outcome challenges 3. ...Boundary partners 2. Mission 1. Vision
    12. 12. Progress marker ladders Vision Boundary partner 1 Boundary partner 2 Outcome challenge 1 Love to see Like to see Expect to see Outcome challenge 2
    13. 13. Method: timeframe
    14. 14. Method: OM <ul><li>Special use of OM for external evaluation </li></ul><ul><ul><li>Mission, vision, progress markers, outcome challenge </li></ul></ul><ul><ul><li>Assessment of planning strategies using strategy maps </li></ul></ul><ul><ul><li>OM questionnaires </li></ul></ul><ul><ul><li>OM interview with boundary partners </li></ul></ul><ul><ul><li>OM interview with implementing partners </li></ul></ul><ul><ul><li>Writing evaluation-start Outcome Journals </li></ul></ul><ul><ul><li>Confirming evaluation-start Outcome Journals </li></ul></ul><ul><ul><li>OM questionnaires (same as evaluation-start) </li></ul></ul><ul><ul><li>OM interview with boundary partners (on the basis of evaluation-start interview) </li></ul></ul><ul><ul><li>OM interview with implementing partners (on the basis of evaluation-start interview) </li></ul></ul><ul><ul><li>Assessment of management progress using existing strategy maps </li></ul></ul><ul><ul><li>Mission and vision: still relevant? Changed focus? </li></ul></ul><ul><ul><li>Gathering contextual information and intervention timeline </li></ul></ul>Baseline interim final assessment
    15. 15. Findings: OM <ul><li>Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision (and  ) </li></ul><ul><li>All made progress towards their vision </li></ul><ul><li>(A lot of information about individual projects) </li></ul>
    16. 16. Method: relevance interviews <ul><li>Aim: to assess project and program relevance </li></ul><ul><li>Sources: </li></ul><ul><ul><li>50 additional interviews at baseline: stakeholder views of poverty, gender and civil society </li></ul></ul><ul><ul><li>+ 50 at final assessment: project relevance. </li></ul></ul><ul><li>Analysis: qualitative methods, specialised software -> stakeholders’ own themes. </li></ul>
    17. 17. Sample: relevance interviews <ul><li>10 representatives of implementing partner organisations </li></ul><ul><ul><li>9 female, 7 urban, 7 participated in tertiary education; half under 30, none over 50. </li></ul></ul><ul><li>34 boundary partners </li></ul><ul><ul><li>20 women, 23 from rural areas, 14 never begun secondary education and 14 over 50. </li></ul></ul>
    18. 18. Findings: relevance interviews: Gender <ul><li>Strong opinions about gender equality </li></ul><ul><ul><li>There is no real gender equality </li></ul></ul><ul><ul><li>Women are capable of taking leading roles in society </li></ul></ul><ul><ul><li>Women are not really interested in change (especially female respondents). </li></ul></ul>
    19. 19. table
    20. 20. Findings: relevance interviews: poverty and change <ul><li>About one respondent in four brought up the concept of &quot;spiritual&quot; or &quot;intellectual&quot; poverty … </li></ul><ul><li>Most very sceptical about the possibility of change. </li></ul><ul><li>Males very frequently mentioned the problem of corruption. </li></ul><ul><li>Implementing partners much more optimistic </li></ul><ul><li>There is no end to it. Here, people are hanging and killing themselves with their own hands. In my village, fifteen days ago, right beneath my window, a man hung himself... He was single. He might have had some small pension, just a little bit... and his health was pretty bad ... that’s what I think. </li></ul>
    21. 21. Assistance? <ul><li>A good part of the population is used to some sort of humanitarian aid, some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything. </li></ul>
    22. 22. Method: quantitative analysis <ul><li>For four projects, a customised questionnaire was made on the basis of progress markers. </li></ul><ul><li>Questionnaire filled in at baseline and again at final assessment </li></ul><ul><li>Results from around 100 baseline and 100 final assessment questionnaires compared on a per-project basis. </li></ul>
    23. 23. Findings: quantitative analysis <ul><li>Results for one project </li></ul><ul><li>Small but significant overall improvement </li></ul><ul><li>Women start lower than men and improve more than men </li></ul><ul><li>In spite of a possible tendency to “raise the bar” during the project </li></ul>male female
    24. 24. Improvement up the progress marker ladder <ul><li>at which points of the progress marker ladder do which boundary partners improve? </li></ul><ul><li>Not only do women improve more than men but they improve strongly at the upper end of the scale </li></ul>Progress marker ladder improvement female male
    25. 25. Conclusions: programming <ul><li>Sida support helps develop civil society and CS activists </li></ul><ul><li>Vision is not always about boundary partner change </li></ul><ul><li>Civil society not a mass movement! Small number of partners; handful of key “career” activists </li></ul><ul><li>Poverty </li></ul><ul><ul><li>Less clear progress with the least educated </li></ul></ul><ul><ul><li>Not clear that civil society development has reduction of poverty as a direct outcome. </li></ul></ul><ul><ul><li>Most successful projects do not connect directly to poverty </li></ul></ul><ul><ul><li>Projects with a stronger connection to poverty have less of a development perspective </li></ul></ul><ul><ul><li>Projects are not directly relevant to the really poor; different language; illiteracy </li></ul></ul><ul><ul><li>For the poor, not with the poor </li></ul></ul><ul><li>Gender equality </li></ul><ul><ul><li>is perceived as lacking </li></ul></ul><ul><ul><li>is not perceived to be a primary need in rural populations </li></ul></ul><ul><li>In CS projects, women may have benefited more than men </li></ul>
    26. 26. Recommendations: programming <ul><li>Think missing links! </li></ul><ul><li>Think hard about motivation of “missing links” </li></ul><ul><ul><li>Differential skills are needed (talking to parliament, then villagers) </li></ul></ul><ul><ul><li>HR for activists? Business support for NGOs? </li></ul></ul><ul><li>Need to demonstrate link between civil society and poverty reduction?? </li></ul><ul><li>Pay more attention to how the poor conceive poverty </li></ul>
    27. 27. Conclusions : OM <ul><li>“ Thinking OM”, e.g. about different groups of boundary partners, makes a dramatic difference </li></ul><ul><li>NGOs enthusiastic!? </li></ul><ul><li>OM for monitoring -> OM for planning </li></ul><ul><li>Focus on contribution is a big relief </li></ul><ul><li>Gives richer feedback </li></ul><ul><li>Strategy maps inspire thinking about different dimensions of planning </li></ul><ul><li>Separate quantitative analysis validates OM: both show progress </li></ul>
    28. 28. Reflection: criticisms of RBM for CS programming <ul><li>RBM: killing the birds of planning, control, monitoring and evaluation with one stone? Data and influence flow upwards. </li></ul><ul><ul><li>Planning: Inspires theories of change which are inaccurate and optimistically mechanistic. Blind to the groups and individuals who implement change. </li></ul></ul><ul><ul><li>Control: gathering data via a chain in which every link has a vested interest in lying </li></ul></ul><ul><ul><li>Monitoring: ticking boxes at the expense of strategic thinking and organisational learning </li></ul></ul><ul><ul><li>Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job? </li></ul></ul>Impacts … outcomes … outputs … Program Project B Project A
    29. 29. Recommendations: OM <ul><li>Better planning: OM could/should be adopted by donors from the project application stage (or as a hybrid). </li></ul><ul><li>Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs. </li></ul><ul><li>Better monitoring: in exchange, require elements of strategic planning and organisational learning. </li></ul><ul><li>Better evaluation: </li></ul><ul><ul><li>Demonstrate donor-relevant results: commission independent, external investigations of society change and why it happens/happened </li></ul></ul><ul><ul><li>Sharpen the focus: help NGOs to do empirical explorations of a selection of specific issues that really interest them </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.