Assessment of Outcome Mapping as a tool for evaluating and monitoring support to civil society organisations EASY-ECO Vienna 2008 Stephen Powell, Joakim Molander & Ivona Čelebičić proMENTE social research, Sarajevo

Who are we Steve Powell & Ivona Čelebičić proMENTE social research, Sarajevo Joakim Molander: at the time of the evaluation, First Secretary at the Swedish Embassy, Sarajevo Report: www.promente.org/sida2eng

Steve Powell & Ivona Čelebičić proMENTE social research, Sarajevo

Joakim Molander: at the time of the evaluation, First Secretary at the Swedish Embassy, Sarajevo

Report: www.promente.org/sida2eng

Background Govt. in B&H barely functional Civil society fills the gap?  Support people and networks How to evaluate?  OM

Govt. in B&H barely functional

Civil society fills the gap?

 Support people and networks

How to evaluate?

 OM

Evaluation TOR Explorative evaluation of six civil society projects Sida-funded civil society programming: lessons on sustainability Exploration of OM as a tool

Explorative evaluation of six civil society projects

Sida-funded civil society programming: lessons on sustainability

Exploration of OM as a tool

Messages OM worked well for civil society evaluation OM plays well with other approaches Sustainability ↔ focus on “key players” OM can be part of a patchwork , rather than monolithic , approach to M&E

OM worked well for civil society evaluation

OM plays well with other approaches

Sustainability ↔ focus on “key players”

OM can be part of a patchwork , rather than monolithic , approach to M&E

Our project in B&H: 1 donor, 3 framework partners, 6 implementing partners, 6 projects: 2007 "Improvement of access to justice – Raising public awareness on access to rights, legislative changes and legal procedures in BiH Vasa Prava Work with judges and journalists on Press Code BiH Press Council (funded by SHC) Swedish Helsinki Committee for Human Rights "Citizen in action"– Project of community-based advocacy campaigns for solving priority citizens problems from "Civic Platform for 2006 Elections". "Local Government leadership building activities" Civil Society Promotion Center – GROZD, "Citizen in action" Academy for political leaders Helsinki Citizens Assembly (HCA) – Academy for political leaders) Olof Palme International Center Village activities (including round tables in towns) Most Womens political lobby Zenski Centar Kvinna til Kvinna Evaluated activity Partner organisation Framework organisation

OM: 12 steps Helping the implementing partner to learn Internal M &E Considering all the dimensions of strategy Focus on outcomes 11. Performance journal 7. Organisational practices Evaluation Outcome & performance monitoring Intentional design 10. Strategy journal 6. …Strategy maps 12. Evaluation plan 8. Monitoring priorities 9. Outcome journals 5. ...Progress markers 4. ...Outcome challenges 3. ...Boundary partners 2. Mission 1. Vision

Method: timeframe Baseline Final assessment

Method: OM Special use of OM for external evaluation Mission, vision, progress markers, outcome challenge Assessment of planning strategies using strategy maps OM questionnaires OM interview with boundary partners (+ relevance interviews ) OM interview with implementing partners (+ relevance interviews ) Writing evaluation-start Outcome Journals Confirming evaluation-start Outcome Journals OM questionnaires (same as evaluation-start) OM interview with boundary partners (on the basis of evaluation-start interview) (+ relevance interviews ) OM interview with implementing partners (on the basis of evaluation-start interview) (+ relevance interviews ) Assessment of management progress using existing strategy maps Mission and vision: still relevant? Changed focus? Gathering contextual information and intervention timeline Baseline interim final assessment

Special use of OM for external evaluation

Mission, vision, progress markers, outcome challenge

Assessment of planning strategies using strategy maps

OM questionnaires

OM interview with boundary partners (+ relevance interviews )

OM interview with implementing partners (+ relevance interviews )

Writing evaluation-start Outcome Journals

Confirming evaluation-start Outcome Journals

OM questionnaires (same as evaluation-start)

OM interview with boundary partners (on the basis of evaluation-start interview) (+ relevance interviews )

OM interview with implementing partners (on the basis of evaluation-start interview) (+ relevance interviews )

Assessment of management progress using existing strategy maps

Mission and vision: still relevant? Changed focus?

Gathering contextual information and intervention timeline

Message 1: OM worked well with civil society evaluation Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision NGOs enthusiastic!? Focus on contribution is a big relief Gives richer feedback Strategy maps inspire and organise thinking about different dimensions of planning OM standard method and questionnaires, (and interviews?) showed projects making progress towards their vision Consider problem-based rather than vision-based programming. Vision is not always about boundary partner change

Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision

NGOs enthusiastic!?

Focus on contribution is a big relief

Gives richer feedback

Strategy maps inspire and organise thinking about different dimensions of planning

OM standard method and questionnaires, (and interviews?) showed projects making progress towards their vision

Consider problem-based rather than vision-based programming. Vision is not always about boundary partner change

Message 2: OM plays well with other approaches

2a: Qualitative methods Gained a lot of additional information with systematic content analysis of independent “relevance interviews” “ A good part of the population is used to some sort of humanitarian aid, some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything. ”

Gained a lot of additional information with systematic content analysis of independent “relevance interviews”

“ A good part of the population is used to some sort of humanitarian aid, some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything. ”

2b: quantitative analysis For four projects, a customised questionnaire was made on the basis of progress markers. Questionnaire filled in at baseline and again at final assessment Results from around 100 baseline and 100 final assessment questionnaires compared on a per-project basis. Validation of OM evaluation methods? Less clear progress with the least educated

For four projects, a customised questionnaire was made on the basis of progress markers.

Questionnaire filled in at baseline and again at final assessment

Results from around 100 baseline and 100 final assessment questionnaires compared on a per-project basis.

Validation of OM evaluation methods?

Less clear progress with the least educated

2b: quantitative analysis Results for one project: average scores on progress marker questionnaires  at baseline  at evaluation end Small but significant overall improvement Corresponds to information from OM interviews & journals Women start lower than men and improve more than men In spite of a possible tendency to “raise the bar” during the project  “soft” interpretation male female

Results for one project: average scores on progress marker questionnaires

 at baseline

 at evaluation end

Small but significant overall improvement

Corresponds to information from OM interviews & journals

Women start lower than men and improve more than men

In spite of a possible tendency to “raise the bar” during the project  “soft” interpretation

2c: implementing partners and boundary partners do not agree on “difficulty” of progress markers  involve boundary partners in planning!! – especially the poor Love to see / like to see / expect to see Average baseline score for one project

 involve boundary partners in planning!! – especially the poor

2d: problems with OM + RBM Avoiding a double workload: OM for monitoring  OM for planning and reporting Better planning: OM could/should be adopted by donors from the project application stage (or as a hybrid) Donors have to want to help organisations to learn, at the cost of demonstrating effectiveness

Avoiding a double workload:

OM for monitoring  OM for planning and reporting

Better planning: OM could/should be adopted by donors from the project application stage (or as a hybrid)

Donors have to want to help organisations to learn, at the cost of demonstrating effectiveness

Message 3: Sustainability ↔ focus on real people!

3a: differentiate your boundary partners Especially in civil society, b oundary partner groups can be usefully divided into "less powerful", "potentially powerful" and "powerful". Even within one group, d istinguish clearly between partners who are at different levels of development towards , or agreement with , the outcome challenge .

Especially in civil society, b oundary partner groups can be usefully divided into "less powerful", "potentially powerful" and "powerful".

Even within one group, d istinguish clearly between partners who are at different levels of development towards , or agreement with , the outcome challenge .

3b: focus on “key players” = boundary partners or implementing partners?? The y believe that change is possible through the efforts of individuals. Civil society not a mass movement! H andful of key ( “career” ) activists and organisations They have “first name and family name” and are not interchangeable or easily replaceable; yet invisible on paper

= boundary partners or implementing partners??

The y believe that change is possible through the efforts of individuals.

Civil society not a mass movement! H andful of key ( “career” ) activists and organisations

They have “first name and family name” and are not interchangeable or easily replaceable; yet invisible on paper

3c: A taboo: motivation Why would these people want to play along? Money a taboo? If we aren’t using sticks, do we have juicy carrots? Careers advice for activists? Business support for NGOs?

Why would these people want to play along? Money a taboo? If we aren’t using sticks, do we have juicy carrots?

Careers advice for activists? Business support for NGOs?

3d: Another taboo: personality E ffective activists are not necessarily effective bureaucrats (Easterly) . Should they be? Personality is hard to change Differential skills are needed (talking to parliament, then villagers)

E ffective activists are not necessarily effective bureaucrats (Easterly) . Should they be?

Personality is hard to change Differential skills are needed (talking to parliament, then villagers)

Message 4: OM can be part of a patchwork , rather than monolithic , approach to M&E

4a: the monolithic approach RBM: killing the birds of project and program planning, control, monitoring and evaluation with one stone? Impacts … outcomes … outputs … Causation Data aggregation Control: gathering data via a chain in which every link has a vested interest in lying Monitoring: ticking boxes at the expense of strategic thinking and organisational learning Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job? Program Project B Project A

RBM: killing the birds of project and program planning, control, monitoring and evaluation with one stone?

Control: gathering data via a chain in which every link has a vested interest in lying

Monitoring: ticking boxes at the expense of strategic thinking and organisational learning

Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job?

4b: OM is only part of a (patchwork) solution Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs. Better planning and monitoring: use OM where appropriate Better management : in exchange, require elements of strategic planning and organisational learning. Better evaluation: Demonstrate donor-relevant impact: commission independent, external investigations of society change and why it happens/happened Sharpen the focus: help NGOs to do empirical explorations of a selection of specific issues that really interest them OM isn’t everything

Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs.

Better planning and monitoring: use OM where appropriate

Better management : in exchange, require elements of strategic planning and organisational learning.

Better evaluation:

Demonstrate donor-relevant impact: commission independent, external investigations of society change and why it happens/happened

Sharpen the focus: help NGOs to do empirical explorations of a selection of specific issues that really interest them

OM isn’t everything