Outcome Mapping BOND Quality Group 11 December 2008 London Simon Hearn (s.hearn@odi.org.uk) ODI, London
Introduce Outcome Mapping Present examples Present the virtual learning community
Origins of Outcome Mapping 1990’s:  IDRC had a need to demonstrate results 1998:  Discovered  Outcome Engineering  (Barry Kibel) – used to plan and measure the results of drug rehabilitation 2000:  publication of the OM manual in English Global training & using OM globally 2006 : www.outcomemapping.ca
 
Focus of Outcome Mapping Beginning Life cycle of the  program End During Objectives Inputs  Activities Results Outputs Impact
Focus of Outcome Mapping Sphere  of  Control Sphere  of  Influence Sphere  of  Interest LFA focus RBM focus OM focus Inputs Activities Outputs Outcomes Impacts
Focus of Outcome Mapping Outcome Mapping Local partners / beneficiaries ownership increases Program influence decreases Inputs Activities Outputs Outcomes Impacts
The Problem with  Impact Impact implies… The reality is… Cause & effect Open system Positive, intended results Unexpected positive & negative results occur Focus on ultimate effects Upstream effects are important Credit goes to a single contributor Multiple actors create results & need credit Story ends when program obtains success Change process never ends
What is Outcome Mapping? A method for planning and assessing the social effects and internal performance of projects, programs, and organisations
 
Useful M&E Depends On Learning & Reporting (Info to users/events/audiences) Clarifying Intent (Both interventions & outcomes) M&E Planning (Choosing what to track) M&E Implementation (Data collection & interpretation)
4 Key Planning Questions Vision Boundary Partners Outcomes Challenges, Progress Markers Mission, Strategy Map, Organizational Practices Why? Who? What? How?
Key ideas in Outcome Mapping
Look at the bigger picture See yourself as a part of an interconnected web of actors, factors and relationships
Constant change “ It’s not possible to see the same river twice”
Recognize that change is… Continuous  Complex Non-linear Multidirectional Not controllable
Keep your eyes wide open “ Being attentive along the journey is as important as arriving at the destination” Michael Quinn Patton
In the face of this complexity: How can we increase our knowledge of the processes we engage in? How can we know if we made a difference? How can we understand social change? How can we recognize contributors and share the credit?
Contribution, not Attribution Change processes involve interactions among multiple actors and factors
Influence, not Impact You can influence but not control change in your partners
Methodologically, what does OM add? Outcomes as behaviour changes Definition of system boundaries Focus on actors who will drive & maintain  change Sets out a pathway of change with markers
The Limits of Influence Programme Sphere of  Control Sphere of  Influence Sphere of  Interest Beneficiaries Stakeholders Boundary Partners
Boundary Partners Those individuals, groups, or organisations with whom the program Interacts directly to effect change Anticipates opportunities for influence Engages in mutual learning
Behaviour Change Behaviour is defined as… Actions Activities Interactions Relationships … of your Boundary Partners
Why Outcomes as Behaviour Change? Measurable and observable Sympathetic to the difference between influence and control,  attribution  and  contribution Recognises the limits of the programme’s influence
Progress markers Expanding influence, helping others, sharing expertise Actively engaged, learning,  commitment   Early encouraging response to program, initial engagement Love to see Like to see Expect to see
Progress Markers move from initial to more profound changes in behaviour show transformation in a single boundary  partner more informative than a  single indicator
Primary Uses PLANNING   Articulate goals & define activities MONITORING Track program performance & partners’ progress EVALUATION Design & conduct a use-oriented evaluation
Other uses SYSTEMISATION Rebuild the story or parts of the intervention or from the partners’ perspective EXTERNAL EVALUATION Existent or re-built intentional design as the base of a traditional evaluation COMBINED WITH OTHER METHODS According to M&E requirements
Principles of use Flexible:  modular to  be adapted to use & context  Complementary : combine  with other methods
Participatory : seeks dialogue and collaboration with partners Evaluative : promotes culture of reflection, results oriented thinking, and social & organizational learning Principles of use
Where is the map? OM is a guide to the journey we take with our partners. We co-create the map. It focuses on the intention and what happens along the way The map is not the territory, it shows the route taken
A few examples…
St 2 eep Secondary teacher training in environmental education Zimbabwe, from 2003 Partnership between Ministry of Higher Education, VVOB, Secondary Teacher Training Colleges
St 2 eep Initially using a Log Frame Did not address the programme’s theory of change Informal self-assessment and monitoring Donor driven M&E divorced from programme planning Limited learning Adopted OM for second phase Integrated M&E into project planning to ensure learning Involved partners in planning process
St 2 eep Factors that helped uptake of OM Realisation of the problem Very applicable manual and OM framework OM worked with culture of St 2 eep programme Core team to push through new ideas Support from VVOB, IDRC and other OM users
VECO Belgian NGO 13 countries Food security, sustainable agriculture Policy advocacy and networking
VECO Problems with logical framework Strong emphasis on quantitative indicators Results and reporting focus of M&E Limited learning and mutual accountability
VECO New system New strategic plan gave opportunity to embed an OM approach in its M&E Planning, Learning and Accountability Tested in Indonesia first Problems Donor required LFA Developed a ‘bricolage’ framework between OM and LFA OM for internal planning LFA for reporting
VECO
VECO Benefits of OM OM facilitates critical reflection Greater variety of partners Wider scope of strategies to build capacity Assessment of internal processes – CO’s were BPs of Head office
Further Information Visit the OM learning community: www.outcomemapping.ca Contact: Simon Hearn [email_address]

OM Introduction

  • 1.
    Outcome Mapping BONDQuality Group 11 December 2008 London Simon Hearn (s.hearn@odi.org.uk) ODI, London
  • 2.
    Introduce Outcome MappingPresent examples Present the virtual learning community
  • 3.
    Origins of OutcomeMapping 1990’s: IDRC had a need to demonstrate results 1998: Discovered Outcome Engineering (Barry Kibel) – used to plan and measure the results of drug rehabilitation 2000: publication of the OM manual in English Global training & using OM globally 2006 : www.outcomemapping.ca
  • 4.
  • 5.
    Focus of OutcomeMapping Beginning Life cycle of the program End During Objectives Inputs Activities Results Outputs Impact
  • 6.
    Focus of OutcomeMapping Sphere of Control Sphere of Influence Sphere of Interest LFA focus RBM focus OM focus Inputs Activities Outputs Outcomes Impacts
  • 7.
    Focus of OutcomeMapping Outcome Mapping Local partners / beneficiaries ownership increases Program influence decreases Inputs Activities Outputs Outcomes Impacts
  • 8.
    The Problem with Impact Impact implies… The reality is… Cause & effect Open system Positive, intended results Unexpected positive & negative results occur Focus on ultimate effects Upstream effects are important Credit goes to a single contributor Multiple actors create results & need credit Story ends when program obtains success Change process never ends
  • 9.
    What is OutcomeMapping? A method for planning and assessing the social effects and internal performance of projects, programs, and organisations
  • 10.
  • 11.
    Useful M&E DependsOn Learning & Reporting (Info to users/events/audiences) Clarifying Intent (Both interventions & outcomes) M&E Planning (Choosing what to track) M&E Implementation (Data collection & interpretation)
  • 12.
    4 Key PlanningQuestions Vision Boundary Partners Outcomes Challenges, Progress Markers Mission, Strategy Map, Organizational Practices Why? Who? What? How?
  • 13.
    Key ideas inOutcome Mapping
  • 14.
    Look at thebigger picture See yourself as a part of an interconnected web of actors, factors and relationships
  • 15.
    Constant change “It’s not possible to see the same river twice”
  • 16.
    Recognize that changeis… Continuous Complex Non-linear Multidirectional Not controllable
  • 17.
    Keep your eyeswide open “ Being attentive along the journey is as important as arriving at the destination” Michael Quinn Patton
  • 18.
    In the faceof this complexity: How can we increase our knowledge of the processes we engage in? How can we know if we made a difference? How can we understand social change? How can we recognize contributors and share the credit?
  • 19.
    Contribution, not AttributionChange processes involve interactions among multiple actors and factors
  • 20.
    Influence, not ImpactYou can influence but not control change in your partners
  • 21.
    Methodologically, what doesOM add? Outcomes as behaviour changes Definition of system boundaries Focus on actors who will drive & maintain change Sets out a pathway of change with markers
  • 22.
    The Limits ofInfluence Programme Sphere of Control Sphere of Influence Sphere of Interest Beneficiaries Stakeholders Boundary Partners
  • 23.
    Boundary Partners Thoseindividuals, groups, or organisations with whom the program Interacts directly to effect change Anticipates opportunities for influence Engages in mutual learning
  • 24.
    Behaviour Change Behaviouris defined as… Actions Activities Interactions Relationships … of your Boundary Partners
  • 25.
    Why Outcomes asBehaviour Change? Measurable and observable Sympathetic to the difference between influence and control, attribution and contribution Recognises the limits of the programme’s influence
  • 26.
    Progress markers Expandinginfluence, helping others, sharing expertise Actively engaged, learning, commitment Early encouraging response to program, initial engagement Love to see Like to see Expect to see
  • 27.
    Progress Markers movefrom initial to more profound changes in behaviour show transformation in a single boundary partner more informative than a single indicator
  • 28.
    Primary Uses PLANNING Articulate goals & define activities MONITORING Track program performance & partners’ progress EVALUATION Design & conduct a use-oriented evaluation
  • 29.
    Other uses SYSTEMISATIONRebuild the story or parts of the intervention or from the partners’ perspective EXTERNAL EVALUATION Existent or re-built intentional design as the base of a traditional evaluation COMBINED WITH OTHER METHODS According to M&E requirements
  • 30.
    Principles of useFlexible: modular to be adapted to use & context Complementary : combine with other methods
  • 31.
    Participatory : seeksdialogue and collaboration with partners Evaluative : promotes culture of reflection, results oriented thinking, and social & organizational learning Principles of use
  • 32.
    Where is themap? OM is a guide to the journey we take with our partners. We co-create the map. It focuses on the intention and what happens along the way The map is not the territory, it shows the route taken
  • 33.
  • 34.
    St 2 eepSecondary teacher training in environmental education Zimbabwe, from 2003 Partnership between Ministry of Higher Education, VVOB, Secondary Teacher Training Colleges
  • 35.
    St 2 eepInitially using a Log Frame Did not address the programme’s theory of change Informal self-assessment and monitoring Donor driven M&E divorced from programme planning Limited learning Adopted OM for second phase Integrated M&E into project planning to ensure learning Involved partners in planning process
  • 36.
    St 2 eepFactors that helped uptake of OM Realisation of the problem Very applicable manual and OM framework OM worked with culture of St 2 eep programme Core team to push through new ideas Support from VVOB, IDRC and other OM users
  • 37.
    VECO Belgian NGO13 countries Food security, sustainable agriculture Policy advocacy and networking
  • 38.
    VECO Problems withlogical framework Strong emphasis on quantitative indicators Results and reporting focus of M&E Limited learning and mutual accountability
  • 39.
    VECO New systemNew strategic plan gave opportunity to embed an OM approach in its M&E Planning, Learning and Accountability Tested in Indonesia first Problems Donor required LFA Developed a ‘bricolage’ framework between OM and LFA OM for internal planning LFA for reporting
  • 40.
  • 41.
    VECO Benefits ofOM OM facilitates critical reflection Greater variety of partners Wider scope of strategies to build capacity Assessment of internal processes – CO’s were BPs of Head office
  • 42.
    Further Information Visitthe OM learning community: www.outcomemapping.ca Contact: Simon Hearn [email_address]

Editor's Notes

  • #2 Many slides originate from members of the OM community: www.outcomemapping.ca and shared freely. Thanks to Terry Smutylo for sharing many of his slides.
  • #3 BvLF Workshop. the Hague. Nov 2007
  • #4 OM developed by IDRC in Canada They needed to demonstrate results In the late 90’s they came across a tool called Outcome Engineering which (Barry Kibel) was being used to plan and measure drug rehabilitation, from the perspective of behaviour changes as outcomes Over the next few years, the Evaluation Unit adapted this approach, piloted this methodology in several development projects. In 2000, they released the OM manual Since then, use of OM has been supported globally through trainings and through the virtual learning community – more later
  • #5 The problem IDRC were trying to address with OM was this: a development intervention starts with an input and ends with an output but the processes that turn the input into the output were either being ignored or weren’t understood.
  • #6 Another way to think about this is that at the beginning of a project you have objectives, inputs and activities At the end you have results, outputs and impact And there’s a process to get from one to the other – but this is often a black box, what goes on here is a mystery Outcome Mapping aims to de-mystify this black box
  • #7 We can think about the environment which is project exists in from three perspectives… This is what we call the nested spheres diagram, Sphere of control = operational environment DIRECT CONTROL Sphere of Interest = part of the world that the programme is interested in. beneficiaries, communities. Changes in socio environmental state & conditions INDIRET INFLUENCE Sphere of Influence = Relationships & Interactions DIRECT INFLUENCE And if we look at our results chain, we can see that the different elements fall into different spheres As far as methods and tools are concerned, LFA focusses on SOC, RBM on SOI, OM on SOI
  • #8 Important to focus on outcomes… Taking again our simplified results chain, we can map the ownership of our local partners or beneficiaries. We see that this increases. At the level of inputs, our partners don’t have a lot of say – decisions about money, who to work with and where to work are largely made by the program or even the donor. But at the impact level, it all depends on the partners and the benficiaries. Outcome mapping is focussed at the level of outcomes. This is because the partner ownership is high enough to be able to see an effect, some observe changes and the programme influence is high enough to be able to say with some confidence that those changes originated from the intervention.
  • #9 conclusion: « impact » is a highly politicized concept in development. OM focuses on outcomes not impact there are other methods to do impact assessment at OM African Users Workshop in Niamey, January 2007: OM not only about P,M&E but about the way you conceptulize development
  • #10 Main Messages In a results’ chain OM helps you think about the « outcomes » section This image is useful because just like geographic mapping, conceptual mapping also needs tools. OM helps by providing some tools for this. Workshop talking about the program level Definition of Program: A group of related projects and activities with a specific set of resources (human, financial, capital) directed to the achievement of a set of goals within a specified period of time Can be used by projects, organizations, and communities too
  • #11 Planning: What are we trying to accomplish? Why? How? Who? Monitoring: What do we want to know? Evaluation: What do we want to learn?
  • #12 Appendix 2: Workshop slides
  • #13 How does OM respond to these? Not go through these now – three day workshop
  • #15 an approach for designing M&E in relation to the broader development context but assessment is within your sphere of influence.
  • #16 - The river is always change; change is always there – pay attention and you will see it
  • #17 Complex (multiple actors and factors) Continuous (not limited to the life of the project) Non-linear (unexpected results occur) Beyond the control of the project (but subject to its influence) Two-way (program also changes)
  • #18 The journey may be more important than the destination OM helps you pay attention to the journey on the way to the destination Need to give them equal weight In OM the journey is part of the destination - different from other evaluation methods
  • #19 In the face of such complexity: How can we show that we have made a difference with our research / project? Proving causality is difficult, especially when there are several factors and actors at work How can we reduce the unknowns regarding our contribution? How can we share the credit?
  • #27 Not a lock step progression Indicate DEPTH of change; signals that a transformation is taking place Milestones Spectrum of transformation
  • #28 Important thing: see movement
  • #29 other uses: individual transformation process: leadership development for executives in NGOs in Canada framework for social research UHIN PRA tool for participatory development work with women`s self help groups in India Point out the 7 planning steps as intro to next, step-by-step, sections. Tell short story about Nagaland?
  • #32 Appendix 2: Workshop slides
  • #33 Quick check-in plenary question: how do you use the term “outcome” in your context? OM helps us map and monitor relationships A road map facilitates being clear & specific about where you are going and by what route; and helps track progress. OM does the same: clarity re intent & the mean to be used; milestones to track progress; facilitates modifications in response to what emerges along the way.
  • #38 Africa, Asia and South-America BvLF Workshop. the Hague. Nov 2007