Measuring policy influence:
   like measuring thin air?
             John Young: j.young@odi.org.uk
       Arnaldo Pellini: a.pellini.ra@odi.org.uk
RAPID



 Power, Politics and evidence use




Knowledge intermediaries
and interactions                    Evidence production and
                                    communication
Policy processes are...

                                                   Cabinet
   Donors                          Policy
                                Formulation

                    Agenda                                   Parliament
                    Setting
                                                  Decision
                                                  Making
Civil Society


                    Monitoring and                        Ministries
                     Evaluation               Policy
                                         Implementation
          Private
          Sector
The Cynefn Framework




4
Policy change

    • Discursive: Client-
      focused services
    • Attitudinal: Farmers
      have good ideas
    • Procedural: Participatory
      approaches to service development
    • Content: UU20, UU25. New guidelines
    • Behavioural: Approach being applied in
      practice

5
Focus on behaviour change


Project
Team




Other
Actors
     Inputs   Activities   Outputs   Outcomes
                                      Outcome
                                      Outcomes
                                                  Impact
                                                 Impact
                                                 Impact




                                     Behaviour
 6                                    Change
RAPID Outcome Mapping Approach

                  Media strategy

                  Online
                  communications

                  Develop a network
                  or partnership

                  Academic research
                  communications

                  Policy advocacy
                  coalition


                  More research
Why do M&E?

• To learn about what works
• To manage better
• To account:
   – to donors
   – to recipients




8
Methods

• Classical case studies (IDRC, IFPRI)
• Episode studies (ODI/RAPID)
• Stories of Change (Denning)
• Micro-Narratives (Snowden)
• Impact matrices (Davies)
• Peer evaluations (CHSRF)
• Systematic reviews?
• RCTs?


9
Outcome Mapping

                          OUTCOME
                          MAPPING:
                          Building
                          Learning and
                          Reflection into
                          Development
                          Programs
                          Sarah Earl, Fred
                          Carden, and Terry
                          Smutylo
     http://www.idrc.ca/en/ev-9330-201-1-
     DO_TOPIC.html



10
Social Network Analysis




11
RAPID Outcome Assessment




12
A systematic approach
1. Strategy and direction –are you
   doing the right thing?
2. Management –are you doing what
   you planned to do?
3. Outputs – are the outputs
   appropriate for the audience?
4. Uptake – are people aware of your
   work?
5. Outcomes and impacts –are you
   having any impact?



13
M&E in ODI
Impact Logs & SoC




15
3ie PIM Project Evaluation Framework




16
TTI Evaluation Framework

Monitoring & Evaluation - the RAPID approach

  • 1.
    Measuring policy influence: like measuring thin air? John Young: j.young@odi.org.uk Arnaldo Pellini: a.pellini.ra@odi.org.uk
  • 2.
    RAPID Power, Politicsand evidence use Knowledge intermediaries and interactions Evidence production and communication
  • 3.
    Policy processes are... Cabinet Donors Policy Formulation Agenda Parliament Setting Decision Making Civil Society Monitoring and Ministries Evaluation Policy Implementation Private Sector
  • 4.
  • 5.
    Policy change • Discursive: Client- focused services • Attitudinal: Farmers have good ideas • Procedural: Participatory approaches to service development • Content: UU20, UU25. New guidelines • Behavioural: Approach being applied in practice 5
  • 6.
    Focus on behaviourchange Project Team Other Actors Inputs Activities Outputs Outcomes Outcome Outcomes Impact Impact Impact Behaviour 6 Change
  • 7.
    RAPID Outcome MappingApproach Media strategy Online communications Develop a network or partnership Academic research communications Policy advocacy coalition More research
  • 8.
    Why do M&E? •To learn about what works • To manage better • To account: – to donors – to recipients 8
  • 9.
    Methods • Classical casestudies (IDRC, IFPRI) • Episode studies (ODI/RAPID) • Stories of Change (Denning) • Micro-Narratives (Snowden) • Impact matrices (Davies) • Peer evaluations (CHSRF) • Systematic reviews? • RCTs? 9
  • 10.
    Outcome Mapping OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo http://www.idrc.ca/en/ev-9330-201-1- DO_TOPIC.html 10
  • 11.
  • 12.
  • 13.
    A systematic approach 1.Strategy and direction –are you doing the right thing? 2. Management –are you doing what you planned to do? 3. Outputs – are the outputs appropriate for the audience? 4. Uptake – are people aware of your work? 5. Outcomes and impacts –are you having any impact? 13
  • 14.
  • 15.
  • 16.
    3ie PIM ProjectEvaluation Framework 16
  • 17.