Towards a more ‘impact-oriented’
institutional M&E system:
common challenges and potential solutions
from a UN perspective...
Outline
Characteristics of IE practices in the UN
system
Challenges and solutions in developing
more ‘impact-oriented’ M...
Characteristics of IE practices in
the UN (UNEG, 2009)…
 IEs carried out by only 9 (out of 27) agencies
 Many of these e...
…Characteristics of IE
practices in the UN
 Diversity in the UN system (UNEG, 2012), some
frontrunners
 Most agencies ha...
Examples of pathways of
institutionalization of IE
- Conducting high-profile rigorous impact evaluations to
create institu...
Challenges of strengthening IE
practices in the UN system
 Demand side constraints (e.g. resources)
 ‘Evaluability’ bias...
Potential solutions: towards more
‘impact-oriented’ M&E systems
 Improve causal logic (e.g. ToC) at
project/activity leve...
Example 1: Review of Outcome to Impact model
in the GEF (ROtI)
1. A theory-based approach as a framework for
assessing lin...
Example 2: UNDP meta -review
1. Meta review of evaluations of country programmes
(commissioned by EO)
2. 30 evaluations we...
Example 3: articulating the nested intervention
logic of UNESCO’s World Heritage Convention
Ratification of convention
n1 ...
Inventory of cultural /
natural heritage
Inclusion of site in WH
list
ratification of 1972
convention
integration of princ...
Example 3 (cont’d): improving M&E systems
 Data collection at four levels in the nested
intervention logic
 Support clai...
Concluding remarks
 Higher gains in more ‘impact-oriented’ M&E
systems rather than ‘simply’ doing more IE
 Focus on inst...
Upcoming SlideShare
Loading in …5
×

Towards a more ‘impact-oriented’ institutional M&E system:

891 views
780 views

Published on

Towards a more ‘impact-oriented’ institutional M&E system: Presentation at IDS Impact, Innovation and Learning Conference, March 2013

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
891
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Towards a more ‘impact-oriented’ institutional M&E system:

  1. 1. Towards a more ‘impact-oriented’ institutional M&E system: common challenges and potential solutions from a UN perspective Brighton, March 26-27, 2013 Jos Vaessen (UNESCO) Oscar Garcia (UNDP) Juha Uitto (UNDP)
  2. 2. Outline Characteristics of IE practices in the UN system Challenges and solutions in developing more ‘impact-oriented’ M&E systems Illustrations of (potential) solutions
  3. 3. Characteristics of IE practices in the UN (UNEG, 2009)…  IEs carried out by only 9 (out of 27) agencies  Many of these evaluations reported as IEs are not particularly strong on addressing the attribution challenge  Prevalent methodologies: non-experimental quantitative approaches, theory of change approaches, (combinations of) qualitative methods
  4. 4. …Characteristics of IE practices in the UN  Diversity in the UN system (UNEG, 2012), some frontrunners  Most agencies have little funding available for IE (UNEG, 2013, forthcoming)  Recently, the number of agencies undertaking IE (and the number of IEs conducted) appears to have increased
  5. 5. Examples of pathways of institutionalization of IE - Conducting high-profile rigorous impact evaluations to create institutional demand which may lead to institutionalization of the practice (middle income country examples; IEG, 2009) - Mandatory rigorous IE as a prerequisite for continuation / increase in public funding for programmes (US example: Epstein and Klerman, 2012; Latin America examples: Briceño and Gaarder, 2010) - Strengthening institutional M&E systems towards becoming more ‘impact-oriented’ and creating the conditions for low-cost and high value for money impact evaluation (Kusek and Rist, 2004; this presentation)
  6. 6. Challenges of strengthening IE practices in the UN system  Demand side constraints (e.g. resources)  ‘Evaluability’ bias in current IEs  Many interventions aimed at changes at institutional level (e.g. policy, normative, catalytic)  Multi-actor, multi-stranded, multi-site interventions  ‘Demand’ for evidence on impact at multiple levels (triple A challenge; White, 2003)
  7. 7. Potential solutions: towards more ‘impact-oriented’ M&E systems  Improve causal logic (e.g. ToC) at project/activity level as a framework for data collection ex ante/interim/ex post  Develop generic (nested) causal frameworks of how individual projects/activities fit into larger programmes  Develop analytical tools to aggregate/synthesize impact-related evidence from project/activity to higher levels
  8. 8. Example 1: Review of Outcome to Impact model in the GEF (ROtI) 1. A theory-based approach as a framework for assessing linkages between project outcomes and potential or actual impacts 2. Desk ROtI: 1. Stage 1: Developing project Theory of Change (TOC) models 2. Stage 2: Assessing outcomes-impacts pathways 3. Stage 3: Rating the Project 3. Ratings can be aggregated
  9. 9. Example 2: UNDP meta -review 1. Meta review of evaluations of country programmes (commissioned by EO) 2. 30 evaluations were assessed on the criteria effectiveness, efficiency, sustainability 3. Aggregate frequency analysis of criteria-specific ratings 4. QCA: common factors explaining performance criteria 5. Challenges: 1. Rating is ex post 2. Gaps in evidence available from reports 3. Unit of analysis (country programme)) and comparability
  10. 10. Example 3: articulating the nested intervention logic of UNESCO’s World Heritage Convention Ratification of convention n1 countries Convention reflected in policy strategies n2 (< n1) countries Convention integrated into concrete interventions n3 (< n2) countries Preservation and sustainable use of World Heritage sites UNESCO support UNESCO support UNESCO support effects effects effects effects
  11. 11. Inventory of cultural / natural heritage Inclusion of site in WH list ratification of 1972 convention integration of principles of convention in national strategies, policies , legislation implementation of policy, legislation and regulations for WH protection and conservation (and specific programs, projects that relate to the convention) UNESCO (WH Centre; WH Fund) contributes through: • capacity- building, training and awareness- raising • policy advice • technical assistance • convening actors and facilitating dialogue • sharing knowledge and best practices Institutional framework (at national, regional, local level) for WH protection and conservation Awareness raising, advocacy, training, education on protection / conservation of WH Better protected / conserved WH sites Increased knowledge and awareness about natural / cultural heritage Increase in economic activity incl. tourism, financial partnerships, etc. Increase / improvement Social cohesion, social capital, cultural identities strengthened Enhanced conservation and sustainable use of natural and cultural heritage Economic livelihoods strengthened governance of natural and cultural heritage strengthened: • awareness • capacities • policy framework • regulatory framework • institutional architecture Context: • Civil society involvement • Tourism industry and potential • Environmental threats and pressure on land use • Climate change • Migration • Urbanization • Etc. Management systems and plans for natural and cultural heritage sites (incl. multiple uses of sites) Advisory Bodies ICCROM, ICOMOS, IUCN Member States Research and scientific inquiry Inter- national recognition
  12. 12. Example 3 (cont’d): improving M&E systems  Data collection at four levels in the nested intervention logic  Support claims of attribution and aggregation (magnitude)  Strengthen learning and accountability through impact evaluation covering different countries and assessing causal links at different levels of the nested intervention logic
  13. 13. Concluding remarks  Higher gains in more ‘impact-oriented’ M&E systems rather than ‘simply’ doing more IE  Focus on institutional change level (attribution and ‘measurement’)  Addressing the aggregation challenge:  Portfolio-wide synthetic analysis (review) of intervention types  Strengthened (nested) intervention logics  Systematic data collection on implementation and delivery aligned to higher-level evaluative exercises (including IE) to analyze processes of change

×