This document outlines common challenges faced by the UN in developing more impact-oriented monitoring and evaluation systems, and provides potential solutions. It notes that while some UN agencies conduct impact evaluations, many evaluations do not fully address attribution of impacts. It then presents examples of approaches that strengthen causal logic and frameworks to better assess attribution and aggregate evidence of impacts across levels, such as the GEF's Review of Outcome to Impact model and UNDP's meta-reviews of country program evaluations.

1. Towards a more ‘impact-oriented’
institutional M&E system:
common challenges and potential solutions
from a UN perspective
Brighton, March 26-27, 2013
Jos Vaessen (UNESCO)
Oscar Garcia (UNDP)
Juha Uitto (UNDP)

2. Outline
Characteristics of IE practices in the UN
system
Challenges and solutions in developing
more ‘impact-oriented’ M&E systems
Illustrations of (potential) solutions

3. Characteristics of IE practices in
the UN (UNEG, 2009)…
 IEs carried out by only 9 (out of 27) agencies
 Many of these evaluations reported as IEs are not
particularly strong on addressing the attribution
challenge
 Prevalent methodologies: non-experimental
quantitative approaches, theory of change
approaches, (combinations of) qualitative
methods

4. …Characteristics of IE
practices in the UN
 Diversity in the UN system (UNEG, 2012), some
frontrunners
 Most agencies have little funding available for IE
(UNEG, 2013, forthcoming)
 Recently, the number of agencies undertaking IE
(and the number of IEs conducted) appears to
have increased

5. Examples of pathways of
institutionalization of IE
- Conducting high-profile rigorous impact evaluations to
create institutional demand which may lead to
institutionalization of the practice (middle income country
examples; IEG, 2009)
- Mandatory rigorous IE as a prerequisite for continuation /
increase in public funding for programmes (US example:
Epstein and Klerman, 2012; Latin America examples:
Briceño and Gaarder, 2010)
- Strengthening institutional M&E systems towards
becoming more ‘impact-oriented’ and creating the
conditions for low-cost and high value for money impact
evaluation (Kusek and Rist, 2004; this presentation)

6. Challenges of strengthening IE
practices in the UN system
 Demand side constraints (e.g. resources)
 ‘Evaluability’ bias in current IEs
 Many interventions aimed at changes at
institutional level (e.g. policy, normative,
catalytic)
 Multi-actor, multi-stranded, multi-site interventions
 ‘Demand’ for evidence on impact at multiple
levels (triple A challenge; White, 2003)

7. Potential solutions: towards more
‘impact-oriented’ M&E systems
 Improve causal logic (e.g. ToC) at
project/activity level as a framework for data
collection ex ante/interim/ex post
 Develop generic (nested) causal frameworks of
how individual projects/activities fit into larger
programmes
 Develop analytical tools to aggregate/synthesize
impact-related evidence from project/activity to
higher levels

8. Example 1: Review of Outcome to Impact model
in the GEF (ROtI)
1. A theory-based approach as a framework for
assessing linkages between project outcomes
and potential or actual impacts
2. Desk ROtI:
1. Stage 1: Developing project Theory of Change (TOC) models
2. Stage 2: Assessing outcomes-impacts pathways
3. Stage 3: Rating the Project
3. Ratings can be aggregated

9. Example 2: UNDP meta -review
1. Meta review of evaluations of country programmes
(commissioned by EO)
2. 30 evaluations were assessed on the criteria
effectiveness, efficiency, sustainability
3. Aggregate frequency analysis of criteria-specific
ratings
4. QCA: common factors explaining performance
criteria
5. Challenges:
1. Rating is ex post
2. Gaps in evidence available from reports
3. Unit of analysis (country programme)) and comparability

10. Example 3: articulating the nested intervention
logic of UNESCO’s World Heritage Convention
Ratification of convention
n1 countries
Convention reflected in
policy strategies
n2 (< n1) countries
Convention integrated into
concrete interventions
n3 (< n2) countries
Preservation and sustainable
use of World Heritage sites
UNESCO
support
UNESCO
support
UNESCO
support
effects
effects
effects
effects

11. Inventory of cultural /
natural heritage
Inclusion of site in WH
list
ratification of 1972
convention
integration of principles of
convention in national
strategies, policies , legislation
implementation of policy,
legislation and regulations
for WH protection and
conservation (and specific
programs, projects that
relate to the convention)
UNESCO (WH
Centre; WH Fund)
contributes
through:
• capacity-
building,
training and
awareness-
raising
• policy advice
• technical
assistance
• convening
actors and
facilitating
dialogue
• sharing
knowledge and
best practices
Institutional
framework (at
national,
regional, local
level) for WH
protection and
conservation
Awareness raising,
advocacy, training,
education on
protection /
conservation of
WH
Better protected /
conserved WH sites
Increased
knowledge and
awareness about
natural / cultural
heritage
Increase in
economic activity
incl. tourism,
financial
partnerships, etc.
Increase / improvement
Social cohesion,
social capital,
cultural identities
strengthened
Enhanced
conservation and
sustainable use of
natural and cultural
heritage
Economic
livelihoods
strengthened
governance
of natural and
cultural
heritage
strengthened:
• awareness
• capacities
• policy
framework
• regulatory
framework
• institutional
architecture
Context:
• Civil society involvement
• Tourism industry and
potential
• Environmental threats and
pressure on land use
• Climate change
• Migration
• Urbanization
• Etc.
Management
systems and plans
for natural and
cultural heritage
sites (incl. multiple
uses of sites)
Advisory Bodies
ICCROM,
ICOMOS, IUCN
Member States
Research
and
scientific
inquiry
Inter-
national
recognition

12. Example 3 (cont’d): improving M&E systems
 Data collection at four levels in the nested
intervention logic
 Support claims of attribution and aggregation
(magnitude)
 Strengthen learning and accountability through
impact evaluation covering different countries
and assessing causal links at different levels of
the nested intervention logic

13. Concluding remarks
 Higher gains in more ‘impact-oriented’ M&E
systems rather than ‘simply’ doing more IE
 Focus on institutional change level (attribution
and ‘measurement’)
 Addressing the aggregation challenge:
 Portfolio-wide synthetic analysis (review) of intervention types
 Strengthened (nested) intervention logics
 Systematic data collection on implementation and delivery aligned
to higher-level evaluative exercises (including IE) to analyze
processes of change