This document discusses the challenges of measuring sustainability, partnerships, and health systems changes. It notes that these concepts are hard to measure due to complicated implementing structures, multi-dimensional problems, abstract concepts, and multiple pathways to outcomes. The document provides some examples of more straightforward metrics like training outcomes and sustainability measures. It also examines issues like assessing related but separate organizations and roles, and determining the contributors to strengthened health systems and population health. Throughout, it emphasizes the need for prioritizing what can reasonably be measured given limitations of feasibility, plausibility, and utility.

1. Assessing sustainability,
partnerships and health systems
changes
Anne Buffardi
THET Partnerships for Global Health
23-25 October 2017, London

2. Dimensions of hard to measure stuff
1. Challenging settings
2. Complicated implementing structures
3. Multi-dimensional problems & abstract concepts
4. Multiple, uncertain pathways to multiple outcomes
(equifinality, multifinality)
Technical & political

3. More straightforward to measure
• Training with specific modules aimed at improving specific skills
sets of a particular group of people
• Sustainability: To what extent did the practice continue once the
project ended? To what extent did the benefits continue?
Follow-up 1,3,5 years after a project closes
Selected institutional relationships over time

4. Complicated implementing structures
Portfolio
Project ProjectProject Project
Org C
Org BOrg A
Programme
• What should (not)
be measured
together?
• How do the roles of
different partners
relate to one
another?

5. Multi-dimensional, abstract concepts – strengthened partnerships

6. Multi-
component
interventions,
multiple
pathways &
outcomes
What factors
contribute to
strengthened
health systems?
To improved
population-level
health?

7. Assessing sustainability, partnerships and health systems changes
• Utility: what are the primary uses and users of information?
• Plausibility: what can reasonably be expected to observe, when?
• Feasibility: what is possible to measure?
Processes of prioritisation, discussion of underlying, more
fundamental questions
www.odi.org/publications/10378-methods-lab-evaluation-toolkit
http://www.betterevaluation.org/

9. ‘Pretending to evaluate comprehensive policy
interventions without telling policymakers
that what they’re asking for can’t be done,
but we’ll take a shot at it anyway and blame
the inadequate results on complexity’
- Michael Quinn Patton 2015, p.8