This document discusses measuring the performance of capacity development efforts. It presents a theory of change model showing how capacity development activities can strengthen organizations, improve their performance, and ultimately impact communities. The document then describes frameworks for defining and operationalizing performance, including effectiveness, efficiency, relevance, and sustainability. It shows results from applying the Organizational Performance Index tool globally and over time, finding acceptable levels of reliability. The purpose is to have a reliable way to measure the impact of capacity development work.

1. Measuring Capacity Development
Performance
March 19, 2013
Jennifer Mulik, MA, MPH– Global Director Health
jmulik@pactworld.org

2. The problem
“The monitoring and evaluation of
capacity building is as much a
challenge now as it was two
decades ago.”
Simister & Smith, 2010, INTRAC

3. Theory of Change for Capacity Development
(a)Stronger local organizations;
(b)Improve their performance; and
(c) Have greater impact in their target communities.
Outputs
Change in the internal
Outcomes
systems, skills and
Improved external
Impact
policies of
organizations, networks performance of
and systems organizations, networks Improved
and systems health, environment
and/or livelihoods in
the communities served
by
organizations, networks
and systems
• OCA • OPI
• OD Roadmap • Outcome Mapping • Evaluations
• ISP Tracking • Most Significant
Change

4. Defining Performance - Framework
IDRC/Universalia
History
CD Outcomes Mission Culture
Framework Incentives Rewards
Political Effectiveness Structure
Efficiency Leadership
Economic
Relevance Financial
Technological Sustainability
Technology
Administrative
Infrastructure
Social / cultural Human resources
Stakeholder Program / services
Linkages

5. Operationalizing Performance
Effectiveness: Efficiency:
Results Delivery
Standards Reach
Relevance: Sustainability:
Target Population Resources
Learning Social Capital

6. Benchmarks
Level 4
Level 3
Level 2
Level 1
Level 1 Level 2 Level 3 Level 4
Effective
The organization is in The organization has
The organization has set The organization has
Results
the process of met over 75% of
clearly defined outcome met over 50% of
developing outcome outcome level targets
level targets for its outcome level targets for
level targets for its for its programs &
programs & services. its programs & services.
programs & services. services.

7. Global OPI Scores by Length of Partnership
Results
4.0
Social Capital 3.0
Standards
2.0
Resources 1.0 Delivery
Less than 1 year
Learning Reach
1-3 Years
Target
Population More than 3 years

8. Global OPI Scores Time Series
Results
4.0
Social Capital Standards
3.0
2.0
Resources 1.0 Delivery
2011
Learning Reach 2012
Target
Population

9. Ethiopia OPI Scores by Length of Partnership
4.0
3.5
3.1 3.1
3.0 2.8 2.8
2.6
2.5 2.5 2.5 Less than 1 year
2.5
1-3 Years
More than 3 years
2.0
1.8 1.8
1.5
1.5
1.3
1.0
Effectiveness Efficiency Relevance Sustainability

10. Organizational Performance Index (OPI)
Testing Project
The purpose of the OPI Testing Project
funded by the Rockefeller Foundation
is to check the OPI reliability, i.e.
whether the Index generates the same
set of scores when it’s reviewed by two
people, separately.

11. Kappa statistic inter-score reliability
Country Pact 1 v Pact 2 Pact 1 v Partner Pact 2 v Partner
Vietnam .804 .498 .556
Ethiopia .706 .454 .576
South Sudan .605 .571 .437
Swaziland .504 .495 .583
Nigeria .415 .502 .299
Zimbabwe .357 .724 .587

12. Result
Our OPI tool and methodology
of application meets the
reliability standard