The document discusses the benefits of simultaneously designing social programs and impact evaluations through examples from Mexico, Nigeria, Zambia, and Zimbabwe. It notes that simultaneous design allows the evaluation to establish the causal impact of the program, inform improvements, ensure broad coverage for validity, and increase local ownership and political support. Close collaboration between operational and evaluation teams is emphasized, including jointly defining interventions, outcomes, and data collection to maximize learning.

1. SIMULTANEOUSLY
DESIGNING PROJECTS
AND IMPACT
EVALUATIONS: THE
BENEFITS AND
CHALLENGES
EXAMPLES FROM NIGERIA, ZAMBIA, AND
ZIMBABWE

2. EXAMPLE FROM RECENT HISTORY:
PROGRESA (OPORTUNIDADES)
• In 1998, the Mexican government adopted a new social
protection program: Progresa (a conditional cash transfer)
• provides a small amount of cash assistance to the poor while
simultaneously making progress towards goals of improving
children’s health and education
• Policy design team had substantial concerns about program
design and performance
• most social programs do not survive government changeover
• Policy makers felt evidence assessed by international experts
would help program survive.
• Program survived after change in government as evaluation
evidence was too politically costly to ignore
• even though new government campaigned on ending the
program, new head of state was convinced by evidence and
decided to honor his pledge by renaming it: Oportunidades

3. EXAMPLE FROM RECENT HISTORY:
PROGRESA (OPORTUNIDADES)
The team has attributed the following benefits to the simultaneous
design of the program and the IE:
1. Political hardiness– PROGRESA is a pillar of Mexican SP policy.
2. Increased and diverse funding sources–donors were attracted to
the program due to the clean, non-partisan evidence of its impact.
3. Significant expansion-- not only has PROGRESA continued over the
years, it has expanded vastly from the initial pilot phase.
4. Explicit codification of requirement for independently-run
evaluation of all Mexican social programs; CONEVAL established.
5. Impact on how donors such as DfID and WB program and evaluate
support
 Conclusion – policy makers anticipating benefits of
program were able to design a robust evaluation that
led to increased political support

4. CHIEF PURPOSES OF IE
1. Establish whether the program is having a significant
effect on targeted outcomes.
2. Carefully-designed IE can tell you why something works
or doesn’t work.
3. Inform policy makers of program performance and gaps,
including in gaps implementation for improvement.
4. Most reliable way to achieve this is consider IE design /
simultaneously with program design
5. But there are many paths to the same destination, and we
will explore at least three stories, their challenges, and
the lessons learned

5. FIVE “GUIDING PRINCIPLES” TO
ENHANCE IE POLICY RELEVANCE
Generalizable to scale – typically a need to replicate successful
interventions on a wider scale than study context
Effectiveness studies with operational orientation – evaluations
of programs implemented within existing system capabilities
Regular feed-back to implementers – process monitoring
validates the program model and identifies areas that need
improvement
Include a cost focus – to inform policy choices and trade-offs
Local ownership – core involvement of government and local
investigators is critical both for effective implementation as well
as policy adoption
All of these principles much easier to attain when IE and program
are jointly designed and planned

6. IE DESIGN TO SATISFY THE
FIVE PRINCIPLES: RBF IN
ZAMBIA
Wide population coverage of key MCH services
remains a challenge in Zambia – MoH wants to
increase provision of services while
simultaneously strengthening health system
After extensive discussion, results based financing
(RBF) reform takes the form of fee-for-selected-
services paid to facilities with:
• quality-of-care assessments, enhanced supervision
and monitoring, devolved autonomy to clinic
• incentivized activities focus on MCH services

7. ZAMBIA, WHAT DOES
THE IE LOOK LIKE
Q: What is the causal impact of the RBF on service provision and
population health indicators of interest?
Intervention is a government sponsored system-level reform with a
central role for district supervision and monitoring
Treatment arm: Facilities and patients residing in districts that
introduce the RBF program
Control: facilities and households in matched “comparison”
districts
A difference-in-difference estimator between matched facilities in
treatment and control will estimate program impact

8. BROAD GEOGRAPHIC
COVERAGE FACILITATES
EXTERNAL VALIDITY
In Zambia, MoH with strong desire to pilot RBF in every
province. For policy learning, select districts
“representative” for that province, based on district health
systems capacity and service coverage. 27 out of 62 total
districts in study

9. ZAMBIA, HOW DID THE
TWO SIDES WORK
TOGETHER
Operational and IE team were integrated from the very
beginning – the first design discussions with government
involved both teams
Repeated consultations with RBF experts, IE experts, and
government lead to design of RBF pilot as well as created
constituency for IE
As a result, IE extended to answer additional questions
• what is the cost-effectiveness of offering higher incentive levels
• randomly selected subset of facilities offered additional 25%
bonus
• what is cost-effective audit rate in external verification
• facilities presented with randomized likelihood of audit:
100%, 30%, or 10%

10. ZIMBABWE, AN
OVERVIEW OF THE RBF
AND THE IE
Zimbabwe created a quality adjusted fee-for-service RBF
pilot with a focus on improving priority MCH outcomes
Evaluation of RBF grouped comparable pairs of districts and
randomly selected one of each pair to receive the RBF pilot
and the other to serve as a comparison
Pilot districts stratified by province, so every Provincial
Health Management Team gains experience in RBF, as well as
involved in the IE planning
Strong constituency for IE within Ministry – creation of a
Technical Evaluation Working Group to advise on all steps of
IE and to monitor progress

11. BROAD GEOGRAPHIC SCOPE IN
ZIMBABWE FACILITATES EXTERNAL
VALIDITY AND INVOLVEMENT OF
ALL PMDS IN IE

12. GOVERNMENTAL IE WORKING
GROUP ADVISES ON ALL DATA
ACTIVITIES
Survey data regularly collected from multiple perspectives:
• Household
• Facility and health worker
• Community
• District and provincial management
Routine administrative data also critical due to scope and
frequency of observation
Qualitative assessment with complementary focus on
complex subjective qualia: understanding, motivation, and
innovative actions
Collection, integration, and analysis of multiple data sources
(both continuous and sporadic) underscores benefit of
locally based investigators and government counterparts

13. NIGERIA: AN OVERVIEW
OF RBF PROGRAM
• Three states purposively selected:
Adamawa, Nasarawa and Ondo.
• Selected health facilities in local government (LGAs)
within each state receive one of the two interventions:
PBF and DFF. LGAs are randomized to receive PBF or DFF.
• Local and state governments receive bonus payments tied
to performance
• Control LGAs are randomly selected from matched states
for each project state

14. NIGERIA: ADVANTAGES OF DESIGNING
THE PROJECT AND IE SIMULTANEOUSLY
Simultaneous design of the project interventions and IE
required careful definition of interventions and empirical
approaches of measuring impacts.
• Simultaneous development of the interventions and IE
ensured that the distribution of LGAs into PBF and DFF
were random so that impact of the interventions can be
measured retrospectively.
• Joint definition of interventions and components.
• The adaptation of the questionnaires were guided by the
definitions of interventions as well as the impacts to be
measured.
• Joint development of routine monitoring tools (quality
checklist, counter verification, mhealth questionnaires etc)
with the intention of setting up a platform where all data,
including HMIS, could be triangulated in the future

15. NIGERIA: DEFINING
THE
INTERVENTIONS,
PBF AND DFF

16. NIGERIA: AN OVERVIEW OF IE
DESIGN
The IE design of Nigeria will be able to address the following:
• Evaluate conditionality: Impacts of PBF package versus DFF
• Evaluate DFF: Impacts of DFF package versus ‘business as
usual’
• Evaluate PBF: PBF package versus ‘business as usual’

17. NIGERIA: HOW IE AND OPERATIONAL
TEAMS WORKED TOGETHER TO
INTEGRATE ACTIVITIES AND
MAXIMIZE LEARNING
• Random selection of LGAs and facilities were
possible without significant political pressure
• Helped in the revision and adaptation of
questionnaires: various aspects of interventions,
measures of impacts and intermediate outcomes,
capturing potential spill-over effects.
• Baseline survey will provide some preliminary
indications of potential effects of interventions
(comparing pre-pilot areas to matched areas), even
though baseline information will not allow pre-post,
baseline-midline comparisons.