SIMULTANEOUSLY
DESIGNING PROJECTS
AND IMPACT
EVALUATIONS: THE
BENEFITS AND
CHALLENGES
EXAMPLES FROM NIGERIA, ZAMBIA, AND
Z...
EXAMPLE FROM RECENT HISTORY:
PROGRESA (OPORTUNIDADES)
• In 1998, the Mexican government adopted a new social
protection pr...
EXAMPLE FROM RECENT HISTORY:
PROGRESA (OPORTUNIDADES)
The team has attributed the following benefits to the simultaneous
d...
CHIEF PURPOSES OF IE
1.  Establish whether the program is having a significant
effect on targeted outcomes.
2.  Carefully-...
FIVE “GUIDING PRINCIPLES” TO
ENHANCE IE POLICY RELEVANCE
Generalizable to scale – typically a need to replicate successful...
IE DESIGN TO SATISFY THE
FIVE PRINCIPLES: RBF IN
ZAMBIA
Wide population coverage of key MCH services
remains a challenge i...
ZAMBIA, WHAT DOES
THE IE LOOK LIKE
Q: What is the causal impact of the RBF on service provision and
population health indi...
BROAD GEOGRAPHIC
COVERAGE FACILITATES
EXTERNAL VALIDITY
In Zambia, MoH with strong desire to pilot RBF in every
province. ...
ZAMBIA, HOW DID THE
TWO SIDES WORK
TOGETHER
Operational and IE team were integrated from the very
beginning – the first de...
ZIMBABWE, AN
OVERVIEW OF THE RBF
AND THE IE
Zimbabwe created a quality adjusted fee-for-service RBF
pilot with a focus on ...
BROAD GEOGRAPHIC SCOPE IN
ZIMBABWE FACILITATES EXTERNAL
VALIDITY AND INVOLVEMENT OF
ALL PMDS IN IE
GOVERNMENTAL IE WORKING
GROUP ADVISES ON ALL DATA
ACTIVITIES
Survey data regularly collected from multiple perspectives:
•...
NIGERIA: AN OVERVIEW
OF RBF PROGRAM
•  Three states purposively selected:
Adamawa, Nasarawa and Ondo.
•  Selected health f...
NIGERIA: ADVANTAGES OF DESIGNING
THE PROJECT AND IE SIMULTANEOUSLY
Simultaneous design of the project interventions and IE...
NIGERIA: DEFINING
THE
INTERVENTIONS,
PBF AND DFF
NIGERIA: AN OVERVIEW OF IE
DESIGN
The IE design of Nigeria will be able to address the following:
•  Evaluate conditionali...
NIGERIA: HOW IE AND OPERATIONAL
TEAMS WORKED TOGETHER TO
INTEGRATE ACTIVITIES AND
MAXIMIZE LEARNING
•  Random selection of...
Upcoming SlideShare
Loading in …5
×

Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges

212 views

Published on

A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.
Examples from Nigeria, Zambia and Zimbabwe.

Published in: Healthcare
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
212
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges

  1. 1. SIMULTANEOUSLY DESIGNING PROJECTS AND IMPACT EVALUATIONS: THE BENEFITS AND CHALLENGES EXAMPLES FROM NIGERIA, ZAMBIA, AND ZIMBABWE
  2. 2. EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) • In 1998, the Mexican government adopted a new social protection program: Progresa (a conditional cash transfer) •  provides a small amount of cash assistance to the poor while simultaneously making progress towards goals of improving children’s health and education • Policy design team had substantial concerns about program design and performance •  most social programs do not survive government changeover • Policy makers felt evidence assessed by international experts would help program survive. • Program survived after change in government as evaluation evidence was too politically costly to ignore •  even though new government campaigned on ending the program, new head of state was convinced by evidence and decided to honor his pledge by renaming it: Oportunidades
  3. 3. EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) The team has attributed the following benefits to the simultaneous design of the program and the IE: 1.  Political hardiness– PROGRESA is a pillar of Mexican SP policy. 2.  Increased and diverse funding sources–donors were attracted to the program due to the clean, non-partisan evidence of its impact. 3.  Significant expansion-- not only has PROGRESA continued over the years, it has expanded vastly from the initial pilot phase. 4.  Explicit codification of requirement for independently-run evaluation of all Mexican social programs; CONEVAL established. 5.  Impact on how donors such as DfID and WB program and evaluate support  Conclusion – policy makers anticipating benefits of program were able to design a robust evaluation that led to increased political support
  4. 4. CHIEF PURPOSES OF IE 1.  Establish whether the program is having a significant effect on targeted outcomes. 2.  Carefully-designed IE can tell you why something works or doesn’t work. 3.  Inform policy makers of program performance and gaps, including in gaps implementation for improvement. 4.  Most reliable way to achieve this is consider IE design / simultaneously with program design 5.  But there are many paths to the same destination, and we will explore at least three stories, their challenges, and the lessons learned
  5. 5. FIVE “GUIDING PRINCIPLES” TO ENHANCE IE POLICY RELEVANCE Generalizable to scale – typically a need to replicate successful interventions on a wider scale than study context Effectiveness studies with operational orientation – evaluations of programs implemented within existing system capabilities Regular feed-back to implementers – process monitoring validates the program model and identifies areas that need improvement Include a cost focus – to inform policy choices and trade-offs Local ownership – core involvement of government and local investigators is critical both for effective implementation as well as policy adoption All of these principles much easier to attain when IE and program are jointly designed and planned
  6. 6. IE DESIGN TO SATISFY THE FIVE PRINCIPLES: RBF IN ZAMBIA Wide population coverage of key MCH services remains a challenge in Zambia – MoH wants to increase provision of services while simultaneously strengthening health system After extensive discussion, results based financing (RBF) reform takes the form of fee-for-selected- services paid to facilities with: • quality-of-care assessments, enhanced supervision and monitoring, devolved autonomy to clinic • incentivized activities focus on MCH services
  7. 7. ZAMBIA, WHAT DOES THE IE LOOK LIKE Q: What is the causal impact of the RBF on service provision and population health indicators of interest? Intervention is a government sponsored system-level reform with a central role for district supervision and monitoring Treatment arm: Facilities and patients residing in districts that introduce the RBF program Control: facilities and households in matched “comparison” districts A difference-in-difference estimator between matched facilities in treatment and control will estimate program impact
  8. 8. BROAD GEOGRAPHIC COVERAGE FACILITATES EXTERNAL VALIDITY In Zambia, MoH with strong desire to pilot RBF in every province. For policy learning, select districts “representative” for that province, based on district health systems capacity and service coverage. 27 out of 62 total districts in study
  9. 9. ZAMBIA, HOW DID THE TWO SIDES WORK TOGETHER Operational and IE team were integrated from the very beginning – the first design discussions with government involved both teams Repeated consultations with RBF experts, IE experts, and government lead to design of RBF pilot as well as created constituency for IE As a result, IE extended to answer additional questions •  what is the cost-effectiveness of offering higher incentive levels •  randomly selected subset of facilities offered additional 25% bonus •  what is cost-effective audit rate in external verification •  facilities presented with randomized likelihood of audit: 100%, 30%, or 10%
  10. 10. ZIMBABWE, AN OVERVIEW OF THE RBF AND THE IE Zimbabwe created a quality adjusted fee-for-service RBF pilot with a focus on improving priority MCH outcomes Evaluation of RBF grouped comparable pairs of districts and randomly selected one of each pair to receive the RBF pilot and the other to serve as a comparison Pilot districts stratified by province, so every Provincial Health Management Team gains experience in RBF, as well as involved in the IE planning Strong constituency for IE within Ministry – creation of a Technical Evaluation Working Group to advise on all steps of IE and to monitor progress
  11. 11. BROAD GEOGRAPHIC SCOPE IN ZIMBABWE FACILITATES EXTERNAL VALIDITY AND INVOLVEMENT OF ALL PMDS IN IE
  12. 12. GOVERNMENTAL IE WORKING GROUP ADVISES ON ALL DATA ACTIVITIES Survey data regularly collected from multiple perspectives: •  Household •  Facility and health worker •  Community •  District and provincial management Routine administrative data also critical due to scope and frequency of observation Qualitative assessment with complementary focus on complex subjective qualia: understanding, motivation, and innovative actions Collection, integration, and analysis of multiple data sources (both continuous and sporadic) underscores benefit of locally based investigators and government counterparts
  13. 13. NIGERIA: AN OVERVIEW OF RBF PROGRAM •  Three states purposively selected: Adamawa, Nasarawa and Ondo. •  Selected health facilities in local government (LGAs) within each state receive one of the two interventions: PBF and DFF. LGAs are randomized to receive PBF or DFF. •  Local and state governments receive bonus payments tied to performance •  Control LGAs are randomly selected from matched states for each project state
  14. 14. NIGERIA: ADVANTAGES OF DESIGNING THE PROJECT AND IE SIMULTANEOUSLY Simultaneous design of the project interventions and IE required careful definition of interventions and empirical approaches of measuring impacts. •  Simultaneous development of the interventions and IE ensured that the distribution of LGAs into PBF and DFF were random so that impact of the interventions can be measured retrospectively. •  Joint definition of interventions and components. •  The adaptation of the questionnaires were guided by the definitions of interventions as well as the impacts to be measured. •  Joint development of routine monitoring tools (quality checklist, counter verification, mhealth questionnaires etc) with the intention of setting up a platform where all data, including HMIS, could be triangulated in the future
  15. 15. NIGERIA: DEFINING THE INTERVENTIONS, PBF AND DFF
  16. 16. NIGERIA: AN OVERVIEW OF IE DESIGN The IE design of Nigeria will be able to address the following: •  Evaluate conditionality: Impacts of PBF package versus DFF •  Evaluate DFF: Impacts of DFF package versus ‘business as usual’ •  Evaluate PBF: PBF package versus ‘business as usual’
  17. 17. NIGERIA: HOW IE AND OPERATIONAL TEAMS WORKED TOGETHER TO INTEGRATE ACTIVITIES AND MAXIMIZE LEARNING •  Random selection of LGAs and facilities were possible without significant political pressure •  Helped in the revision and adaptation of questionnaires: various aspects of interventions, measures of impacts and intermediate outcomes, capturing potential spill-over effects. •  Baseline survey will provide some preliminary indications of potential effects of interventions (comparing pre-pilot areas to matched areas), even though baseline information will not allow pre-post, baseline-midline comparisons.

×