Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges
Upcoming SlideShare
Loading in...5
×
 

Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges

on

  • 151 views

A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.
Examples from Nigeria, Zambia and Zimbabwe.

Statistics

Views

Total Views
151
Views on SlideShare
151
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges Annual Results and Impact Evaluation Workshop for RBF - Day Five - Simultaneously Designing Projects and Impact Evaluations - The Benefits and Challenges Presentation Transcript

  • SIMULTANEOUSLY DESIGNING PROJECTS AND IMPACT EVALUATIONS: THE BENEFITS AND CHALLENGES EXAMPLES FROM NIGERIA, ZAMBIA, AND ZIMBABWE
  • EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) • In 1998, the Mexican government adopted a new social protection program: Progresa (a conditional cash transfer) •  provides a small amount of cash assistance to the poor while simultaneously making progress towards goals of improving children’s health and education • Policy design team had substantial concerns about program design and performance •  most social programs do not survive government changeover • Policy makers felt evidence assessed by international experts would help program survive. • Program survived after change in government as evaluation evidence was too politically costly to ignore •  even though new government campaigned on ending the program, new head of state was convinced by evidence and decided to honor his pledge by renaming it: Oportunidades
  • EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) The team has attributed the following benefits to the simultaneous design of the program and the IE: 1.  Political hardiness– PROGRESA is a pillar of Mexican SP policy. 2.  Increased and diverse funding sources–donors were attracted to the program due to the clean, non-partisan evidence of its impact. 3.  Significant expansion-- not only has PROGRESA continued over the years, it has expanded vastly from the initial pilot phase. 4.  Explicit codification of requirement for independently-run evaluation of all Mexican social programs; CONEVAL established. 5.  Impact on how donors such as DfID and WB program and evaluate support  Conclusion – policy makers anticipating benefits of program were able to design a robust evaluation that led to increased political support
  • CHIEF PURPOSES OF IE 1.  Establish whether the program is having a significant effect on targeted outcomes. 2.  Carefully-designed IE can tell you why something works or doesn’t work. 3.  Inform policy makers of program performance and gaps, including in gaps implementation for improvement. 4.  Most reliable way to achieve this is consider IE design / simultaneously with program design 5.  But there are many paths to the same destination, and we will explore at least three stories, their challenges, and the lessons learned
  • FIVE “GUIDING PRINCIPLES” TO ENHANCE IE POLICY RELEVANCE Generalizable to scale – typically a need to replicate successful interventions on a wider scale than study context Effectiveness studies with operational orientation – evaluations of programs implemented within existing system capabilities Regular feed-back to implementers – process monitoring validates the program model and identifies areas that need improvement Include a cost focus – to inform policy choices and trade-offs Local ownership – core involvement of government and local investigators is critical both for effective implementation as well as policy adoption All of these principles much easier to attain when IE and program are jointly designed and planned
  • IE DESIGN TO SATISFY THE FIVE PRINCIPLES: RBF IN ZAMBIA Wide population coverage of key MCH services remains a challenge in Zambia – MoH wants to increase provision of services while simultaneously strengthening health system After extensive discussion, results based financing (RBF) reform takes the form of fee-for-selected- services paid to facilities with: • quality-of-care assessments, enhanced supervision and monitoring, devolved autonomy to clinic • incentivized activities focus on MCH services
  • ZAMBIA, WHAT DOES THE IE LOOK LIKE Q: What is the causal impact of the RBF on service provision and population health indicators of interest? Intervention is a government sponsored system-level reform with a central role for district supervision and monitoring Treatment arm: Facilities and patients residing in districts that introduce the RBF program Control: facilities and households in matched “comparison” districts A difference-in-difference estimator between matched facilities in treatment and control will estimate program impact
  • BROAD GEOGRAPHIC COVERAGE FACILITATES EXTERNAL VALIDITY In Zambia, MoH with strong desire to pilot RBF in every province. For policy learning, select districts “representative” for that province, based on district health systems capacity and service coverage. 27 out of 62 total districts in study
  • ZAMBIA, HOW DID THE TWO SIDES WORK TOGETHER Operational and IE team were integrated from the very beginning – the first design discussions with government involved both teams Repeated consultations with RBF experts, IE experts, and government lead to design of RBF pilot as well as created constituency for IE As a result, IE extended to answer additional questions •  what is the cost-effectiveness of offering higher incentive levels •  randomly selected subset of facilities offered additional 25% bonus •  what is cost-effective audit rate in external verification •  facilities presented with randomized likelihood of audit: 100%, 30%, or 10%
  • ZIMBABWE, AN OVERVIEW OF THE RBF AND THE IE Zimbabwe created a quality adjusted fee-for-service RBF pilot with a focus on improving priority MCH outcomes Evaluation of RBF grouped comparable pairs of districts and randomly selected one of each pair to receive the RBF pilot and the other to serve as a comparison Pilot districts stratified by province, so every Provincial Health Management Team gains experience in RBF, as well as involved in the IE planning Strong constituency for IE within Ministry – creation of a Technical Evaluation Working Group to advise on all steps of IE and to monitor progress
  • BROAD GEOGRAPHIC SCOPE IN ZIMBABWE FACILITATES EXTERNAL VALIDITY AND INVOLVEMENT OF ALL PMDS IN IE
  • GOVERNMENTAL IE WORKING GROUP ADVISES ON ALL DATA ACTIVITIES Survey data regularly collected from multiple perspectives: •  Household •  Facility and health worker •  Community •  District and provincial management Routine administrative data also critical due to scope and frequency of observation Qualitative assessment with complementary focus on complex subjective qualia: understanding, motivation, and innovative actions Collection, integration, and analysis of multiple data sources (both continuous and sporadic) underscores benefit of locally based investigators and government counterparts
  • NIGERIA: AN OVERVIEW OF RBF PROGRAM •  Three states purposively selected: Adamawa, Nasarawa and Ondo. •  Selected health facilities in local government (LGAs) within each state receive one of the two interventions: PBF and DFF. LGAs are randomized to receive PBF or DFF. •  Local and state governments receive bonus payments tied to performance •  Control LGAs are randomly selected from matched states for each project state
  • NIGERIA: ADVANTAGES OF DESIGNING THE PROJECT AND IE SIMULTANEOUSLY Simultaneous design of the project interventions and IE required careful definition of interventions and empirical approaches of measuring impacts. •  Simultaneous development of the interventions and IE ensured that the distribution of LGAs into PBF and DFF were random so that impact of the interventions can be measured retrospectively. •  Joint definition of interventions and components. •  The adaptation of the questionnaires were guided by the definitions of interventions as well as the impacts to be measured. •  Joint development of routine monitoring tools (quality checklist, counter verification, mhealth questionnaires etc) with the intention of setting up a platform where all data, including HMIS, could be triangulated in the future
  • NIGERIA: DEFINING THE INTERVENTIONS, PBF AND DFF
  • NIGERIA: AN OVERVIEW OF IE DESIGN The IE design of Nigeria will be able to address the following: •  Evaluate conditionality: Impacts of PBF package versus DFF •  Evaluate DFF: Impacts of DFF package versus ‘business as usual’ •  Evaluate PBF: PBF package versus ‘business as usual’
  • NIGERIA: HOW IE AND OPERATIONAL TEAMS WORKED TOGETHER TO INTEGRATE ACTIVITIES AND MAXIMIZE LEARNING •  Random selection of LGAs and facilities were possible without significant political pressure •  Helped in the revision and adaptation of questionnaires: various aspects of interventions, measures of impacts and intermediate outcomes, capturing potential spill-over effects. •  Baseline survey will provide some preliminary indications of potential effects of interventions (comparing pre-pilot areas to matched areas), even though baseline information will not allow pre-post, baseline-midline comparisons.