• Save
Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations
Upcoming SlideShare
Loading in...5

Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations



Presented by Janine Barden-O'Fallon at the International Conference on Family Planning in November 2013.

Presented by Janine Barden-O'Fallon at the International Conference on Family Planning in November 2013.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations Presentation Transcript

  • Evolution of Family Planning Impact Evaluation New context and methodological considerations Janine Barden-O’Fallon Sian Curtis Jessica Levy Ilene Speizer
  • Outline of Presentation  Background and purpose  Definition of impact evaluation  Historical context  Current considerations  Discussion and recommendations
  • Addressing the Information “Gap”:Addressing the Information “Gap”: Increased Focus on Impact EvaluationIncreased Focus on Impact Evaluation WHEN WILL WE EVER LEARN? IMPROVING LIVES THROUGH IMPACT EVALUATION DFID Department for International Development Working Paper 38 U.S. Global Health Initiative
  • Definition of Impact Evaluation “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured.” -USAID 2011
  • Ideal Impact Assessment
  • Impact Evaluation in FP: Historical context The Evaluation Project (1991-1997) Randomized experimental designs (RCTs) Quasi-experimental designs Multilevel regression models
  • Experimental Designs in FP  Use of RCTs in FP is relatively rare  Bauman (1997) identified 16 RCTs of FP programs between 1960-1993  Mwaikambo et al. (2011) identified 6 RCTs of FP programs between 1995-2009  QEDs more common  Groups are often the level of analysis  Matlab and Navrongo sites
  • Methodological Constraints to Experimental Designs  Pure comparison areas may not exist – other programs in comparison areas or cross-over of interventions to comparison areas  Non-random placement of programs – intervention areas and control areas often not comparable  Need/ability to control for other factors beyond the program that might affect outcomes  External validity can be an issue when considering replication in a different context or replication at scale Source: Victora et al., 2010.
  • Multilevel Regression Models + - Hierarchical design is well suited for evaluations pertaining to social context and its effect on behavior Access to reliable data for all levels is necessary Multi-equation modeling can control for endogenous program placement Linking data sets is often necessary but not always possible A current example is Gates-funded MLE Project collecting data in urban Senegal from individual women and at the facilities they attend
  • Impact Evaluation in FP: Current Considerations Experimental and multilevel models answer questions like: “Did the intervention work?” Not: “How could the intervention be improved?” or “Will the intervention work elsewhere” or “What are the factors that contributed to the success of the intervention?”
  • Impact Evaluation in FP: Current Considerations  Since 1990s, context for FP evaluation has evolved: emphasis on rights and vulnerable populations emphasis on integrating programs emphasis on expanding coverage, or ‘scaling up’ emphasis on structural interventions
  • Theory-based Approaches “Theory based evaluation is an approach to evaluation (i.e., a conceptual analytical model) and not a specific method or technique. It is a way of structuring and undertaking analysis in an evaluation.” -Centre of Excellence for Evaluation, Treasury Board of Canada Secretariat, 2012 Plausible program logic that identifies key service components, expected outcomes, and the hypothesized links between them Often require mixed-methods and/or links across information sources
  • Theory-based Approaches  National Platform Approach (Victora et al., 2010)  Large scale evaluations  District as unit of design and analysis  Continuous monitoring  Multiple sources of data  Variety of evaluation methodologies  Political and technical constraints to implementation = remains largely untested
  • Systems-based Approaches Systems-based approaches place programs within an environment (physical, social, political, etc.) and assess the interaction of the environment on the program itself or on the anticipated outcomes. Multilevel Identify underlying mechanisms Often require complex modeling
  • Systems-based Approaches Example: An organizational network analysis in Ethiopia identified gaps and barriers to HIV/FP client referral networks -MEASURE Evaluation, 2012 Client referral network in Kirkos, Ethiopia
  • Implementation Science  To identify the factors in the implementation process that lead to success or failure and to develop and test practical solutions  To identify if and how interventions should be modified to achieve sustainable impacts  To determine the best way to facilitate full-scale implementation  Ex: How can the barriers to scaling up a FP promotion program be overcome so that it reaches all women with unmet need?
  • Discussion Points  There is no one-size-fits-all impact evaluation method  The evaluation question and programmatic attributes should drive the choice of evaluation design and methodology  All methods have strengths and weaknesses and any evaluation can be poorly or well executed
  • Recommendations  Accept a wide range of evaluation designs. See Realworld Evaluation: Working under budget, time, data, and political constraints (Baumberger, Rugh and Mabry, 2012)  Many options exist for addressing design limitations, including the use of multiple methods  The rigor of evaluation designs should meet but not exceed the needs of stakeholders
  • MEASURE Evaluation is a MEASURE project funded by the U.S. Agency for International Development and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. Government. MEASURE Evaluation is the USAID Global Health Bureau's primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.