Evolution of Family Planning
Impact Evaluation
New context and methodological
considerations
Janine Barden-O’Fallon
Sian C...
Outline of Presentation
 Background and purpose
 Definition of impact evaluation
 Historical context
 Current consider...
Addressing the Information “Gap”:Addressing the Information “Gap”:
Increased Focus on Impact EvaluationIncreased Focus on ...
Definition of Impact Evaluation
“Impact evaluations measure the change in a
development outcome that is attributable to a ...
Ideal Impact Assessment
Impact Evaluation in FP:
Historical context
The Evaluation Project (1991-1997)
Randomized experimental designs (RCTs)
Qu...
Experimental Designs in FP
 Use of RCTs in FP is relatively rare
 Bauman (1997) identified 16 RCTs of FP programs betwee...
Methodological Constraints to
Experimental Designs
 Pure comparison areas may not exist –
other programs in comparison ar...
Multilevel Regression Models
+ -
Hierarchical design is well
suited for evaluations
pertaining to social context
and its e...
Impact Evaluation in FP: Current
Considerations
Experimental and multilevel models answer
questions like:
“Did the interve...
Impact Evaluation in FP: Current
Considerations
 Since 1990s, context for FP evaluation has
evolved:
emphasis on rights a...
Theory-based Approaches
“Theory based evaluation is an approach to
evaluation (i.e., a conceptual analytical model) and
no...
Theory-based Approaches
 National Platform Approach (Victora et al., 2010)
 Large scale evaluations
 District as unit o...
Systems-based Approaches
Systems-based approaches place programs within an
environment (physical, social, political, etc.)...
Systems-based Approaches
Example:
An organizational network
analysis in Ethiopia
identified gaps and
barriers to HIV/FP cl...
Implementation Science
 To identify the factors in the implementation
process that lead to success or failure and to
deve...
Discussion Points
 There is no one-size-fits-all impact evaluation
method
 The evaluation question and programmatic
attr...
Recommendations
 Accept a wide range of evaluation designs. See
Realworld Evaluation: Working under budget,
time, data, a...
MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Car...
Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations
Upcoming SlideShare
Loading in …5
×

Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations

492 views

Published on

Presented by Janine Barden-O'Fallon at the International Conference on Family Planning in November 2013.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
492
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Evolution of Family Planning Impact Evaluation: New contexts and methodological considerations

  1. 1. Evolution of Family Planning Impact Evaluation New context and methodological considerations Janine Barden-O’Fallon Sian Curtis Jessica Levy Ilene Speizer
  2. 2. Outline of Presentation  Background and purpose  Definition of impact evaluation  Historical context  Current considerations  Discussion and recommendations
  3. 3. Addressing the Information “Gap”:Addressing the Information “Gap”: Increased Focus on Impact EvaluationIncreased Focus on Impact Evaluation WHEN WILL WE EVER LEARN? IMPROVING LIVES THROUGH IMPACT EVALUATION DFID Department for International Development Working Paper 38 U.S. Global Health Initiative
  4. 4. Definition of Impact Evaluation “Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured.” -USAID 2011
  5. 5. Ideal Impact Assessment
  6. 6. Impact Evaluation in FP: Historical context The Evaluation Project (1991-1997) Randomized experimental designs (RCTs) Quasi-experimental designs Multilevel regression models
  7. 7. Experimental Designs in FP  Use of RCTs in FP is relatively rare  Bauman (1997) identified 16 RCTs of FP programs between 1960-1993  Mwaikambo et al. (2011) identified 6 RCTs of FP programs between 1995-2009  QEDs more common  Groups are often the level of analysis  Matlab and Navrongo sites
  8. 8. Methodological Constraints to Experimental Designs  Pure comparison areas may not exist – other programs in comparison areas or cross-over of interventions to comparison areas  Non-random placement of programs – intervention areas and control areas often not comparable  Need/ability to control for other factors beyond the program that might affect outcomes  External validity can be an issue when considering replication in a different context or replication at scale Source: Victora et al., 2010.
  9. 9. Multilevel Regression Models + - Hierarchical design is well suited for evaluations pertaining to social context and its effect on behavior Access to reliable data for all levels is necessary Multi-equation modeling can control for endogenous program placement Linking data sets is often necessary but not always possible A current example is Gates-funded MLE Project collecting data in urban Senegal from individual women and at the facilities they attend
  10. 10. Impact Evaluation in FP: Current Considerations Experimental and multilevel models answer questions like: “Did the intervention work?” Not: “How could the intervention be improved?” or “Will the intervention work elsewhere” or “What are the factors that contributed to the success of the intervention?”
  11. 11. Impact Evaluation in FP: Current Considerations  Since 1990s, context for FP evaluation has evolved: emphasis on rights and vulnerable populations emphasis on integrating programs emphasis on expanding coverage, or ‘scaling up’ emphasis on structural interventions
  12. 12. Theory-based Approaches “Theory based evaluation is an approach to evaluation (i.e., a conceptual analytical model) and not a specific method or technique. It is a way of structuring and undertaking analysis in an evaluation.” -Centre of Excellence for Evaluation, Treasury Board of Canada Secretariat, 2012 Plausible program logic that identifies key service components, expected outcomes, and the hypothesized links between them Often require mixed-methods and/or links across information sources
  13. 13. Theory-based Approaches  National Platform Approach (Victora et al., 2010)  Large scale evaluations  District as unit of design and analysis  Continuous monitoring  Multiple sources of data  Variety of evaluation methodologies  Political and technical constraints to implementation = remains largely untested
  14. 14. Systems-based Approaches Systems-based approaches place programs within an environment (physical, social, political, etc.) and assess the interaction of the environment on the program itself or on the anticipated outcomes. Multilevel Identify underlying mechanisms Often require complex modeling
  15. 15. Systems-based Approaches Example: An organizational network analysis in Ethiopia identified gaps and barriers to HIV/FP client referral networks -MEASURE Evaluation, 2012 Client referral network in Kirkos, Ethiopia
  16. 16. Implementation Science  To identify the factors in the implementation process that lead to success or failure and to develop and test practical solutions  To identify if and how interventions should be modified to achieve sustainable impacts  To determine the best way to facilitate full-scale implementation  Ex: How can the barriers to scaling up a FP promotion program be overcome so that it reaches all women with unmet need?
  17. 17. Discussion Points  There is no one-size-fits-all impact evaluation method  The evaluation question and programmatic attributes should drive the choice of evaluation design and methodology  All methods have strengths and weaknesses and any evaluation can be poorly or well executed
  18. 18. Recommendations  Accept a wide range of evaluation designs. See Realworld Evaluation: Working under budget, time, data, and political constraints (Baumberger, Rugh and Mabry, 2012)  Many options exist for addressing design limitations, including the use of multiple methods  The rigor of evaluation designs should meet but not exceed the needs of stakeholders
  19. 19. MEASURE Evaluation is a MEASURE project funded by the U.S. Agency for International Development and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. Government. MEASURE Evaluation is the USAID Global Health Bureau's primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.

×