Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Understanding causal pathways within health
systems policy evaluation through mediation
analysis: an application to paymen...
Rationale
• Programme evaluation has mainly focused on measuring the
impact on outcomes, with little attention to the caus...
Causal mediation analysis
• A causal mechanism is a process through which a programme or
intervention influences an outcom...
Program Outcome
Mediator 1
P4P Indirect
effect
P4P Measured effect =
P4P Direct Effect + P4P Indirect effect 1 + P4P Indir...
P4P Programme in Tanzania
• P4P Scheme introduced in 2011 by the MoH in the Pwani region
• Target on Maternal and Child He...
P4P theory of change in a
Causal Mediation Analyses Framework
P4P Outcome
P4P indirect effect
through Governance
P4P indir...
Methods
• Step1: Estimating the impact of P4P on outcomes (DiD)
P4Pt indicator of P4P district
δt time indicator
Xijt wome...
Results: Potential mediators (Step 2)
Potential mediators Effect of P4P (% change)
Financing
Proportion of women who paid ...
Results: P4P direct and indirect effect (Step 3)
• Facility based delivery
P4P total effect: +8.2%
P4P indirect effect thr...
Sensitivity analysis
• Semiparametric mediation analysis to quantify the sensitivity of results
to the assumption of no co...
Summary: Indirect effects of P4P
• P4P significantly affects a number of financing, governance and
human resources factors...
Some reflections on mediation analysis
• Mediation analysis rarely applied using difference in difference
• Quasi–experime...
Conclusions
• Mediation analysis is helpful to quantify causal direct and
indirect effects and the relative relevance of c...
Thank you!
Acknowledgements to the whole “P4P team”
Upcoming SlideShare
Loading in …5
×

Understanding the causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania - Laura Anselmi

This presentation was given at the pay-for-performance workshop in Tanzania, November 2015

  • Be the first to comment

Understanding the causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania - Laura Anselmi

  1. 1. Understanding causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania Laura Anselmi, Peter Binyaruka, Masuma Mamdani Josephine Borghi Payment for Performance: a health systems perspective A workshop for scientists and practitioners Dar es Salaam, 26 November 2015
  2. 2. Rationale • Programme evaluation has mainly focused on measuring the impact on outcomes, with little attention to the causal pathways • The relevance of undertaking process evaluation to be integrated with outcome evaluation is increasingly recognised • Process evaluation is particularly relevant for complex interventions:  It increases confidence in the plausibility of outcome effects  It increases the external validity of the evaluation • Process evaluations have been carried out, but without formal assessment of causal pathways
  3. 3. Causal mediation analysis • A causal mechanism is a process through which a programme or intervention influences an outcome • It can be identified by specifying intermediate outcomes or variables (mediators) that are on the causal pathway between intervention and outcome • Causal mediation analysis has been employed to test change pathways within the evaluation of public health programmes • Mediators have been limited to individual level indicators, psychological or physical • Health system mediators which are relevant to the evaluation of health systems or services interventions have not been considered
  4. 4. Program Outcome Mediator 1 P4P Indirect effect P4P Measured effect = P4P Direct Effect + P4P Indirect effect 1 + P4P Indirect Effect 2 Confounder Mediator 2 P4P indirect effect Sequential ignorability b): Given treatment status and pre-treatment confounders the mediators are ignorable (no confounders affecting both mediators and outcome) Programme Measured effect Confounder Confounder Causal mediation analysis Sequential ignorability a): Given pre-treatment confounders the treatment is assigned independently of potential outcomes and mediators P4P direct effect
  5. 5. P4P Programme in Tanzania • P4P Scheme introduced in 2011 by the MoH in the Pwani region • Target on Maternal and Child Health Care outcomes • Outcome evaluation: • 8.2% increase in coverage of institutional deliveries (ID) • 6.5% increase in delivery in public health facilities • 10.3% increase in the uptake of two doses of anti-malarial (IPT) during pregnancy • The effect of P4P on a number of governance, financing and human resources factors has been identified • Data: • Health facility survey: 150 HFs (75 P4P + 75 control) • 1-2 interviews with health workers per HF • 1,500 exit interviews • 3,000 household survey • Baseline: Jan-March 2012, Endline: February 2013
  6. 6. P4P theory of change in a Causal Mediation Analyses Framework P4P Outcome P4P indirect effect through Governance P4P indirect effect through Human Resources P4P indirect effect through Financing P4P direct effect
  7. 7. Methods • Step1: Estimating the impact of P4P on outcomes (DiD) P4Pt indicator of P4P district δt time indicator Xijt women socio-economic characteristics γj HF fixed effects • Step 2: Identifying effect of P4P on potential mediators (DiD) • Step 3: Identifying direct and indirect causal effects (DiD) 𝛽1 3 P4P direct effect 𝛽1 2 X 𝛽4 3 P4P indirect effect through mediator M 𝑌𝑖𝑗𝑡 = 𝛽0 3 + 𝛽1 3 (𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2 3 𝛿𝑡 + 𝛽3 3 𝑋𝑖𝑗𝑡 + 𝛽4 3 𝑀𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡 3 𝑌𝑖𝑗𝑡 = 𝛽0 1 + 𝛽1 1 (𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2 1 𝛿𝑡 + 𝛽3 1 𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡 1 𝑀𝑖𝑗𝑡 = 𝛽0 2 + 𝛽1 2 (𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2 2 𝛿𝑡 + 𝛽3 2 𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡 2
  8. 8. Results: Potential mediators (Step 2) Potential mediators Effect of P4P (% change) Financing Proportion of women who paid for delivery in a HF (1) -8.0** Proportion of women who paid for delivery in a public HF (1) -7.5***BS Service delivery disrupted due to broken equipment last 90days -149** Drug stock-out index-general (0-1 index) -17.2***BS Medical supplies stock-out index (0-1 index) -14.8***BS Oxytocin injection stock-out last 90days -36.2***BS Ergometrine injection stock-out last 90days -26.1** Drugs at delivery stock-out index (0-1 index) -27.0***BS Mean all financing indicators(0-1 index) -8.3** Factor analysis weighted score (2) -60.0***BS Governance Max time from external supervision: 90 days ago -18.0** Dist/Regional supervision provided positive feed-back 23.8** Dist/Regional supervision provided negative feed-back 28.2** Dist/Regional supervision delivered supply -19.3** Dist/Regional checked records 1.5** Dist/Regional observed consultation 0.8** Human resources Mean patient satisfaction with interpersonal care (0-1 scale) (1) 6.7***BS Mean kindness ranks for HW at delivery (0-1 scale)(1) 10.3***BS * p<0.10, ** p<0.05, *** p<0.01 , BS: Significant at 5% level with Bonferroni adjusted p-value for multiple outcomes: Bonferroni adjusted p-value Financing 0.0047, Governance 0.0017, Human resources 0.0414, (1) Out of all women deliveringin a HF in same catchment area (2) equipment, vaccines, drugs, medical supply
  9. 9. Results: P4P direct and indirect effect (Step 3) • Facility based delivery P4P total effect: +8.2% P4P indirect effect through mean of all financing indicators: +1 % P4P indirect effect through reduction in stock-out of oxytocine: +1.8 % P4P direct effect: +7.2% or +6.4% • Delivery in public health facility P4P total effect: +6.5 % P4P indirect effect through reduction in stock-out of oxytocine: +1.9 % P4P direct effect: +4.6% • Uptake of two doses of IPT during pregnancy P4P total effect: +10.3 % P4P indirect effect through reduction in last supervision being 90 days ago: +1.5 % P4P direct effect: +8.8%
  10. 10. Sensitivity analysis • Semiparametric mediation analysis to quantify the sensitivity of results to the assumption of no confounders affecting mediator and outcome • Analysis carried out at the HF level • Estimate a logit model for binary mediators • Multiple hypothesis testing adjustment of p-values for families of mediators • DiD with district fixed effects
  11. 11. Summary: Indirect effects of P4P • P4P significantly affects a number of financing, governance and human resources factors which could potentially mediate its effect on maternal care outcomes • The effect of P4P on the reduction of oxytocine injection stock- out mediates the effect of P4P on institutional deliveries (22%) and deliveries in a public health facility (30%) • The effect of P4P on the frequency of supervisions mediates 15% of the effect of P4P on the uptake of at least two doses of IPT during pregnancy
  12. 12. Some reflections on mediation analysis • Mediation analysis rarely applied using difference in difference • Quasi–experimental setting + Difference-in-Difference provide confidence that the assumption of no pre-treatment confounders is satisfied • But how plausible are the assumptions of no confounders between mediator and outcome? • Data availability limits testing pre-trends for mediators • Possible differences in results according to the level of the analysis • Little analysis of the role of individual level factors or other moderating factors • Possibly simplified description of the causal chain
  13. 13. Conclusions • Mediation analysis is helpful to quantify causal direct and indirect effects and the relative relevance of change pathways • It requires assumptions to identify causality and these can not be tested formally • Quantify the P4P indirect effects helps in thinking about relative cost-effectiveness compared to alternative interventions
  14. 14. Thank you! Acknowledgements to the whole “P4P team”

×