Trend analyses of nationallyrepresentative survey data: What story
can be told and what is missing.
Lia Florey, MEASURE DH...
Acknowledgements
The Multiagency Malaria Control Impact Evaluations are
a joint effort of many partners.
The core team inc...
Outline
•
•
•
•
•

Introduction to Impact Evaluation Project
General plausibility approach
Challenges to the plausibility ...
Introduction to Impact Evaluation

Goal: Determine if scale-up of malaria control
interventions has had an impact on malar...
Evaluation Framework

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Core Analytic
Questions
Input

Process

Output

Outcome

Impact

Question 1: Has the availability of services for malaria ...
Plausibility approach
• Show trends in scale-up of malaria control
interventions (ITNs, IRS, IPTp, Effective Case Manageme...
Why use a Plausibility Approach?
• Data on malaria-specific outcomes poor or lacking
• Difficult to measure cause-specific...
Impact Model

Sixth MIM Pan-African Malaria
Conference October 9, 2013, Durban,
South Africa
Plausibility Scenario

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Challenges to determining plausibility using survey data

Low levels of coverage throughout evaluation period
– Maybe insu...
Challenges to determining plausibility using survey data
Mortality trend
Malaria intervention coverage

2000

2010

Lack o...
Example from Angola Impact Evaluation

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Challenges to determining plausibility using survey data

Mortality decline began before intervention scale-up

Sixth MIM ...
Challenges to determining plausibility using survey data

Mortality trend scenario 1

Malaria intervention coverage

2000
...
Example from Malawi Impact Evaluation

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Challenges to determining plausibility using survey data
Seasonal variation in data collection, Malawi

– DHS low transmis...
Challenges to determining plausibility using survey data
Ecological fallacy

Sixth MIM Pan-African Malaria Conference

Oct...
Challenges to determining plausibility using survey data
Ecological fallacy

Sixth MIM Pan-African Malaria Conference

Oct...
Examples

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Rwanda

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Ethiopia

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
What else can be done?

Tell the story with more detail
– Stratifications – by malaria risk, wealth, urban/rural, age

Six...
What else can be done?

Tell the story with more detail
– Accessibility, health systems, specific intervention campaigns
What else can be done?
Use other methodological approaches
– District-level ecological analyses - Malawi
• Requires large ...
What else can be done?
Use other methodological approaches
– Decomposition analyses - Rwanda
•
•
•
•

Survival models
Indi...
What else can be done?
• Use other sources of data
– Subnational Anemia & Parasitemia surveys

Sixth MIM Pan-African Malar...
What else can be done?
• Use other sources of data
– Demographic Surveillance Systems (DSS)

Sixth MIM Pan-African Malaria...
Questions?

Sixth MIM Pan-African Malaria Conference

October 9, 2013, Durban, South Africa
Upcoming SlideShare
Loading in …5
×

Trend Analyses of Nationally-representative Survey Data: What story can be told and what is missing

845 views

Published on

Presented by Lia Florey, MEASURE DHS/ICF International, as part of a symposium organized by MEASURE Evaluation and MEASURE DHS at the 6th MIM Pan-African Malaria Conference.

Published in: Health & Medicine, Sports
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
845
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Timing of the change
    Space
    Dose-response
    Age-pattern
    Correspondence malaria morbidity – mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Reference Rowe paper
  • Conceptual Framework
  • These are both plausible patterns for impact. Scenario 1 shows a negative linear relationship between intervention coverage and mortality. Scenario 2 shows a threshold effect.
  • Timing of collection of intervention data compared to measurement of outcomes
    Changing drug policies make it difficult to link trends in treatment with trends in morbidity/mortality
    Early surveys did not contain standard questions necessary for calculating some of these indicators (i.e. ITN use).
    Difficult to assess a trend with few data points, especially with five-year intervals
    Data not always available for the required period
    Plausibility versus causality
    Time series data on interventions not available
    Accounting for possible contextual factors
  • Alternative scenario are possible with very different implications for the plausibility argument.
    Treated nets
    Few ITNs in 2000
    LLINs distributed en masse ~2007/8
    IPTp in 39 countries by 2007
    Malawi is an exception (1993)
    ACTs
    2006/2007
    IRS
    Usually in targeted areas and not useful as a national measure
  • ACCM not specific to malaria and therefore cannot be attributed fully to malaria interventions
  • We see this especially with IPTp, and with ITNs in some countries.
  • Steep declines during the period before intense intervention began.
  • Timing of collection of intervention data compared to measurement of outcomes
    Changing drug policies make it difficult to link trends in treatment with trends in morbidity/mortality
    Early surveys did not contain standard questions necessary for calculating some of these indicators (i.e. ITN use).
    Difficult to assess a trend with few data points, especially with five-year intervals
    Data not always available for the required period
    Plausibility versus causality
    Time series data on interventions not available
    Accounting for possible contextual factors
  • This is hypothetical. Can’t test it without disaggregated data.
  • Significant mortality decline
    Significant and rapid intervention coverage
    Timing
    IPTp not implemented throughout
    Data on % of households with ITNs not available from 2000
    So successful (look at such high coverage) what next? Will need another approach as coverage gets so high and morbidity so low.
  • Rapid declines in mortality before uptake of interventions
    Rapid uptake of interventions followed by stagnation yet mortality declines continue. How to interpret this?
    MIS vs. DHS potential effects
    Potential effects of varied epidemiologic conditions (elevation etc.)
  • Timing of the change
    Space, dose-response, age pattern
    Correspondence malaria morbidity and mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Timing of the change
    Space, dose-response, age pattern
    Correspondence malaria morbidity and mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Timing of the change
    Space, dose-response, age pattern
    Correspondence malaria morbidity and mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Timing of the change
    Space, dose-response, age pattern
    Correspondence malaria morbidity and mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Timing of the change
    Space, dose-response, age pattern
    Correspondence malaria morbidity and mortality change
    Other factors
    LiST deaths averted (magnitude expected)
  • Trend Analyses of Nationally-representative Survey Data: What story can be told and what is missing

    1. 1. Trend analyses of nationallyrepresentative survey data: What story can be told and what is missing. Lia Florey, MEASURE DHS - ICF International Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa Symposium 38
    2. 2. Acknowledgements The Multiagency Malaria Control Impact Evaluations are a joint effort of many partners. The core team includes members from the following organizations: •PMI/USAID – Erin Eckert, Christine Hershey, Rene Salgado •PMI/CDC – Achuyt Bhattarai, Carrie Nielsen, Steven Yoon •MEASURE DHS – Fred Arnold, Lia Florey, Cameron Taylor •MEASURE Evaluation – Ana Claudia Franca-Koh, Samantha Herrera, Jui Shah, Yazoume Ye Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    3. 3. Outline • • • • • Introduction to Impact Evaluation Project General plausibility approach Challenges to the plausibility approach Examples What else can be done Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    4. 4. Introduction to Impact Evaluation Goal: Determine if scale-up of malaria control interventions has had an impact on malaria outcomes 1. What impact have malaria control interventions had on malaria-related morbidity and mortality? 2. Can we demonstrate and quantify plausible association between intervention and impact? 3. What else could have contributed? Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    5. 5. Evaluation Framework Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    6. 6. Core Analytic Questions Input Process Output Outcome Impact Question 1: Has the availability of services for malaria prevention and treatment increased and are services equitably distributed? a. Funding/spending for malaria programs (from X partner organizations and/or domestic funding) b. Vector management (ITNs, IRS) X X c. Case management X X d. IPTp X X Question 2: Has mortality decreased? a. All-cause under-five mortality (ACCM)         X b. Malaria-specific under-five mortality         X Question 3: Have the malaria incidence and prevalence decreased? a. Morbidity (anemia prevalence, parasite prevalence, malaria cases) b. Is there anecdotal evidence suggesting additional potential impacts of malaria control (burden placed on health facilities, etc.) Question 4: Have other health programs (non-malaria) been scaled up in recent years? a. Vitamin A, immunizations, etc. X X X X X
    7. 7. Plausibility approach • Show trends in scale-up of malaria control interventions (ITNs, IRS, IPTp, Effective Case Management) • Show trends in malaria outcomes (Morbidity, Mortality) • Show trends in other factors that could have influenced trends in outcomes (Contextual Factors) • Conclude whether it is plausible that malaria control interventions reduced malaria-related deaths Malaria related interventions Morbidity Mortality Contextual factors Contextual factors Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    8. 8. Why use a Plausibility Approach? • Data on malaria-specific outcomes poor or lacking • Difficult to measure cause-specific mortality in most of Africa – Weak vital registration system – Cause of death difficult to verify • We do not have individual-level data needed for directly measuring causal relationships – ITN use questions ask about previous night – Mortality is measured over a five year period – Exposures to interventions do not always precede outcomes Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    9. 9. Impact Model Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    10. 10. Plausibility Scenario Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    11. 11. Challenges to determining plausibility using survey data Low levels of coverage throughout evaluation period – Maybe insufficient to expect impact on mortality Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    12. 12. Challenges to determining plausibility using survey data Mortality trend Malaria intervention coverage 2000 2010 Lack of baseline data for interventions – Started measuring half way through evaluation period – New/improved interventions introduced during period Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    13. 13. Example from Angola Impact Evaluation Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    14. 14. Challenges to determining plausibility using survey data Mortality decline began before intervention scale-up Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    15. 15. Challenges to determining plausibility using survey data Mortality trend scenario 1 Malaria intervention coverage 2000 2010 Intervention coverage plateaued but mortality trends continued Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    16. 16. Example from Malawi Impact Evaluation Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    17. 17. Challenges to determining plausibility using survey data Seasonal variation in data collection, Malawi – DHS low transmission season, MIS high transmission season – Affects use of interventions as well as outcomes
    18. 18. Challenges to determining plausibility using survey data Ecological fallacy Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    19. 19. Challenges to determining plausibility using survey data Ecological fallacy Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    20. 20. Examples Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    21. 21. Rwanda Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    22. 22. Ethiopia Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    23. 23. What else can be done? Tell the story with more detail – Stratifications – by malaria risk, wealth, urban/rural, age Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    24. 24. What else can be done? Tell the story with more detail – Accessibility, health systems, specific intervention campaigns
    25. 25. What else can be done? Use other methodological approaches – District-level ecological analyses - Malawi • Requires large number of sampled districts • Allows inclusion of contextual factors • Few national surveys representative at district level Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    26. 26. What else can be done? Use other methodological approaches – Decomposition analyses - Rwanda • • • • Survival models Individual level Allows inclusion of contextual factors Timing issue with exposure data Decomposition models show that the observed increase in household bed net ownership, from 8% to 94% could have explained as much as 45% of the observed decline in ACCM between 2000 and 2010, equivalent to a reduction of 37 deaths per 1,000 live births. Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    27. 27. What else can be done? • Use other sources of data – Subnational Anemia & Parasitemia surveys Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    28. 28. What else can be done? • Use other sources of data – Demographic Surveillance Systems (DSS) Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa
    29. 29. Questions? Sixth MIM Pan-African Malaria Conference October 9, 2013, Durban, South Africa

    ×