Multi-Agency Impact Evaluation

GHI Presentation, March 22nd , 2012, 1:00pm-2:00pm
USAID/Washington DC

Multi-Agency Impact Evaluation

Challenges in Evaluating the Impact of the Scale-up
of Malaria Interventions: The Tanzania Impact
Evaluation

Yazoume Ye, MEASURE Evaluation/ICF International

Content of the Presentation

 Overview of the Multiagency Impact Evaluation
 Mainland Tanzania impact evaluation
o Scope
o Process and methodological approach

GHI Presentation, March 22nd , 2012, USAID/Washington DC

Overview of the Multiagency Impact
Evaluation

Rationale
 Substantial increase in
funding for malaria
control in the recent
decade
 Progressing scaled up of
key interventions
 What is the effect of the
scale-up of key
interventions on malaria
burden? Source: Donor report & Gvt of Tanzania budget


Institutions Involved
 PMI
o USAID - Washington, DC and Country RA
o CDC - Atlanta, GA and Country RA
 ICF International
o MEASURE Evaluation
o MEASURE DHS
 National Malaria Control Programs
 National Statistic Bureaus
 In country Research Institution


Evaluation Questions

 What impact have malaria control interventions had on
malaria-related morbidity and mortality?

 Can we demonstrate and quantify plausible association
between intervention and impact?


Key Indicators
Program-based Data Population-based Data

Inputs Processes Outputs Outcomes Impact

• Strategies • Human • Service • Intervention • Disease
• Policies resources delivery coverage burden
• Guidelines • Training • Knowledge, • Use of • Mortality
• Financing • Commodities skills, practice intervention

Focus


What is the Story?
Linking Morbidity and Mortality Change to Interventions

Malaria intervention coverage

Mortality trend scenario 1

Mortality trend scenario 2

1999 2010


Evaluation Scope

 Focus on impact evaluation, NOT program
effectiveness and efficiency
 Not restricted to PMI-funded malaria control
 Focus on period of malaria control scale-up
and change in outcomes (1999-present)


Achieving Collective Ownership
 Prior consultation:
o Common ownership of SINGLE impact evaluation report
o Workshop involved NMCP, RBM-MERG, WHO, GFATM, and
PMI
 Stakeholders review preliminary findings
 TAG: Comprised of RBM members and other international
experts review first draft
 Disseminate final report nationally and internationally


Evaluation Design

 Describe trends in intervention coverage
 Describe trends in morbidity and mortality
 Linking these trends
o “Plausibility” approach
o Temporal, spatial, age-pattern, “dose-response” associations
o Use models (LisT) to estimate number of deaths averted
 Multivariate models when possible with available data


Plausibility Framework
Indicators

• ITN Ownership
• Parasitemia All cause under-five
• ITN use
• Anaemia (<8g/dL) mortality (5q0)
• IPTp
• Treatment

Decreased
Increase in effective Decreased malaria-
intervention coverage morbidity associated
mortality
Contextual

• Health intervention • Socioeconomic
factors

• Climatic factor • Health care • Education
• Rainfall utilization • Fertility risk
• Temperature • ANC, EPI, Vit A, • Housing condition
PMTCT • Nutrition


Data Sources

 DHS, MIS, MICS surveys
 Other national/large scale surveys
 HMIS data
 HDSS (if any in the country) - case study
 Studies and reports


Evaluation Timeline
Angola GF PMI Nov Jun

Benin GF PMI Feb Feb

Ethiopia GF PMI Feb Feb

Ghana GF PMI Feb Feb

Kenya GF PMI
Liberia GF PMI
Madagascar GF PMI
GF PMI Nov Jun
Malawi
Mali GF PMI
Mozambique GF PMI Feb Feb

Rwanda GF PMI Feb Dec

Senegal GF PMI Feb Dec

Tanzania GF PMI Feb

Uganda GF PMI Feb Feb
Zambia GF PMI
Zanzibar Feb Dec

2003/04 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Mainland Tanzania Impact Evaluation
The Tanzania Impact Evaluation Team

Alex Mwita Fabrizio Molteni Monica Olewe Rita Njau
Angelica Rugarabamu Frank Chaky Paul Smithson Rose Lusinde
Achuyt Bhattarai Fred Arnold Peter McElroy Salim Abdulla
Christine Hershey Honorathy Masanja Renata Mandike Yazoume Ye
Erin Eckert Lia Florey Rene Salgado

Evaluation Team
 National Malaria Control Program -Tanzania
 National Bureau of Statistics
 Ifakara Health Institute
 PMI
o USAID -Washington, DC and Tanzania
o CDC – Atlanta GA and Tanzania
 ICF International
o MEASURE Evaluation
o MEASURE DHS


Data Sources

 “DHS” surveys: 1992, 1996, 1999, 2004/5, 2007/8, 2009/10
 Other national/large scale surveys
o HBS 2001, 2007; NMCP 2006, 2008; others
 Facility-based data: Rapid impact assessment 2001-2007
 Un/Published studies and report


Outcome Indicators
Intervention Indicator Trends
ITN “Proportion of Households with at Least One ITN” 1999-2009/10
“Proportion of children under five reporting sleeping 1999-2009/10
under an ITN the night prior to interview”
IPTp “Proportion of women with a live birth in the 2 years 2004/5-2009/10
prior to survey who received ≥2 doses of SP during
her most recent pregnancy, ≥1 via ANC”

Assess the effect of SP stock-out (program data)

Treatment “Proportion of under-fives with self-reported fever 1999-2009/10
in the previous two weeks treated with first line anti-
malarial same/next day after fever onset”


Impact Indicators
Health outcome Indicator Trends
Morbidity Parasitemia prevalence: Proportion of children 2007/08 only +
aged 6-59 months with malaria infection NMCP surveys

Anemia prevalence: Proportion of children age 2004/5, 2007/8,
6-59 months with a hemoglobin measurement 2009/10
of <8 g/dL
Fever prevalence: Proportion of children aged 1999, 2004/05,
6-59 month with reported fever 2007/8, 2009/10

Mortality All-cause under five mortality rate (5q0) 1990-2009/10


Contextual Factors
 Fertility risk, female education, literacy, access to media
 Housing condition
 Nutrition status
 Health care utilization
 Health interventions: ANC, Vaccination, Vit A, PMTCT
 Co-morbidities (ARI, diarrhea)
 Climatic factors-Rainfall pattern from 1990
 Change in GDP


Plausibility-Morbidity and Mortality

 Timing of the change
 Space, dose-response, age pattern
 Correspondence malaria morbidity and mortality
change
 Other factors
 LiST deaths averted (magnitude expected)


Additional Analysis

 District level analysis- HMIS :
o Factors associated with malaria case incidence
o Factors associated malaria deaths
 Analysis of cross-sectional datasets to assess the
association of ITN exposure (and other control
measures) with malaria health outcomes; and
 Survival analysis - ITN exposure and child survival


Challenges - Technical
 Timing of collection of intervention data compared with
measurement of outcomes
 Changing drug policies make it difficult to link trends in
treatment with trends in morbidity/mortality
 Early surveys did not contain standard questions necessary
for calculating some of these indicators (i.e ITN use)
 Data were not always available for the required period
 Plausibility versus causality


Challenges - organizational

 Country ownership - Involving country- NMCP
 Getting all partners to agree one approach
 Review process:
o Consider time to get feedback from the TAG
o Having the report approved by all partners involved takes time


Lessons learned
 Country ownership, but need external oversight to move the
process
 Plausibility approach yes, but further district level analysis
is needed (where possible with available data) to strengthen
the results
 National level estimates from Survey are good, but HMIS
and HDSS data could complement and help tell the story


MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill, in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.


Thank You!


Multi-Agency Impact Evaluation

More Related Content

Similar to Multi-Agency Impact Evaluation

More from removed_62798267384a091db5c693ad7f1cc5ac

Recently uploaded

Multi-Agency Impact Evaluation