Your SlideShare is downloading. ×
GHI Presentation, March 22nd , 2012, 1:00pm-2:00pm                             USAID/Washington DC          Multi-Agency I...
Content of the Presentation         Overview of the Multiagency Impact Evaluation         Mainland Tanzania impact evalu...
Overview of the Multiagency Impact            Evaluation
Rationale   Substantial increase in    funding    for    malaria    control in the recent    decade   Progressing scaled...
Institutions Involved         PMI               o USAID - Washington, DC and Country RA               o CDC - Atlanta, GA...
Evaluation Questions       What impact have malaria control interventions had on        malaria-related morbidity and mor...
Key Indicators                            Program-based Data                                    Population-based Data     ...
What is the Story?       Linking Morbidity and Mortality Change to Interventions                                          ...
Evaluation Scope            Focus on impact evaluation, NOT program             effectiveness and efficiency            ...
Achieving Collective Ownership     Prior consultation:           o Common ownership of SINGLE impact evaluation report   ...
Evaluation Design       Describe trends in intervention coverage       Describe trends in morbidity and mortality      ...
Plausibility Framework   Indicators                •   ITN Ownership                                             • Parasit...
Data Sources          DHS, MIS, MICS surveys          Other national/large scale surveys          HMIS data          H...
Evaluation TimelineAngola                 GF     PMI                                  Nov         JunBenin          GF    ...
Mainland Tanzania Impact Evaluation              The Tanzania Impact Evaluation Team Alex Mwita            Fabrizio Molten...
Evaluation Team         National Malaria Control Program -Tanzania         National Bureau of Statistics         Ifakar...
Data Sources      “DHS” surveys: 1992, 1996, 1999, 2004/5, 2007/8, 2009/10      Other national/large scale surveys      ...
Outcome Indicators    Intervention Indicator                                                           Trends    ITN      ...
Impact Indicators   Health outcome                Indicator                                       Trends   Morbidity      ...
Contextual Factors        Fertility risk, female education, literacy, access to media        Housing condition        N...
Plausibility-Morbidity and Mortality      Timing of the change      Space, dose-response, age pattern      Corresponden...
Additional Analysis           District level analysis- HMIS :                 o Factors associated with malaria case inci...
Challenges - Technical    Timing of collection of intervention data compared with     measurement of outcomes    Changin...
Challenges - organizational    Country ownership - Involving country- NMCP    Getting all partners to agree one approach...
Lessons learned    Country ownership, but need external oversight to move the     process    Plausibility approach yes, ...
MEASURE Evaluation is a MEASURE project funded by the       U.S. Agency for International Development and implemented by  ...
Thank You!GHI Presentation, March 22nd , 2012, USAID/Washington DC
Upcoming SlideShare
Loading in...5
×

Multi-Agency Impact Evaluation

1,571

Published on

GHI Presentation by Yazoume Ye

Published in: Health & Medicine, Travel
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,571
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Multi-Agency Impact Evaluation"

  1. 1. GHI Presentation, March 22nd , 2012, 1:00pm-2:00pm USAID/Washington DC Multi-Agency Impact EvaluationChallenges in Evaluating the Impact of the Scale-up of Malaria Interventions: The Tanzania Impact Evaluation Yazoume Ye, MEASURE Evaluation/ICF International
  2. 2. Content of the Presentation  Overview of the Multiagency Impact Evaluation  Mainland Tanzania impact evaluation o Scope o Process and methodological approachGHI Presentation, March 22nd , 2012, USAID/Washington DC
  3. 3. Overview of the Multiagency Impact Evaluation
  4. 4. Rationale  Substantial increase in funding for malaria control in the recent decade  Progressing scaled up of key interventions  What is the effect of the scale-up of key interventions on malaria burden? Source: Donor report & Gvt of Tanzania budgetGHI Presentation, March 22nd , 2012, USAID/Washington DC
  5. 5. Institutions Involved  PMI o USAID - Washington, DC and Country RA o CDC - Atlanta, GA and Country RA  ICF International o MEASURE Evaluation o MEASURE DHS  National Malaria Control Programs  National Statistic Bureaus  In country Research InstitutionGHI Presentation, March 22nd , 2012, USAID/Washington DC
  6. 6. Evaluation Questions  What impact have malaria control interventions had on malaria-related morbidity and mortality?  Can we demonstrate and quantify plausible association between intervention and impact?GHI Presentation, March 22nd , 2012, USAID/Washington DC
  7. 7. Key Indicators Program-based Data Population-based Data Inputs Processes Outputs Outcomes Impact • Strategies • Human • Service • Intervention • Disease • Policies resources delivery coverage burden • Guidelines • Training • Knowledge, • Use of • Mortality • Financing • Commodities skills, practice intervention FocusGHI Presentation, March 22nd , 2012, USAID/Washington DC
  8. 8. What is the Story? Linking Morbidity and Mortality Change to Interventions Malaria intervention coverage Mortality trend scenario 1 Mortality trend scenario 2 1999 2010GHI Presentation, March 22nd , 2012, USAID/Washington DC
  9. 9. Evaluation Scope  Focus on impact evaluation, NOT program effectiveness and efficiency  Not restricted to PMI-funded malaria control  Focus on period of malaria control scale-up and change in outcomes (1999-present)GHI Presentation, March 22nd , 2012, USAID/Washington DC
  10. 10. Achieving Collective Ownership  Prior consultation: o Common ownership of SINGLE impact evaluation report o Workshop involved NMCP, RBM-MERG, WHO, GFATM, and PMI  Stakeholders review preliminary findings  TAG: Comprised of RBM members and other international experts review first draft  Disseminate final report nationally and internationallyGHI Presentation, March 22nd , 2012, USAID/Washington DC
  11. 11. Evaluation Design  Describe trends in intervention coverage  Describe trends in morbidity and mortality  Linking these trends o “Plausibility” approach o Temporal, spatial, age-pattern, “dose-response” associations o Use models (LisT) to estimate number of deaths averted  Multivariate models when possible with available dataGHI Presentation, March 22nd , 2012, USAID/Washington DC
  12. 12. Plausibility Framework Indicators • ITN Ownership • Parasitemia All cause under-five • ITN use • Anaemia (<8g/dL) mortality (5q0) • IPTp • Treatment Decreased Increase in effective Decreased malaria- intervention coverage morbidity associated mortality Contextual • Health intervention • Socioeconomic factors • Climatic factor • Health care • Education • Rainfall utilization • Fertility risk • Temperature • ANC, EPI, Vit A, • Housing condition PMTCT • NutritionGHI Presentation, March 22nd , 2012, USAID/Washington DC
  13. 13. Data Sources  DHS, MIS, MICS surveys  Other national/large scale surveys  HMIS data  HDSS (if any in the country) - case study  Studies and reportsGHI Presentation, March 22nd , 2012, USAID/Washington DC
  14. 14. Evaluation TimelineAngola GF PMI Nov JunBenin GF PMI Feb FebEthiopia GF PMI Feb FebGhana GF PMI Feb FebKenya GF PMILiberia GF PMIMadagascar GF PMI GF PMI Nov JunMalawiMali GF PMIMozambique GF PMI Feb FebRwanda GF PMI Feb DecSenegal GF PMI Feb DecTanzania GF PMI FebUganda GF PMI Feb FebZambia GF PMIZanzibar Feb Dec 2003/04 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
  15. 15. Mainland Tanzania Impact Evaluation The Tanzania Impact Evaluation Team Alex Mwita Fabrizio Molteni Monica Olewe Rita Njau Angelica Rugarabamu Frank Chaky Paul Smithson Rose Lusinde Achuyt Bhattarai Fred Arnold Peter McElroy Salim Abdulla Christine Hershey Honorathy Masanja Renata Mandike Yazoume Ye Erin Eckert Lia Florey Rene Salgado
  16. 16. Evaluation Team  National Malaria Control Program -Tanzania  National Bureau of Statistics  Ifakara Health Institute  PMI o USAID -Washington, DC and Tanzania o CDC – Atlanta GA and Tanzania  ICF International o MEASURE Evaluation o MEASURE DHSGHI Presentation, March 22nd , 2012, USAID/Washington DC
  17. 17. Data Sources  “DHS” surveys: 1992, 1996, 1999, 2004/5, 2007/8, 2009/10  Other national/large scale surveys o HBS 2001, 2007; NMCP 2006, 2008; others  Facility-based data: Rapid impact assessment 2001-2007  Un/Published studies and reportGHI Presentation, March 22nd , 2012, USAID/Washington DC
  18. 18. Outcome Indicators Intervention Indicator Trends ITN “Proportion of Households with at Least One ITN” 1999-2009/10 “Proportion of children under five reporting sleeping 1999-2009/10 under an ITN the night prior to interview” IPTp “Proportion of women with a live birth in the 2 years 2004/5-2009/10 prior to survey who received ≥2 doses of SP during her most recent pregnancy, ≥1 via ANC” Assess the effect of SP stock-out (program data) Treatment “Proportion of under-fives with self-reported fever 1999-2009/10 in the previous two weeks treated with first line anti- malarial same/next day after fever onset”GHI Presentation, March 22nd , 2012, USAID/Washington DC
  19. 19. Impact Indicators Health outcome Indicator Trends Morbidity Parasitemia prevalence: Proportion of children 2007/08 only + aged 6-59 months with malaria infection NMCP surveys Anemia prevalence: Proportion of children age 2004/5, 2007/8, 6-59 months with a hemoglobin measurement 2009/10 of <8 g/dL Fever prevalence: Proportion of children aged 1999, 2004/05, 6-59 month with reported fever 2007/8, 2009/10 Mortality All-cause under five mortality rate (5q0) 1990-2009/10GHI Presentation, March 22nd , 2012, USAID/Washington DC
  20. 20. Contextual Factors  Fertility risk, female education, literacy, access to media  Housing condition  Nutrition status  Health care utilization  Health interventions: ANC, Vaccination, Vit A, PMTCT  Co-morbidities (ARI, diarrhea)  Climatic factors-Rainfall pattern from 1990  Change in GDPGHI Presentation, March 22nd , 2012, USAID/Washington DC
  21. 21. Plausibility-Morbidity and Mortality  Timing of the change  Space, dose-response, age pattern  Correspondence malaria morbidity and mortality change  Other factors  LiST deaths averted (magnitude expected)GHI Presentation, March 22nd , 2012, USAID/Washington DC
  22. 22. Additional Analysis  District level analysis- HMIS : o Factors associated with malaria case incidence o Factors associated malaria deaths  Analysis of cross-sectional datasets to assess the association of ITN exposure (and other control measures) with malaria health outcomes; and  Survival analysis - ITN exposure and child survivalGHI Presentation, March 22nd , 2012, USAID/Washington DC
  23. 23. Challenges - Technical  Timing of collection of intervention data compared with measurement of outcomes  Changing drug policies make it difficult to link trends in treatment with trends in morbidity/mortality  Early surveys did not contain standard questions necessary for calculating some of these indicators (i.e ITN use)  Data were not always available for the required period  Plausibility versus causalityGHI Presentation, March 22nd , 2012, USAID/Washington DC
  24. 24. Challenges - organizational  Country ownership - Involving country- NMCP  Getting all partners to agree one approach  Review process: o Consider time to get feedback from the TAG o Having the report approved by all partners involved takes timeGHI Presentation, March 22nd , 2012, USAID/Washington DC
  25. 25. Lessons learned  Country ownership, but need external oversight to move the process  Plausibility approach yes, but further district level analysis is needed (where possible with available data) to strengthen the results  National level estimates from Survey are good, but HMIS and HDSS data could complement and help tell the storyGHI Presentation, March 22nd , 2012, USAID/Washington DC
  26. 26. MEASURE Evaluation is a MEASURE project funded by the U.S. Agency for International Development and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. Government. MEASURE Evaluation is the USAID Global Health Bureaus primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.GHI Presentation, March 22nd , 2012, USAID/Washington DC
  27. 27. Thank You!GHI Presentation, March 22nd , 2012, USAID/Washington DC

×