Your SlideShare is downloading. ×
Advances in Outcome Monitoring
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Advances in Outcome Monitoring

744
views

Published on

Presented by Scott McKeown, Janine Barden O’Fallon and Jennifer Chapman at the MEASURE Evaluation End-of-Phase-III Event.

Presented by Scott McKeown, Janine Barden O’Fallon and Jennifer Chapman at the MEASURE Evaluation End-of-Phase-III Event.


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
744
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Introduction to Session on Advances in Outcome Monitoring Scott McKeown, MPH DrPH Jack Hazerjian, MPH MEASURE Evaluation End-of-Phase-III Event, May 22, 2014
  • 2. The basic tracking of variables that have been adapted as measures or “indicators” of the desired program outcomes. Outcome monitoring does not infer causality; changes in outcome could be attributable to multiple factors, not just the program. Definition: Outcome Monitoring Global HIV M&E Information – TWG UNAIDS MERG
  • 3. Organized2006meetingforappraisal ofmethodologiestomonitorprogram outcomesas acomplementtotheroutine monitoringofprogramoutputs Phase II: Review by Field Experts of Outcome Monitoring Methodologies  Lot Quality Assurance Sample (LQAS) Survey  30 x “n” Cluster Sampling  Cluster Sampling with Stratification  Rider/Omnibus Survey (concurrently conducted with larger household survey)  Longitudinal Demographic and Community Surveillance
  • 4. Consensus on Use of Sampling Methodologies to Measure Health Program Outcomes Sampling Methodologies Health Program Study Areas Lot Quality Assurance Sampling * Immunization rates, MCH service access, HIV risk factors * Operations research, quality management 30 x “n” Cluster Sampling * Immunization, MCH,FP and HIV/AIDS rates Other Cluster Sampling Methods * Widely used for multiple indicators Rider/Omnibus Surveys * FP use, prevalence and contraceptive security *Willingness to pay for FP Longitudinal Community Surveillance * Demographic & health surveillance * Evaluation studies * Health worker training in M&E
  • 5.  Continue use of outcomemonitoring methodologies,as indicated by health programcontext  Develop/refineindicators and toolsfor outcome monitoring Phase III: Outcome Monitoring Priorities o Where good indicators exist – develop improved data collection tools (e.g., for maternal and child health programs) o Where program areas are new – develop indicators and data collection tools (e.g., for orphans and vulnerable children programs)
  • 6. Lot Quality Assurance Sampling  Kenya: o Child health outcomes study (2009, 2010, 2011-2012) o Malaria health behavior outcomes study (2010)  Liberia: o Maternal & child health, water & sanitation outcomes study (2011, 2012, 2013) Phase III Field Work in Outcome Monitoring (1)
  • 7. Cluster Sampling  Zambia: o Sexual behavior study (2009) Stratified Cluster Sampling  Nigeria: o Household survey on women’s reproductive health and children’s health and primary education (2009) Phase III Field Work in Outcome Monitoring (2) Wave 1 May 2009 Wave 2 July 2009 Kano State % urban and rural respondents rejecting common misconceptions about HIV transmission
  • 8. Phase III Field Work in Outcome Monitoring (3) Rider/Omnibus Survey  Angola: o Gender-based violence sub-study within PLACE Survey (2010-2011) Longitudinal Community Surveillance  Uganda: o Study of effect of mobile phone text messaging on uptake of family planning services (2012) Eric Lafforgue (2010) Health Child, Uganda (2013)
  • 9.  Design of appropriate indicators and survey instruments remains within work scope as new health program interventions are introduced  Collaboration with partners in design and implementation of outcome monitoring studies provides a valuable, learning-by-doing opportunity in capacity building Parting Thoughts Outcome Monitoring Under MEASURE Evaluation
  • 10. Rapid Assessments for Monitoring MNCH Knowledge and Behavior Experiences with LQAS Janine Barden-O’Fallon, PhD MEASURE Evaluation End-of-Phase-III Event, May 22, 2014
  • 11. Why Use Rapid Assessments? Benefits  Savings in cost, human resource commitment, and time  Can be conducted as often as needed  Information is optimal for program management Considerations  Low statistical power  Not tools for measuring incremental change
  • 12. The Rapid Household Survey Handbook How To Obtain Reliable Data on Health at the Local Level Two-stage cluster sampling (30 x “n”) and Lot Quality Assurance Sampling (LQAS) Two Methodologies
  • 13. Key Features of LQAS  Goal is to improve project performance  Designed to assist project management  Flags issues needing attention  Provides local level information  Uses a Decision Rule
  • 14. Key Features Of LQAS Cont’d Decision Rule = thepoint at which a sub-project area (or “Lot”) satisfactorily meets the target Example LQAS with 5 lots Sample size = 19 per lot (Total=95) Target for indicator = 80% Decision Rule = 13
  • 15. LQAS for OM of Child Health in Kenya  Pilot in 2 provinces (2009)  Rollout to 6 additional provinces (2010)  Return to 2 provinces (2011/12)  Reflected integrated programming  Focused on strengthening capacity at the district and program management level; “learning by doing”
  • 16. Example Results: Kenya 2009 Indicator: Percentage of mothers of children aged 0-23 months who boil or chlorinate their drinking water to make it safe Western Province Coverage: 44.8% Decision Rule: 6 CI: 33.2-56.4 Districts Sample Size Correct Responses Meets decision rule? Budalangi 19 10 Yes Teso North 19 9 Yes Teso South 19 4 No Busia 19 10 Yes Samia 19 8 Yes
  • 17.  Further train CHWs  Focus on water and sanitation initiatives:  Tablets  Educate community members  Water tanks  VIP latrines  Hand washing Use Of Results: Teso South District, Kenya
  • 18. Example Results: Kenya 2012 Indicator: Percentage of mothers of children aged 0-23 months who boil or chlorinate their drinking water to make it safe Western Province Coverage: 55.9% Decision Rule: 9 CI: 44.2-67.6 Districts Sample Size Correct Responses Meets decision rule? Bunyala 19 11 Yes Teso North 19 11 Yes Teso South 19 11 Yes Busia 19 11 Yes Samia 19 8 No
  • 19. LQAS for OM of Child Health in Liberia  Pilotin 4counties(2011)  Rolloutto7counties(2012)  Capacitybuilding exercises (2013/14)  Reflected integrated programming  Focusedonstrengthening capacityatthe countyandprogrammanagement level; “learning bydoing”
  • 20. LQAS Capacity Building: Liberia  Data use workshops(2013)  MOH-led in 2 counties(2013/14)  CB practicum for MOHSW central and county levels(2014)  Interviewerscomprisedofcounty healthteams(2012)  CB practicum for MOHSWcentral and county levels(2013)
  • 21. Example Results: Liberia 2013 Indicator: Percentage of mothers of children aged 0-23 months who received second dose of IPT for malaria during pregnancy Lofa County Target: 80% Coverage: 61.2% Decision Rule: 13 CI: 51.5-70.9 Health District Sample Size Correct Responses Meets decision rule? A 19 12 No B 19 13 Yes C 19 12 No D 19 6 No E 19 11 No F 19 12 No
  • 22. Rapid Assessments for OM and HIS Basic requirements  Good planning  Understanding type of information  Intention to use information  Human and financial resources
  • 23. Evaluating OVC Outcomes Jenifer Chapman, PhD MEASURE Evaluation End-of-Phase-III Event, May 22, 2014 Global Indicators and Tools for Assessing Child Well-being
  • 24. The Problem  High investment  BUT “what works” in improving household well-being?  Challenge: lack of standardized measures and tools
  • 25. A Proposed Solution Standardized questionnaires for a survey of children and their adult caregivers
  • 26. The Purpose  Standardize population-level data beyond what is available from routine surveys  Produce actionable data to inform programs  Enable comparative assessments
  • 27. Focus on PEPFAR OVC Programs  Indicators needtoreflect&be amenabletochangebyPEPFAR programintervention  HH intervention by home visitors  Low direct funding per target, focus on linkages  Often inadequate services in vicinity
  • 28. Who Are These Tools For?  Local and international research institutions  USAID Forward
  • 29. Guiding Principles 1. Focus on measuring program outcomes 2. Data collection by trained data collectors, not service providers 3. Documented protocol is required 4. Protocol with tools needs to undergo ethical approval 5. Tools require pilot testing in new settings
  • 30. Developing the Tools  Two-phase process  Stakeholder-driven, multi-agency Result:  3 tools  Pilot tested
  • 31. Core questions Optional  Household schedule* (10)  Changes in household composition (4)  Demographic information* (7)  Work* (3)  Access to money (3)  Shelter (1)  Household food security (6)  General health (2)  Caregiver support (4)  Parental self-efficacy (1)  Basic HIV/AIDS knowledge* (7)  HIV testing* (3)  Attitudes to condom education (1)  Household access to services (1)  Household Economic Status  Progress out of Poverty Index or similar (country specific)  Dietary Diversity (1)  Perceptions and experience of abuse, exploitation and violence  Gender roles and decision making power* (9)  HIV/AIDS attitudes* (4) *DHS, bold=core indicator Caregiver Questionnaire
  • 32. Core questions Optional  Confirm demographics (5)  General health & disability (4)  Birth certificate (2)  Vaccinations (11)  Fever (<5 years)* (1)  Diarrhea (<5 years)* (1)  Shelter (1)  Experience of neglect (2)  Slept under mosquito net* (1)  HIV testing experience* (2)  School attendance*, progression/ repeats, drop-outs, missed school days (5+ years) (9)  Work for wages (2)  Early childhood stimulation (2)  Food consumption (2+ years) (8)  Child access to services (1)  Weight*, Height*, MUAC  Fever: extended* (4)  Diarrhea: extended* (3)  Dietary diversity (1) Child Questionnaire (Ages 0-9)
  • 33. Child Questionnaire (Ages 10-17) Core questions Optional  Confirm demographics* (5)  Identity of caregiver (1)  Daily log (6)  School attendance*, progress/ repeats, drop-outs (9)  Chores (3)  Work (7)  Food consumption (8)  Alcohol consumption (3)  Birth certificate (2)  General health & disability (3)  General support (4)  Basic HIV/AIDS knowledge* (7)  HIV testing* (3)  Child access to services (1)  Weight, Height, MUAC  Dietary diversity (1)  Perceptions/ experience of abuse, exploitation, violence  Child development knowledge (6)  HIV/AIDS attitudes and beliefs (4)  Sexual behavior (13-17 yrs) (5)
  • 34. The Toolkit – Where It All Comes Together  Tools & Manual  Template protocol with consent/assent forms  Data analysis guide  Data collector training manual and materials  French translations
  • 35. So What?  We know what we’re reaching for  No more reinventing the wheel  We’re accountable – MER reporting
  • 36. Where can I find out more? Go to our website: http://www.measureevaluation.or g/our-work/ovc Keep in touch on ChildStatusNet: http://childstatus.net/ Email: Jenifer Chapman: jchapman@futuresgroup.com
  • 37. www.measureevaluation.org/eop