More Related Content

Slideshows for you(20)

Similar to Validating Intervention Coverage Indicators for Maternal Postnatal Care(20)


More from MEASURE Evaluation(20)


Validating Intervention Coverage Indicators for Maternal Postnatal Care

  2. Overview • Study 1 – immediate postnatal care (PNC) (within 1 hour of birth) • Study 2 – return PNC visit (24hrs – 10 weeks of birth) • Summary of overall findings
  3. Health Risks in Postnatal Period • More maternal deaths occur in period between first 24 hours of birth up to 6 weeks after delivery than any other phase of pregnancy (36%) • 2 of 5 child deaths <age 5 occur within first 28 days of life • Healthy behaviors (e.g., breastfeeding, immunization visits, use of postnatal HIV services) can have lifelong effects on health • Despite health risks during this period, only one-third of women in LMIC receive a PNC visit within the first 2 days of birth (UN, 2015) (1) United Nations, Millennium Development Goals Report, 2015.
  4. Maternal PNC Coverage Indicators in DHS/MICS - After delivery, whether anyone checked on woman’s health (if so who, when, and where check occurred) - After left health facility [or two months following birth if a home birth], whether anyone checked on woman’s health (if so health (if so who, when, and where check occurred)
  5. Newborn PNC Coverage Indicators in DHS/MICS - After delivery, whether anyone checked on newborn’s health (if so who, when and where check occurred) - After left health facility, whether anyone checked on newborn’s health (if so who, when and where check occurred) - Early initiation of breastfeeding (within 1 hour of birth) - Newborn vaccination (DHS) - Content of PNC check: examine cord, newborn temperature, temperature, counsel woman on danger signs for newborns, counsel woman on breastfeeding, observe breastfeeding (DHS R7) - Newborn thermal care: drying, skin-to-skin contact, delayed bathing - Cord care (DHS Pregnancy & PNC Module)* * Module is optional and currently being piloted
  6. Research Question How accurately can women report the content of PNC received (1) immediately postpartum or (2) during a return facility visit? • Study 1: Immediate PNC Interventions (1hr of delivery) – Studies in Kenya and Mexico (2012 - 2013) compare observations of labor and delivery to women’s exit interview [1,2] – In Kenya re-interviewed women 13-15 months later to examine accuracy of reporting over time [3] • Study 2: PNC Return Visit (24 hr - 10 wks of delivery) – Secondary analysis collected under Integra Study in Kenya and Swaziland (2009 – 2012) – Postnatal women’s visit to a health facility (1) Blanc et al., J Global Health. 2016: 6(1). (2) Blanc et al. BMC Preg & Childbirth. 2016:16(255). (3) McCarthy et al., J Global Health. 2016:6(2).
  7. Validation Analysis 1. Individual Level Accuracy: • Sensitivity (Se) and specificity (Sp), summarized as area under receiver operating curve (AUC) • 0 – 1 scale 2. Population Level Accuracy: • Psurvey-based = (true coverage*Se) +(true coverage*(1 – Sp)) • Inflation factor (IF)- ratio of estimated survey-based prevalence to true coverage - AUC<0.60 = low validity - 0.60<AUC<0.7 = moderate - AUC>0.70 = high validity Acceptable overall performance: AUC>0.60 and 0.75<IF<1.25 - 0.75 < IF < 1.25 = low bias
  8. Study 1: PNC Interventions Received Immediately Postpartum (within 1 hour of birth) • Kenya and Mexico • 2012 – 2013
  9. Study 1 Design. Immediate PNC Women with matched baseline data N = 662 Successfully located N = 568 Location Rate: 93.7% Not located, moved or death N = 38 Non-Location Rate: 6.3% Completed interview N = 515; Follow- up Rate: 90.7% Refused or did not complete interview N = 53; Refusal Rate: 9.3% 515 women with matched baseline and follow-up data. Sample: • Women ages 15-49 whose labor & delivery was observed • 2 hospitals in Central and Eastern Kenya; 1 hospital in Mexico City • In Kenya re- interviewed women who consented to follow-up at their Mexico Women who consented to home- based follow-up interview N = 606 (Acceptance rate: 91%) 13-15monthspostLaboranddelivery Consented to study participation N = 616 Successfully observed during labor N = 609 (Observation rate: 98.8%) Completed exit interview N = 597 Follow- up Rate: 98% 597 women with matched data Lost to follow-up or refused survey N=12; Loss to FU rate 2% Could not be located or sent home N=7; Loss to FU rate 1.2% Kenya Consented to study participation N = 1039 Did not progress into labor, sent home, lost to FU N=373; Loss to FU rate 35.8%
  10. Study 1. Sample Characteristics Kenya Sample: N=662 baseline; N=515 follow-up • Mean age: 26 ± 0.22 • 48% 1 prior birth, range: 1-8 prior births • 44% primary school is highest education • 85% married or living together • 13% cesarean rate • Mean age: 24 ± 0.31 • 52% 1 prior birth, range: 1-7 prior births • 92% completed secondary school or higher • 74% married or living together Mexico Sample: N=597 baseline
  11. Aspects of Immediate Maternal PNC Measured (within 1 hour of birth) Uterine massage performed following delivery of placenta Physical exam following delivery to: • Check for bleeding • Take blood pressure • Check perineum • Take temperature • Examine whether belly was becoming firm
  12. Immediate PNC Interventions for Mother (w/in 1 hr), Indicator Accuracy Immediate PNC for Mother (up to 1 hr after birth) KENYA MEXICO Individual Accuracy R0 | R1 Population Accuracy R0 | R1 Individual Accuracy Population Accuracy After delivery of placenta, provider firmly massaged lower abdomen to help womb become firm - | - - | + - + In the first physical examination after delivery, did a health provider…. Look for or ask you about bleeding - | - - | + - + Check your blood pressure - | - - | + NA NA Do a perineal exam - | - - | - - + Take your temperature + | - - | + - - Check belly to see if it was becoming firm (involution) - | - + | - - + R0 = Exit interview at hospital discharge; R1 = Re-interview 13-15 months postpartum. Individual Accuracy: ⎼ =AUC<0.60; + = 0.60<AUC<0.70; ++ = 0.70<AUC; Population Accuracy: + = 0.75<IF<1.25; NA = insufficient sample size for country.
  13. Qualitative Interviews: How Difficult is it for Women to Recall Immediate PNC Care? • Kenya: In-depth interviews (N=20) with sub-sample of women at re-interview (13 to 15 months postpartum) • Relative to interventions received in the first stage of labor, women had difficulty recalling immediate postnatal care, particularly for the mother “You know that time [after the birth] I was over excited so after the caesarean section I was happy to see my child like this and I gave God my thanks, so I cannot know because once I saw the baby I was tired so whatever happened after that I don’t know”.
  14. Women’s Immediate Postnatal Experiences in Kenya “…When you asked me if the baby was placed on my chest against my skin, that was hard for me to remember because because at that time I was tired because I had gone without gone without sleep for two days.”
  15. Study 2: PNC Interventions Received 24 Hours – 10 weeks of Birth • Kenya and Swaziland • 2009 – 2012
  16. Study 2 Design. PNC Interventions Received within 24hrs - 10 weeks of birth 1. Direct Observation of Health Visit 12 health facilities in Kenya (KY); 8 health facilities in Swaziland (SZ) • KY: Central/ Eastern provinces; SZ: Lubombo, Manzini, Shiselweni regions • Part of SRH and HIV care integration study • High volume (>50 infant immunizations, >100 FP clients/ month) • Offered PNC, FP, voluntary HTC, STI treatment, PMTCT 2. Exit Interview with Women on Received Care at Facility Discharge • Women ages 15 - 44 years • Presenting at study health facility for health check for themselves and/or their infant (aged >24 hrs – 10 wks) • Informed consent Assess validity of self- reports
  17. Study 2. Sample Kenya N=545 Swaziland N=319 % % Age of Client 15-24 44.2 54.9 25-34 44.8 38.4 35-44 11.0 7.3 Age of Baby <2wks 21.3 16.6 2-6wks 66.2 76.4 7-10wks 12.5 7.0 Married/live together 86.0 44.7 Prior Parity 1 29.9 34.2 2 23.2 25.3 3+ 47.0 40.5 Education Level None /pre-primary 39.8 7.5 Primary 41.1 18.5 Secondary+ 19.1 74.0
  18. Study 2. Aspects of Maternal PNC Measured • Maternal physical exam • Maternal danger signs advice • Return to fertility and birth spacing information • Family planning methods discussion / provision • STI/HIV risk assessment
  19. Study 2. Validation Results • 18 maternal PNC indicators attempted • 13 indicators in KY and 15 in SZ had adequate sample size for validation
  20. Study 2. Indicators of PNC Examination for Mother INDICATORS Kenya Individual | Population-level Accuracy Swaziland Individual | Population-level Accuracy Met Both (AUC & IF) KY | SZ Take blood pressure ++ | + + | + KY | SZ Examine breasts ++ | + ++ | + KY | SZ Examine abdomen ++ | + ++ | + KY | SZ Examine vagina + | + ++ | + KY | SZ Check anemia (pallor or refer for HB test) ++ | + - | + KY Screen for cervical cancer NA | NA + | - KY = Kenya; SZ = Swaziland; NA = Insufficient sample size for country Individual Accuracy: ⎼ =AUC<0.60; + = 0.60<AUC<0.70; ++ = 0.70<AUC Population Accuracy: + = 0.75<IF<1.25
  21. Study 2. Indicators of PNC Provider Contact and Health Counseling for Mother INDICATORS Kenya Individual | Population-level Accuracy Swaziland Individual | Population-level Accuracy Met Both (AUC & IF) KY | SZ Contact with nurse or nurse/midwife NA | NA NA | NA NA Contact with doctor NA | NA NA | NA NA Ask about excessive bleeding ++ | + - | + KY Discuss danger signs after birth ++ | + - | + KY Discussed STIs or HIV/AIDS ++ | + - | + KY KY = Kenya; SZ = Swaziland; NA = Insufficient sample size for country Individual Accuracy: ⎼ =AUC<0.60; + = 0.60<AUC<0.70; ++ = 0.70<AUC Population Accuracy: + = 0.75<IF<1.25
  22. INDICATORS Kenya Individual | Population-level Accuracy Swaziland Individual | Population-level Accuracy Met Both (AUC & IF) KY | SZ Discuss how soon after delivery a woman can get pregnant + | + - | + KY Discuss return to fertility + | + - | - KY Discuss benefits of birth spacing ++ | + + | + KY | SZ Discuss return to sexual activity ++ | + + | - KY Discussed a FP method (incl natural) ++ | + - | + KY Received any modern FP method NA ++| + NA | SZ Explains advan/disad of chosen FP method NA + | + NA | SZ Study 2. Indicators of PNC Return to Fertility & Family Planning Counseling Individual Accuracy: ⎼ =AUC<0.60; + = 0.60<AUC<0.70; ++ = 0.70<AUC Population Accuracy: + = 0.75<IF<1.25; NA = Insufficient sample size for country
  23. Summary of Maternal PNC Indicators Immediate PNC: • No indicators met both accuracy criteria in Kenya or Mexico • 4 indicators were appropriate for population-level measurement only in each country • Whether (1) received uterine massage and (2) provider checked for bleeding met at population level accuracy only in both countries Return PNC: • 13/13 maternal PNC indicators in KY and 7/15 in SZ met individual and population-level accuracy • 5 PNC indicators met both criteria in both countries Whether during the PNC consultation the provider… • Checked the woman’s blood pressure • Did a breast exam • Examined the woman’s abdomen • Examined the woman’s vagina • Discussed the benefits of birth spacing
  24. Summary of Maternal PNC Indicators • In general, women are able to report on maternal PNC interventions received at a return visit with higher accuracy than those received immediately (within 1 hour) postpartum • The same trend was true for newborn PNC indicators (not presented) • Influence of delivery context? • Findings differ by country, lower overall accuracy in Mexico and Swaziland vs. Kenya • Limitation: Can’t distinguish between differences in understanding of the question, wording, recall, or error introduced by poor interviewing • Findings suggest additional content of care PNC indicators can be accurately measured
  25. This work was supported by the Maternal Health Task Force through the Bill & Melinda Gates Foundation and the Intervention Coverage Measurement Group, also through the Bill & Melinda Gates Foundation. Findings of these studies are based on work in collaboration with Council colleagues Karla Berdichevsky, Claudia Diaz, James Kimani, Brian Mdawida, Charity Ndwidga and Charlotte Warren.
  26. The Population Council conducts research and delivers solutions that improve lives around the world. Big ideas supported by evidence: It’s our model for global change. Ideas. Evidence. Impact.
  27. Questions • There are only a handful of validation studies; almost all conducted in hospitals or high-volume clinics. Do we need to examine accuracy of reporting among women who gave birth in other types of facilities (or at home)? • Do we need to test longer recall periods? Do we need to examine validity in other settings? • What are alternatives to population-based surveys for collecting data on intervention coverage for PNC? • Can we identify a (small) set of critical and valid indicators that can be recommended for inclusion in household surveys (e.g., DHS, MICS)?
  28. © 2014The Population Council. All rights reserved. Use of these materials is permitted only for noncommercial purposes. The following full source citation must be included: This presentation may contain materials owned by others. User is responsible for obtaining permissions for use from third parties as needed. Ann K. Blanc, Charlotte Warren, Katharine McCarthy, Brian Mdawida. Validating intervention coverage indicators for maternal postnatal care. Webinar hosted by MEASURE Evaluation. September 2017.

Editor's Notes

  1. The analysis involved measuring validity by comparing, for each indicators, what the observers reported (which we define as ‘true’ or the ‘gold standard’) against what the woman reported. The basic measures are: Sensitivity (or the proportion of true positives) - generally this means cases where the observer and the woman both reported that something occurred Specificity (or the portion of true negatives) – generally this means cases where both the observer and the woman reported that something didn’t occur The area under receiver operating curve or AUC- summarizes sensitivity and specificity information in one indicator of individual level accuracy. AUC varies from on a zero to one scale with AUC of 0.5 equivalent to a ‘random guess’ No established criteria in terms of cut-off benchmarks for AUC. We adopted an approach that uses 3 categories that divide the results into low, moderate, and high validity depending on the value of AUC following what others have done in the literature. (Stanton, PLOS One, 2013) For a measure of population level accuracy, we use the Inflation Factor. We also applied the sensitivity and specificity measures calculated in this study to each indicator’s “true coverage” in the sample (aka- the prevalence as calculated from the observer data) to assess the estimated prevalence of the indicator that would be achieved by conducting a survey of women. A factor of 1 would mean that the true prevalence and the survey prevalence (or coverage) are the same. A factor that is substantially different from 1 means that the true prevalence and the survey prevalence vary. We have classified indicators with an Inflation Factor between 0.75 and 1.25 as having low bias. Using an equation by Vecchio, each indicator’s estimated sensitivity (SE) and specificity (SP) was applied to its true prevalence (P) (i.e., observer report) to estimate the prevalence that would be obtained using a population-based survey (Pr) Pr = (P * SE) +(P*(1-SP)) - essentially this equation is counting all “true” and “false” positives in the sample (i.e., true positive rate * coverage + false positive rate * coverage) Next we compared the ratio of the estimated survey-based prevalence to its true prevalence, to estimate the degree to which each indicator would be over or under-estimated if assessed using a population-based survey = known as Inflation Factor Est. survey-based prevalence / True coverage or (Pr/P) Complex relationship between bias, validity and recorded coverage (Liu, 2013) – when coverage is high, the TAP ratio approx. equals sensitivity and is independent in specificity. In cases of high coverage and high sensitivity indicators – IF will not deviate greatly from 1 Therefore- high coverage could limit power to accurately estimate specificity, difficult to measure low coverage indicators