This document discusses heterogeneity in trial data and how analyzing differences within and across studies can provide more useful information for policymaking. It provides examples from water, sanitation and hygiene (WASH) trials that showed impacts varied by gender, socioeconomic status, behavior, location and other factors. Accounting for these differences can help target interventions more effectively and generalize trial results to diverse settings.

1. Heterogeneity in Trial Data: Learning from Difference Rick Rheingans, PhD University of Florida SHARE Research Consortium

2. What Works Best? It Depends Sector debate at the interface of science and policy Based on reasonable questions: what will work best here? Differences between studies Meta-analyses Differences within studies Analytical focus on main effects Differences outside of studies

3. Meta-analyses and Heterogeneity

4. Variability across and within studies Depends on behaviors Depends on who – higher protection among the most vulnerable Depends on initial water quality and other exposures

5. Sources of Variability within Trials Different effect levels in different sub-populations due to behavior or vulnerability Opportunity for targeting Spatial differences environmental conditions affecting exposure Opportunity for geographic targeting Differences between settings based on implementation Opportunity to adjust adjust the intervention

6. Analytical Tools for Teasing Out Difference Random effects models Did the intervention work differently in different communities – especially for cluster randomized trials Effect modification Are there characteristics of individuals or communities that change the impact of the intervention Stratification to look at discrete populations Focus is usually on the main effect

7. Grappling with Differences: School Water, Sanitation and Hygiene Impacts SWASH+ Collaborative applied research and advocacy project led by CARE in western Kenya Cluster-randomized trial in 185 schools Included hygiene promotion, water treatment, sanitation infrastructure, and water supply Objective: Estimate the impact of school WASH interventions on health (helminthes and diarrhea), educational outcomes (absenteeism and performance), and behaviors (e.g., diffusion to homes)

8. What’s the Question? National and global policy and advocacy interest in estimating the main effects Days of absence avoided Percent reduction in diarrhea Compare it to other school investments Compare it to other WASH investments What if the most important answer is - it depends?

9. Differential Impacts of School Water, Sanitation and Hygiene Absenteeism (Freeman et al, 2011) Strong impact for girls (Odds Ratio 0.4), no measureable impact for boys Helminthsreinfections - Differences by gender Ascaris for girls; especially poorest Hookworm for boys; especially poorest Differences by behavior Reduced hookworm reinfection among boys without shoes Diffusion of behavior change (water treatment) to homes Strongest effect among the poorest households Differences between schools and regions

10. Conduct across 3 Districts in western Kenya Differing socio-economic and exposure conditions

11. Trying to Explain Differences in School-Cluster Performance Reveals challenges in sustaining hand washing facilities and water treatment In compliance adjusted analysis, both having HW facilities and treated water are associated with reduced absence

12. Trying to Explain Differences: New Pathways In schools receiving new latrines, children had increases in fecal hand contamination Suggests Importance of latrine cleanliness Interdependence of hand-washing and sanitation Need for anal cleansing materials

13. Implications? What to invest in: De-worming? School uniforms? More teachers? School WASH?

14. Different Conditions and Impact Variability: A Hypothetical Exercise Overall impact estimates provide us with the ‘average’ setting, but what will it be in a particular setting? Assume a setting where on average 35% of under-5 diarrhea preventable through improved sanitation Part of diarrhea burden is due to non-sanitation related exposures Some due to whether the household has sanitation Some due to whether they have to share that facility Some due depending how community’s coverage

15. Different Conditions and Impact Variability: A Hypothetical Exercise Overall impact estimates provide us with the ‘average’ setting, but what will it be in a particular setting? Assume a setting where on average 35% of under-5 diarrhea preventable through improved sanitation Part of diarrhea burden is due to non-sanitation related exposures Some due to whether the household has sanitation Some due to whether they have to share that facility Some due depending how community’s coverage Comm 35% Preventable Household Other

16. What Happens with Greater Community Exposures 65% Preventable? Comm If there is heterogeneity in level of community exposure – Would severe diarrhea rates go up? Would the preventable fraction with sanitation go up? 50% Preventable? Comm 35% Preventable 25% Preventable? Comm Household Household Household Household Other Other Other Other

17. Variability in Community Level Exposure One measure may be population density of people without sanitation Based on cluster-level coverage and population density Varies widely within countries and provinces

18. Variability in Community Level Exposure One measure may be population density of people without sanitation Based on cluster-level coverage and population density Varies widely within countries and provinces Does sanitations impact change?

19. Other Sources of Differences Could also consider heterogeneity in vulnerability (e.g., nutritional status) Increased odds diarrhea mortality with low weight for age (Caulfield et al, 2004) Increased risk of illness, for a given exposure Increased risk of mortality, given illness May not affect the fraction preventable through sanitation, but would increase the number of severe cases preventable

20. Differences in Impact? Median Median 16.7%

21. Differences in Impact? How does sanitation impact vary across the space? Could heterogeneity in impact trial data help us understand how much? Same is likely true for other WASH interventions

22. Variability within Provinces

23. Salvador, Brazil Sanitation Trial Not a randomized trial – repeated cross-sectional study before and after city-wide sanitation project (Barreto et al, 2010) Took advantage of different levels of household and neighborhood change to estimate impact on childhood diarrhea and helminth infections Lessons from heterogeneity within the trial Changes in community sanitation coverage were more important than whether households received a connection Impacts on helminth reduction were strongest among the poorest Showed that intervention reduced the impact of SES on diarrhea disparities

24. Trial Heterogeneity and Generalizability Trials often focus on settings with high levels of burden and homogeneity Increase the measureable impact Reduce the size of the intervention needed However lack of heterogeneity within the trial can make it hard to generalize to a broader setting External validity

25. Example Deworming and Soil Transmitted Helminths Miguel and Kremer tested the impact of deworming for STH on educational outcomes in western Kenya Found that deworming can significantly reduce absenteeism (Miguel and Kremer, 2004); spillover effects; and increased long-term earnings (Baird et al, 2011) However the prevalence of STH was uniformly high within the study, compared to the rest of the country

26. Translating to Heterogeneous Settings Pullan and colleagues developed spatial estimates of national burden to identify where mass treatment would be most appropriate

27. Schistosomiasis Control in China Liang et al examined the impact environmental and chemotherapy interventions for Schisto control Developed mathematical models of transmission Used data from intervention trials to calibrate the models in different settings Identified patterns for generalizing

28. Using Variability to Make a Difference Getting more out of trials Analyzing factors modifying intervention effect Better characterizing mechanisms – connecting interventions to outcomes Exposure and environmental studies Modeling and new analytical techniques Deliberate attention to external validity and generalizability in trial design Better translation Using non-trial outcome data to better understand what happens in more diverse settings Better characterizing contexts for which we would like to know the effect of interventions Better policy signals - to encourage more effective intervention selection