Recommended
More Related Content
What's hot
What's hot (20)
Similar to Easier said than done: experiences and challenges in natural experimental evaluations
Similar to Easier said than done: experiences and challenges in natural experimental evaluations (20)
Recently uploaded
Recently uploaded (20)
Easier said than done: experiences and challenges in natural experimental evaluations
- 1. Easier said than done: Experiences and challenges in natural experimental evaluations Jenna Panter MRC Epidemiology Unit | University of Cambridge
- 3. To elucidate the potential of population-level approaches to physical activity behaviour change by: • Evaluating the effects of environmental and policy interventions • Understanding related patterns and mechanisms of behaviour change Aims of our research programme
- 5. Many research questions in public health remain unanswered due to the insidious nature of the prevailing evaluation research paradigm. Nutbeam, 2016
- 7. Craig et al., J Epidemiol Community Health 2012 What is a natural experiment? Exposure to the intervention is:
- 8. Craig et al., J Epidemiol Community Health 2012 What is a natural experiment? Exposure to the intervention is: • Not controlled by evaluator
- 9. Craig et al., J Epidemiol Community Health 2012 What is a natural experiment? Exposure to the intervention is: • Not controlled by evaluator • Usually not randomised
- 10. A ‘natural experiment’ is something that happens and can be evaluated.
- 11. A ‘natural experiment’ is something that happens and can be evaluated. It is not a study design or method of analysis as such.
- 12. An ‘experiment of opportunity’ Morris, Uses of epidemiology 1957
- 14. Mortality 0 per 100,000 Mortality 114 per 100,000 Mortality 60 per 100,000
- 15. All outcomes in the best available natural experiments that have investigated the causal effect of changes to the built environment on physical activity had either an overall critical (n = 12) or serious (n = 3) risk of bias. Thus, according to principles of the ACROBAT-NRSI, four fifths of included outcomes are ‘too problematic to provide any useful evidence on the effects of intervention’ and one fifth ‘have some important problems’ Benton et al., 2016
- 18. …using tools like ACROBAT-NRSI without careful consideration of the practical difficulties in generating research evidence in certain areas of public health may serve to set the bar of methodological acceptability too high Humphreys et al., 2016
- 19. • Lack of control over intervention • Accessing v collecting data • Assessing exposure to the intervention • Causal inference Challenges
- 20. • Lack of control over intervention • Accessing v collecting data • Assessing exposure to the intervention • Causal inference Challenges
- 21. Health impacts of the Cambridgeshire Guided Busway: the Commuting and Health in Cambridge study
- 27. 2009 Pre 2012 Post 2010 Pre Intercept survey In depth studies In depth studies In depth studies In depth studies Cohort study Cohort study Cohort study Cohort study 2011
- 28. Panter et al., under review Cycling Walking Commuting 1.34 (1.03, 1.76) 0.90 (0.69, 1.18) Adjusted relative risk ratios (95% confidence intervals) for an increase in weekly duration of the given behaviour per unit of proximity (square root of distance) to busway. N=469 Effects on cycling and walking time
- 29. 1 km4 km 9 km 4 km 34% more likely 34% more likely
- 30. Panter et al., under review RRR (95% CI) Cycling Walking Commuting 1.34 (1.03, 1.76) 0.90 (0.69, 1.18) Mean increase +86 min·wk-1 Effects on cycling and walking time Adjusted for age, sex, education, car ownership, home ownership, children, health condition, body mass index, urban-rural classification, distance to work, car parking provision at work, baseline level of active commuting and home or work relocation
- 31. Cycling Walking Commuting 1.34 (1.03, 1.76) 0.90 (0.69, 1.18) Mean increase +86 min·wk-1 Commuting plus recreation 1.32 (1.04, 1.68) Effects on cycling and walking time Adjusted for age, sex, education, car ownership, home ownership, children, health condition, body mass index, urban-rural classification, distance to work, car parking provision at work, baseline level of active commuting and home or work relocation Panter et al., AmJ Prev Med 2016
- 32. • Lack of control over intervention • Accessing v collecting data • Assessing exposure to the intervention • Causal inference Challenges
- 35. Cycling to workPrevalence(%,95%CI) 02468 1981 1991 2001 Intervention towns Intervention period Goodman et al., SocSci Med 2013 2011
- 36. Cycling to workPrevalence(%,95%CI) 02468 1981 1991 2001 Matched comparison Intervention towns National comparison Unfunded comparison Intervention period Goodman et al., SocSci Med 2013 2011
- 37. Cycling to work Difference-in-differences 0.69 (0.60, 0.77) Ratio-of-ratios 1.09 (1.06, 1.11) Prevalence(%,95%CI) 02468 1981 1991 2001 Matched comparison Intervention towns National comparison Unfunded comparison Intervention period Goodman et al., SocSci Med 2013 2011
- 39. • Lack of control over intervention • Accessing v collecting data • Assessing exposure to the intervention • Causal inference Challenges
- 43. Data collected • Participants sampled from local area & followed over time. • Self-reported demographic, health, activity & travel data • Exposure to the intervention was defined as proximity to the nearest access point. 2010 2012 1510 participants1510 participants 3156 participants Goodman et al., 2014 Am J Public Health ; Ogilvie et al., 2012 BMJ Open
- 44. Data analysis Step 3: Test entire model in path analysis Step 2: Define mediation model to be tested Step 1: Factor structure of physical environmental items
- 45. Step 1: Factor structure * * * * * * * * safety * infrastructure
- 46. Step 2: Identifying plausible causal pathways Adjusting for baseline age, sex, site, ethnicity, education, income, employment status, car and bicycle ownership, general health, and time spent walking and cycling
- 47. Step 2: Identifying plausible causal pathways Adjusting for baseline age, sex, site, ethnicity, education, income, employment status, car and bicycle ownership, general health, and time spent walking and cycling
- 48. Step 3: Testing causal pathways Adjusting for baseline age, sex, site, ethnicity, education, income, employment status, car and bicycle ownership, general health, and time spent walking and cycling
- 49. Step 3: Testing causal pathways Adjusting for baseline age, sex, site, ethnicity, education, income, employment status, car and bicycle ownership, general health, and time spent walking and cycling
- 50. Panter and Ogilvie, 2015 BMJ Open
- 53. Take home messages • It’s much easier to critique studies than do them • These studies are difficult but important • The answer to these evaluations are often not easily quantifiable • Internal validity of studies are important but its important to balance rigour and pragmatism • Flexibility from everyone is key
- 54. Sam Hajna Anna Le Gouais David Ogilvie Jenna Panter Lindsey Smith tinyurl.com/cam-paph The Physical Activity and Public Health research programme is supported by the Medical Research Council.The research described in this talk was carried out as part of the Centre for Diet andActivity Research (CEDAR), a UKCRC Public Health Research Centre of Excellence funded by the British Heart Foundation, Economic and Social Research Council, Medical Research Council, National Institute for Health Research (NIHR) andWellcomeTrust under the auspices of the UKClinical Research Collaboration. The evaluability framework was an independent report commissioned and funded by the Policy Research Programme in the Department of Health.The iConnect study was funded by the Engineering and Physical Sciences Research Council.The M74 study was funded by the NIHR Public Health Research programme. Anna Goodman and Jenna Panter were supported by NIHR Postdoctoral Fellowships. The views expressed are those of the author and not necessarily those of the Public Health Research programme, NHS, NIHR or Department of Health.