This document discusses how predictive risk models can help evaluate interventions in observational studies. It notes challenges with observational studies, such as regression to the mean and lack of control groups. It proposes several solutions to address these challenges, including before-after studies, regression adjustment, matching controls, and regression discontinuity designs. Matched controls are described as a way to reduce dependence on regression model specification by "data pre-processing" to compare intervention patients to similar control patients. The document concludes by surveying the current state of telehealth studies, finding most are descriptive or use before-after designs, with few employing more rigorous controlled designs.

1. © Nuffield Trust
Evaluation methods – where can
predictive risk models help?
Adam Steventon
Nuffield Trust
8 July 2013

2. © Nuffield Trust
The problem with observational studies
Eligible patients
All patients Intervention
patients
n 54,990 556
% aged 85+ 21.6 46.2
Prior emergency admissions 0.5 1.4
Number chronic conditions 1.0 1.1
Predictive risk score 22.3 33.6
Intervention
patients
Source: Steventon et al (2012)

3. © Nuffield Trust
Solutions, 1) before-after study

4. © Nuffield Trust
Solutions, 2) regression adjustment
Y = f(age, number of chronic conditions,
prior emergency admissions,
intervention status)

5. © Nuffield Trust
Eligible patients
Intervention
patients
Matched
controls
All patients Matched
controls
Intervention
patients
n 54,990 556 556
% aged 85+ 21.6 46.2 46.2
Prior emergency
admissions
0.5 1.4 1.4
Number chronic
conditions
1.0 1.3 1.1
Predictive risk score 22.3 33.5 33.6
Solutions, 3) Matched controls
Source: Steventon et al (2012)

6. © Nuffield Trust
How to select matched controls
Propensity score (Rosenbaum and Rubin 1983)
- Predictive risk of receiving the intervention
Prognostic score (Hansen 2008)
- Predictive risk of experiencing the outcome (e.g. emergency
hospitalisation), in the absence of the intervention
Genetic matching (Sekhon and Grieve 2012)

7. © Nuffield Trust
Advantages / disadvantages
Disadvantage – only allows for observed variables
But
Matching as ‘data pre-processing’ – reduces dependence of estimated
intervention effects on regression model specification
Intuitive?

8. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

9. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

10. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

11. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

12. © Nuffield Trust
Solutions, 4) regression discontinuity
Winningthenextelection
Fraction of votes awarded to Democrats in the previous election
Source: Lee and Lemieux (2009)

13. © Nuffield Trust
What is being done at the moment?
Telehealth studies in Pubmed, 2006-2012
Descriptive Before after
Dose
response
Controlled All
Number of studies 3 24 1 16 44
Median number of patients in
telemonitored group (range)
45
(40 to 851)
35
(7 to 17,025)
246
102
(19 to 1,767)*
45
(7 to 17,025)*
Endpoints
Mortality 2 - 1 3 6
Hospital use (or costs) 3 6 - 12 21
Clinical (e.g. HbA1c) - 17 - 4 21
Patient reported outcomes
(e.g. quality of life)
1 10 - 3 14
Source: Steventon, Krief and Grieve (work in progress)

14. © Nuffield Trust
References
Lee DS, Lemieux T. Regression discontinuity designs in economics. 2009.
Available from: http://www.nber.org/papers/w14723.pdf?new_window=1
Sekhon JS, Grieve RD. A matching method for improving covariate balance in
cost-effectiveness analyses. Health economics 2012;21:695–714.
Rosenbaum P, Rubin D. The central role of the propensity score in observational
studies for causal effects. Biometrika 1983;70:41–55.
Hansen BB. The prognostic analogue of the propensity score. Biometrika
2008;95:481–8.

15. © Nuffield Trust8 July 2013
www.nuffieldtrust.org.uk
Sign-up for our newsletter:
www.nuffieldtrust.org.uk/newsletter
Follow us on Twitter: http://twitter.com/NuffieldTrust
© Nuffield Trust
adam.steventon@nuffieldtrust.org.uk