Effects of e-Rx on potential ADEs:*Sig. RR from 35% to 98% (6/9, RR=0.02-0.65)*Sig. RR of 29% (1/9, RR=1.29, 95%CI: 1.04-1.60): [Mitchell D, et al. J Inform Tech Healthcare 2004;2(1): 19-29] small study*Inconclusive result (2/9): RR 0.63, 95%CI: 0.38-1.05 [Gandhi 2005] & RR 0.97, 95%CI: 0.16-5.82 [Bizovi 2002] small studies [Gandhi TK, et al. J Gen Intern Med 2005;20(9):837-841, Bizovi KE, et al. AcadEmerg Med 2002;9(11):1168-1175]Effects of e-Rx on ADEs:*Sig. RR from 30% to 84% (4/6, RR=0.16-0.70)*Non-sigRR of 13% (1/6, RR=0.87, 95%CI: 0.39-1.94) [Mullett CJ, et al. Pediatrics 2001;108(4):E75]*Small & non-sig. RR of 9% (1/6, RR=1.09, 95%CI: 0.92-1.29): [Bates DW, et al. JAMA 1998:280(15):1311-1316]
Home-grown systemsvs Commercial systems:* Higher relative risk reduction (RRR) by home-grown than commercial systems.*Home-grown CPOEs (11/23): Sig. RR from 13% to 99% (RR=0.01-0.87, 95%CI: 0.00-0.94)*Commercial CPOEs (12/23): Non-sig changes, ranges from 96% RR to 26% RR (RR=0.04-1.26, 95%CI: 0.00-1.55)** Reason: Home-grown systems can be modified & adapted more easily and quickly to local needs.
Evidence in Digital Healthcare
The evidence for digital healthcare Jeremy Wyatt and Kevin Yap
Questions about digital health technologies Question Information needed Type of study1. Can it be done ? That a prototype exists Feasibility study2. Is it safe ? Hazard analysis; accuracy of Inspection eg. HAZOP; pilot data, advice etc. tests of accuracy, usability3. Is it acceptable? User attitudes, usage rates Interviews, survey, log file analysis4. Is it clinically Size of benefit vs. harms Pragmatic randomised trial effective ?5. Is it cost Incremental cost NICE technology appraisal effective? effectiveness ratio
“We know it works”“… Full advanced life-support did not decrease mortality ormorbidity... during advanced life-support, mortality was greateramong patients with Glasgow Coma Scale scores < 9”Stiell IG et al. CMAJ. 2008; 178: 1141-52Solution: do a trial - Liu & Wyatt, JAMIA 2011
Plausible DHC technologies that failed Diagnostic decision support (Wyatt, MedInfo ‘89) Integrated medicines management for a children’s hospital (Koppel, JAMA 2005) MSN messenger triage (Eminovic, JTT 2006) Telehealth for COPD (Polisena, JMIR 2010) Smart home applications ? (Martin, CDSR 2008): “The effects of smart technologies to support people in their homes are not known. Better quality research is needed.”
The serious games evidence base 1 randomised trial: – 91 learners about triage randomly allocated to serious game or card sort control – Outcomes: skill at triage of 8 simulated cases – Results: • Accuracy (0 or 1 errors on 8 cases): 91% game, 80% control (p=0.02) • Time taken: game 456, card sort control 435, p=0.155 (Knight, de Freitas, Dunwell et al. Resuscitation 2010; 81: 1175-9) 0 Systematic reviews
Cochrane review on phone consultation / triage Range of study types: 5 RCTs, 1 CCT, 3 ITS up to 2007 Results: 3/5 studies on GP visits showed drop, but 2 showed rise in later visits Of 7 studies on A&E usage, 6 showed no change & 1 showed an increase 2 studies looked at deaths – no difference “Phone consultation appears to reduce the no. of surgery contacts & OOH visits by GPs, but questions remain on service use, safety, cost & pt. satisfaction”(Burin et al, CDSR 2010)
e-prescribing & adverse drug events 4 studies showed significant reduction in ADEs by 30-84% 1 study showed non-sig. increase in risk by 9% Ammenwerth E, et al. JAMIA 2008;15(5):585-600.
TeleHealth in diabetes, bronchitis & heart failureDiabetes (Farmer et al SR, 2005): – Slight reduction of HbA1C by 0.1% (95% CI -0.4% to 0.04%) – Use of services no different or increased with telehealthBronchitis (Polisena et al SR, 2010): – Mortality may be greater in telephone-support group (RR = 1.2; 95% CI 0.84 to 1.75) – Reduced hospitalization and A&E visits, but impact on hospital bed days variedHeart failure (Inglis et al, CDSR 2010): – Reduced mortality by 44% (RR 0.66, CI 0.54-0.81, p < 0.001) – Reduced CHF-related admissions by 23% (RR 0.77) – However, recent large RCT negative (Chaudry NEJMed, Dec 2010)
The evidence base for digital healthcare ? Quality Safety Prevention Productivity Electronic records + ++ ++ +/- Phone consultations ++ +/- - ++ Email, SMS + +/- - +/- Decision support +++ ++ ++ ++ Telemedicine + + - + Remote monitoring +/- - - + Serious games + + + +/- Virtual reality + + +/- +Importance of context:• Setting: community, primary, secondary care• Users: nurses, doctors, therapists, the patient• Care groups – age, disease, severity…• Other benefits: improved patient access, education
Case study: decision support systems“A knowledge-rich system that processes two or moreitems of patient data to generate encounter-specificadvice or interpretation” Wyatt & Spiegelhalter, 1991Used to generate: Advice about diagnosis Prescribing alerts & reminders Interpretation of lab test results…
New errors introduced by DSS Inappropriate drug form selected for route – e.g. capsules for IV administration Inappropriate product selected Incorrect dose, frequency, formulation from dropdown menu Inappropriate selection of default doses Missed drug allergies / high severity interactions – high override rate Duplicate orders – system could not distinguish regular, one off & PRN orders Failure to stop drugs that are no longer required Increased drug monitoring errors – eg. fail to include diluents if CV line not selected
Reasons for negative ACORN trial ACORN was too slow (1986 !) so advice given too late Nurses not empowered to act on advice Cardiac care unit always full ACORN also used in 15% of controls Patients, not nurses, randomised – learning effect
When do decision support systems work ? Success rates across trialsTarget clinical practice Clinical practice Patient outcomesDiagnosis 40% 4/10 0% 0/5Disease management 62% 23/37 18% 5/27Single drug prescribing, 62% 15/24 11% 2/18dosingPrevention 76% 16/21 0% 0/1Multi-drug prescribing 80% 4/5 0% 0/4Overall 64% 62/97 13% 7/52 Garg et al, JAMA 2005, 293: 1223-38
Home-grown vs commercial systems Commercial: 96% reduction to 26% increase (NS) Home grown: strong effect 99% to 16% reduction (p<0.05)Ammenwerth E, et al. JAMIA 2008;15(5):585-600.
Cost effectiveness of DSS to adviseon warfarin dosage Clinically effective & promote safety: trials show 3- 13% increase in patient time within therapeutic range Very cost effective: incremental cost effectiveness ratio £2200 per quality adjusted life year (NICE technology appraisal 2005)
Conclusions1. Technology can harm as well as help2. Evidence is often context sensitive3. Some evidence of effectiveness, little on cost effectiveness / savings4. The wrong kind of evaluation studies (eg. Chaudry) can mislead5. Working with EU Commission to improve design of evaluation studies in Framework 7/8 projects6. IDH will be publishing evidence reviews on selected technologies over next few months