CATCH-IT Journal Club Holbrook et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009 Jul 7;181(1-2):37-44 James Mullen October 19th, 2009 HAD 5726

Outline Background and context of the paper Methodological issues The good, the bad, and the ugly! Questions for the authors

Background and context of the paper

Methodological issues

The good, the bad, and the ugly!

Questions for the authors

The Paper Purpose: “ There have been few randomized trials to confirm that computerized decision support systems can reliably improve patient outcomes” Interesting to: eHealth professionals Healthcare decision makers Policy makers Clinicians Consumers (ie. patients with chronic diseases)

Purpose:

“ There have been few randomized trials to confirm that computerized decision support systems can reliably improve patient outcomes”

Interesting to:

eHealth professionals

Healthcare decision makers

Policy makers

Clinicians

Consumers (ie. patients with chronic diseases)

Study investigators • Anne Holbrook (principal investigator), Division of Clinical Pharmacology and Therapeutics, McMaster University, Hamilton, Ont. • Lisa Dolovich, Department of Family Medicine, McMaster University, Hamilton, Ont. • Hui Lee (deceased), Group Health Centre, Sault Ste. Marie, Ont. • Robert Bernstein, Department of Family Medicine, University of Toronto, Toronto, Ont. • David Chan, Department of Family Medicine, McMaster University, Hamilton, Ont. • Hertzel Gerstein, Department of Medicine, McMaster University, Hamilton, Ont. • Dereck Hunt, Department of Medicine, McMaster University, Hamilton, Ont. • Rolf Sebaldt, Department of Medicine, McMaster University, Hamilton, Ont. • Karim Keshavjee, InfoClin, Toronto, Ont. • Lehana Thabane, Department of Clinical Epidemiology

• Anne Holbrook (principal investigator), Division of Clinical

Pharmacology and Therapeutics, McMaster University, Hamilton, Ont.

• Lisa Dolovich, Department of Family Medicine, McMaster University,

Hamilton, Ont.

• Hui Lee (deceased), Group Health Centre, Sault Ste. Marie, Ont.

• Robert Bernstein, Department of Family Medicine, University of

Toronto, Toronto, Ont.

• David Chan, Department of Family Medicine, McMaster University,

Hamilton, Ont.

• Hertzel Gerstein, Department of Medicine, McMaster University,

Hamilton, Ont.

• Dereck Hunt, Department of Medicine, McMaster University, Hamilton,

Ont.

• Rolf Sebaldt, Department of Medicine, McMaster University, Hamilton,

Ont.

• Karim Keshavjee, InfoClin, Toronto, Ont.

• Lehana Thabane, Department of Clinical Epidemiology

Principle author Dr. Anne Holbrook MD, PharmD, B.Sc, M.Sc, FRCPC, FISPE Holbrook AM, Thabane L, Shcherbatykh IY, O'Reilly D. E-health interventions as complex interventions: improving the quality of methods of assessment . AMIA.Annu.Symp.Proc. 2006:952. Mollon B, Chong J,Jr, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med.Inform.Decis.Mak. 2009 Feb 11;9:11. Mollon B, Holbrook AM, Keshavjee K, Troyan S, Gaebel K, Thabane L, et al. Automated telephone reminder messages can assist electronic diabetes care. J.Telemed.Telecare 2008;14(1):32-36. Shcherbatykh I, Holbrook A, Thabane L, Dolovich L, COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J.Am.Med.Inform.Assoc. 2008 Sep-Oct;15(5):575-580.

Dr. Anne Holbrook MD, PharmD, B.Sc, M.Sc, FRCPC, FISPE

Hypothesis “ The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”

“ The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”

Patients Diagnosed with Diabetes Mellitus >18 years of age Fluent in English Rostered with a community-based primary care providers n = 1610 n = 511 n = 258 n = 253 Total patients identified Total patients eligible Control Intervention

Diagnosed with Diabetes Mellitus

>18 years of age

Fluent in English

Rostered with a community-based primary care providers

Intervention Web-based diabetes tracker of the ‘Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness study II (COMPETE II). Interfaces with the providers EMR Interfaces with an automated telephone reminder system

Web-based diabetes tracker of the ‘Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness study II (COMPETE II).

Interfaces with the providers EMR

Interfaces with an automated telephone reminder system

Intervention COMPETE II EMR Telephone 5 different types of EMRs

Patient Screen *http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/2

Physician Screen *http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/1

Physician Screen *http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/1

Intervention 13 variables selected by a clinical subcommittee Based on practices by CDA, ADA, and internal literature review Mailed out tracker page 4x/year (twice per patient) Phone reminders once a month Tracker updated nightly Patients had the option of personal data entry Trend data was available

13 variables selected by a clinical subcommittee

Based on practices by CDA, ADA, and internal literature review

Mailed out tracker page 4x/year (twice per patient)

Phone reminders once a month

Tracker updated nightly

Patients had the option of personal data entry

Trend data was available

Context COMPETE I : focused on successful implementation of EMRs in small, community-based primary care offices COMPETE II : this study COMPETE III : broadened to vascular risk: diabetes, hypertension, cholesterol, previous heart attacks or stroke.

COMPETE I : focused on successful implementation of EMRs in small, community-based primary care offices

COMPETE II : this study

COMPETE III : broadened to vascular risk: diabetes, hypertension, cholesterol, previous heart attacks or stroke.

Context COMPETE II considered a Pilot project *“A deliberate goal of the project was to have patients regularly visit their family physician” *http://www.hc-sc.gc.ca/hcs-sss/pubs/chipp-ppics/2003-compete/final-eng.php

COMPETE II considered a Pilot project

*“A deliberate goal of the project was to have patients regularly visit their family physician”

Methods Recruit primary care providers who were already using EMRs in their practices Have providers identify patients Selection of patient participants Allocation concealment and central computer generation of group assignment Participants and providers complete baseline questionnaires Patients in intervention sent for set of relevant blood tests (pre-determined) then make appt with FP. (NOT CONTROL GROUP)

Recruit primary care providers who were already using EMRs in their practices

Have providers identify patients

Selection of patient participants

Allocation concealment and central computer generation of group assignment

Participants and providers complete baseline questionnaires

Patients in intervention sent for set of relevant blood tests (pre-determined) then make appt with FP. (NOT CONTROL GROUP)

Timelines Study conducted from late 2002 to the end of 2003 Study follow up was 6 months (mean follow up was 5.9 months)

Study conducted from late 2002 to the end of 2003

Study follow up was 6 months (mean follow up was 5.9 months)

Scores Process composite score Sum of the quality monitoring of the 8 variables Glycated hemoglobin; BP; LDL cholesterol; Albuminuria; BMI; Foot surveillance; Exercise; Smoking; ABC composite Clinical composite score (8 variables) Clinical targets for the composite scores E.g., BP < 130/80 mm Hg

Process composite score

Sum of the quality monitoring of the 8 variables

Glycated hemoglobin; BP; LDL cholesterol; Albuminuria; BMI; Foot surveillance; Exercise; Smoking; ABC composite

Clinical composite score (8 variables)

Clinical targets for the composite scores

E.g., BP < 130/80 mm Hg

Analysis Alpha = 0.05 2-tailed test (1:1 allocation) Used t tests to assess the difference between groups in terms of change in the process composite score

Alpha = 0.05

2-tailed test (1:1 allocation)

Used t tests to assess the difference between groups in terms of change in the process composite score

Analysis They analyzed clinical outcomes in 2 ways: Change in variable (+1 if positive, 0 if no change, -1 if negative change) Whether outcome met predefined targets

They analyzed clinical outcomes in 2 ways:

Change in variable (+1 if positive, 0 if no change, -1 if negative change)

Whether outcome met predefined targets

Process Outcomes Improvement for intervention group for 7 out of the 9 variables (no improvement in exercise of smoking) “ Number of visits to primary care provider increased significantly more in the intervention group than in the control group (diff of 0.66, 95% CI 0.37 to 1.02. P< 0.001).”

Improvement for intervention group for 7 out of the 9 variables (no improvement in exercise of smoking)

“ Number of visits to primary care provider increased significantly more in the intervention group than in the control group (diff of 0.66, 95% CI 0.37 to 1.02. P< 0.001).”

Clinical Outcomes Intervention had an improvement on a number of clinical composite variables that were on target (though not significantly significant!) Statistically significant improvements in BP and glycated hemoglobin

Intervention had an improvement on a number of clinical composite variables that were on target (though not significantly significant!)

Statistically significant improvements in BP and glycated hemoglobin

Other Outcomes Patients were more ‘optimistic’ 75.9% of intervention patients were ‘more satisfied’ with their care No statistically significant changes in quality-of-life measures (SF-12 and Diabetes-39)

Patients were more ‘optimistic’

75.9% of intervention patients were ‘more satisfied’ with their care

No statistically significant changes in quality-of-life measures (SF-12 and Diabetes-39)

Interpretations “ Care of complex chronic disease can be improved with electronic tracking and decision support shared by patients and providers”

“ Care of complex chronic disease can be improved with electronic tracking and decision support shared by patients and providers”

Hypothesis validated? “The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care” Is the result valid?

“The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”

Is the result valid?

What can: Health professionals learn from this? Consumers learn from this? Policy makers learn from this? Researchers learn from this?

Health professionals learn from this?

Consumers learn from this?

Policy makers learn from this?

Researchers learn from this?

Thoughts… Overall… this was a useful in that it adds to the literature base. But I want to know if the benefits outweigh the investment in time, money, and resources.

Overall… this was a useful in that it adds to the literature base. But I want to know if the benefits outweigh the investment in time, money, and resources.

What others are saying *Pros: Successful implementation of an RCT without major flaws such as: temporal trends participatory bias (Hawthorne effect) Environment of the study Not based in a research hospital *Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.

*Pros:

Successful implementation of an RCT without major flaws such as:

temporal trends

participatory bias (Hawthorne effect)

Environment of the study

Not based in a research hospital

What others are saying *Cons: “ The authors ascribe the positive impact of their intervention to the influence of individualized decision support and to the role (again) of reminders” Generalizability *Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.

*Cons:

“ The authors ascribe the positive impact of their intervention to the influence of individualized decision support and to the role (again) of reminders”

Generalizability

Class comments More than 50% of participants never used computers or Internet. What about the patients’ view? Usability of the tracker Did they educate the participants in using computers and the tracker? Is there any data on the usage amount on the tracker by the participants? (interaction) How was composite score validated?

More than 50% of participants never used computers or Internet.

What about the patients’ view?

Usability of the tracker

Did they educate the participants in using computers and the tracker?

Is there any data on the usage amount on the tracker by the participants? (interaction)

How was composite score validated?

Questions If the control group had as many physicians visits as the Intervention group, would there be a difference in care? How would this system improve care with at least a neutral effect on time. (follow up time was only 6 months, what about 2 years?) Were the results (improved score) from the decision support system or from the increase in physician visits - better coordination of care? In other words, what was the causal agent? Decision support? Reminders to patient?

If the control group had as many physicians visits as the Intervention group, would there be a difference in care?

How would this system improve care with at least a neutral effect on time. (follow up time was only 6 months, what about 2 years?)

Were the results (improved score) from the decision support system or from the increase in physician visits - better coordination of care? In other words, what was the causal agent? Decision support? Reminders to patient?

Questions What about the difficulties involved with integrating with COMPETE II with the PCP’s EMRs. What about the technical difficulties. 51% of the Intervention group NEVER used the internet. Did this have an effect on the results? Scoring was +1 if there was improvement, 0 with no change, and -1 without change. How much improvement was measured? Was this improvement a result of the decision support (COMPETE II), automated reminders, or coincidence?

What about the difficulties involved with integrating with COMPETE II with the PCP’s EMRs. What about the technical difficulties.

51% of the Intervention group NEVER used the internet. Did this have an effect on the results?

Scoring was +1 if there was improvement, 0 with no change, and -1 without change.

How much improvement was measured? Was this improvement a result of the decision support (COMPETE II), automated reminders, or coincidence?

Questions Why didn’t the researchers do a repeated measures design t test. Why didn’t the researchers do baseline lab tests on the control group Qualitative data was not explained? Was it anecdotal or statistically calculated? How did they measure if patients were actually using the system? Attrition, adoption, acceptance, and user not really addressed

Why didn’t the researchers do a repeated measures design t test.

Why didn’t the researchers do baseline lab tests on the control group

Qualitative data was not explained? Was it anecdotal or statistically calculated?

How did they measure if patients were actually using the system?

Attrition, adoption, acceptance, and user not really addressed

The end

The end

Questions What about confounders: Age Education Socio-economic status Region (three regions used in the study) Morbidity What about negative outcomes? Why did patients dropout

What about confounders:

Age

Education

Socio-economic status

Region (three regions used in the study)

Morbidity

What about negative outcomes?

Why did patients dropout