Measuring the effectiveness of e-health initiatives in hospitals Prof Johanna Westbrook Health Informatics Research & Evaluation Unit The University of Sydney

Health Informatics Research & Evaluation Unit 17 research staff most funded by grants Aims: Develop and test rigorous and innovative evaluation tools & approaches. Produce research evidence about impact of ICT on health care delivery, professionals’ work and patient outcomes. Disseminate evidence to inform policy, system design, integration and effective use of ICT in health care.

17 research staff most funded by grants

Aims:

Develop and test rigorous and innovative evaluation tools & approaches.

Produce research evidence about impact of ICT on health care delivery, professionals’ work and patient outcomes.

Disseminate evidence to inform policy, system design, integration and effective use of ICT in health care.

Research Questions Do pathology order entry systems deliver more efficient care? Do electronic medication management systems make health care safer? Do clinical systems make clinical work more efficient and release clinicians to spend more time with patients? What is the role of mobile technologies in supporting clinical work in hospitals? Approaches, results to date, methodological challenges

Do pathology order entry systems deliver more efficient care?

Do electronic medication management systems make health care safer?

Do clinical systems make clinical work more efficient and release clinicians to spend more time with patients?

What is the role of mobile technologies in supporting clinical work in hospitals?

Approaches, results to date, methodological challenges

Is care delivery more efficient? Few studies all specialised units all reported improved turnaround times. Computerised test ordering Turnaround time = Time from receipt of specimen in laboratory to report of result

Few studies

all specialised units

all reported improved turnaround times.

AIMS Do turnaround times decrease in the first 12 months following system introduction and are improvements sustained? What is the impact on pathology staff?

Do turnaround times decrease in the first 12 months following system introduction and are improvements sustained?

What is the impact on pathology staff?

Methods 650 teaching hospital Measurement of TAT pre & post CPOE -Cerner Millennium PowerChart Periods Jul – Aug 2003 Jul – Aug 2004 (post 1) Jul – Aug 2005 (post 2) Westbrook JI, et al. (2006) Computerised pathology test order-entry reduces laboratory turnaround times and influences tests ordered by hospital clinicians: A controlled before and after study. Journal of Clinical Pathology, 59, 533-536.

650 teaching hospital

Measurement of TAT pre & post CPOE -Cerner Millennium PowerChart

Periods

Jul – Aug 2003

Jul – Aug 2004 (post 1)

Jul – Aug 2005 (post 2)

Westbrook JI, et al. (2006) Computerised pathology test order-entry reduces laboratory turnaround times and influences tests ordered by hospital clinicians: A controlled before and after study. Journal of Clinical Pathology, 59, 533-536.

Test turnaround time significantly declined Year 1 by 18.6% , Year 2 by 12.6% Average number of tests per patient did not change: 92.5 assays/pt vs 103.2 (P=0.23)

Average number of tests per patient did not change:

92.5 assays/pt vs 103.2 (P=0.23)

Changes in TAT post CPOE in four hospitals

Effectiveness – Does a reduction in TAT really matter? Is there a relationship between TATs and lengths of stays in an emergency department prior to CPOE? Regression analyses - TAT was a significant factor contributing to patients’ length of stay in ED (p<0.0001). Westbrook JI , et al (2009) Does computerised provider order entry reduce test turnaround times?: a before and after study at four hospitals. Stud Technol Inform; 150: 527-531.

Is there a relationship between TATs and lengths of stays in an emergency department prior to CPOE?

Regression analyses - TAT was a significant factor contributing to patients’ length of stay in ED (p<0.0001).

Qualitative studies to assess the impact pathology work Focus groups & interviews with management, pathology, clinical and IT department staff Observational video study of pathology staff over several months

Focus groups & interviews with management, pathology, clinical and IT department staff

Observational video study of pathology staff over several months

“… I don’t have figures to prove this, but in my estimation it has made the turnaround time longer.” (Senior scientist, 2004)

Implementing Systems Changes in roles & responsibilities Elimination of some tasks but creation of new tasks Failure of one group to use the system as expected impacts upon the work of others These elements of system impact are as important as quantitative indicators!

Changes in roles & responsibilities

Elimination of some tasks but creation of new tasks

Failure of one group to use the system as expected impacts upon the work of others

Benefits realisation framework Georgiou A, et al (2007) The impact of computerised physician order entry systems on pathology services: a systematic review. Intern J Med Informatics 76 (7), 514-529. Georgiou et al. (2008) Electronic test management systems and hospital pathology services – a framework for investigating their impact. Encyclopaedia of Healthcare Information Systems Efficiency Effectiveness Quality Test costs Redundant test rates Turn around times Work practices Patient safety Compliance with guidelines Patient management Length of stay Test volumes Communication

Will electronic medication management systems make our health services safer?

Do e-prescribing systems reduce prescribing errors in hospital inpatients? 13 papers (US 6, UK 4, Europe 2, Israel 1) 9 showed significant decrease 2 decrease in some categories 2 an increase in errors Limitations in study designs, eg only specific drugs Only 5 studies examined severity of errors – 2 defined their scales Very limited evidence of effectiveness to reduce serious errors Reckmann, Westbrook et al (2009) Does computerized order entry reduce prescribing errors for hospital inpatients? A systematic review. Journal of American Medical Informatics Association . 16 (5) 613-623.

13 papers (US 6, UK 4, Europe 2, Israel 1)

9 showed significant decrease

2 decrease in some categories

2 an increase in errors

Limitations in study designs, eg only specific drugs

Only 5 studies examined severity of errors – 2 defined their scales

Very limited evidence of effectiveness to reduce serious errors

Reckmann, Westbrook et al (2009) Does computerized order entry reduce prescribing errors for hospital inpatients? A systematic review. Journal of American Medical Informatics Association . 16 (5) 613-623.

Controlled Before & After study 2 Hospitals 2 Systems 6 wards

Methods Prospective medication chart review pre & post. Inter-rater reliability, kappa = 0.82-0.84 Classification of: error types severity – 5 point scale Clinical Documentation System-related 2006 – pre 2008/9 - post Prescribing error types

Prospective medication chart review pre & post.

Inter-rater reliability, kappa = 0.82-0.84

Classification of:

error types

severity – 5 point scale

Clinical

Documentation

System-related

2006 – pre 2008/9 - post

Do electronic medication administration records reduce errors? Few studies – all flawed methods Perceptions of staff Examination of voluntary incident reports

Few studies – all flawed methods

Perceptions of staff

Examination of voluntary incident reports

Observational Medication Administration Error Study Observe nurses as they prepare & administer medications Record interruptions & multi-tasking Compare observed data with patients’ charts to identify errors

Observe nurses as they prepare & administer medications

Record interruptions & multi-tasking

Compare observed data with patients’ charts to identify errors

Study Methods 6 wards at 2 hospitals Information sessions to recruit nurses - 82% response rate (n=98 nurses pre) Researchers arrived on the wards at peak medication times (7:00-19:30) Approx 8 administrations/observation Hr Inter-rater reliability – Kappa score 0.94-0.96 Serious error protocol used 10 times

6 wards at 2 hospitals

Information sessions to recruit nurses

- 82% response rate (n=98 nurses pre)

Researchers arrived on the wards at peak medication times (7:00-19:30)

Approx 8 administrations/observation Hr

Inter-rater reliability – Kappa score 0.94-0.96

Serious error protocol used 10 times

How does system use impact upon patterns of work? Will these systems save time? Do drs & nurses spend more time with patients?

Will these systems save time?

Do drs & nurses spend more time with patients?

Aim: To develop a reliable method for observing and recording time spent by clinicians in different work tasks Work Observation Method By Activity Timing (WOMBAT) Westbrook JI, Ampt A (2009) Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication. International Journal of Medical Informatics. 78S, S25-S33.

PDA data collection tool What task? With whom? With what? Interruptions Multi-tasking

Controlled before and after study nurses and doctors 4 wards at baseline 1 or 2 intervention wards 2 control wards post Completion date Dec 2009

Controlled before and after study nurses and doctors

4 wards at baseline

1 or 2 intervention wards

2 control wards post

Completion date Dec 2009

Proportions of observed time in tasks across four wards ( Before)

Time with patients & interruptions (Baseline data) Nurses = 34.5%, interrupted 1/49mins, 12% multi-tasking Ward Drs = 15.0%, interrupted 1/21mins, 20% multi-tasking On average nurses spend 8.4 mins/shift talking with a Dr. Westbrook JI, et al (2008) Medical Journal of Australia. 188(9): 506-509.

Nurses = 34.5%, interrupted 1/49mins, 12% multi-tasking

Ward Drs = 15.0%, interrupted 1/21mins, 20% multi-tasking

On average nurses spend 8.4 mins/shift talking with a Dr.

Distribution of doctors’ tasks over the day 2006

Distribution of doctors’ tasks over the day including social tasks 2006

Data Analysis Changes in distribution of time across tasks average time for each task frequency of each task times of the day when tasks completed with whom tasks are completed A lot more to come …….

Changes in

distribution of time across tasks

average time for each task

frequency of each task

times of the day when tasks completed

with whom tasks are completed

A lot more to come …….

Challenges of integrating the use of technology into everyday work practices

Poor mobility workarounds may result in less safe practices

Paper is a highly mobile technology!

Capturing what happens on a ward Structured observations Video observations Talking to staff 80 hours observation, 2 wards Aim: To measure which devices nurses and doctors select Andersen P, Lindgaard A, Prgomet M, Creswick N, Westbrook JI (2009) Is selection of hardware device related to clinical task?: A multi-method study of mobile and fixed computer use by doctors and nurses on hospital wards. J Medical Internet Research . 11(3) Available devices on each ward: Two forms of COWs (n=5 & 6) Two forms of tablets – (Motion computing C5 & LE1700) (n=2/ward) Fixed PCs (n=7)

Structured observations

Video observations

Talking to staff

Available devices on each ward:

Two forms of COWs (n=5 & 6)

Two forms of tablets – (Motion computing C5

& LE1700) (n=2/ward)

Fixed PCs (n=7)

Computers on wheels 82% of nurses’ tasks 3% of nurses’ work tasks

Doctors’ on ward rounds 57% of tasks completed using a generic COW 36% of tasks completed using a tablet Only 3% of tasks completed at the patient’s bedside

57% of tasks completed using a generic COW

36% of tasks completed using a tablet

Only 3% of tasks completed at the patient’s bedside

Conclusions Recognise the limitations of existing evidence-base Use explicit indicators & measure them Engagement of academics/clinicians/ vendors Feedback impact data to staff Create a market for evidence of impact Share & compare between systems, organisations

Recognise the limitations of existing evidence-base

Use explicit indicators & measure them

Engagement of academics/clinicians/ vendors

Feedback impact data to staff

Create a market for evidence of impact

Share & compare between systems, organisations

Acknowledgements HIREU Team Andrew Georgiou Joanne Callen Amanda Woods Margaret Reckmann Connie Lo Yvonne Koh Fiona Ray Nerida Creswick Marilyn Rob Mirela Prgomet Antonia Hordern Fiona McWhinney Pia Andersen Anne-Mette Lingaard Funding Bodies Australian Research Council NH & MRC NSW Health HCF Research Foundation SSWAHS Hospital staff at our study sites Publications available at : www.fhs.usyd.edu.au/hireu/ [email_address]

HIREU Team

Andrew Georgiou

Joanne Callen

Amanda Woods

Margaret Reckmann

Connie Lo

Yvonne Koh

Fiona Ray

Nerida Creswick

Marilyn Rob

Mirela Prgomet

Antonia Hordern

Fiona McWhinney

Pia Andersen

Anne-Mette Lingaard

Funding Bodies

Australian Research Council

NH & MRC

NSW Health

HCF Research Foundation

SSWAHS