Measuring the effectiveness of e-health initiatives in hospitals Prof Johanna Westbrook Health Informatics Research & Eval...
Health Informatics Research & Evaluation Unit <ul><li>17 research staff most funded by grants </li></ul><ul><li>Aims: </li...
Research Questions <ul><li>Do pathology order entry systems deliver more efficient care? </li></ul><ul><li>Do electronic m...
Is care delivery more efficient? <ul><li>Few studies  </li></ul><ul><ul><li>all specialised units </li></ul></ul><ul><ul><...
AIMS <ul><li>Do turnaround times decrease in the first 12 months following system introduction and are improvements sustai...
Methods <ul><li>650 teaching hospital </li></ul><ul><li>Measurement of TAT pre & post CPOE -Cerner Millennium PowerChart  ...
Test turnaround time significantly declined   Year 1 by 18.6% ,  Year 2 by 12.6% <ul><li>Average  number of tests  per pat...
Changes in TAT post CPOE in four hospitals
Effectiveness – Does a reduction in TAT really matter? <ul><li>Is there a relationship between TATs and lengths of stays i...
Qualitative studies to assess the impact pathology work <ul><ul><li>Focus groups & interviews with management, pathology, ...
“… I don’t have figures to prove this, but in my estimation it has made the turnaround time longer.” (Senior scientist, 20...
Implementing Systems <ul><li>Changes in roles & responsibilities  </li></ul><ul><li>Elimination of some tasks but creation...
Benefits realisation framework Georgiou A, et al (2007) The impact of computerised physician order entry systems on pathol...
Will electronic medication management systems make our health services safer?
Do e-prescribing systems reduce prescribing errors in hospital inpatients? <ul><li>13 papers (US 6, UK 4, Europe 2, Israel...
Controlled Before & After study 2 Hospitals  2 Systems  6 wards
Methods <ul><li>Prospective medication chart review pre & post. </li></ul><ul><ul><li>Inter-rater reliability,  kappa = 0....
Do electronic medication administration records reduce errors? <ul><li>Few studies – all flawed methods  </li></ul><ul><ul...
Observational Medication Administration Error Study   <ul><li>Observe nurses as they prepare & administer  medications </l...
Study Methods <ul><li>6 wards at 2 hospitals  </li></ul><ul><li>Information sessions to recruit nurses </li></ul><ul><ul><...
How does system use impact upon patterns of work?  <ul><li>Will these systems save time? </li></ul><ul><li>Do drs & nurses...
Aim:  To develop a reliable method for observing and recording time spent by clinicians in different work tasks Work Obser...
PDA data collection tool What task? With whom? With what? Interruptions Multi-tasking
<ul><li>Controlled before and after study nurses and doctors </li></ul><ul><li>4 wards at baseline </li></ul><ul><li>1 or ...
Proportions of observed time in tasks across four wards ( Before)
Time with patients & interruptions (Baseline data) <ul><li>Nurses = 34.5%, interrupted 1/49mins, 12% multi-tasking </li></...
Distribution of doctors’ tasks over the day 2006
Distribution of doctors’ tasks over the day including social tasks 2006
Data Analysis <ul><li>Changes in   </li></ul><ul><ul><li>distribution of time across tasks </li></ul></ul><ul><ul><li>aver...
Challenges of integrating the use of technology into everyday work practices
Poor mobility  workarounds may result in less safe practices
Paper is a highly mobile technology!
Capturing what happens on a ward <ul><li>Structured observations </li></ul><ul><li>Video observations </li></ul><ul><li>Ta...
Computers on wheels 82% of nurses’ tasks 3% of nurses’ work tasks
Doctors’ on ward rounds <ul><li>57% of tasks completed using a generic COW </li></ul><ul><li>36% of tasks completed using ...
Conclusions <ul><li>Recognise the limitations of existing evidence-base </li></ul><ul><li>Use explicit indicators & measur...
Acknowledgements <ul><li>HIREU Team </li></ul><ul><li>Andrew Georgiou </li></ul><ul><li>Joanne Callen </li></ul><ul><li>Am...
Upcoming SlideShare
Loading in...5
×

Measuring the Effectiveness of eHealth Initiatives in Hospitals

1,231

Published on

Prof Johanna Westbrook
Health Informatics Research & Evaluation Unit, The University of Sydney
www.fhs.usyd.edu.au/hireu/
(2/10/09, Forum, 10.50)

Published in: Health & Medicine, Business
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,231
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide
  • We are undertaking an observational study of medication preparation and administration processes in a large teaching hospitals in order to identify the extent of medication errors and the relationship of interruptions to error rates. We are focusing on IV medications due to the reported overseas data on error rates. Only 1 small scale Aust study has been published. We have a 2 stage method - Researchers follow nurses on the ward and record rugs administered, procedural errors and any interruptions Stage 2 observed data is compared with patients medication charts to identify errors. We are using a PDA we designed especially for this study. One of our recently technical problems with this tool was that we had anticipated that any nurse would be interrupted more than 13 times while trying to administer one drug, but we were wrong and have had to adjust the system accordingly.
  • Measuring the Effectiveness of eHealth Initiatives in Hospitals

    1. 1. Measuring the effectiveness of e-health initiatives in hospitals Prof Johanna Westbrook Health Informatics Research & Evaluation Unit The University of Sydney
    2. 2. Health Informatics Research & Evaluation Unit <ul><li>17 research staff most funded by grants </li></ul><ul><li>Aims: </li></ul><ul><ul><li>Develop and test rigorous and innovative evaluation tools & approaches. </li></ul></ul><ul><ul><li>Produce research evidence about impact of ICT on health care delivery, professionals’ work and patient outcomes. </li></ul></ul><ul><ul><li>Disseminate evidence to inform policy, system design, integration and effective use of ICT in health care. </li></ul></ul>
    3. 3. Research Questions <ul><li>Do pathology order entry systems deliver more efficient care? </li></ul><ul><li>Do electronic medication management systems make health care safer? </li></ul><ul><li>Do clinical systems make clinical work more efficient and release clinicians to spend more time with patients? </li></ul><ul><li>What is the role of mobile technologies in supporting clinical work in hospitals? </li></ul><ul><li>Approaches, results to date, methodological challenges </li></ul>
    4. 4. Is care delivery more efficient? <ul><li>Few studies </li></ul><ul><ul><li>all specialised units </li></ul></ul><ul><ul><li>all reported improved turnaround times. </li></ul></ul>Computerised test ordering Turnaround time = Time from receipt of specimen in laboratory to report of result
    5. 5. AIMS <ul><li>Do turnaround times decrease in the first 12 months following system introduction and are improvements sustained? </li></ul><ul><li>What is the impact on pathology staff? </li></ul>
    6. 6. Methods <ul><li>650 teaching hospital </li></ul><ul><li>Measurement of TAT pre & post CPOE -Cerner Millennium PowerChart </li></ul><ul><li>Periods </li></ul><ul><ul><li>Jul – Aug 2003 </li></ul></ul><ul><ul><li>Jul – Aug 2004 (post 1) </li></ul></ul><ul><ul><li>Jul – Aug 2005 (post 2) </li></ul></ul><ul><ul><li>Westbrook JI, et al. (2006) Computerised pathology test order-entry reduces laboratory turnaround times and influences tests ordered by hospital clinicians: A controlled before and after study. Journal of Clinical Pathology, 59, 533-536. </li></ul></ul>
    7. 7. Test turnaround time significantly declined Year 1 by 18.6% , Year 2 by 12.6% <ul><li>Average number of tests per patient did not change: </li></ul><ul><ul><li>92.5 assays/pt vs 103.2 (P=0.23) </li></ul></ul>
    8. 8. Changes in TAT post CPOE in four hospitals
    9. 9. Effectiveness – Does a reduction in TAT really matter? <ul><li>Is there a relationship between TATs and lengths of stays in an emergency department prior to CPOE? </li></ul><ul><li>Regression analyses - TAT was a significant factor contributing to patients’ length of stay in ED (p<0.0001). </li></ul>Westbrook JI , et al (2009) Does computerised provider order entry reduce test turnaround times?: a before and after study at four hospitals. Stud Technol Inform; 150: 527-531.
    10. 10. Qualitative studies to assess the impact pathology work <ul><ul><li>Focus groups & interviews with management, pathology, clinical and IT department staff </li></ul></ul><ul><ul><li>Observational video study of pathology staff over several months </li></ul></ul>
    11. 11. “… I don’t have figures to prove this, but in my estimation it has made the turnaround time longer.” (Senior scientist, 2004)
    12. 12. Implementing Systems <ul><li>Changes in roles & responsibilities </li></ul><ul><li>Elimination of some tasks but creation of new tasks </li></ul><ul><li>Failure of one group to use the system as expected impacts upon the work of others </li></ul>These elements of system impact are as important as quantitative indicators!
    13. 13. Benefits realisation framework Georgiou A, et al (2007) The impact of computerised physician order entry systems on pathology services: a systematic review. Intern J Med Informatics 76 (7), 514-529. Georgiou et al. (2008) Electronic test management systems and hospital pathology services – a framework for investigating their impact. Encyclopaedia of Healthcare Information Systems Efficiency Effectiveness Quality Test costs Redundant test rates Turn around times Work practices Patient safety Compliance with guidelines Patient management Length of stay Test volumes Communication
    14. 14. Will electronic medication management systems make our health services safer?
    15. 15. Do e-prescribing systems reduce prescribing errors in hospital inpatients? <ul><li>13 papers (US 6, UK 4, Europe 2, Israel 1) </li></ul><ul><ul><li>9 showed significant decrease </li></ul></ul><ul><ul><li>2 decrease in some categories </li></ul></ul><ul><ul><li>2 an increase in errors </li></ul></ul><ul><li>Limitations in study designs, eg only specific drugs </li></ul><ul><li>Only 5 studies examined severity of errors – 2 defined their scales </li></ul><ul><li>Very limited evidence of effectiveness to reduce serious errors </li></ul><ul><li>Reckmann, Westbrook et al (2009) Does computerized order entry reduce prescribing errors for hospital inpatients? A systematic review. Journal of American Medical Informatics Association . 16 (5) 613-623. </li></ul>
    16. 16. Controlled Before & After study 2 Hospitals 2 Systems 6 wards
    17. 17. Methods <ul><li>Prospective medication chart review pre & post. </li></ul><ul><ul><li>Inter-rater reliability, kappa = 0.82-0.84 </li></ul></ul><ul><li>Classification of: </li></ul><ul><ul><li>error types </li></ul></ul><ul><ul><li>severity – 5 point scale </li></ul></ul><ul><ul><li>Clinical </li></ul></ul><ul><ul><li>Documentation </li></ul></ul><ul><ul><li>System-related </li></ul></ul><ul><li>2006 – pre 2008/9 - post </li></ul>Prescribing error types
    18. 18. Do electronic medication administration records reduce errors? <ul><li>Few studies – all flawed methods </li></ul><ul><ul><li>Perceptions of staff </li></ul></ul><ul><ul><li>Examination of voluntary incident reports </li></ul></ul>
    19. 19. Observational Medication Administration Error Study <ul><li>Observe nurses as they prepare & administer medications </li></ul><ul><li>Record interruptions & multi-tasking </li></ul><ul><li>Compare observed data with patients’ charts to identify errors </li></ul>
    20. 20. Study Methods <ul><li>6 wards at 2 hospitals </li></ul><ul><li>Information sessions to recruit nurses </li></ul><ul><ul><li>- 82% response rate (n=98 nurses pre) </li></ul></ul><ul><li>Researchers arrived on the wards at peak medication times (7:00-19:30) </li></ul><ul><li>Approx 8 administrations/observation Hr </li></ul><ul><li>Inter-rater reliability – Kappa score 0.94-0.96 </li></ul><ul><li>Serious error protocol used 10 times </li></ul>
    21. 21. How does system use impact upon patterns of work? <ul><li>Will these systems save time? </li></ul><ul><li>Do drs & nurses spend more time with patients? </li></ul>
    22. 22. Aim: To develop a reliable method for observing and recording time spent by clinicians in different work tasks Work Observation Method By Activity Timing (WOMBAT) Westbrook JI, Ampt A (2009) Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication. International Journal of Medical Informatics. 78S, S25-S33.
    23. 23. PDA data collection tool What task? With whom? With what? Interruptions Multi-tasking
    24. 24. <ul><li>Controlled before and after study nurses and doctors </li></ul><ul><li>4 wards at baseline </li></ul><ul><li>1 or 2 intervention wards </li></ul><ul><li>2 control wards post </li></ul><ul><li>Completion date Dec 2009 </li></ul>
    25. 25. Proportions of observed time in tasks across four wards ( Before)
    26. 26. Time with patients & interruptions (Baseline data) <ul><li>Nurses = 34.5%, interrupted 1/49mins, 12% multi-tasking </li></ul><ul><li>Ward Drs = 15.0%, interrupted 1/21mins, 20% multi-tasking </li></ul><ul><li>On average nurses spend 8.4 mins/shift talking with a Dr. </li></ul>Westbrook JI, et al (2008) Medical Journal of Australia. 188(9): 506-509.
    27. 27. Distribution of doctors’ tasks over the day 2006
    28. 28. Distribution of doctors’ tasks over the day including social tasks 2006
    29. 29. Data Analysis <ul><li>Changes in </li></ul><ul><ul><li>distribution of time across tasks </li></ul></ul><ul><ul><li>average time for each task </li></ul></ul><ul><ul><li>frequency of each task </li></ul></ul><ul><ul><li>times of the day when tasks completed </li></ul></ul><ul><ul><li>with whom tasks are completed </li></ul></ul><ul><li>A lot more to come ……. </li></ul>
    30. 30. Challenges of integrating the use of technology into everyday work practices
    31. 31. Poor mobility workarounds may result in less safe practices
    32. 32. Paper is a highly mobile technology!
    33. 33. Capturing what happens on a ward <ul><li>Structured observations </li></ul><ul><li>Video observations </li></ul><ul><li>Talking to staff </li></ul>80 hours observation, 2 wards Aim: To measure which devices nurses and doctors select Andersen P, Lindgaard A, Prgomet M, Creswick N, Westbrook JI (2009) Is selection of hardware device related to clinical task?: A multi-method study of mobile and fixed computer use by doctors and nurses on hospital wards. J Medical Internet Research . 11(3) <ul><li>Available devices on each ward: </li></ul><ul><li>Two forms of COWs (n=5 & 6) </li></ul><ul><li>Two forms of tablets – (Motion computing C5 </li></ul><ul><li>& LE1700) (n=2/ward) </li></ul><ul><li>Fixed PCs (n=7) </li></ul>
    34. 34. Computers on wheels 82% of nurses’ tasks 3% of nurses’ work tasks
    35. 35. Doctors’ on ward rounds <ul><li>57% of tasks completed using a generic COW </li></ul><ul><li>36% of tasks completed using a tablet </li></ul><ul><li>Only 3% of tasks completed at the patient’s bedside </li></ul>
    36. 36. Conclusions <ul><li>Recognise the limitations of existing evidence-base </li></ul><ul><li>Use explicit indicators & measure them </li></ul><ul><li>Engagement of academics/clinicians/ vendors </li></ul><ul><li>Feedback impact data to staff </li></ul><ul><li>Create a market for evidence of impact </li></ul><ul><ul><li>Share & compare between systems, organisations </li></ul></ul>
    37. 37. Acknowledgements <ul><li>HIREU Team </li></ul><ul><li>Andrew Georgiou </li></ul><ul><li>Joanne Callen </li></ul><ul><li>Amanda Woods </li></ul><ul><li>Margaret Reckmann </li></ul><ul><li>Connie Lo </li></ul><ul><li>Yvonne Koh </li></ul><ul><li>Fiona Ray </li></ul><ul><li>Nerida Creswick </li></ul><ul><li>Marilyn Rob </li></ul><ul><li>Mirela Prgomet </li></ul><ul><li>Antonia Hordern </li></ul><ul><li>Fiona McWhinney </li></ul><ul><li>Pia Andersen </li></ul><ul><li>Anne-Mette Lingaard </li></ul><ul><li>Funding Bodies </li></ul><ul><li>Australian Research Council </li></ul><ul><li>NH & MRC </li></ul><ul><li>NSW Health </li></ul><ul><li>HCF Research Foundation </li></ul><ul><li>SSWAHS </li></ul>Hospital staff at our study sites Publications available at : www.fhs.usyd.edu.au/hireu/ [email_address]

    ×