This document discusses the need for rigorous evaluation of e-health and mhealth technologies. It notes that while such technologies are widely used, their actual impacts on productivity and health outcomes are unclear. Evaluations of healthcare IT have found unintended consequences like increased clinician workload and new types of medical errors. The document calls for investment in e-health to be based on evidence from well-designed trials and evaluations. It identifies key questions around measuring impacts, designing technologies to improve health, and supporting organizational changes. Finally, it presents guidelines for good evaluation practices in health informatics.

1. Fostering Research in e-Health and mHealth Evaluation
FACULTY OF
ENGINEERING &
INFORMATION
TECHNOLOGIES
CPC e-Health showcase 6th Nov 2013
Dr. Simon Poon

2. The Problem
› Evaluation of Technological Impacts:
› The Productivity Paradox Debate
- “You can see the computer age everywhere but in the productivity statistics”
(Robert Solow 1987)
- “Despite differences in methodology and data sources, a consensus is building
that the remarkable behaviour of IT prices provides the key to the surge in
economic growth”
(Dale Jorgenson, 2001)
› Evaluations in Healthcare IT:
(Seldling et al. 2013)
2

3. Unintended Consequences
› Hinders clinical practice (The Telegraph UK, 14 December 2009)
› is not working as it should in UK (BBC News, 18 May 2011)
› “provide little support for the cognitive tasks of the clinicians or the
workflow that can increase the chance of error” (National Research Council
Report on Healthcare 2009, USA)
Types of Unintended Consequences
of EMR
More work for clinicians
Workflow Issues
Never ending system demands
Paper persistence
Change in communication
Emotions
New kinds of errors
Changes in power structure
Overdependence on technology
TOTAL
% Frequency
N= 324 (Campbell et al 2006)
Sample size
(Ash et al 2007a)
19.8
17.6
14.8
10.8
10.1
7.7
7.1
6.8
5.2
100.0
125
149
143
N/A
146
140
82
61
138
984
Rahadhan & Poon (2012)
3

4. Evaluation Research in e-Health
› Investing in e-Health on the basis on its evidence, not on its potential
- Urquhart (2009): found limited evidence of effective interventions… more well
designed trials are needed
- Shojania (2011): evaluated the effects of on-screen computer reminders…
median improvement of 4%
- Bright et al. (2012): reviewed 148 randomised trials… 86% assessed process
measures, 20% assessed outcomes and 15% accessed costs
› Key questions:
- What is valid and reliable measures of e-Health solutions?
- How to design and configure e-Health to improve health outcomes?
- How to identify complementary organisational changes and channel resource for
training?
4

5. Guideline for good evaluation practice in health
informatics (GEP-HI)
Nykänen et al. (2011)
5

6. Thank you!
Email: simon.poon@sydney.edu.au
6