This document summarizes a presentation on health informatics evaluation and health technology assessment. It outlines the foundations, emergent issues, opportunities for collaboration, and resources in these areas. The foundations section contrasts the focus of health technology assessment (HTA), which evaluates broad health interventions but not IT, with health informatics evaluation, which focuses on software and system factors like usability, safety, and cost-effectiveness. Emergent issues discussed include problems deploying large health IT systems and ensuring their socio-technical fit, adoption, and interoperability. The document advocates collaboration between HTA and health informatics experts to identify issues of shared interest and develop mutually beneficial methodologies.

1. Health Informatics Evaluation and Health Technology
Assessment: Lessons and Warnings
Juzwishin, D., Borycki, E., Kushniruk, Nohr, C
University of Victoria & Aalborg University
June 2010
Dublin, Ireland

2. Objectives
• Foundations
• Emergent issues
• Compare and contrast
• Strengths and
weaknesses
• Opportunities for
collaboration
2

3. Foundations of HTA & HIT Evaluation
• HTA • HIT Evaluation
– Focus on broad spectrum – Focus on IT and software
of health care interventions – System safety, effectiveness and
efficiency
but non-IT focus
– System Usability
– Safety – Integration and interoperability
– Efficacy – Cost effectiveness
– Clinical effectiveness – Accuracy and Correctness of
Information/Data and Processing
– Cost effectiveness – Reusability and Extensibility
– Social contextualization – System Reliability
– Perspective – Outcome as well as increasing focus
on Process Oriented Evaluation (i.e.
– Outcomes oriented – e.g. both quantitative and qualitative)
clinical controlled trials
3

4. Emergent issues
• HTA • HIT
– Integration of medical – Problems with Large Scale
devices and information Deployments (e.g. local,
systems regional and national efforts)
– In Canada patient – Socio-technical fit
management software – Adoption issues
fits the definition of a – Usability
medical device
– System interoperability
– Uncertainty surrounding
– Integration with medical
effectiveness
devices
– Examples of harm
– Proof of financial benefits
emerging from IS elements
of health interventions – Slow progress
4

5. 2010 World Health IT Conference and the EU
Ministerial High Level Conference on eHealth
• "eHealth Conference Declaration" adopted.
• "... we consider that the following points should be
addressed:
• 2a ... To evaluate eHealth, as much as possible in a
standardised manner, with respect to health outcomes,
benefits and cost effectiveness, including patient safety,
accessibility to care and quality of care"
5

6. Strengths from collaboration
• Establish a working linkage
between International HIT and
International HTA community
• Undertake an exploratory
paper to Identify issues of
common interest,
responsibility, and opportunity
• Identify mutually beneficial
methodologies and
approaches
• Test the level of interest
6

7. Resources
• STARE-HI—Statement on reporting of evaluation studies in
Health Informatics
– http://www.imia.org/endorsed/Stare-HI_as_published.pdf
• Health IT Evaluation Database
– http://evaldb.umit.at
– 1400 evaluation papers 1980 – 2009
• Bad Health Informatics can Kill
– http://iig.umit.at/efmi
• INAHTA Checklist
– http://www.inahta.org/HTA/Checklist/
7