As investments in healthcare are expanding and medical imaging technology are maturing, the need for structured measurement approaches and holistic evaluation methods is expanding. The wider adoption of Technology Assessments (TA), and the fact that hospitals are re-evaluating their current PACS implementations, reflects this demand. No recognized work structurally defines what PACS performance constitutes, however. Based on TA models for diagnostic imaging technology and levels of (clinical) efficacy [1-3], and balanced evaluation models [4], we define PACS performance as: the multi-factorial impacts and benefits produced by the application of PACS in terms of hospital efficiency and clinical effectiveness with respect to PACS workflow and patients’ clinical journey. In this paper, the authors review PACS performance according to a predefined literature search protocol, and apply a meta-analytic approach that synthesizes these sources.

1. 1
Defining and Formalizing:
a synthesized review on the
multi-factorial nature of PACS
performance
Rogier van de Wetering and Ronald Batenburg
Dept. of Information and Computing Sciences, Utrecht University
Utrecht, The Netherlands
Geneva, June 24, 2010

2. 2
Outline
 Background
 Methods
 Results
 Conclusion
 Questions & discussion

3. 3
Background
 Previous studies indicated the need for holistic approaches
and performance measurement from different perspectives
 Adoption of Technology Assessments (TA) and re-
evaluation trends reflects this demand
 No recognized work structurally defines what PACS
performance constitutes
 Objective is to review literature and apply a meta-analytic
approach

4. 4 4
Background
 PACS performance as
“the multi-factorial impacts and benefits produced by the
application of PACS in terms of hospital efficiency and
clinical effectiveness with respect to PACS workflow and
patients’ clinical journey”
X1
X2
X3
X4
X5
PACS
performance
PACS
alignment
Yc1
Yc2
Yc3
Yc4
a1
a2
a3
a4
a5
γ1
b1
b2
b3
b4
Figure 1. PACS research agenda

5. 5
Methods
Structure literature review
 Using Omega, central data repositories (e.g., PubMed) and
Scholar (June 2009 - October 2009)
 Query: Picture Archiving and Communication System OR PACS OR
digital radiology OR image management OR filmless radiology OR
diagnostic images + {performance | impact | measure | indicator |
benefits | improvements | assessment | improvement | effect |
efficiency | evaluation | indicator | measure | efficacy | experience |
outcome}
 Retrieval of 980 papers
 37 key publications and positioning papers (2000–2009)
Meta-analytic synthesizing
 Following a qualitative meta-analysis approach
 Applied an approach by three interrelated processes (1)
meta-theory, (2) meta-method and (3) meta-data-analysis

6. 6
Results (1-2)
 Included sources used a variety of qualitative and
quantitative methodologies
 Some focused on diagnostic/clinical elements, others aimed
at e.g., evaluating productivity and workflow efficiency
 Four performance constructs are defined (see also PACS-
BSC, Van de Wetering et al.,2006)
 Some measures might not be independent of performance
in another measure (“overlap”)

7. 7
Results (2-2)
Performance constructs Measurements
1) Clinical contribution construct
- Interpretation agility Interpretation time per modality
examination
- Efficacy of PACS Diagnostic accuracy
- Contributions to clinical
communication and collaboration
Communication efficacy
- Therapeutic intervention Patient management contribution
2) Organisational efficiency construct
- Timeliness of radiology report Report-turnaround-time
- Productivity Number of examinations
Cost-effectiveness Budget ratio or cost prise
3) Service construct
- Perceived service to patients Patient satisfaction
- Service delivery to referring
physicians
Referring physician satisfaction
- Service delivery to users User satisfaction
- Patient flow & throughput Patient waiting time
4) Technical Information System construct
- Instantaneous image display of
newly acquired images
Average time-to-display
- Old images availability Average time-to-display
Table 1. PACS performance constructs

8. 8
Conclusions
 PACS performance defined by synthesized review
 Framework applicable addressing PACS impacts
 Debatable how to weight (whether equal or unequal) each of
the measures
 Unlikely to achieve good quality on higher levels of
assessment (see hierarchical TA models), given insufficient
performance on lower levels
 Framework is valuable tool for empirical research and relevant
for radiology practice
 Need for empirical validation (currently under investigation)

9. 9
Contact
drs. R. (Rogier) van de Wetering
Email: R.vandeWetering@cs.uu.nl
Office: CGN-B129
Secretary: +31 (30) 253 9251
Group: Organization and information
section: Information and software
systems
Website: www.pacsmaturity.com