The document discusses the value of information in healthcare and analyzing temporal patterns in patient care and health services. It provides examples of studies that analyzed patterns in test follow-up rates and mortality rates for weekend hospital admissions. While the studies found issues like high rates of unreviewed tests and higher mortality for weekend admissions, further analysis of temporal patterns provided insights into potential causative factors and opportunities for intervention. The value of information is realized when it leads to changes in decisions and care processes that improve outcomes.

1. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
The Value of Information
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Enrico Coiera

2. The vision: “traffic lights” lets us know if patient
is safely within ‘envelope’ of good care
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Alert: Patient is at high
risk of readmission …

3. Source: The Economist, Aug 27 2014
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

4. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
:: time and care
• Temporal patterns are core to clinical medicine
• We use these patterns to disambiguate differential
diagnoses, detect co-morbidities, predict most likely next
event in a sequence
• Patient level patterns:
• Time ordering of events in a patient history
• Dynamic signals e.g. ECG, arterial pressures
• Population level patterns:
• Unfolding of infectious outbreaks, seasonal mortality
rates.

5. (AIHW Bulletin, 2, 2002)
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

6. (AIHW Bulletin, 2, 2002)
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

7. :: health services have temporal
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
patterns too
•Why study patterns of health service delivery?
•Allocation of scare resources:
• Optimise day to day resource allocation
• Assist in longer term workforce and resource planning
•Improve safety and quality of care:
• Identify when we are not providing the services our
patients need
• Minimize avoidable harms due to mismatch between
allocation and need

8. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

9. :: Case study 1 - Follow-up of test
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
orders
• Not every test that is ordered is followed up
• In US, between 20-61% of inpatient tests not followed up
(Callen et al. BMJ Qual Saf. 2011;20(2):194-9)
• Failure to follow-up test results accounts for 45% of US
diagnosis-related malpractice cases.
(Gandhi et al. Ann Intern Med. 2006;145(7):488-96.)
• Many of these results are clinically significant, with
potential to impact patient care.
• Why? Seems a classic co-ordination of care systems
problem. Busy clinicians? Poor training?

10. :: study: patterns in failure to follow
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
up
• Study setting: 370-bed metropolitan teaching hospital.
Lab order entry implemented for all path tests.
• Data: All 664,643 inpatient path and micro tests between
Feb and June 2011. Time stamps for test orders,
posting, and first test result view.
• Internal medicine and surgery accounted for 63.4% and
33% respectively of all inpatient tests. ED 3%.

11. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
:: results
• Of 664,643 tests analyzed, 3.2% not reviewed at the
time of discharge (n=21,141), 1.5% 2 months post
discharge (n = 10,166).
• 40.3% of inpatients had one or more results not
reviewed at discharge (n=2717) , 28.7% 2 months post
discharge (n=1932) .
• Of unreviewed tests, 20.5% outside normal range at
discharge, 10.6% 2 months post discharge.
• Interesting. But this analysis doesn’t explain causality -
what is going on.
• We need a causal hypothesis!

12. Archives of Internal Medicine, 2012;172(17):1347-1349
At discharge, 21.4% of tests ordered not followed
up compared to 1.9% of tests ordered on other
days (p<0.001).
46.8% of all unreviewed tests were ordered on
the day of discharge
Test follow up a function of time available for
review p(follow-up) = f (LOS)
“Wasted” tests that will never be reviewed
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

13. :: can we improve test follow-up?
• Temporal pattern points to main source of the problem:
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
discharge planning and post-discharge follow up.
• Better discharge planning, that test ordering to discharge
planning
• Better use of electronic alerts, at time of order, or to trigger
post-discharge follow-up
• Enabling patients to assist follow up by:
– Informing them of pending tests
– Encourage them to seek GP follow up
– Personally controlled health record (PCEHR)

14. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

15. :: Case study 2 - the weekend
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
effect
•A higher rate of death following weekend admission to
hospital compared to weekday admission.
•Possible causes:
• Selection bias: cohort of patients admitted on weekends
different (e.g. sicker and older) compared to weekdays.
• Quality of weekend services: lower staffing levels, locum
staff, unavailability of tests or procedures.
• ED and ICU in major hospitals relatively protected from
the weekend effect as many run a similar service across
all days.

16. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

17. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

18. :: study: weekly patterns in death
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
rates
• Study setting: Emergency department admissions to all
501 hospitals in New South Wales, Australia, between
2000 and 2007 were linked to the Death Registry and
analysed.
• Data: There were a total of 3,381,962 admissions for
539,122 patients and 64,789 deaths at 1 week after
admission.
• We computed excess mortality risk curves for weekend
over weekday admissions, adjusting for age, sex,
comorbidity (Charlson index) and diagnostic group.

19. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
:: results
• Weekends accounted for 27.1% of all admissions (917
257/3 381 962) and 28.2% of deaths (18 282/64 789).
• Adjusted mortality rates: weekday 1.85% (95% CI 1.85%
to 1.85%), weekend 2.12% (95% CI 2.12% to 2.12%)
(difference 0.27%, p<0.001).
• Sixteen of 430 diagnosis groups (DRGs) had a
significantly increased risk of death following weekend
admission. They accounted for 40% of all deaths.
• Again, initial data analysis has shown a problem but not
helped us understand causation.

20. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
:: results (2)
F70 (Major Arrhythmia and
Cardiac Arrest).
E61 (Pulmonary Embolism)
E64 (Pulmonary Oedema and
Respiratory Failure)
F65 (Peripheral Vascular
Disorders)
I65 (Connective Tissue
Malignancy, including
Pathological Fracture)
R60 (Acute Leukemia)
R61 (Lymphoma and Non-
Acute Leukaemia)
B02 (Craniotomy)
B67 (Degenerative Nervous
System Disorders)
B70 (Stroke and Other
Cerebrovascular Disorders)
E71 (Respiratory Neoplasms)
F62 (Heart Failure and Shock)
G60 (Malignancy)
H61 (Malignancy of
Hepatobiliary System,
Pancreas)
J62 (Malignant Breast
Disorders) L60 (Renal Failure).

21. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Excess Risk of death
s
t
d w
c
s = severity effect
c = care effect
d = delay in care effect
w = washout of care effect
excess(t)=pweekend(t) − pweekday(t)

22. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Care Same Care Different
Cohort Different Cohort Same
t
Risk of death
H0 H1
t
Risk of death
H2 H3
t
Risk of death
t
Risk of death

23. Major Arrhythmia and Cardiac
Arrest
Pulmonary Embolism,
Pulmonary Oedema and
Respiratory Failure, Peripheral
Vascular Disorders
Connective Tissue Malignancy, Acute
Leukemia, Lymphoma and Non-
Acute Leukaemia
Malignant Breast Disorders,
Respiratory Neoplasms, Malignancy
of Hepatobiliary System, Pancreas,
Craniotomy, Stroke, Heart Failure and
Shock, Renal Failure
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

24. :: what weekend effect patterns say
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
about causation
• Pure care effect for Myocardial infarction ie probably due
to variation in care e.g. unavailability of specialist staff,
imaging or stenting services.
• Risk washout e.g. PE, pulmonary oedema. Acute events
requiring access to high quality immediate care, but with
less abrupt risk of mortality. Those who survive the first
48 h fare better when re-exposed to weekday care.
• Cancer patients dominated the steady risk pattern.
Possibly cancer patients with more severe illness are
admitted on the weekend e.g. when community care can
no longer manage them.

25. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION

26. :: the value of information
• Information that is collected but not acted upon has no
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
realized value
• Health analytics becomes useful when it moves from
demonstrating “mere” association, helps us see patterns
that suggest causation and hence intervention
• Information becomes valuable when it is actionable
• The value of information lies in its ability to change what
we decide and for those decisions to change outcomes
• Can we think about the benefits of e-health systems
using a value of information perspective?

27. The value of e-health interventions
•Some broad generalizations from the recent literature:
• Electronic Health Records appear to decrease nurse but increase
doctor data entry times, improve record completeness, but appear to
not be associated with improvements in care quality.
• Care pathways and plans reduce practice variation by increasing
compliance with standards of care, can improve process metrics
(e.g. test ordering, drug order sets) but typically do not impact
outcomes (e.g. LOS, death).
• Telehealth interventions can increase patient satisfaction, and can
improve patient outcomes in some but not all cases (e.g. chronic
care), but in many cases is surprisingly not cost-effective.
• Decision support systems do improve the safety and efficiency of
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
care and improve patient outcomes.
• How do we interpret this? Is this a failure of some intervention
classes or is this exactly what we should expect?

28. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Interac on
Informa on
received
Decision
changed
Outcome
changed
Care process
altered
Number needed to treat (NNT): How many patients must receive this
treatment before 1 patient sees a benefit
Number needed to read (NNR): How many times must this information
be accessed before I see a change?
Expected Utility (EU): probability(event) x its utility
Value of Information (VOI): EU(option 1) – EU(option 2)

29. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Interac on
Informa on
received
Decision
changed
Outcome
changed
Care process
altered
number of events
value per event
value chain

30. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
Interac on Informa on
received
Care pathway
Decision support
Decision
changed
Outcome
changed
Care process
altered
Expected u lity
value chain
Current prac ce
EHR
Teleconsulta on

31. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
different expectations and metrics
along the value chain
Interaction Information Decision Care process Outcome
Electronic
Health
record
n steps in creating
or retrieving a
record, time per
interaction, n queries
to record, n alerts
created or dismissed
n records in EHR, n
records viewed, record
completeness and
accuracy
n correct or incorrect
decisions, decision
velocity
n and type of tests
ordered, medications
prescribed, cost of
care
Morbidity and
mortality,
QALY
Telehealth
system
n conversations, call
time, user
satisfaction
Quality and quantity of
patient level data
shared
n additional correct or
incorrect decisions
Health service
utilization rates,
travel costs
Blood
pressure,
HbA1c, blood
glucose etc.,
Morbidity and
mortality,
QALY

32. AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION
:: summary
• Information becomes valuable when it is actionable
• Information that is collected but not acted upon has no
realized value
• Health analytics becomes useful when it helps us see
patterns in the process of care that suggest causation
and hence intervention
• E-health systems have very different information value
profiles.
• Our expectations of the impact of e-health must be
shaped by understanding where in the value chain the
maximum benefit will be see.

33. Thank You
Email: e.coiera@unsw.edu.au
Twitter: @enricocoiera
AUSTRALIAN INSTITUTE
OF HEALTH INNOVATION