ICN Victoria presents Professor Jack Iwashyna, giving a thought provoking talk on how we may better use data from ANZICS large RCTs to guide management of our critically ill patients.

1. Please Stop Wasting RCT Data
Theodore J. Iwashyna, MD, PhD
University of Michigan
Ann Arbor VA Center for Clinical Management Research
on sabbatical at ANZIC-RC at Monash University
2 December 2015 – Victorian Intensive Care Network

2. Disclosures
• Key Funding:
• U.S. NIH K08 HL091249 (TJI)
• U.S. NIH R21 AG044752 (TJI)
• U.S. VA HSR&D IIR 11-109 (TJI)
• University of Michigan (for sabbatical funds)
• Much of this is joint work with Jim Burke, Jeremy Sussman,
Hallie Prescott, Rod Hayward, and Derek Angus. It would have
been impossible without them. Errors are, however, mine.
• This work does not necessarily represent the views of the
U.S. Government or Department of Veterans Affairs
• I have no relevant financial conflicts of interest to disclose
• This talk is based on PMID: 26177009

3. The Simple Dream:
I will provide this treatment if
the benefits outweigh the
harms.
Net Benefit = Benefit – Harm
If Net Benefit > 0, I treat.

4. Guyatt et al (1994) JAMA 271:59.

5. Guyatt et al (1994) JAMA 271:59.

6. Courtesy of HC Prescott; North American Symptomatic Carotid Endarterctomy Trial Collaboration (1991) NEJM 325:445.

7. Rothwell (1995) The Lancet 345:8965.

8. The Simple Dream:
If Net Benefit > 0, I treat.
If a patient would have been
enrolled in a clinical trial, then
my best guess should be that
Net Benefit > 0
The Extenders:
If Net Benefit > 0, I treat.
If a patient would have been
enrolled in a clinical trial, then
we need more information to
know if Net Benefit > 0

9. The Simple Dream:
If Net Benefit > 0, I treat.
If a patient would have been
enrolled in a clinical trial, then
my best guess should be that
Net Benefit > 0
The Extenders:
If Net Benefit > 0, I treat.
If a patient would have been
enrolled in a clinical trial, then
we need more information to
know if Net Benefit > 0

10. The Implications of
Heterogeneity of Treatment
Effect by Baseline Risk
• Why we should just about
always expect HTE
• HTE and positive trials in
acute respiratory failure
• HTE and negative trials in
acute respiratory failure
• But, maybe…
• So what the heck am I
supposed to do with this?

11. RiskofDeath
If Never Treated
Untreated Risk of Death

12. RiskofDeath
If Never Treated
Untreated Risk of Death
RiskofDeath
If Treated
(and no side-effects)
Untreated Risk of Death

13. RiskofDeath
If Never Treated
Untreated Risk of Death
RiskofDeath
If Treated
(and no side-effects)
Untreated Risk of Death

14. RiskofDeath
If Never Treated
Untreated Risk of Death
RiskofDeath
If Treated
(and no side-effects)
Untreated Risk of Death
ReductioninRiskofDeath
Absolute Mortality Benefit
of Treatment, assuming no
side effects
Untreated Risk of Death

15. RiskofDeath
If Never Treated
Untreated Risk of Death
RiskofDeath
If Treated
(and no side-effects)
Untreated Risk of Death
ReductioninRiskofDeath
Absolute Mortality Benefit
of Treatment, assuming no
side effects
Untreated Risk of Death

16. RiskofDeath
Side Effect Risk of
Treatment
Untreated Risk of Death

17. RiskofDeath
Putting it all together
Untreated Risk of Death
Absolute Mortality Benefit
of Treatment, assuming no
side effects
Side Effect Risk of
Treatment

18. Untreated Risk of Death
A
A: Clearly good, these people should get this
3 Domains of Net Benefit

19. Untreated Risk of Death Untreated Risk of Death
A B
A: Clearly good, these people should get this
B: Clearly bad, these people should not get this
3 Domains of Net Benefit

20. Untreated Risk of Death Untreated Risk of Death
A B
Untreated Risk of Death
C
A: Clearly good, these people should get this
B: Clearly bad, these people should not get this
C: Uncertain, requires a conversation
3 Domains of Net Benefit

21. Untreated Risk of Death Untreated Risk of Death
A B
Untreated Risk of Death
C
A: Clearly good, these people should get this
B: Clearly bad, these people should not get this
C: Uncertain, requires a conversation
Key question at the bedside:
For this patient, for this treatment, where are we?

22. This all hinges on there being a big distribution
of baseline risk. Are my patients that heterogeneous?
0
100
200
300
400
Frequency
0.0 0.2 0.4 0.6 0.8 1.0
Baseline Risk of Death
Non-Surgical Mechanical Ventilation
US Veterans Affairs
Non-Post-Operative Mech Vent
0
500
1000
1500
2000
2500
Frequency
0 .2 .4 .6 .8 1
Apache3RiskOfDeath
Austalian APD
Non-Post-Operative Mech Vent
w/ LOS>24h, not Drug Overdose

23. The Implications of
Heterogeneity of Treatment
Effect by Baseline Risk
• Why we should just about
always expect HTE
• HTE and positive trials in
acute respiratory failure
• HTE and negative trials in
acute respiratory failure
• But, maybe…
• So what the heck am I
supposed to do with this?

24. Simulating a new therapy
• 20% relative risk reduction if no
adverse events
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death

25. Simulating a new therapy
• 20% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death

26. Simulating a new therapy
• 20% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death
0
100
200
300
400
Frequency
0.0 0.2 0.4 0.6 0.8 1.0
Baseline Risk of Death
Non-Surgical Mechanical Ventilation

27. Simulating a new therapy
• 20% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
• Observed RRR = 0.85
(95% CI: 0.77, 0.94)
• Absolute risk reduction from
39.8% to 33.6%
• Wooohooo!
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death

28. Favors Treatment
Favors Control
5%
0
10%
20%
NetMortalityEffectofTreatment
0
.2
.4
.6
.8
Mortality
1 2 3 4 5 6 7 8 9 10
Deciles of Baseline Risk of Death at Randomization
HTE in Positive Trials
Untreated
Treated
Lowest Risk Highest Risk
Simulating a new therapy
• 20% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
• Observed RRR = 0.85
(95% CI: 0.77, 0.94)
• Absolute risk reduction from
39.8% to 33.6%
NNT in highest risk = 8
NNT in decile 2 = 90
Net harm in lowest risk patients.

29. The Implications of
Heterogeneity of Treatment
Effect by Baseline Risk
• Why we should just about
always expect HTE
• HTE and positive trials in
acute respiratory failure
• HTE and negative trials in
acute respiratory failure
• But, maybe…
• So what the heck am I
supposed to do with this?

30. Simulating a new therapy
• 15% relative risk reduction if no
adverse events
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death

31. Simulating a new therapy
• 15% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death

32. Simulating a new therapy
• 15% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death
0
100
200
300
400
Frequency
0.0 0.2 0.4 0.6 0.8 1.0
Baseline Risk of Death
Non-Surgical Mechanical Ventilation

33. Simulating a new therapy
• 15% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
• 45% of these trials are now
negative (p>0.05 for differences
between treated & controls)
RiskofDeath
If Treated
(and no adverse events)
Untreated Risk of Death
RiskofDeath
Adverse Event
Risk of Treatment
Untreated Risk of Death

34. Favors Control
Favors Treatment
5%
0
10%
20%
NetMortalityEffectofTreatment
0
.2
.4
.6
.8
Mortality
1 2 3 4 5 6 7 8 9 10
Deciles of Baseline Risk of Death at Randomization
HTE in Negative Trials
Untreated
Treated
Lowest Risk Highest Risk
Simulating a new therapy
• 15% relative risk reduction if no
adverse events
• 3% risk of adverse events
(known and unknown) if treated
• N=2,500, run in an acute
respiratory failure population
• 45% of these trials are now
negative (p>0.05 for differences
between treated & controls)
But, NNT in highest risk = 9
NNT in 2nd highest risk = 14

35. 0510152025
Frequency
0 .2 .4 .6 .8 1
Severity
Untreated Risk of Death
0102030
Frequency
0 .2 .4 .6 .8 1
Severity
2 RCTs:
Same
Net Benefit Profile
but
Modest Differences
In Baseline Risk
Among Enrolled

36. 0510152025
Frequency
0 .2 .4 .6 .8 1
Severity
RiskofDeath
Untreated Risk of Death
AB
0102030
Frequency
0 .2 .4 .6 .8 1
Severity
Yay!
Positive Trial
Sad!
“Negative” Trial
Kent et al (2010) Trials 11:85; see also Hayward et al (2005) Health Affairs 24:1571.
2 RCTs:
Same
Net Benefit Profile
but
Modest Differences
In Baseline Risk
Among Enrolled

37. The Implications of
Heterogeneity of Treatment
Effect by Baseline Risk
• Why we should just about
always expect HTE
• HTE and positive trials in
acute respiratory failure
• HTE and negative trials in
acute respiratory failure
• But, maybe…
• So what the heck am I
supposed to do with this?

38. Aren’t people doing this already?

39. Aren’t people doing this already?
Sort of. ICNARC does it as part of
their trials, hidden in the online
Appendices. Not so much others.
If you find others, please tell me!

40. But in reality, adverse events are
not perfectly even. Sicker patients
have more adverse events.
NNT in highest risk: 7
NNT in lowest risk: 74

41. But in reality, adverse events are
not perfectly even. Sicker patients
have more adverse events.
NNT in highest risk: 7
NNT in lowest risk: 74
0
.2
.4
.6
.8
Mortality
1 2 3 4 5 6 7 8 9 10
HTE in Positive Trials with Adverse Event Rate that Increases with Baseline Risk
Untreated
Treated
Deciles of Baseline Risk of Death at Randomization
Favors Treatment
Favors Control
5%
0
10%
20%
NetMortalityEffectofTreatment

42. But in reality, many of the things
that kill patients are not even
potentially responsive to treatment.
But if there is too much non-
responsive risk, it becomes really
hard to have a positive trial.

43. But in reality, many of the things
that kill patients are not even
potentially responsive to treatment.
But if there is too much non-
responsive risk, it becomes really
hard to have a positive trial.
FractionofRiskthat
isTreatment-
Responsive
FractionofRiskfrom
OtherCausesof
Death
Treatment-
ResponsiveRelative
RiskReduction100% 0% 20%
75% 25% 27.5%
50% 50% 40%
25% 75% 80%

44. Highest
Risk
Lowest
Risk
Decile of Baseline Risk
for Death
100% -640 239 121 73 49 37 26 19 12 6
75% 1071 151 90 59 44 34 26 19 13 6
50% 284 103 66 46 37 29 23 23 13 8
25% 131 65 45 36 29 23 19 15 13 14
1 2 3 4 5 6 7 8 9 10
ProportionofRiskthatis
TreatmentResponsive
Based on data visualization courtesy of HC Prescott.

45. Highest
Risk
Lowest
Risk
Decile of Baseline Risk
for Death
100% -640 239 121 73 49 37 26 19 12 6
75% 1071 151 90 59 44 34 26 19 13 6
50% 284 103 66 46 37 29 23 23 13 8
25% 131 65 45 36 29 23 19 15 13 14
1 2 3 4 5 6 7 8 9 10
ProportionofRiskthatis
TreatmentResponsive
Based on data visualization courtesy of HC Prescott.
But in reality, many of the things that kill patients are not even
potentially responsive to treatment.
Even when this is true, and you have an incredibly potent therapy,
there is still substantial variability in NNT in positive trials.

46. But Australia has many very low
risk patients—it’s ICU population is
way more skewed than that VA
data you showed.
0
100
200
300
400
Frequency
0.0 0.2 0.4 0.6 0.8 1.0
Baseline Risk of Death
Non-Surgical Mechanical Ventilation
0
500
1000
1500
2000
2500
Frequency
0 .2 .4 .6 .8 1
Apache3RiskOfDeath

47. But Australia has many very low
risk patients—it’s ICU population is
way more skewed than that VA
data you showed.
The more uneven the distribution,
the worse the problem.
0
100
200
300
400
Frequency
0.0 0.2 0.4 0.6 0.8 1.0
Baseline Risk of Death
Non-Surgical Mechanical Ventilation
0
500
1000
1500
2000
2500
Frequency
0 .2 .4 .6 .8 1
Apache3RiskOfDeath

48. The Implications of
Heterogeneity of Treatment
Effect by Baseline Risk
• Why we should just about
always expect HTE
• HTE and positive trials in
acute respiratory failure
• HTE and negative trials in
acute respiratory failure
• But, maybe…
• So what the heck am I
supposed to do with this?

49. Storming of the Bastille, by Jean-Pierre-Louis-Laurent Houel, from Wikipedia.org.
Our journals are letting us down.
RCTs should, at a minimum, be
published with subgroup analyses by
baseline risk of death.
These should be interpreted cautiously,
like any subgroup, but with a high prior
likelihood of variation in effect size.
Demand better!

50. Overall baseline risk, not just specific physiology, fundamentally
shapes each patient’s opportunity for benefit from any therapy.
Higher risk patients may often benefit substantially more from our
therapies than low risk patients—even if high risk patients die more
often anyway.
Our bedside psychology (availability & salience biases) may
mislead us, emphasizing the deaths despite therapy in high risk
patients more than the saves because of it – and
overemphasizing the “saves” despite therapy in low risk groups.
Be willing to withhold “indicated” therapy in very low risk patients, or
in modestly low risk patients with higher likelihoods of adverse
events.
This is not an excuse to willy-nilly ignore RCTs & guidelines.

51. Untreated Risk of Death Untreated Risk of Death
A B
Untreated Risk of Death
C
Key question at the bedside:
For this patient, for this treatment, where are we?
Please email me at jack.iwashyna.on.sabbatical @
gmail.com for copies of my slides or to continue a
conversation. I often tweet @iwashyna.