7. RCTs – best evidence
• RCTs are fundamentally hypothesis testing
instruments (a hypothesis being an educated
guess about how things work)
• RCTs are a type of scientific experiment which
aims to reduce bias when testing new/existing
treatments
8.
9. RCTs – best evidence
• RCTs are fundamentally hypothesis testing
instruments (a hypothesis being an educated
guess about how things work)
• RCTs are a type of scientific experiment which
aims to reduce bias when testing new/existing
treatments
• RCTs permit the use of probability theory to
express the likelihood of a difference in
outcome between treatments/groups
10. P value
• P value - probability of obtaining a result equal
to or more extreme than observed when the
null hypothesis is true
11.
12. P value
• P value - probability of obtaining a result equal
to or more extreme than observed when the
null hypothesis is true
• Null hypothesis H0 – default position – assumed
to be true until evidence indicates otherwise
13. P value
• P value - probability of obtaining a result equal
to or more extreme than observed when the
null hypothesis is true
• Null hypothesis H0 – default position – assumed
to be true until evidence indicates otherwise
• If observed data are unlikely,
then null hypothesis is rejected
• If observed data are consistent,
then null hypothesis is not rejected
14. An analogy might help…?
• Analogous to a criminal trial
• Defendant assumed to be innocent (null)
• Until proven guilty (null is rejected)
• Beyond reasonable doubt (agreed threshold)
15. P value
• P value - probability of obtaining a result equal to
or more extreme than observed when the null
hypothesis is true
• Agreed threshold traditionally set at 5% (P<0.05)
16.
17. Fisher’s exact test
• Used in the analysis of contingency tables
• Used when sample sizes are small
(valid for all sample sizes)
• Calculates significance of deviation from the
null hypothesis exactly
• It can’t be calculated in your head…!
21. P value
• P value - probability of obtaining a result equal to
or more extreme than observed when the null
hypothesis is true
• Agreed threshold traditionally set at 5% (P<0.05)
• A shift of only a few events can change
interpretation…
24. P value
• P value - probability of obtaining a result equal
to or more extreme than observed when the
null hypothesis is true
• Traditionally set at 5% (P<0.05)
• A shift of only a few events can change
interpretation…
• Simple recalculation with the correct statistical
test can change interpretation…
28. P value
• P value - probability of obtaining a result equal to
or more extreme than observed when the null
hypothesis is true
• Traditionally set at 5% (P<0.05)
• A shift of only a few events can change
interpretation…
• Simple recalculation with the correct test can
change interpretation…
• Significant subgroups(?!) – hypothesis generating
46. Walsh et al. 2014 Evaniew et al. 2015 Ridgeon et al. 2016
RCTs in NEJM,
Lancet, JAMA,
Annals, BMJ
(RCT in MeSH)
RCTs on spine
surgery
Multicentre (>1)
RCTs in critically ill
2004-2010 2009-2014 No date restriction
P<0.05 result
in abstract
P<0.05 result
in abstract
P<0.05 result
for mortality
RCTs=399 RCTs= 40 RCTs=56
Patients=
median 682
(IQR 15-112604)
Patients=
median 132
(IQR 79-208)
Patients=
median 127
(IQR 79-326)
47. Walsh et al. 2014 Evaniew et al. 2015 Ridgeon et al. 2016
RCTs in NEJM,
Lancet, JAMA,
Annals, BMJ
(RCT in MeSH)
RCTs on spine
surgery
Multicentre (>1)
RCTs in critically ill
2004-2010 2009-2014 No date restriction
P<0.05 result
in abstract
P<0.05 result
in abstract
P<0.05 result
for mortality
RCTs=399 RCTs= 40 RCTs=56
Patients=
median 682
(IQR 15-112604)
Patients=
median 132
(IQR 79-208)
Patients=
median 127
(IQR 79-326)
66% primary result 58% primary result 52% primary result
48. Walsh et al. 2014 Evaniew et al. 2015 Ridgeon et al. 2016
Fragility Index =
median 8
(IQR 3-18)
Fragility Index =
median 2
(IQR 1-3)
Fragility Index =
median 2
(IQR 1-3.5)
Range = 0-808 Range = 0-39 Range = 0-48
FI zero = 10% FI zero = 20% FI zero = 20%
FI ≤3 = 25% FI ≤3 = 75% FI ≤3 = 75%
FI ≤loss to
follow-up = 53%
FI ≤loss to
follow-up = 65%
FI ≤loss to
follow-up = 87.5%
49. Walsh et al. 2014 Evaniew et al. 2015 Ridgeon et al. 2016
Fragility Index =
median 8
(IQR 3-18)
Fragility Index =
median 2
(IQR 1-3)
Fragility Index =
median 2
(IQR 1-3.5)
Range = 0-808 Range = 0-39 Range = 0-48
FI zero = 10% FI zero = 20% FI zero = 20%
FI ≤3 = 25% FI ≤3 = 75% FI ≤3 = 75%
FI ≤loss to
follow-up = 53%
FI ≤loss to
follow-up = 65%
FI ≤loss to
follow-up = 87.5%
50. Fragility Index
• Significant results of many RCTs hinge on very
few events
• Results could potentially be overturned if
missing/loss to follow-up data were known
56. Fragility Index
• Significant results of many RCTs hinge on very
few events
• Results could potentially be overturned if
missing/loss to follow-up data were known
• Reporting of the Fragility Index may help
interpretation/over-interpretation
• You can calculate it yourself…!
http://fragilityindex.com/
Editor's Notes
RCT Gold standard
Experiment
Random allocation (like tossing a coin)
Minimises selection bias - balances known and unknown characteristics
Groups only differ by randomly allocated intervention
Simple to analyse
