SlideShare a Scribd company logo
Clinical Trials are not Enough
Stephen Senn
(c) Stephen Senn 1
Acknowledgements
(c) Stephen Senn 2
Acknowledgements
This work is partly supported by the European Union’s 7th Framework Programme for
research, technological development and demonstration under grant agreement no.
602552. “IDEAL”
Basic Thesis
• Clinical trials are experiments
• At every stage of drug development, even phase III, they are
unrepresentative of ‘real world’ clinical practice
• The key to using their results to inform ‘real world’ decisions
is not to make the trial more representative
• The key is to use appropriate scales for analysis and transfer
the results into practical real-world decision making
• This will require
– A different attitude
– More modelling
– Extensive use of auxiliary real world data
(c) Stephen Senn 3
An Example
• Dan Moerman’s analysis more than 30 years ago
of Smith-Kline-French’s development of Tagamet®
– Tagamet® (cimetidine) was the best selling drug of its
day
• 31 trials in 1692 patients
– in 17 countries
– Duodenal, Gastric, Mixed
– 13 significant according to Moerman
• Used Chi-square test with Yates’s correction
(c) Stephen Senn 4
(c) Stephen Senn 5
on the risk-
difference
scale and
there is
significant
evidence of
heterogeneity
What the EBM movement used to
conclude from this sort of thing
• The treatment effect varies according to the type
of patient
• We want RCTS that automatically deliver the right
decision
• We can’t rely on the results from RCTs which
involve artificially selected patients
• We need large simple trials in representative
populations
• They need to be reported using numbers needed
to treat (NNTs)
(c) Stephen Senn 6
Problems
• At best you can hope to recommend
treatments that are beneficial on average
• But in practice you can never guarantee that
trials are representative of practice anyway
• You lose the opportunity to study issues more
deeply
• NNTs are a terrible scale for doing analysis
(c) Stephen Senn 7
(c) Stephen Senn 8
Here the
analysis is on
the log-odds
ratio scale and
heterogeneity is
much reduced
and not even
‘significant’.
9
“If you need statistics to prove it I don’t believe it”
You can’t prove it with statistics but everybody believes it
Thanks to Pat
Ballew’s blog site
(c) Stephen Senn 9
That is to say, we see non-random variation too easily.
This example does NOT give strong evidence treatment
by trial interaction
Lessons
• If not carefully, studied random variation can
be underestimated
• Differences from trial to trial in true effect
may be less than one thinks
• Finding a good scale is important
• BUT The additive scale is not necessarily the
relevant one
(c) Stephen Senn 10
What not to do
• The solution is not to attempt to make trials
more representative
• The solution is to measure appropriately and
translate appropriately
• This requires the following
– Good scales
– Good analysis
– Good modelling
– Good supplementary real world data
(c) Stephen Senn 11
(c) Stephen Senn 12
Chasing sub-
groups leads
nowhere
Solution?
(c) Stephen Senn 14
“a possible resolution is to use the additive
measure at the point of analysis and transform
to the relevant scale at the point of
implementation. This transformation at the
point of medical decision-making will require
auxiliary information on the level of background
risk of the patient.”
Senn, Statistics in Medicine, 2004
How we already use modelling, data
and additive scales
• Interspecies scaling
• Bioequivalence
– log relative bioavailability is additive but
difference in absolute bioavailability is not
• Dose proportionality
• Use of additive scales in phase III
– Log hazard
– Log-odds ratio
(c) Stephen Senn 15
(c) Stephen Senn 17Controlled Clinical Trials, 1989
Implications of the Lubsen-Tijssen Model
• We need to study treatment benefit on
disaggregated (of harm) additive scale
• We will need real world data on harms
• We will need real world data on background risk
• We will need models
• We will need cooperation between
– Medics and statisticians working on clinical trials
– Statisticians, epidemiologists, health economists,
medics and others working in real world data
(c) Stephen Senn 18
Example of Atrial Fibrillation
• Such patients are at
higher risk of stroke
• Meta-analysis
(reproduced in Hart et al
2007)concluded that
warfarin has a beneficial
protective effect
• But there is a risk of
intracranial bleeding
• Who should get warfarin?
(c) Stephen Senn 19
(c) Stephen Senn 20
(c) Stephen Senn 21
So you have atrial fibrillation
• Should you take warfarin?
• What else do you need to know?
– The difference in risk taking warfarin or not
– The rate of side effects
– The consequences of side-effects
• These cannot be answered (alone) by analysis of
RCTs with pre-specified efficacy measures on the
additive scale
• The RCTs has to be translated and supplemented
by real world data
(c) Stephen Senn 22
(c) Stephen Senn 23
(c) Stephen Senn 24
Estimate
based on 6 v 3
cases only
The reimburser’s perspective
• What benefit and harm to the population will
accrue from recommending warfarin
prophylaxis?
• How much will it cost?
• How can its use be optimised?
• Who should get it?
(c) Stephen Senn 25
(c) Stephen Senn 26
Reimburser’s needs
Requirements
• A means of separating
patients by risk
• A means of establishing risk
distribution in the
population of patients
above any threshold chosen
• A means of determining
expected benefits and costs
Solutions
• These figures cannot be
delivered by clinical trials
alone but will require
– Cohort studies/case control
studies
– Health surveys
– Economic modelling
(c) Stephen Senn 27
We need to model background risk
• Sort of data set we could use is that provided
by the UK Clinical Practice Research Data Link
CPRD
• Could use this to model
– A) Predictors of risk of stroke
– B) Distribution of risk levels in the population
• Former relevant to individuals to make
decisions
• Latter is relevant to reimbursers
(c) Stephen Senn 28
Is there a Trust Problem?
• Yes
• Clinical trials provide a “template of trust”
whereby regulators can mandate sponsors to
provide the proof
• Modelling + real world data cannot provide these
guarantees
• But this is no excuse
– Whether or not you model, others will
– You need to know as much as possible about your
own drugs and where and when to use them
(c) Stephen Senn 29
Finally
I leave you with this though
(c) Stephen Senn 30
Any damn fool can analyse a clinical trial and frequently does
But doing it properly involves skilful analysis,
understanding what the results mean requires intelligence
insight and experience,
and applying the results intelligently needs more of the
same plus modelling and real world data
And to whom do we look to provide these skills?
Statisticians!

More Related Content

What's hot

Correcting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confoundingCorrecting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confounding
Maarten van Smeden
 
Prognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient healthPrognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient health
Maarten van Smeden
 
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
Ewout Steyerberg
 
P-values in crisis
P-values in crisisP-values in crisis
P-values in crisis
Laure Wynants
 
Personalised medicine a sceptical view
Personalised medicine a sceptical viewPersonalised medicine a sceptical view
Personalised medicine a sceptical view
Stephen Senn
 
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
GaryCollins74
 
Uncertainty in AI
Uncertainty in AIUncertainty in AI
Uncertainty in AI
Maarten van Smeden
 
Introduction to prediction modelling - Berlin 2018 - Part I
Introduction to prediction modelling - Berlin 2018 - Part IIntroduction to prediction modelling - Berlin 2018 - Part I
Introduction to prediction modelling - Berlin 2018 - Part I
Maarten van Smeden
 
What is the point of point estimates
What is the point of point estimates What is the point of point estimates
What is the point of point estimates
StephenSenn2
 
The Seven Habits of Highly Effective Statisticians
The Seven Habits of Highly Effective StatisticiansThe Seven Habits of Highly Effective Statisticians
The Seven Habits of Highly Effective Statisticians
Stephen Senn
 
Development and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutionsDevelopment and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutions
Maarten van Smeden
 
Dichotomania and other challenges for the collaborating biostatistician
Dichotomania and other challenges for the collaborating biostatisticianDichotomania and other challenges for the collaborating biostatistician
Dichotomania and other challenges for the collaborating biostatistician
Laure Wynants
 
Introduction to prediction modelling - Berlin 2018 - Part II
Introduction to prediction modelling - Berlin 2018 - Part IIIntroduction to prediction modelling - Berlin 2018 - Part II
Introduction to prediction modelling - Berlin 2018 - Part II
Maarten van Smeden
 
The basics of prediction modeling
The basics of prediction modeling The basics of prediction modeling
The basics of prediction modeling
Maarten van Smeden
 
Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?
Maarten van Smeden
 
Predictimands
PredictimandsPredictimands
Predictimands
Maarten van Smeden
 
Clinical prediction models
Clinical prediction modelsClinical prediction models
Clinical prediction models
Maarten van Smeden
 
Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead
Maarten van Smeden
 
Statistics and ML 21Oct22 sel.pptx
Statistics and ML 21Oct22 sel.pptxStatistics and ML 21Oct22 sel.pptx
Statistics and ML 21Oct22 sel.pptx
Ewout Steyerberg
 
NNTs, responder analysis & overlap measures
NNTs, responder analysis & overlap measuresNNTs, responder analysis & overlap measures
NNTs, responder analysis & overlap measures
Stephen Senn
 

What's hot (20)

Correcting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confoundingCorrecting for missing data, measurement error and confounding
Correcting for missing data, measurement error and confounding
 
Prognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient healthPrognosis-based medicine: merits and pitfalls of forecasting patient health
Prognosis-based medicine: merits and pitfalls of forecasting patient health
 
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019
 
P-values in crisis
P-values in crisisP-values in crisis
P-values in crisis
 
Personalised medicine a sceptical view
Personalised medicine a sceptical viewPersonalised medicine a sceptical view
Personalised medicine a sceptical view
 
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
QUANTIFYING THE IMPACT OF DIFFERENT APPROACHES FOR HANDLING CONTINUOUS PREDIC...
 
Uncertainty in AI
Uncertainty in AIUncertainty in AI
Uncertainty in AI
 
Introduction to prediction modelling - Berlin 2018 - Part I
Introduction to prediction modelling - Berlin 2018 - Part IIntroduction to prediction modelling - Berlin 2018 - Part I
Introduction to prediction modelling - Berlin 2018 - Part I
 
What is the point of point estimates
What is the point of point estimates What is the point of point estimates
What is the point of point estimates
 
The Seven Habits of Highly Effective Statisticians
The Seven Habits of Highly Effective StatisticiansThe Seven Habits of Highly Effective Statisticians
The Seven Habits of Highly Effective Statisticians
 
Development and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutionsDevelopment and evaluation of prediction models: pitfalls and solutions
Development and evaluation of prediction models: pitfalls and solutions
 
Dichotomania and other challenges for the collaborating biostatistician
Dichotomania and other challenges for the collaborating biostatisticianDichotomania and other challenges for the collaborating biostatistician
Dichotomania and other challenges for the collaborating biostatistician
 
Introduction to prediction modelling - Berlin 2018 - Part II
Introduction to prediction modelling - Berlin 2018 - Part IIIntroduction to prediction modelling - Berlin 2018 - Part II
Introduction to prediction modelling - Berlin 2018 - Part II
 
The basics of prediction modeling
The basics of prediction modeling The basics of prediction modeling
The basics of prediction modeling
 
Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?Algorithm based medicine: old statistics wine in new machine learning bottles?
Algorithm based medicine: old statistics wine in new machine learning bottles?
 
Predictimands
PredictimandsPredictimands
Predictimands
 
Clinical prediction models
Clinical prediction modelsClinical prediction models
Clinical prediction models
 
Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead Why the EPV≥10 sample size rule is rubbish and what to use instead
Why the EPV≥10 sample size rule is rubbish and what to use instead
 
Statistics and ML 21Oct22 sel.pptx
Statistics and ML 21Oct22 sel.pptxStatistics and ML 21Oct22 sel.pptx
Statistics and ML 21Oct22 sel.pptx
 
NNTs, responder analysis & overlap measures
NNTs, responder analysis & overlap measuresNNTs, responder analysis & overlap measures
NNTs, responder analysis & overlap measures
 

Similar to Real world modified

The challenge of small data
The challenge of small dataThe challenge of small data
The challenge of small data
Stephen Senn
 
In search of the lost loss function
In search of the lost loss function In search of the lost loss function
In search of the lost loss function
Stephen Senn
 
What is your question
What is your questionWhat is your question
What is your question
Stephen Senn
 
What is your question
What is your questionWhat is your question
What is your question
StephenSenn2
 
Clinical trials: quo vadis in the age of covid?
Clinical trials: quo vadis in the age of covid?Clinical trials: quo vadis in the age of covid?
Clinical trials: quo vadis in the age of covid?
Stephen Senn
 
To infinity and beyond v2
To infinity and beyond v2To infinity and beyond v2
To infinity and beyond v2
Stephen Senn
 
Minimally important differences
Minimally important differencesMinimally important differences
Minimally important differences
Stephen Senn
 
Minimally important differences v2
Minimally important differences v2Minimally important differences v2
Minimally important differences v2
Stephen Senn
 
Numbers needed to mislead
Numbers needed to misleadNumbers needed to mislead
Numbers needed to mislead
Stephen Senn
 
Clinical trials are about comparability not generalisability V2.pptx
Clinical trials are about comparability not generalisability V2.pptxClinical trials are about comparability not generalisability V2.pptx
Clinical trials are about comparability not generalisability V2.pptx
StephenSenn3
 
Does clinical research help me take care of my patient?
Does clinical research help me take care of my patient?Does clinical research help me take care of my patient?
Does clinical research help me take care of my patient?
scanFOAM
 
Clinical prediction models: development, validation and beyond
Clinical prediction models:development, validation and beyondClinical prediction models:development, validation and beyond
Clinical prediction models: development, validation and beyond
Maarten van Smeden
 
8 screening.pptxscreening.pptxscreening.
8 screening.pptxscreening.pptxscreening.8 screening.pptxscreening.pptxscreening.
8 screening.pptxscreening.pptxscreening.
Wasihun Aragie
 
To infinity and beyond
To infinity and beyond To infinity and beyond
To infinity and beyond
Stephen Senn
 
Understanding randomisation
Understanding randomisationUnderstanding randomisation
Understanding randomisation
Stephen Senn
 
Screening in biomedical sciences ‫‬
Screening in biomedical sciences ‫‬Screening in biomedical sciences ‫‬
Screening in biomedical sciences ‫‬
Dr Abbas Assayed
 
Testing of hypothesis and Goodness of fit
Testing of hypothesis and Goodness of fitTesting of hypothesis and Goodness of fit
Testing of hypothesis and Goodness of fit
Sir Parashurambhau College, Pune
 
Depersonalising medicine
Depersonalising medicineDepersonalising medicine
Depersonalising medicine
Stephen Senn
 
Sampling and Sample Size
Sampling and Sample SizeSampling and Sample Size
Sampling and Sample Size
Dr. Keerti Jain
 
Surveillance and screening-cp.pptx
Surveillance and screening-cp.pptxSurveillance and screening-cp.pptx
Surveillance and screening-cp.pptx
chcjayanagara
 

Similar to Real world modified (20)

The challenge of small data
The challenge of small dataThe challenge of small data
The challenge of small data
 
In search of the lost loss function
In search of the lost loss function In search of the lost loss function
In search of the lost loss function
 
What is your question
What is your questionWhat is your question
What is your question
 
What is your question
What is your questionWhat is your question
What is your question
 
Clinical trials: quo vadis in the age of covid?
Clinical trials: quo vadis in the age of covid?Clinical trials: quo vadis in the age of covid?
Clinical trials: quo vadis in the age of covid?
 
To infinity and beyond v2
To infinity and beyond v2To infinity and beyond v2
To infinity and beyond v2
 
Minimally important differences
Minimally important differencesMinimally important differences
Minimally important differences
 
Minimally important differences v2
Minimally important differences v2Minimally important differences v2
Minimally important differences v2
 
Numbers needed to mislead
Numbers needed to misleadNumbers needed to mislead
Numbers needed to mislead
 
Clinical trials are about comparability not generalisability V2.pptx
Clinical trials are about comparability not generalisability V2.pptxClinical trials are about comparability not generalisability V2.pptx
Clinical trials are about comparability not generalisability V2.pptx
 
Does clinical research help me take care of my patient?
Does clinical research help me take care of my patient?Does clinical research help me take care of my patient?
Does clinical research help me take care of my patient?
 
Clinical prediction models: development, validation and beyond
Clinical prediction models:development, validation and beyondClinical prediction models:development, validation and beyond
Clinical prediction models: development, validation and beyond
 
8 screening.pptxscreening.pptxscreening.
8 screening.pptxscreening.pptxscreening.8 screening.pptxscreening.pptxscreening.
8 screening.pptxscreening.pptxscreening.
 
To infinity and beyond
To infinity and beyond To infinity and beyond
To infinity and beyond
 
Understanding randomisation
Understanding randomisationUnderstanding randomisation
Understanding randomisation
 
Screening in biomedical sciences ‫‬
Screening in biomedical sciences ‫‬Screening in biomedical sciences ‫‬
Screening in biomedical sciences ‫‬
 
Testing of hypothesis and Goodness of fit
Testing of hypothesis and Goodness of fitTesting of hypothesis and Goodness of fit
Testing of hypothesis and Goodness of fit
 
Depersonalising medicine
Depersonalising medicineDepersonalising medicine
Depersonalising medicine
 
Sampling and Sample Size
Sampling and Sample SizeSampling and Sample Size
Sampling and Sample Size
 
Surveillance and screening-cp.pptx
Surveillance and screening-cp.pptxSurveillance and screening-cp.pptx
Surveillance and screening-cp.pptx
 

More from Stephen Senn

Has modelling killed randomisation inference frankfurt
Has modelling killed randomisation inference frankfurtHas modelling killed randomisation inference frankfurt
Has modelling killed randomisation inference frankfurt
Stephen Senn
 
Vaccine trials in the age of COVID-19
Vaccine trials in the age of COVID-19Vaccine trials in the age of COVID-19
Vaccine trials in the age of COVID-19
Stephen Senn
 
Approximate ANCOVA
Approximate ANCOVAApproximate ANCOVA
Approximate ANCOVA
Stephen Senn
 
A century of t tests
A century of t testsA century of t tests
A century of t tests
Stephen Senn
 
Is ignorance bliss
Is ignorance blissIs ignorance bliss
Is ignorance bliss
Stephen Senn
 
What should we expect from reproducibiliry
What should we expect from reproducibiliryWhat should we expect from reproducibiliry
What should we expect from reproducibiliry
Stephen Senn
 
De Finetti meets Popper
De Finetti meets PopperDe Finetti meets Popper
De Finetti meets Popper
Stephen Senn
 
In Search of Lost Infinities: What is the “n” in big data?
In Search of Lost Infinities: What is the “n” in big data?In Search of Lost Infinities: What is the “n” in big data?
In Search of Lost Infinities: What is the “n” in big data?
Stephen Senn
 
Seventy years of RCTs
Seventy years of RCTsSeventy years of RCTs
Seventy years of RCTs
Stephen Senn
 
The Rothamsted school meets Lord's paradox
The Rothamsted school meets Lord's paradoxThe Rothamsted school meets Lord's paradox
The Rothamsted school meets Lord's paradox
Stephen Senn
 
The revenge of RA Fisher
The revenge of RA Fisher The revenge of RA Fisher
The revenge of RA Fisher
Stephen Senn
 
The story of MTA/02
The story of MTA/02The story of MTA/02
The story of MTA/02
Stephen Senn
 
Confounding, politics, frustration and knavish tricks
Confounding, politics, frustration and knavish tricksConfounding, politics, frustration and knavish tricks
Confounding, politics, frustration and knavish tricks
Stephen Senn
 
And thereby hangs a tail
And thereby hangs a tailAnd thereby hangs a tail
And thereby hangs a tail
Stephen Senn
 
The revenge of RA Fisher
The revenge of RA FisherThe revenge of RA Fisher
The revenge of RA Fisher
Stephen Senn
 
P value wars
P value warsP value wars
P value wars
Stephen Senn
 
Thinking statistically v3
Thinking statistically v3Thinking statistically v3
Thinking statistically v3
Stephen Senn
 
Seven myths of randomisation
Seven myths of randomisation Seven myths of randomisation
Seven myths of randomisation
Stephen Senn
 

More from Stephen Senn (18)

Has modelling killed randomisation inference frankfurt
Has modelling killed randomisation inference frankfurtHas modelling killed randomisation inference frankfurt
Has modelling killed randomisation inference frankfurt
 
Vaccine trials in the age of COVID-19
Vaccine trials in the age of COVID-19Vaccine trials in the age of COVID-19
Vaccine trials in the age of COVID-19
 
Approximate ANCOVA
Approximate ANCOVAApproximate ANCOVA
Approximate ANCOVA
 
A century of t tests
A century of t testsA century of t tests
A century of t tests
 
Is ignorance bliss
Is ignorance blissIs ignorance bliss
Is ignorance bliss
 
What should we expect from reproducibiliry
What should we expect from reproducibiliryWhat should we expect from reproducibiliry
What should we expect from reproducibiliry
 
De Finetti meets Popper
De Finetti meets PopperDe Finetti meets Popper
De Finetti meets Popper
 
In Search of Lost Infinities: What is the “n” in big data?
In Search of Lost Infinities: What is the “n” in big data?In Search of Lost Infinities: What is the “n” in big data?
In Search of Lost Infinities: What is the “n” in big data?
 
Seventy years of RCTs
Seventy years of RCTsSeventy years of RCTs
Seventy years of RCTs
 
The Rothamsted school meets Lord's paradox
The Rothamsted school meets Lord's paradoxThe Rothamsted school meets Lord's paradox
The Rothamsted school meets Lord's paradox
 
The revenge of RA Fisher
The revenge of RA Fisher The revenge of RA Fisher
The revenge of RA Fisher
 
The story of MTA/02
The story of MTA/02The story of MTA/02
The story of MTA/02
 
Confounding, politics, frustration and knavish tricks
Confounding, politics, frustration and knavish tricksConfounding, politics, frustration and knavish tricks
Confounding, politics, frustration and knavish tricks
 
And thereby hangs a tail
And thereby hangs a tailAnd thereby hangs a tail
And thereby hangs a tail
 
The revenge of RA Fisher
The revenge of RA FisherThe revenge of RA Fisher
The revenge of RA Fisher
 
P value wars
P value warsP value wars
P value wars
 
Thinking statistically v3
Thinking statistically v3Thinking statistically v3
Thinking statistically v3
 
Seven myths of randomisation
Seven myths of randomisation Seven myths of randomisation
Seven myths of randomisation
 

Recently uploaded

Get Covid Testing at Fit to Fly PCR Test
Get Covid Testing at Fit to Fly PCR TestGet Covid Testing at Fit to Fly PCR Test
Get Covid Testing at Fit to Fly PCR Test
NX Healthcare
 
R3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
R3 Stem Cell Therapy: A New Hope for Women with Ovarian FailureR3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
R3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
R3 Stem Cell
 
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
Levi Shapiro
 
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
SamboGlo
 
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
gjsma0ep
 
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
DR Jag Mohan Prajapati
 
National Rural Health Mission(NRHM).pptx
National Rural Health Mission(NRHM).pptxNational Rural Health Mission(NRHM).pptx
National Rural Health Mission(NRHM).pptx
Jyoti Chand
 
Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.
Vishal kr Thakur
 
Top massage center in ajman chandrima Spa
Top massage center in ajman chandrima  SpaTop massage center in ajman chandrima  Spa
Top massage center in ajman chandrima Spa
Chandrima Spa Ajman
 
nurs fpx 4050 assessment 4 final care coordination plan.pdf
nurs fpx 4050 assessment 4 final care coordination plan.pdfnurs fpx 4050 assessment 4 final care coordination plan.pdf
nurs fpx 4050 assessment 4 final care coordination plan.pdf
Carolyn Harker
 
Sectional dentures for microstomia patients.pptx
Sectional dentures for microstomia patients.pptxSectional dentures for microstomia patients.pptx
Sectional dentures for microstomia patients.pptx
SatvikaPrasad
 
The Importance of Black Women Understanding the Chemicals in Their Personal C...
The Importance of Black Women Understanding the Chemicals in Their Personal C...The Importance of Black Women Understanding the Chemicals in Their Personal C...
The Importance of Black Women Understanding the Chemicals in Their Personal C...
bkling
 
Pediatric Emergency Care for Children | Apollo Hospital
Pediatric Emergency Care for Children | Apollo HospitalPediatric Emergency Care for Children | Apollo Hospital
Pediatric Emergency Care for Children | Apollo Hospital
Apollo 24/7 Adult & Paediatric Emergency Services
 
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
SGRT Community
 
Professional Secrecy: Forensic Medicine Lecture
Professional Secrecy: Forensic Medicine LectureProfessional Secrecy: Forensic Medicine Lecture
Professional Secrecy: Forensic Medicine Lecture
DIVYANSHU740006
 
Bath patient Fundamental of Nursing.pptx
Bath patient Fundamental of Nursing.pptxBath patient Fundamental of Nursing.pptx
Bath patient Fundamental of Nursing.pptx
MianProductions
 
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdfVEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
Vedanta A
 
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COMHUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
priyabhojwani1200
 
Sexual Disorders.gender identity disorderspptx
Sexual Disorders.gender identity  disorderspptxSexual Disorders.gender identity  disorderspptx
Sexual Disorders.gender identity disorderspptx
Pupayumnam1
 
English Drug and Alcohol Commissioners June 2024.pptx
English Drug and Alcohol Commissioners June 2024.pptxEnglish Drug and Alcohol Commissioners June 2024.pptx
English Drug and Alcohol Commissioners June 2024.pptx
MatSouthwell1
 

Recently uploaded (20)

Get Covid Testing at Fit to Fly PCR Test
Get Covid Testing at Fit to Fly PCR TestGet Covid Testing at Fit to Fly PCR Test
Get Covid Testing at Fit to Fly PCR Test
 
R3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
R3 Stem Cell Therapy: A New Hope for Women with Ovarian FailureR3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
R3 Stem Cell Therapy: A New Hope for Women with Ovarian Failure
 
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
Michigan HealthTech Market Map 2024 with Policy Makers, Academic Innovation C...
 
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
COLOUR CODING IN THE PERIOPERATIVE NURSING PRACTICE.
 
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
 
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
DELIRIUM BY DR JAGMOHAN PRAJAPATI.......
 
National Rural Health Mission(NRHM).pptx
National Rural Health Mission(NRHM).pptxNational Rural Health Mission(NRHM).pptx
National Rural Health Mission(NRHM).pptx
 
Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.
 
Top massage center in ajman chandrima Spa
Top massage center in ajman chandrima  SpaTop massage center in ajman chandrima  Spa
Top massage center in ajman chandrima Spa
 
nurs fpx 4050 assessment 4 final care coordination plan.pdf
nurs fpx 4050 assessment 4 final care coordination plan.pdfnurs fpx 4050 assessment 4 final care coordination plan.pdf
nurs fpx 4050 assessment 4 final care coordination plan.pdf
 
Sectional dentures for microstomia patients.pptx
Sectional dentures for microstomia patients.pptxSectional dentures for microstomia patients.pptx
Sectional dentures for microstomia patients.pptx
 
The Importance of Black Women Understanding the Chemicals in Their Personal C...
The Importance of Black Women Understanding the Chemicals in Their Personal C...The Importance of Black Women Understanding the Chemicals in Their Personal C...
The Importance of Black Women Understanding the Chemicals in Their Personal C...
 
Pediatric Emergency Care for Children | Apollo Hospital
Pediatric Emergency Care for Children | Apollo HospitalPediatric Emergency Care for Children | Apollo Hospital
Pediatric Emergency Care for Children | Apollo Hospital
 
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
Surface guided deep inspiration breath hold (SG-DIBH) in ultra-hypofractionat...
 
Professional Secrecy: Forensic Medicine Lecture
Professional Secrecy: Forensic Medicine LectureProfessional Secrecy: Forensic Medicine Lecture
Professional Secrecy: Forensic Medicine Lecture
 
Bath patient Fundamental of Nursing.pptx
Bath patient Fundamental of Nursing.pptxBath patient Fundamental of Nursing.pptx
Bath patient Fundamental of Nursing.pptx
 
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdfVEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
VEDANTA AIR AMBULANCE SERVICES IN REWA AT A COST-EFFECTIVE PRICE.pdf
 
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COMHUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
 
Sexual Disorders.gender identity disorderspptx
Sexual Disorders.gender identity  disorderspptxSexual Disorders.gender identity  disorderspptx
Sexual Disorders.gender identity disorderspptx
 
English Drug and Alcohol Commissioners June 2024.pptx
English Drug and Alcohol Commissioners June 2024.pptxEnglish Drug and Alcohol Commissioners June 2024.pptx
English Drug and Alcohol Commissioners June 2024.pptx
 

Real world modified

  • 1. Clinical Trials are not Enough Stephen Senn (c) Stephen Senn 1
  • 2. Acknowledgements (c) Stephen Senn 2 Acknowledgements This work is partly supported by the European Union’s 7th Framework Programme for research, technological development and demonstration under grant agreement no. 602552. “IDEAL”
  • 3. Basic Thesis • Clinical trials are experiments • At every stage of drug development, even phase III, they are unrepresentative of ‘real world’ clinical practice • The key to using their results to inform ‘real world’ decisions is not to make the trial more representative • The key is to use appropriate scales for analysis and transfer the results into practical real-world decision making • This will require – A different attitude – More modelling – Extensive use of auxiliary real world data (c) Stephen Senn 3
  • 4. An Example • Dan Moerman’s analysis more than 30 years ago of Smith-Kline-French’s development of Tagamet® – Tagamet® (cimetidine) was the best selling drug of its day • 31 trials in 1692 patients – in 17 countries – Duodenal, Gastric, Mixed – 13 significant according to Moerman • Used Chi-square test with Yates’s correction (c) Stephen Senn 4
  • 5. (c) Stephen Senn 5 on the risk- difference scale and there is significant evidence of heterogeneity
  • 6. What the EBM movement used to conclude from this sort of thing • The treatment effect varies according to the type of patient • We want RCTS that automatically deliver the right decision • We can’t rely on the results from RCTs which involve artificially selected patients • We need large simple trials in representative populations • They need to be reported using numbers needed to treat (NNTs) (c) Stephen Senn 6
  • 7. Problems • At best you can hope to recommend treatments that are beneficial on average • But in practice you can never guarantee that trials are representative of practice anyway • You lose the opportunity to study issues more deeply • NNTs are a terrible scale for doing analysis (c) Stephen Senn 7
  • 8. (c) Stephen Senn 8 Here the analysis is on the log-odds ratio scale and heterogeneity is much reduced and not even ‘significant’.
  • 9. 9 “If you need statistics to prove it I don’t believe it” You can’t prove it with statistics but everybody believes it Thanks to Pat Ballew’s blog site (c) Stephen Senn 9 That is to say, we see non-random variation too easily. This example does NOT give strong evidence treatment by trial interaction
  • 10. Lessons • If not carefully, studied random variation can be underestimated • Differences from trial to trial in true effect may be less than one thinks • Finding a good scale is important • BUT The additive scale is not necessarily the relevant one (c) Stephen Senn 10
  • 11. What not to do • The solution is not to attempt to make trials more representative • The solution is to measure appropriately and translate appropriately • This requires the following – Good scales – Good analysis – Good modelling – Good supplementary real world data (c) Stephen Senn 11
  • 12. (c) Stephen Senn 12 Chasing sub- groups leads nowhere
  • 13. Solution? (c) Stephen Senn 14 “a possible resolution is to use the additive measure at the point of analysis and transform to the relevant scale at the point of implementation. This transformation at the point of medical decision-making will require auxiliary information on the level of background risk of the patient.” Senn, Statistics in Medicine, 2004
  • 14. How we already use modelling, data and additive scales • Interspecies scaling • Bioequivalence – log relative bioavailability is additive but difference in absolute bioavailability is not • Dose proportionality • Use of additive scales in phase III – Log hazard – Log-odds ratio (c) Stephen Senn 15
  • 15. (c) Stephen Senn 17Controlled Clinical Trials, 1989
  • 16. Implications of the Lubsen-Tijssen Model • We need to study treatment benefit on disaggregated (of harm) additive scale • We will need real world data on harms • We will need real world data on background risk • We will need models • We will need cooperation between – Medics and statisticians working on clinical trials – Statisticians, epidemiologists, health economists, medics and others working in real world data (c) Stephen Senn 18
  • 17. Example of Atrial Fibrillation • Such patients are at higher risk of stroke • Meta-analysis (reproduced in Hart et al 2007)concluded that warfarin has a beneficial protective effect • But there is a risk of intracranial bleeding • Who should get warfarin? (c) Stephen Senn 19
  • 20. So you have atrial fibrillation • Should you take warfarin? • What else do you need to know? – The difference in risk taking warfarin or not – The rate of side effects – The consequences of side-effects • These cannot be answered (alone) by analysis of RCTs with pre-specified efficacy measures on the additive scale • The RCTs has to be translated and supplemented by real world data (c) Stephen Senn 22
  • 22. (c) Stephen Senn 24 Estimate based on 6 v 3 cases only
  • 23. The reimburser’s perspective • What benefit and harm to the population will accrue from recommending warfarin prophylaxis? • How much will it cost? • How can its use be optimised? • Who should get it? (c) Stephen Senn 25
  • 25. Reimburser’s needs Requirements • A means of separating patients by risk • A means of establishing risk distribution in the population of patients above any threshold chosen • A means of determining expected benefits and costs Solutions • These figures cannot be delivered by clinical trials alone but will require – Cohort studies/case control studies – Health surveys – Economic modelling (c) Stephen Senn 27
  • 26. We need to model background risk • Sort of data set we could use is that provided by the UK Clinical Practice Research Data Link CPRD • Could use this to model – A) Predictors of risk of stroke – B) Distribution of risk levels in the population • Former relevant to individuals to make decisions • Latter is relevant to reimbursers (c) Stephen Senn 28
  • 27. Is there a Trust Problem? • Yes • Clinical trials provide a “template of trust” whereby regulators can mandate sponsors to provide the proof • Modelling + real world data cannot provide these guarantees • But this is no excuse – Whether or not you model, others will – You need to know as much as possible about your own drugs and where and when to use them (c) Stephen Senn 29
  • 28. Finally I leave you with this though (c) Stephen Senn 30 Any damn fool can analyse a clinical trial and frequently does But doing it properly involves skilful analysis, understanding what the results mean requires intelligence insight and experience, and applying the results intelligently needs more of the same plus modelling and real world data And to whom do we look to provide these skills? Statisticians!