Improving predictions: Lasso, Ridge and Stein's paradoxMaarten van Smeden
Slides of masterclass "Improving predictions: Lasso, Ridge and Stein's paradox" at the (Dutch) National Institute for Public Health and the Environment (RIVM)
Improving epidemiological research: avoiding the statistical paradoxes and fa...Maarten van Smeden
Keynote at Norwegian Epidemiological Association conference, October 26 2022. Discussing absence of evidence fallacy, Table 2 fallacy, Winner's curse and Stein's paradox.
Clinical trials are about comparability not generalisability V2.pptxStephenSenn2
Lecture delivered at the September 2022 EFSPI meeting in Basle in which I argued that the patients in a clinical trial should not be viewed as being a representative sample of some target population.
How to combine results from randomised clinical trials on the additive scale with real world data to provide predictions on the clinically relevant scale for individual patients
Improving predictions: Lasso, Ridge and Stein's paradoxMaarten van Smeden
Slides of masterclass "Improving predictions: Lasso, Ridge and Stein's paradox" at the (Dutch) National Institute for Public Health and the Environment (RIVM)
Improving epidemiological research: avoiding the statistical paradoxes and fa...Maarten van Smeden
Keynote at Norwegian Epidemiological Association conference, October 26 2022. Discussing absence of evidence fallacy, Table 2 fallacy, Winner's curse and Stein's paradox.
Clinical trials are about comparability not generalisability V2.pptxStephenSenn2
Lecture delivered at the September 2022 EFSPI meeting in Basle in which I argued that the patients in a clinical trial should not be viewed as being a representative sample of some target population.
How to combine results from randomised clinical trials on the additive scale with real world data to provide predictions on the clinically relevant scale for individual patients
Presentation on similarities and differences between statistical and machine learning research fields for the @UM_MiCHAMP Big Data & AI in Health Seminar Series; October 21, 2022
Development and evaluation of prediction models: pitfalls and solutionsMaarten van Smeden
Slides for the statistics in practice session for the Biometrisches Kolloqium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 18 March 2021
Developing and validating statistical models for clinical prediction and prog...Evangelos Kritsotakis
Talk on clinical prediction models presented at the Joint Seminar Series in Translational and Clinical Medicine organised by the University of Crete Medical School, the Institute of Molecular Biology and Biotechnology of the Foundation for Research and Technology Hellas (IMBB-FORTH), and the University of Crete Research Center (UCRC), Heraklion [online], Greece, April 7, 2021.
Whatever happened to design based inferenceStephenSenn2
Given as the Sprott lecture, University of Waterloo September 2022
Abstract
What exactly should we think about appropriate analyses for designed experiments and why? If conditional inference trumps marginal inference, why should we care about randomisation? Isn’t everything just modelling? The Rothamsted School held that design matters. Taking an example of applying John Nelder’s general balance approach to a notorious problem, Lord’s paradox, I shall show that there may be some lessons for two fashionable topics: causal analysis and big data. I shall conclude that if we want not only to make good estimates but estimate how good our estimates are, design does matter.
Unfortunately, some have interpreted Numbers Needed to Treat as indicating the proportion of patients on whom the treatment has had a causal effect. This interpretation is very rarely, if ever, necessarily correct. It is certainly inappropriate if based on a responder dichotomy. I shall illustrate the problem using simple causal models.
One also sometimes encounters the claim that the extent to which two distributions of outcomes overlap from a clinical trial indicates how many patients benefit. This is also false and can be traced to a similar causal confusion.
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Ewout Steyerberg
Title"Clinical prediction models in the age of artificial intelligence and big data", presented at the Basel Biometrics Society seminar Nov 1, 2019, Basel, by Ewout Steyerberg, with substantial inout from Maarten van Smeden and Ben van Calster
The history of p-values is covered to try and shed light on a mystery: why did Student and Fisher agree numerically but disagree in terms of interpretation.?
Clinical trials are about comparability not generalisability V2.pptxStephenSenn3
It is a fundamental but common mistake to regard clinical trials as being a form of representative inference. The key issue is comparability. Experiments do not involve typical material. In clinical trials; it is concurrent control that is key and randomisation is a device for calculating standard errors appropriately that should reflect the design.
Generalisation beyond the clinical trial always involves theory.
The Seven Habits of Highly Effective StatisticiansStephen Senn
If you know why the title of this talk is extremely stupid, then you clearly know something about control, data and reasoning: in short, you have most of what it takes to be a statistician. If you have studied statistics then you will also know that a large amount of anything, and this includes successful careers, is luck.
In this talk I shall try share some of my experiences of being a statistician in the hope that it will help you make the most of whatever luck life throws you, In so doing, I shall try my best to overcome the distorting influence of that easiest of sciences hindsight. Without giving too much away, I shall be recommending that you read, listen, think, calculate, understand, communicate, and do. I shall give you some example of what I think works and what I think doesn’t
In all of this you should never forget the power of negativity and also the joy of being able to wake up every day and say to yourself ‘I love the small of data in the morning’.
Presentation on similarities and differences between statistical and machine learning research fields for the @UM_MiCHAMP Big Data & AI in Health Seminar Series; October 21, 2022
Development and evaluation of prediction models: pitfalls and solutionsMaarten van Smeden
Slides for the statistics in practice session for the Biometrisches Kolloqium (organized by the Deutsche Region der Internationalen Biometrischen Gesellschaft), 18 March 2021
Developing and validating statistical models for clinical prediction and prog...Evangelos Kritsotakis
Talk on clinical prediction models presented at the Joint Seminar Series in Translational and Clinical Medicine organised by the University of Crete Medical School, the Institute of Molecular Biology and Biotechnology of the Foundation for Research and Technology Hellas (IMBB-FORTH), and the University of Crete Research Center (UCRC), Heraklion [online], Greece, April 7, 2021.
Whatever happened to design based inferenceStephenSenn2
Given as the Sprott lecture, University of Waterloo September 2022
Abstract
What exactly should we think about appropriate analyses for designed experiments and why? If conditional inference trumps marginal inference, why should we care about randomisation? Isn’t everything just modelling? The Rothamsted School held that design matters. Taking an example of applying John Nelder’s general balance approach to a notorious problem, Lord’s paradox, I shall show that there may be some lessons for two fashionable topics: causal analysis and big data. I shall conclude that if we want not only to make good estimates but estimate how good our estimates are, design does matter.
Unfortunately, some have interpreted Numbers Needed to Treat as indicating the proportion of patients on whom the treatment has had a causal effect. This interpretation is very rarely, if ever, necessarily correct. It is certainly inappropriate if based on a responder dichotomy. I shall illustrate the problem using simple causal models.
One also sometimes encounters the claim that the extent to which two distributions of outcomes overlap from a clinical trial indicates how many patients benefit. This is also false and can be traced to a similar causal confusion.
Prediction, Big Data, and AI: Steyerberg, Basel Nov 1, 2019Ewout Steyerberg
Title"Clinical prediction models in the age of artificial intelligence and big data", presented at the Basel Biometrics Society seminar Nov 1, 2019, Basel, by Ewout Steyerberg, with substantial inout from Maarten van Smeden and Ben van Calster
The history of p-values is covered to try and shed light on a mystery: why did Student and Fisher agree numerically but disagree in terms of interpretation.?
Clinical trials are about comparability not generalisability V2.pptxStephenSenn3
It is a fundamental but common mistake to regard clinical trials as being a form of representative inference. The key issue is comparability. Experiments do not involve typical material. In clinical trials; it is concurrent control that is key and randomisation is a device for calculating standard errors appropriately that should reflect the design.
Generalisation beyond the clinical trial always involves theory.
The Seven Habits of Highly Effective StatisticiansStephen Senn
If you know why the title of this talk is extremely stupid, then you clearly know something about control, data and reasoning: in short, you have most of what it takes to be a statistician. If you have studied statistics then you will also know that a large amount of anything, and this includes successful careers, is luck.
In this talk I shall try share some of my experiences of being a statistician in the hope that it will help you make the most of whatever luck life throws you, In so doing, I shall try my best to overcome the distorting influence of that easiest of sciences hindsight. Without giving too much away, I shall be recommending that you read, listen, think, calculate, understand, communicate, and do. I shall give you some example of what I think works and what I think doesn’t
In all of this you should never forget the power of negativity and also the joy of being able to wake up every day and say to yourself ‘I love the small of data in the morning’.
Statistics for UX Professionals - Jessica CameronUser Vision
Are you looking to expand your research toolkit to include some quantitative methods, such as survey research or A/B testing? Have you been asked to collect some usability metrics, but aren’t sure how best to go about that? Or do you just want to be more aware of all of the UX research possibilities? If your answer to any of those questions is yes, then this session is for you.
You may know that without statistics, you won’t know if A is really better than B, if users are truly more satisfied with your new site than with your old one, or which changes to your site have actually impacted conversion rates. However, statistics can also help you figure out how to report satisfaction and other metrics you collect during usability tests. And they’re essential for making sense of the results of quantitative usability tests.
This session will focus on the statistical concepts that are most useful for UX researchers. It won’t make you a quant, but it will give you a good grounding in quantitative methods and reporting. (For example, you will learn what a margin of error is, how to report quantitative data collected during a usability test - and how not to - and how many people you really need to fill out a survey.)
The absence of a gold standard: a measurement error problemMaarten van Smeden
Talk about gold standard problems and solutions in medicine and epidemiology. Invited by the department of infectious disease epidemiology, University Medical Center Utrecht
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...University of Maribor
Slides from:
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artificial Intelligence
https://www.etran.rs/2024/en/home-english/
Earliest Galaxies in the JADES Origins Field: Luminosity Function and Cosmic ...Sérgio Sacani
We characterize the earliest galaxy population in the JADES Origins Field (JOF), the deepest
imaging field observed with JWST. We make use of the ancillary Hubble optical images (5 filters
spanning 0.4−0.9µm) and novel JWST images with 14 filters spanning 0.8−5µm, including 7 mediumband filters, and reaching total exposure times of up to 46 hours per filter. We combine all our data
at > 2.3µm to construct an ultradeep image, reaching as deep as ≈ 31.4 AB mag in the stack and
30.3-31.0 AB mag (5σ, r = 0.1” circular aperture) in individual filters. We measure photometric
redshifts and use robust selection criteria to identify a sample of eight galaxy candidates at redshifts
z = 11.5 − 15. These objects show compact half-light radii of R1/2 ∼ 50 − 200pc, stellar masses of
M⋆ ∼ 107−108M⊙, and star-formation rates of SFR ∼ 0.1−1 M⊙ yr−1
. Our search finds no candidates
at 15 < z < 20, placing upper limits at these redshifts. We develop a forward modeling approach to
infer the properties of the evolving luminosity function without binning in redshift or luminosity that
marginalizes over the photometric redshift uncertainty of our candidate galaxies and incorporates the
impact of non-detections. We find a z = 12 luminosity function in good agreement with prior results,
and that the luminosity function normalization and UV luminosity density decline by a factor of ∼ 2.5
from z = 12 to z = 14. We discuss the possible implications of our results in the context of theoretical
models for evolution of the dark matter halo mass function.
Seminar of U.V. Spectroscopy by SAMIR PANDASAMIR PANDA
Spectroscopy is a branch of science dealing the study of interaction of electromagnetic radiation with matter.
Ultraviolet-visible spectroscopy refers to absorption spectroscopy or reflect spectroscopy in the UV-VIS spectral region.
Ultraviolet-visible spectroscopy is an analytical method that can measure the amount of light received by the analyte.
Richard's entangled aventures in wonderlandRichard Gill
Since the loophole-free Bell experiments of 2020 and the Nobel prizes in physics of 2022, critics of Bell's work have retreated to the fortress of super-determinism. Now, super-determinism is a derogatory word - it just means "determinism". Palmer, Hance and Hossenfelder argue that quantum mechanics and determinism are not incompatible, using a sophisticated mathematical construction based on a subtle thinning of allowed states and measurements in quantum mechanics, such that what is left appears to make Bell's argument fail, without altering the empirical predictions of quantum mechanics. I think however that it is a smoke screen, and the slogan "lost in math" comes to my mind. I will discuss some other recent disproofs of Bell's theorem using the language of causality based on causal graphs. Causal thinking is also central to law and justice. I will mention surprising connections to my work on serial killer nurse cases, in particular the Dutch case of Lucia de Berk and the current UK case of Lucy Letby.
Professional air quality monitoring systems provide immediate, on-site data for analysis, compliance, and decision-making.
Monitor common gases, weather parameters, particulates.
Slide 1: Title Slide
Extrachromosomal Inheritance
Slide 2: Introduction to Extrachromosomal Inheritance
Definition: Extrachromosomal inheritance refers to the transmission of genetic material that is not found within the nucleus.
Key Components: Involves genes located in mitochondria, chloroplasts, and plasmids.
Slide 3: Mitochondrial Inheritance
Mitochondria: Organelles responsible for energy production.
Mitochondrial DNA (mtDNA): Circular DNA molecule found in mitochondria.
Inheritance Pattern: Maternally inherited, meaning it is passed from mothers to all their offspring.
Diseases: Examples include Leber’s hereditary optic neuropathy (LHON) and mitochondrial myopathy.
Slide 4: Chloroplast Inheritance
Chloroplasts: Organelles responsible for photosynthesis in plants.
Chloroplast DNA (cpDNA): Circular DNA molecule found in chloroplasts.
Inheritance Pattern: Often maternally inherited in most plants, but can vary in some species.
Examples: Variegation in plants, where leaf color patterns are determined by chloroplast DNA.
Slide 5: Plasmid Inheritance
Plasmids: Small, circular DNA molecules found in bacteria and some eukaryotes.
Features: Can carry antibiotic resistance genes and can be transferred between cells through processes like conjugation.
Significance: Important in biotechnology for gene cloning and genetic engineering.
Slide 6: Mechanisms of Extrachromosomal Inheritance
Non-Mendelian Patterns: Do not follow Mendel’s laws of inheritance.
Cytoplasmic Segregation: During cell division, organelles like mitochondria and chloroplasts are randomly distributed to daughter cells.
Heteroplasmy: Presence of more than one type of organellar genome within a cell, leading to variation in expression.
Slide 7: Examples of Extrachromosomal Inheritance
Four O’clock Plant (Mirabilis jalapa): Shows variegated leaves due to different cpDNA in leaf cells.
Petite Mutants in Yeast: Result from mutations in mitochondrial DNA affecting respiration.
Slide 8: Importance of Extrachromosomal Inheritance
Evolution: Provides insight into the evolution of eukaryotic cells.
Medicine: Understanding mitochondrial inheritance helps in diagnosing and treating mitochondrial diseases.
Agriculture: Chloroplast inheritance can be used in plant breeding and genetic modification.
Slide 9: Recent Research and Advances
Gene Editing: Techniques like CRISPR-Cas9 are being used to edit mitochondrial and chloroplast DNA.
Therapies: Development of mitochondrial replacement therapy (MRT) for preventing mitochondrial diseases.
Slide 10: Conclusion
Summary: Extrachromosomal inheritance involves the transmission of genetic material outside the nucleus and plays a crucial role in genetics, medicine, and biotechnology.
Future Directions: Continued research and technological advancements hold promise for new treatments and applications.
Slide 11: Questions and Discussion
Invite Audience: Open the floor for any questions or further discussion on the topic.
What is greenhouse gasses and how many gasses are there to affect the Earth.moosaasad1975
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
Mammalian Pineal Body Structure and Also Functions
Why the EPV≥10 sample size rule is rubbish and what to use instead
1. Maarten van Smeden, PhD
2 november 2020
Why the EPV≥10 sample size rule is rubbish
and what to use instead
2. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
• Statistician at Julius Center for Health Sciences and Primary Care
• Main interests (but not limited to):
• prognostic and diagnostic modeling
• measurement error
• missing data
Today’s topic:
EPV≥10 sample size rule (aka 1 in 10 rule) has be one of the leading
sample size rules in prognostic/diagnostic prediction modeling
5. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Outline
• The EPV≥10 rule-of-thumb: where does it come from?
• Evidence the EPV≥10 rule has no rationale
• Evidence that sample size is important (even if you use the fancier methods)
• Actual sample size calculations for prediction models
6. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Ever wondered if AD/BC gives the “best” estimate of the odds ratio?
What if I told you that AD/BC is biased?
7. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Let’s say we have fitted a logistic regression model to a dataset, and obtain
ln
𝑝𝑝𝑖𝑖
1 − 𝑝𝑝𝑖𝑖
= 𝛼𝛼� + 𝛽𝛽̂1 𝑋𝑋1𝑖𝑖 + 𝛽𝛽̂2 𝑋𝑋2𝑖𝑖 + ⋯ + 𝛽𝛽̂𝑘𝑘 𝑋𝑋𝑘𝑘𝑖𝑖
I’m very sorry, but 𝛽𝛽̂1 is a biased estimator, and 𝛽𝛽̂2 too, ….
…. actually they are all finite sample biased
8. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Epidemiology text-books:
• Confounding bias
• Information bias
• Selection bias
… nothing about finite sample bias
9. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Important: bias vs consistency
• Consistency ≈ as sample size increases, estimate converges to truth
• Bias ≈ with repeated samples, the average estimate converges to truth
10. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Log(odds) is consistent but finite sample biased
17. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Illustration by simulation
• Simulate 4 normal covariates with equal multivariable log-odds-ratios of 2
• 1,000 simulation samples of N = 50
• Consistency: create 1,000 meta-dataset of increasing size: meta-dataset
r consists of each created dataset up to r;
• Bias: calculate difference estimate of exposure effect and true value for
each of the created datasets up to r;
20. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Average of 400 studies
with N = 50
1 study with N = 20,000
21. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
With decreasing sample size
How we usually think
22. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
With decreasing sample size
But actually with odds ratios
(and other ratios)
23. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
The origin of the 1 in 10 rule
“For EPV values of 10 or greater, no major problems occurred. For EPV
values less than 10, however, the regression coefficients were biased in
both positive and negative directions”
25. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
More simulation studies
Citations based on Google Scholar, Oct 30 2020
citations: 5,736
“a minimum of 10 EPV […] may be too conservative”
“substantial problems even if the number of EPV exceeds 10”
For EPV values of 10 or greater, no major problems
citations: 2,438
citations: 216
26. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
More simulation studies
Citations based on Google Scholar, Oct 30 2020
citations: 5,736
“a minimum of 10 EPV […] may be too conservative”
“substantial problems even if the number of EPV exceeds 10”
For EPV values of 10 or greater, no major problems
citations: 2,438
citations: 216
!?!
27. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
• Examine the reasons for substantial differences between the earlier EPV
simulation studies
• Evaluate a possible solution to reduce the finite sample bias
28. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
• Examine the reasons for substantial differences between the earlier EPV
simulation studies (simulation technicality: handling of “separation”)
• Evaluate a possible solution to reduce the finite sample bias
29. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
• Examine the reasons for substantial differences between the earlier EPV
simulation studies (simulation technicality: handling of “separation”)
• Evaluate a possible solution to reduce the finite sample bias
30. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
• Firth’s ”correction” aims to reduce finite sample bias in maximum
likelihood estimates, applicable to logistic regression
• It makes clever use of the “Jeffries prior” (from Bayesian literature) to
penalize the log-likelihood, which shrinks the estimated coefficients
• It has a nice theoretical justifications, but does it work well?
31. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Standard
Averaged over 465 simulation conditions with 10,000 replications each
32. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
StandardFirth’scorrection
Averaged over 465 simulation conditions with 10,000 replications each
33. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Firth’s correction
Difficult? No
Example R code:
> require(“logistf”)
> logistf(Y~X1+X2+X3+X4, firth=T, data=df)
Compared to default (maxlik) logistic regression, Firth’s correction generally:
• Narrower confidence intervals
• Lower MSE
• Better predictions*
*requires adjustment of the intercept using flic=TRUE option in logistf
34. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Sample issue size solved?
… not quite!
• Precision of regression coefficients
• Variable selection and functional form
• Ensure predictions are adequate
35. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Sample issue size solved?
… not quite!
• Precision of regression coefficients
• Variable selection and functional form
• Ensure predictions are adequate
• Why would a one-solution fits all rule-of-thumb be appropriate?
• Think of sample size for a randomized clinical trial
Would be odd to suggest all trials should have 100 patients in each arm?
36. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
TRIPOD Item 8. Explain how the study size was arrived at
Moons et al. Ann Intern Med 2015 (TRIPOD Explanation & Elaboration)
“Although there is a consensus on the importance
of having an adequate sample size for model
development, how to determine what counts as
‘adequate’ is not clear …”
37. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Why is sample size important?
• We want to have a large enough sample size to develop a model that
provides accurate risk predictions in new individuals from target
population
• Many (most?) models do not perform well when checked in new data
• small sample sizes
• overfitting
• lack of (internal) validation
• …
38. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Recent example
• Reviewed 232 prediction models
• “All models were rated at high or
unclear risk of bias”
• Sample size: median 338; IQR 134 to 707
• Number of events: median 69; IQR 37 to 160
Living review, doi: 10.1136/bmj.m1328 (these numbers from a soon to appear review update)
39. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Recent example
• External validation 22 COVID-19 related
prognostic models
• Performance: poor to very poor
• “Admission oxygen saturation on room air and patient age are strong
predictors of deterioration and mortality among hospitalised adults with
COVID-19, respectively. None of the prognostic models evaluated here
offered incremental value for patient stratification to these univariable
predictors.”
40. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Small sample size and overfitting
• Spurious predictor-outcome associations
• Important predictors can be missed
• Unimportant predictors can be selected
• Regression coefficients too large and uncertain
• Model doesn’t predict well in new data
• Disappointing discrimination
• Often calibration slope < 1
https://twitter.com/LesGuessing/status/997146590442799105
41. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
With small N: calibration slope often < 1
Predictions too extreme
42. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
“Modern” methods aim to circumvent overfitting
• Penalised regression: e.g. lasso, ridge regression, elastic net
• Standard regression followed by uniform (global) shrinkage
• Target calibrated predicted risks in new data: shrinkage and penalty
terms estimated using bootstrapping or cross-validation
• Sample size problem solved?
43. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
“shrinkage works on the average but may fail in the particular unique
problem on which the statistician is working.”
• Required shrinkage is hard to estimate
• Often large uncertainty correct value to use, especially in small datasets (!)
44. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
“We conclude that, despite improved performance on average, shrinkage often
worked poorly in individual datasets, in particular when it was most needed.
The results imply that shrinkage methods do not solve problems associated
with small sample size or low number of events per variable.”
47. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Our proposal
• Calculate sample size that is needed to
• minimise potential overfitting
• estimate probability (risk) precisely
• Sample size formula’s for
• Continuous outcomes
• Time-to-event outcomes
• Binary outcomes (focus today)
48. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Example
• COVID-19 prognosis hospitalized
patients
• Composite outcome: “deterioration”
(in-hospital death, ventilator support,
ICU)
A priori expectations
• Event fraction at least 30%
• 40 candidate predictor parameters
• C-statistic of 0.71(conservative est)
-> Cox-Snell R2 of 0.24
MedRxiv Preprint (not peer reviewed): 10.1101/2020.10.09.20209957
49. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Restricted cubic splines
with 4 knots: 3 degrees of
freedom
Note: EPV rule also
calculates degrees of
freedom of candidate
predictors, not variables!
50. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Calculate required sample size
Criterion 1. Shrinkage: expected heuristic shrinkage factor, S ≥ 0.9
(calibration slope, target < 10% overfitting)
Criterion 2. Optimism: Cox-Snell R2 apparent - Cox-Snell R2 validation < 0.05
(overfitting)
Criterion 3: A small margin of error in overall risk estimate < 0.05 absolute error
(precision estimated baseline risk)
(Criterion 4: a small margin of absolute error in the estimated risks)
51. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Calculation
R code:
> require(pmsampsize)
> pmsampsize(type="b",rsquared=0.24,parameters=40,prevalence=0.3)
52. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
A few alternative scenarios
• rsquared=0.24,parameters=40,prevalence=0.3 -> EPV≥9.7
• rsquared=0.12,parameters=40,prevalence=0.3 -> EPV≥21.0
• rsquared=0.12,parameters=40,prevalence=0.5 -> EPV≥35.0
• rsquared=0.36,parameters=40,prevalence=0.2 -> EPV≥5
53. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
The sample size that meets all criteria is the MINIMUM required
• Why minimum? Other criteria may be important
e.g. missing data, clustering, variable selection
• May raise required sample size further
• Simulation based approaches
Preprint (not peer reviewed) doi: 10.21203/rs.3.rs-87100/v1
54. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Summary
• Default logistic regression produces finite sample biased estimates
• Finite sample bias can be substantial; easily solved using Firth’s correction
• “Modern” approaches (e.g. Firth, Lasso, Ridge) no compensation for low N
• New sample size criteria to replace the one-size-fits-all EPV≥10 rule
55. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
https://www.prognosisresearch.com/
New website by Richard Riley and Kym Snell
56. M.vanSmeden@umcutrecht.nl | Twitter: @MaartenvSmedenWhy the EPV ≥ 10 rule is rubbish and what to use instead
Work in collaboration with:
• Carl Moons
• Hans Reitsma
• Richard Riley (Keele, materials for this presentation)
• Gary Collins (Oxford)
• Ben Van Calster (Leuven)
• Ewout Steyerberg (Leiden)
• Rishi Gupta (UCL)
• Many others
Contact: M.vanSmeden@umcutrecht.nl