A short introduction to some problems in science, with a focus on reproducibility in medical research, including some examples from COVID research, and some hints for improvement in the research process

1. July 12, 2022
Reproducibility in medical research
Ewout W. Steyerberg, PhD
Professor of Clinical Biostatistics and
Medical Decision Making
Dept of Biomedical Data Sciences
Leiden University Medical Center
Thanks to many for assistance and inspiration

2. Disclaimer
12-Jul-22
2 Insert > Header & footer
As a biostatistician, I like numbers

3. Reproducibility and p-values
12-Jul-22
3 Insert > Header & footer

4. Reproducibility and p-values
12-Jul-22
4 Insert > Header & footer

5. Reproducibility crisis
• Fabio tekst:
Science journalists can help citizens to notice not only [..]
the scandals involving lack of integrity and bad methodology,
but also the [..] initiatives that are trying to fix the current system of
incentives/disincentives that does not value reproducibility studies and
instead often leaves room for rushed, hyped, unreliable research.
1. Rushed / hyped
2. Unreliable
3. Value reproducibility

6. 1. Rushed &hyped research: COVID
12-Jul-22
6 Insert > Header & footer

7. 12-Jul-22
7 Insert > Header & footer

8. Hyped research: COVID risk prediction
12-Jul-22
8 Insert > Header & footer

9. Conclusion 1
Rushed & hyped research happens
• Medical context: urgency
• Reward context: newness
12-Jul-22
9 Insert > Header & footer

10. 2. Unreliable research
“Research reliability is the degree to which research methods produces stable
and consistent results”
Psychology:
test-retest / parellel forms / inter-rater / internal consistency reliability
Epidemiology / biostatistics:
Bias + variance
12-Jul-22
10 Insert > Header & footer

11. Bias & variance
Reduce variance
• Increase sample size
• Better measurement
 Beware of ““Absence of evidence is not evidence of absence”
Reduce bias
• Strong study design costly
• Big Data everywhere
 Beware of overinterpretation
12-Jul-22
11 Insert > Header & footer

12. Conclusion 2
Unreliable research happens
• Ethics: “not too many patients”
• Resources: limited  “efficient research”
12-Jul-22
12 Insert > Header & footer

13. 12-Jul-22
13 Insert > Header & footer
Prediction models
Development – validation – impact assessment / implementation
3. More value on reproducibility

14. Incentives in medical research
12-Jul-22
14 Insert > Header & footer

15. Some key principles
• Contribute to societal needs
• Publish completely and transparently, regardless of the results
• Facilitate dissemination and use of research data and results by others
12-Jul-22
15 Insert > Header & footer

16. Conclusion 3
Incentives need to change
• Classic: publish many papers in high IF journals
• Modern: registered reports; share data; share reproducible results;
collaborative research; peer review;
12-Jul-22
16 Insert > Header & footer

17. 12-Jul-22
17 Insert > Header & footer