Is Open Science Better Science? Ewout W. Steyerberg, PhD Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands Abstract The Open Science movement has many components, including Open Access to scientific publications, sharing of research data, and providing open source software. These components are expected to contribute to better science. In this seminar I aim to reflect on the strength and limitations of Open Science in the context of epidemiological research. First, I note that by making research more open, the scale of research increases; this might enable addressing some research questions better. This allows us to recognize that different researchers use different scientific approaches; Open science makes that we become increasingly aware of different styles in research. Second, we may hope to learn more about the value of modern approaches to data analysis such as machine learning. Indeed, neutral comparison studies benefit from the open availability of multiple data sets that can be analyzed with standardized approaches to the analysis, adding realism compared to analytical and simulation studies. Third, I note that more data sharing is a positive development, especially to highlight heterogeneity between settings. In sum, I remain optimistic that open science will lead to better science, with the caveat that we recognize complexities that limit the interpretation of increasing amounts of data, such as the medical context, study design, measurement and data analysis. These slides were presented in a series of lectures organized by Prof Marianna Huebner, June 2, 2022

1. June, 2022
Is Open Science Better Science?
Ewout W. Steyerberg, PhD
Professor of Clinical Biostatistics and
Medical Decision Making
Thanks to many for assistance and inspiration,
including the GAP3 consortium, CENTER-TBI Study
Yes, but …

2. Open vs closed science
Long ago
- Performed by few, elitarian scientists
- Doing private experiments
- Discussion in small, closed communities

3. Probabilities to quantify uncertainty
• Christiaan Huygens 1657:
'Van rekeningh in spelen van geluck'
• Thomas Bayes 1763:
An Essay towards solving a Problem in the Doctrine of Chances”
(read to the Royal Society by Richard Price)
• Pierre Laplace 1812:
Théorie analytique des probabilités
6-Jun-22
3 Insert > Header & footer

4. Open vs closed science
Long ago
- Performed by few, elitarian scientists
- Doing private experiments
- Discussion in small, closed communities
Recent
- Science as a profession
- Protect data + code as intellectual property
- Aim for shocking findings in high IF journals
https://www.sciencemag.org/news/2020/06/whos-blame-these-three-scientists-are-heart-surgisphere-covid-19-scandal

5. Overall claim
“Open Science will make research better”
Vote pro / neutral / con
“More data is better”
Vote pro / neutral / con
6-Jun-22
5 Insert > Header & footer

6. Today
Aims:
- Highlight some strong points in Open Science
- Hint at some challenges in Open Science
Reflections based on personal 30-yr research experience,
specific focus on prediction research / decision making
6-Jun-22
6 Insert > Header & footer

7. Open Science to better address
Big Research questions

8. Open science research questions: case 1
Example 1: Red cards and dark skin soccer players
https://psyarxiv.com/qkwst/
6-Jun-22
8 Insert > Header & footer

9. Open science research questions: case 1
• 29 teams involving 61 analysts; same dataset; same research question:
whether soccer referees are more likely to give red cards to dark skin
toned players than light skin toned players
• Estimated odds ratios 0.89 –2.93 (median 1.3)
• 20 teams: statistically significant positive effect, 9: non-significant relation
6-Jun-22
9 Insert > Header & footer

10. Estimated odds ratios by 29 research teams
6-Jun-22
10 Insert > Header & footer

11. “Logistic regression”
6-Jun-22
11 Insert > Header & footer

12. Open science research questions: case 1
• 29 teams involving 61 analysts; same dataset; same research question:
whether soccer referees are more likely to give red cards to dark skin toned
players than light skin toned players
• Estimated odds ratios 0.89 –2.93 (median 1.3).
• 20 teams: statistically significant positive effect, 9: non-significant relation.
• 21 unique combinations of covariates
• “Variation in analysis of complex data may be difficult to
avoid, even by experts with honest intentions”
6-Jun-22
12 Insert > Header & footer

13. Open science research questions: case 2
6-Jun-22
13 Insert > Header & footer
Example from Maarten van Smeden
@MaartenvSmeden

14. Predicting mortality – the media

15. Findings not convincing
Cox, #4, 30 vars, max c =0.793
RF, #7, 600 vars, c=0.797
Elastic, #9, 600 vars, c=0.801
6-Jun-22
15 Insert > Header & footer

16. Machine learning vs conventional modeling
1. Findings convincing?
“We found that random forests did not outperform Cox models despite their
inherent ability to accommodate nonlinearities and interactions. …
Elastic nets achieved the highest discrimination performance …, demonstrating
the ability of regularisation to select relevant variables and optimise model
coefficients in an EHR context.”
6-Jun-22
16 Insert > Header & footer

17. Machine learning vs conventional modeling
1. Findings convincing? Not in case-study
2. Systematic / ”it depends” ?
6-Jun-22
17 Insert > Header & footer

18. 6-Jun-22
18 Insert > Header & footer

19. 6-Jun-22
19 Insert > Header & footer

20. Open science research questions: case 2
• 243 real datasets from “the OpenML database”
• RF performed better than LR:
mean difference between RF and LR was 0.041 (95%-CI =[0.031,0.053]) for
the Area Under the ROC Curve
• Results were dependent on the inclusion criteria used to select the example
datasets
• ES: Results rely on 10 x 10-fold cross-validation
6-Jun-22
20 Insert > Header & footer

21. Open science research questions: case 2
• More clarification needed when ML / RF works best; at least large N needed
6-Jun-22
21 Insert > Header & footer

22. Systematic review on ML vs classic modeling
6-Jun-22
22 Insert > Header & footer

23. Differences in discrimination

24. Thanks to Maarten van Smeden

25. Summary on examples of Open Science
to better address Big research questions
• 1 data set
• multiple modelers
• Multiple modeling options
• 1 neutral comparison; 243 OpenML databases
• Review of 282 comparative studies: meta-research
6-Jun-22
25 Insert > Header & footer

26. Open Science: data sharing
 Collaboration vs giving

27. 6-Jun-22
27 Insert > Header & footer

28. Heterogeneity in data .. ignored
6-Jun-22
28 Insert > Header & footer

29. Data sharing
• Pro:
• Allowed for larger sample size in a rare disease
• Cons:
• Heterogeneity?
• Substantial politics / efforts
6-Jun-22
29 Insert > Header & footer

30. Open Science: analyses and interpretation

31. Analyses: ODHSI model
6-Jun-22
31 Insert > Header & footer

32. OHDSI: COVID and other research topics
6-Jun-22
32 Insert > Header & footer

33. The power of OHDSI
6-Jun-22
33 Insert > Header & footer

34. OMOP common data model enables sharing of
model development code
6-Jun-22
34 Insert > Header & footer

35. Performance for different outcomes in multiple cohorts
6-Jun-22
35 Insert > Header & footer

36. OHDSI: bridging data sharing - analyses
• Keep data local
• Run locally started, centrally available analyses
• Share results centrally

37. Open Science: analyses and interpretation

38. Open Science challenge:
dealing with heterogeneity for prediction research
Heterogeneity
• Study design
• Selection of subjects
• Measurement of covariates
• Measurement of outcomes
• Associations of covariates with outcome
• Overall outcome rates
• Performance of prediction models

39. Analyses: dealing with heterogeneity
6-Jun-22
39 Insert > Header & footer

40. 15 cohorts: 11 RCTs, 4 Observational studies
6-Jun-22
40 Insert > Header & footer

41. Heterogeneous case-mix
6-Jun-22
41 Insert > Header & footer

42. Heterogeneous predictor effects
6-Jun-22
42 Insert > Header & footer

43. Heterogeneous predictions
6-Jun-22
43 Insert > Header & footer

44. Heterogeneity  uncertainty in individual predictions
given that a prespecified logistic model is fitted
6-Jun-22
44 Insert > Header & footer

45. “Open Science is Better Science”
1. Research questions in competitions
• Red cards
• Neutral comparisons / meta-research
2. Data sharing
• Collaborative efforts most successful
3. Analyses
• OHDSI: modern, keep data local
• Heterogeneity
6-Jun-22
45 Insert > Header & footer