How 'publish or perish' has failed us and how we can improve the current publication process While it is crucial that machine learning models are reproducible, the issue is not sufficiently addressed in current research. Models can be irreproducible due to a variety of non-intentional and intentional reasons ranging from time pressure to deliberate fraud. To assure validity of published models, some venues demand public availability of source code or trained models. However, these approaches have shown to be insufficient, for example, because there are often no adequate quality checks on source code and functionality during the publication process. We discuss the need as well as the challenges of open source research on machine learning. We address frequently occurring problems and offer (also technical) solutions. These solutions are integrated in a novel open source framework. The applicability of this framework is discussed in the context of data generation and anomaly detection in electrocardiographic data and its specific requirements concerning data samples and model validity. Finally, we present how such frameworks enable more reliable progress in research.

1. Heading towards reproducible machine
learning research for medical data
How ‘publish or perish’ has failed us and how we can
improve the current publication process
Fiete Lüer
11. November 2022
eMundo GmbH

2. e.mundo © 2022
TABLE OF CONTENTS
1. WHAT’S WRONG?
2. FOSTERING REPRODUCIBILITY
3. CASE STUDY: ECGAN FRAMEWORK
4. THE WAY FORWARD
2

3. e.mundo © 2022
What’s wrong?
Why ‘Publish or Perish’ has failed us
● ‘Publish or Perish’ in (academic) research:
● Funding, personal career and fame significantly depend on publication metrics
3

4. e.mundo © 2022
What’s wrong?
Why ‘Publish or Perish’ has failed us
● ‘Publish or Perish’ in (academic) research:
● Funding, personal career and fame significantly depend on publication metrics
● This shapes the working environment and research practices:
● Time pressure: curation of data and code takes time
● Fraud: p-hacking, synthetic data with desirable attributes
● Speed over stability: studies for reproduction (on different datasets) difficult to publish
● Publication “overload”: difficult to keep up-to-date
● Reviewer “overload”: reviews get worse, hinders good research which gets rejected
● Obscurity: thin line between plagiarism and novelty
→ Most of this is a grey area
4

5. e.mundo © 2022
What’s wrong?
Why current open science practices are not sufficient
● Repeated runs of an algorithm on the same dataset should lead to same or similar results
5

6. e.mundo © 2022
What’s wrong?
Why current open science practices are not sufficient
● Repeated runs of an algorithm on the same dataset should lead to same or similar results
● Reproduction is necessary for validity
● “Replication crisis” common in sciences, e.g. psychology
6

7. e.mundo © 2022
What’s wrong?
Why current open science practices are not sufficient
● Repeated runs of an algorithm on the same dataset should lead to same or similar results
● Reproduction is necessary for validity
● “Replication crisis” common in sciences, e.g. psychology
● Some venues demand public availability of source code or trained models to assure reproducibility and validity
● While an important step, this is insufficient for various reasons:
■ Not prevalent (proprietary code, models and/or data)
■ No adequate quality checks on source code and functionality during the publication process
■ Many practices are non-deterministic if used out-of-the-box → difficult to reproduce, cherry-picked
■ Training models depend on various “hyperparameters”, i.e. parameters relevant to the learning
process itself which are often incomplete in the paper itself and not necessarily correct in the code
■ Medicine: Difficult to obtain data, usually small and often old datasets or internal data
7

8. e.mundo © 2022
Fostering Reproducibility
8
Randomization Parameters
(Model initialization, Noise,
Augmentation, data
split/shuffling, dropout,...)
GPU/CPU
(non-)deterministic
algorithms
Source Code
Data
Environment
Stratified cross-validation
Hyperparameters

9. e.mundo © 2022
Case Study
ECGAN Framework
● Joint work between eMundo GmbH and LMU Munich, co-funded by the Bavarian Research Foundation focusing on
arrhythmia detection using generative adversarial networks
9

10. e.mundo © 2022
Case Study
ECGAN Framework
● Joint work between eMundo GmbH and LMU Munich, co-funded by the Bavarian Research Foundation focusing on
arrhythmia detection using generative adversarial networks
● One configuration per experiment (training configuration, evaluation configuration). Configuration steers
● Data:
■ Download and preprocessing → Publish the preprocessed data or make it reproducible
● Code and environment:
■ Reproducibility parameters (random seeds, GPU/CPU,...), model architectures, training, saving & loading
10

11. e.mundo © 2022
Case Study
ECGAN Framework
● Joint work between eMundo GmbH and LMU Munich, co-funded by the Bavarian Research Foundation focusing on
arrhythmia detection using generative adversarial networks
● One configuration per experiment (training configuration, evaluation configuration). Configuration steers
● Data:
■ Download and preprocessing → Publish the preprocessed data or make it reproducible
● Code and environment:
■ Reproducibility parameters (random seeds, GPU/CPU,...), model architectures, training, saving & loading
● Runs can significantly differ depending on these settings
● Publish your config and results of the run → Public and reproducible runs
● Plenty of metrics and visualizations
● Validity checks possible across datasets and on same dataset
11

12. e.mundo © 2022
ecgan-init ecgan-preprocess ecgan-train ecgan-detect
Initialization of
config
based on dataset
and model
Data Preprocessing
dataset-dependent
preprocessing including
data cleaning,
resampling,..
Data Retrieval
Model Training
lightweight processing
(e.g. data transformation
and splitting) and model
training in PyTorch
Evaluation
Config
Model
Evaluation

13. e.mundo © 2022
The way forward
● Current common impact metrics are faulty, further metrics for a better understanding are necessary
● No “one-fits-it-all”
13

14. e.mundo © 2022
The way forward
● Current common impact metrics are faulty, further metrics for a better understanding are necessary
● No “one-fits-it-all”
● Changes to publication process and reward system are required: incentives and reviews are flawed
14

15. e.mundo © 2022
The way forward
● Current common impact metrics are faulty, further metrics for a better understanding are necessary
● No “one-fits-it-all”
● Changes to publication process and reward system are required: incentives and reviews are flawed
● Medicine
● Foster creation of larger open source datasets
● Is it possible to at least publish the test set?
■ In itself: terrible. But better than nothing? ¯_(ツ)_/¯
● Should evaluations on additional datasets be demanded?
■ Publish evaluation on publicly available datasets
● Problem-specific model would not necessarily generalize well on different domain or little data
● Not useful if code and configuration are closed source
15

16. e.mundo © 2022
The way forward
● Clear steps for reproduction
● Public source code or trained models cannot guarantee reproducibility but are a first step
■ Overhead to check for “was train data in test data” too large in current process
■ Enforce basic requirements for functionality and code quality (please less “aaa=aa*b*bb^2”)
● MLOps virtually non-existent in many areas related to publication/research
● Pipelines for common scenarios → some standardization possible
● If review is quick (because: standardized) and configuration is meaningful: Easy to question settings
● Easy-to-use technical means (e.g. use of non-deterministic algorithms)
● If possible: Docker container with all relevant code, package versions and data
16

17. e.mundo © 2022
The way forward
● Clear steps for reproduction
● Public source code or trained models cannot guarantee reproducibility but are a first step
■ Overhead to check for “was train data in test data” too large in current process
■ Enforce basic requirements for functionality and code quality (please less “aaa=aa*b*bb^2”)
● MLOps virtually non-existent in many areas related to publication/research
● Pipelines for common scenarios → some standardization possible
● If review is quick (because: standardized) and configuration is meaningful: Easy to question settings
● Easy-to-use technical means (e.g. use of non-deterministic algorithms)
● If possible: Docker container with all relevant code, package versions and data
● Avoid bitterness: try your best, mistakes are human, be open to corrections
17

18. Fiete Lüer
fiete.lueer@e-mundo.de
+49 (0) 172 1398993
Do not hesitate to contact me!
● Code: https://github.com/emundo/ecgan
● More information:
https://blog.e-mundo.de/post/ecgan-a-framework-f
or-reproducible-research-on-ecg-data/