This document discusses reproducibility in machine learning experiments and provides a checklist to improve reproducibility. It contains the following key points in 3 sentences: The document introduces the topic of reproducibility in machine learning and discusses the importance of making machine learning experiment results more reproducible. It then provides and explains in detail the "Machine Learning Reproducibility Checklist" created by Joelle Pineau, which contains steps researchers should take to clearly describe their models, algorithms, data, hyperparameters and results to enable other researchers to understand and replicate their work. The checklist aims to improve reproducibility by ensuring researchers provide all necessary information and details to allow other to understand, evaluate and build upon their findings.

1. Reproducibility in machine
learning
1
Payam Barnaghi
Centre for Vision, Speech and Signal Processing (CVSSP), University of
Surrey
Care Technology & Research Centre, The UK Dementia Research Institute

2. 2

3. 3

4. 4
Source: Odd Erik Gundersen, the Norwegian University of Science and Technology in
Trondheim,
Science Magazine

5. 5

6. 6

7. What is machine learning?
How can we make the results of machine
learning experiments more reproducible?
7

8. 8
Leia: Yaté. Yaté. Yotó.
(SUBTITLE: “I have come for the bounty on this Wookiee.”)
C-3PO relays this message and Jabba says he’ll offer 25,000 for
Chewie.
Leia: Yotó. Yotó. (SUBTITLE: “50,000, no less.”)
C-3PO relays this message and Jabba asks why he should pay so
much.
Leia: Eí yóto.
The above isn’t subtitled, but Leia pulls out a bomb and activates it.
c-3po: Because he’s holding a thermal detonator!
Jabba is impressed by this and offers 35,000.
Leia: Yató cha.
The above isn’t subtitled, but Leia deactivates the bomb and puts it
away.
c-3po: He agrees.
Order is restored.
Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by

9. 9
Leia: Yaté. Yaté. Yotó.
(SUBTITLE: “I have come for the bounty on this Wookiee.”)
C-3PO relays this message and Jabba says he’ll offer 25,000 for
Chewie.
Leia: Yotó. Yotó. (SUBTITLE: “50,000, no less.”)
C-3PO relays this message and Jabba asks why he should pay so
much.
Leia: Eí yóto.
The above isn’t subtitled, but Leia pulls out a bomb and activates it.
c-3po: Because he’s holding a thermal detonator!
Jabba is impressed by this and offers 35,000.
Leia: Yató cha.
The above isn’t subtitled, but Leia deactivates the bomb and puts it
away.
c-3po: He agrees.
Order is restored.
Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by

10. 10

11. 11Source:
https://www.omniglot.com/conscripts/tengwarhv.htm

12. However, don’t underestimate the human brain
− For example, in German:
−you have to distinguish between “the” and “a” articles
−and then each one has four case forms
−three genders
−and singular and plural forms
−and then the adjectives have to agree
−and then there are verbs!
−or, take this as an example:
12
Adapted from: The Art of Language Invention: From Horse-Lords to Dark Elves, The Words Behind World-Building, by
(biáng)

13. Peterson
13
/ (ˈpiːtəsən) /
Pear Peter/ Peter’s Son
Bear Meter
Ear Greta
… …

14. Payam
14
/ (ˈpæːjʌm) /
Pay + am or Pa +
yam
‫پیام‬

15. 15
Source: https://www.originofalphabet.com/meow-is-just-another-name-for-
cat-2/

16. 16
/ (ˈpiːtəsən) /
/ (ˈpiːtəsən) /
P
ˈp
e
i
…
…
n
n

17. 17
“It is a capital mistake to
theorise before one has
data. Insensibly one begins
to twist facts to suit theories,
instead of theories to suit
facts.”
Arthur Conan Doyle, Sherlock
Holmes

18. 18

19. 19

20. 20

21. 21

22. 22

23. The Machine Learning Reproducibility Checklist
(Version 1.2, Mar.27 2019)- Joelle Pineau
−For all models and algorithms presented, check
if you include:
−A clear description of the mathematical setting,
algorithm, and/or model.
−An analysis of the complexity (time, space, sample
size) of any algorithm.
−A link to a downloadable source code, with
specification of all dependencies, including external
libraries.
23
Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

24. The Machine Learning Reproducibility Checklist
− For any theoretical claim, check if you include:
−A statement of the result.
−A clear explanation of any assumptions.
−A complete proof of the claim.
24
Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

25. The Machine Learning Reproducibility Checklist
− For all figures and tables that present empirical results,
check if you include:
− A complete description of the data collection process, including
sample size.
− A link to a downloadable version of the dataset or simulation
environment.
− An explanation of any data that were excluded, description of
any pre-processing step.
− An explanation of how samples were allocated for training/
validation/ testing.
25
Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

26. The Machine Learning Reproducibility Checklist
− The range of hyper-parameters considered, method to select the
best hyper-parameter configuration, and specification of all hyper-
parameters used to generate results.
− The exact number of evaluation runs.
− A description of how experiments were run.
− A clear definition of the specific measure or statistics used to report
results.
− Clearly defined error bars.
− A description of results with central tendency (e.g. mean) &
variation(e.g. stddev).
− A description of the computing infrastructure used.
26
Reproduced from: www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

27. 27
https://arxiv.org/pdf/1810.03993.pdf

28. Reproducibility in ML
”The point of reproducibility isn’t to replicate the results
exactly. That would be nearly impossible given the
natural randomness in neural networks and variations in
hardware and code. Instead, the idea is to offer a road
map to reach the same conclusions as the original
research, especially when that involves deciding which
machine-learning system is best for a particular task.”
Jesse Dodge, Carnegie Mellon University
28
Reproduced from: https://www.wired.com/story/artificial-intelligence-confronts-
reproducibility-crisis/

29. 29

30. Thank you.
30