Computational practices for reproducible science

1. Computational practices for reproducible science Ga¨el Varoquaux import science science.reproduce()

2. Computational practices for reproducible science Ga¨el Varoquaux This talk: High-level advice on code in science Pointers to good software practices

3. Computational practices for reproducible science Ga¨el Varoquaux Outline: 1 Coding for science, reproducibly 2 Making libraries 3 From software back to science

4. Where I come from PhD in physics Computers: Matlab Instrument control Python for Science: Startup guy Mayavi scikit-learn joblib nilearn Applied maths / machine learning for neuroscience G Varoquaux 2

5. 1 Coding for science, reproducibly G Varoquaux 3

6. 1 Computational reproducibility Computational reproducibility: Automate everything Control the environment G Varoquaux 4

7. 1 Automate everything Just a simple matter of programming G Varoquaux 5

8. 1 Automate everything Just a simple matter of programming Scientiﬁc workﬂow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment G Varoquaux 6

9. 1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility G Varoquaux 7

10. 1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility Lost in iterations G Varoquaux 8

11. 1 Reproducible computational science Harder than it seemed Impediments Missing steps / ﬁles Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is G Varoquaux 9

12. 1 Reproducible computational science Harder than it seemed Impediments Missing steps / ﬁles Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is Technical Human Code quality matters Manual steps are evil G Varoquaux 9

13. 1 How I work progressive consolidation Start with a script playing to understand the problem G Varoquaux 10

14. 1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function Use functions Obstacle: local scope requires identifying input and output variables That’s a good thing Interactive debugging / understanding inside a function: %debug in IPython Functions are the basic reusable abstraction G Varoquaux 10

15. 1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Modules enable sharing between experiments ⇒ avoid 1000 lines scripts + commented code enable testing Fast experiments as tests ⇒ gives conﬁdence, hence refactorings G Varoquaux 10

16. 1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & ﬁles you have version control Attentional load makes it impossible to ﬁnd or understand things Where’s Waldo?G Varoquaux 10

17. 1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & ﬁles you have version control Why is it hard? Long compute times make us unadventurous Know your tools Refactoring editor Version control G Varoquaux 10

18. 1 Mitigating long compute times by storing intermediate results Makeﬁles Accessible intermediate results Break control ﬂow & error management G Varoquaux 11

19. 1 Mitigating long compute times by storing intermediate results Makeﬁles Accessible intermediate results Break control ﬂow & error management Memoization / caching g = mem.cache(f) # ”joblib” Python module b = g(a) # computes a using f c = g(a) # retrieves results from store Fits in coding / experimentation loop Black-boxy, persistence only implicit Discourages refactoring (avoid recomputing) G Varoquaux 11

20. 1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view Photo-editing software Filters Canvas Tool palettes Typical web application Database Web-pages URLs G Varoquaux 12

21. 1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops G Varoquaux 12

22. 1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control G Varoquaux 12

23. 1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control Decouple steps Goals: Reuse code Mitigate compute time G Varoquaux 12

24. Adopting software-engineering best practices G Varoquaux 13

25. 1 The ladder of code quality Use static checking in your editor seriously G Varoquaux 14

26. 1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming “Any fool can write code that a computer can understand. Good programmers write code that humans can understand” – Martin Fowler Your future self will thank you G Varoquaux 14

27. 1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... G Varoquaux 14

28. 1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil G Varoquaux 14

29. 1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil Over versus under engineering Our goal is scientiﬁc discovery Experimentation to develop intuitions ⇒ new ideas As the path becomes clear: consolidation Heavy engineering too early freezes bad ideas G Varoquaux 14

30. 1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library G Varoquaux 15

31. 1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library Code = Software Well-deﬁned problem = research Functions should work as long as input is valid Targets users, not the authors Made for reuse The workhorse of reproducibility G Varoquaux 15

32. 1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git G Varoquaux 16

33. 1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Copying scripts scales very poorly: Accumulation of half-dead code ⇒ cognitive overload No consolidation across studies Each study has diﬀerent variants of diﬀerent bugs Garden of forking code G Varoquaux 16

34. 1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Kirstie Whitaker G Varoquaux 16

35. 1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Frozen food Reusability Apply the approach to a new problem Being able to understand, modify, run in new settings Generalization is the true scientiﬁc test Reusable computational science = libraries G Varoquaux 16

36. 2 Making libraries to enable reproducibility G Varoquaux 17

37. 2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18

38. 2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18

39. 2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Deﬁne project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch G Varoquaux 18

40. 2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Deﬁne project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch Your research (PhD) probably does not qualify ⇒ need to cherry-pick contributions G Varoquaux 18

41. 2 Short cycles, limited ambitions Keep coming back to your users Release early, release often G Varoquaux 19

42. 2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodﬁeld 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity G Varoquaux 20

43. 2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodﬁeld 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity The 80/20 rule 80% of the usecases can be solved with 20% of the lines of code Avoid feature creep G Varoquaux 20

44. 2 Quality Quality is free This is a book, by Philip Crosby G Varoquaux 21

45. 2 You need quality Quality will give you users Bugs give you bad rap Quality will give you developers Contribute to learn and improve Quality avoids scientiﬁc errors Do less, do better Goes against the grant-system incentive G Varoquaux 22

46. 2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of diﬀerent concepts and objects Good numerics Write tests based on mathematical properties When a user ﬁnds an instability, write a new test G Varoquaux 23

47. 2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of diﬀerent concepts and objects Good numerics Write tests based on mathematical properties When a user ﬁnds an instability, write a new test Quality enables reuse Beyond mere reproducibility G Varoquaux 23

48. 2 Quality assurance Code review: pull requests Can include newcomers We read each others code Everything is discussed: - Should the algorithm go in? - Are there good defaults? - Are names meaningfull? - Are the numerics stable? - Could it be faster? G Varoquaux 24

49. 2 Quality assurance Unit testing Everything is tested Great for numerics Overall tests enforce on all estimators - consistency with the API - basic invariances - good handling of various inputs G Varoquaux 25

50. 2 Dependencies Beyond installation, a challenge is to ensure package versions play way together: correctness of the code Breakage of backward compability yields irreconcilable dependencies G Varoquaux 26

51. 2 Dependencies and their upgrade It’s a fact: users hate upgrading If it ain’t broken, don’t ﬁx it even if it is, apparently G Varoquaux 27

52. 2 Declaring undependence? Monolythic packages with no dependencies... But: Scaling is hard Complexity grows as square of codebase size [Woodﬁeld 1979] User support grows with userbase size G Varoquaux 28

53. 3 From software back to science G Varoquaux 29

54. 3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge G Varoquaux 30

55. 3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge Estimating the reproducibility of psychological science [Science 2015] 36% of eﬀects replicate Reasons: Statistical challenges — analysis degrees of freedom Weak insentives — winner’s curse in publication Seldom computational reproducibility G Varoquaux 30

56. In practice, the best way to improve research is to use the right (conceptual) tools. G Varoquaux 31

57. 3 Managing complexity In practice, the best way to improve research is to use the right (conceptual) tools. The everyday roadblock is cognitive load Machine learning, brain anatomy, psychology R, Python, shell scripts Funding agencies, reviewer 3, the Manuscript G Varoquaux 32

58. 3 Tools that reduce cognitive overload It’s a design problem G Varoquaux 33

59. 3 Tools that reduce cognitive overload Jonathan Ive, an industrial designer, is #4 at Apple Code diﬀerent. G Varoquaux 34

60. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 35

61. 3 Some API design principles for the scipy stack Consistency, consistency, consistency np.save(file, obj) pickle.dump(obj, file) fmin(...maxiter=10) lsq linear(...max iter=10) Creates cognitive overload Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 36

62. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes Objects have hidden states, Objects have no universal interface, entry point, output A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 37

63. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts How much do usage patterns carry out across the library? Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 38

64. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Facilitates working with multiple libraries together Easier to get up to speed with a given library Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 39

65. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Change of behavior depending on input type Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 40

66. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Interfaces deﬁne objects Incompatible behaviors lead to bugs (eg np.matrix) Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 41

67. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Properties obfuscate the data model of the object Properties can create hidden compute costs Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 42

68. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Objects are understood by their surface Composition creates cognitive overload Error messages matter Be Pythonic G Varoquaux 43

69. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Explain the problem Print the oﬀending value Be Pythonic G Varoquaux 44

70. 3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic Avoid syntax hacks G Varoquaux 45

71. 3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) G Varoquaux 46

72. 3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) The estimator is a “contract” (slightly more elaborate than above) It has created an ecosystem of packages Based on duck-typing, not inheritence G Varoquaux 46

73. 3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple G Varoquaux 47

74. 3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User ﬂow on the scikit-learn website: Examples G Varoquaux 47

75. 3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User ﬂow on the nilearn website: Examples G Varoquaux 47

76. 3 Building great documentation Focus on explaining concepts (hint: write plain English) Less is more: prioritize, avoid redundancy Code examples must be short (link to full tutorial examples) Links everywhere: users will land at the wrong place Teach with the docs Plan for maintenance of docs: Continuous integration Check links Runs examples Doctests G Varoquaux 48

77. 3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html G Varoquaux 49

78. 3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Resource intensive Continuous integration: Data ⇒ Fight for good open data Computation ⇒ Find good algorithms and tradeoﬀs Forces us to distill the literature (as a review) G Varoquaux 49

79. 3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab G Varoquaux 49

80. 3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab scipy-lectures: living book for Python in science G Varoquaux 49

81. Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload G Varoquaux 50

82. Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload In the industry Reusable Veriﬁable? Not for silicon valley, but in insurance, healthcare, banking... Moving data-scientist code to production? Software projects going over budget? G Varoquaux 50

83. @GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive

84. @GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced

85. @GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization

86. @GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization Reproducible research must borrow from agile development

Computational practices for reproducible science

Gael Varoquaux

Computational practices for reproducible science