Computational practices for reproducible science

Computational practices for reproducible science
Computational practices for reproducible science Gaël Varoquaux import science science.reproduce()
Computational practices for reproducible science Gaël Varoquaux This talk: High-level advice on code in science Pointers to good software practices
Computational practices for reproducible science Gaël Varoquaux Outline: 1 Coding for science, reproducibly 2 Making libraries 3 From software back to science
Where I come from PhD in physics Computers: Matlab Instrument control Python for Science: Startup guy Mayavi scikit-learn joblib nilearn Applied maths / machine learning for neuroscience G Varoquaux 2
1 Coding for science, reproducibly G Varoquaux 3
1 Computational reproducibility Computational reproducibility: Automate everything Control the environment G Varoquaux 4
1 Automate everything Just a simple matter of programming G Varoquaux 5
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment G Varoquaux 6
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility G Varoquaux 7
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility Lost in iterations G Varoquaux 8
1 Reproducible computational science Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is G Varoquaux 9
1 Reproducible computational science Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is Technical Human Code quality matters Manual steps are evil G Varoquaux 9
1 How I work progressive consolidation Start with a script playing to understand the problem G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function Use functions Obstacle: local scope requires identifying input and output variables That’s a good thing Interactive debugging / understanding inside a function: %debug in IPython Functions are the basic reusable abstraction G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Modules enable sharing between experiments ⇒ avoid 1000 lines scripts + commented code enable testing Fast experiments as tests ⇒ gives confidence, hence refactorings G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Attentional load makes it impossible to find or understand things Where’s Waldo?G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Why is it hard? Long compute times make us unadventurous Know your tools Refactoring editor Version control G Varoquaux 10
1 Mitigating long compute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management G Varoquaux 11
1 Mitigating long compute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management Memoization / caching g = mem.cache(f) # ”joblib” Python module b = g(a) # computes a using f c = g(a) # retrieves results from store Fits in coding / experimentation loop Black-boxy, persistence only implicit Discourages refactoring (avoid recomputing) G Varoquaux 11
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view Photo-editing software Filters Canvas Tool palettes Typical web application Database Web-pages URLs G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control Decouple steps Goals: Reuse code Mitigate compute time G Varoquaux 12
Adopting software-engineering best practices G Varoquaux 13
1 The ladder of code quality Use static checking in your editor seriously G Varoquaux 14
1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming “Any fool can write code that a computer can understand. Good programmers write code that humans can understand” – Martin Fowler Your future self will thank you G Varoquaux 14
1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... G Varoquaux 14
1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil G Varoquaux 14
1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil Over versus under engineering Our goal is scientific discovery Experimentation to develop intuitions ⇒ new ideas As the path becomes clear: consolidation Heavy engineering too early freezes bad ideas G Varoquaux 14
1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library G Varoquaux 15
1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library Code = Software Well-defined problem = research Functions should work as long as input is valid Targets users, not the authors Made for reuse The workhorse of reproducibility G Varoquaux 15
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Copying scripts scales very poorly: Accumulation of half-dead code ⇒ cognitive overload No consolidation across studies Each study has different variants of different bugs Garden of forking code G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Kirstie Whitaker G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Frozen food Reusability Apply the approach to a new problem Being able to understand, modify, run in new settings Generalization is the true scientific test Reusable computational science = libraries G Varoquaux 16
2 Making libraries to enable reproducibility G Varoquaux 17
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch Your research (PhD) probably does not qualify ⇒ need to cherry-pick contributions G Varoquaux 18
2 Short cycles, limited ambitions Keep coming back to your users Release early, release often G Varoquaux 19
2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity G Varoquaux 20
2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity The 80/20 rule 80% of the usecases can be solved with 20% of the lines of code Avoid feature creep G Varoquaux 20
2 Quality Quality is free This is a book, by Philip Crosby G Varoquaux 21
2 You need quality Quality will give you users Bugs give you bad rap Quality will give you developers Contribute to learn and improve Quality avoids scientific errors Do less, do better Goes against the grant-system incentive G Varoquaux 22
2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test G Varoquaux 23
2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test Quality enables reuse Beyond mere reproducibility G Varoquaux 23
2 Quality assurance Code review: pull requests Can include newcomers We read each others code Everything is discussed: - Should the algorithm go in? - Are there good defaults? - Are names meaningfull? - Are the numerics stable? - Could it be faster? G Varoquaux 24
2 Quality assurance Unit testing Everything is tested Great for numerics Overall tests enforce on all estimators - consistency with the API - basic invariances - good handling of various inputs G Varoquaux 25
2 Dependencies Beyond installation, a challenge is to ensure package versions play way together: correctness of the code Breakage of backward compability yields irreconcilable dependencies G Varoquaux 26
2 Dependencies and their upgrade It’s a fact: users hate upgrading If it ain’t broken, don’t fix it even if it is, apparently G Varoquaux 27
2 Declaring undependence? Monolythic packages with no dependencies... But: Scaling is hard Complexity grows as square of codebase size [Woodfield 1979] User support grows with userbase size G Varoquaux 28
3 From software back to science G Varoquaux 29
3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge G Varoquaux 30
3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge Estimating the reproducibility of psychological science [Science 2015] 36% of effects replicate Reasons: Statistical challenges — analysis degrees of freedom Weak insentives — winner’s curse in publication Seldom computational reproducibility G Varoquaux 30
In practice, the best way to improve research is to use the right (conceptual) tools. G Varoquaux 31
3 Managing complexity In practice, the best way to improve research is to use the right (conceptual) tools. The everyday roadblock is cognitive load Machine learning, brain anatomy, psychology R, Python, shell scripts Funding agencies, reviewer 3, the Manuscript G Varoquaux 32
3 Tools that reduce cognitive overload It’s a design problem G Varoquaux 33
3 Tools that reduce cognitive overload Jonathan Ive, an industrial designer, is #4 at Apple Code different. G Varoquaux 34
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 35
3 Some API design principles for the scipy stack Consistency, consistency, consistency np.save(file, obj) pickle.dump(obj, file) fmin(...maxiter=10) lsq linear(...max iter=10) Creates cognitive overload Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 36
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes Objects have hidden states, Objects have no universal interface, entry point, output A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 37
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts How much do usage patterns carry out across the library? Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 38
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Facilitates working with multiple libraries together Easier to get up to speed with a given library Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 39
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Change of behavior depending on input type Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 40
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Interfaces define objects Incompatible behaviors lead to bugs (eg np.matrix) Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 41
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Properties obfuscate the data model of the object Properties can create hidden compute costs Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 42
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Objects are understood by their surface Composition creates cognitive overload Error messages matter Be Pythonic G Varoquaux 43
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Explain the problem Print the offending value Be Pythonic G Varoquaux 44
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic Avoid syntax hacks G Varoquaux 45
3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) G Varoquaux 46
3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) The estimator is a “contract” (slightly more elaborate than above) It has created an ecosystem of packages Based on duck-typing, not inheritence G Varoquaux 46
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple G Varoquaux 47
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the scikit-learn website: Examples G Varoquaux 47
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the nilearn website: Examples G Varoquaux 47
3 Building great documentation Focus on explaining concepts (hint: write plain English) Less is more: prioritize, avoid redundancy Code examples must be short (link to full tutorial examples) Links everywhere: users will land at the wrong place Teach with the docs Plan for maintenance of docs: Continuous integration Check links Runs examples Doctests G Varoquaux 48
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Resource intensive Continuous integration: Data ⇒ Fight for good open data Computation ⇒ Find good algorithms and tradeoffs Forces us to distill the literature (as a review) G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab scipy-lectures: living book for Python in science G Varoquaux 49
Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload G Varoquaux 50
Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload In the industry Reusable Verifiable? Not for silicon valley, but in insurance, healthcare, banking... Moving data-scientist code to production? Software projects going over budget? G Varoquaux 50
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization Reproducible research must borrow from agile development

Computational practices for reproducible science

Check these out next

Computational practices for reproducible science

Recommended

More Related Content

Slideshows for you(9)

Similar to Computational practices for reproducible science(20)

More from Gael Varoquaux(20)

Recently uploaded(20)

Computational practices for reproducible science