Gael Varoquaux, profile picture
Uploaded byGael Varoquaux
4,850 views

Computational practices for reproducible science

The document presents guidelines for computational practices aimed at achieving reproducibility in scientific research. It emphasizes the importance of automating workflows, using proper coding conventions, and incorporating software engineering best practices such as version control and unit testing. Additionally, it discusses the development of libraries to facilitate code reuse and ensure reproducible results in computational experiments.

Embed presentation

Downloaded 61 times
Computational practices for reproducible science Ga¨el Varoquaux import science science.reproduce()
Computational practices for reproducible science Ga¨el Varoquaux This talk: High-level advice on code in science Pointers to good software practices
Computational practices for reproducible science Ga¨el Varoquaux Outline: 1 Coding for science, reproducibly 2 Making libraries 3 From software back to science
Where I come from PhD in physics Computers: Matlab Instrument control Python for Science: Startup guy Mayavi scikit-learn joblib nilearn Applied maths / machine learning for neuroscience G Varoquaux 2
1 Coding for science, reproducibly G Varoquaux 3
1 Computational reproducibility Computational reproducibility: Automate everything Control the environment G Varoquaux 4
1 Automate everything Just a simple matter of programming G Varoquaux 5
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment G Varoquaux 6
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility G Varoquaux 7
1 Automate everything Just a simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility Lost in iterations G Varoquaux 8
1 Reproducible computational science Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is G Varoquaux 9
1 Reproducible computational science Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is Technical Human Code quality matters Manual steps are evil G Varoquaux 9
1 How I work progressive consolidation Start with a script playing to understand the problem G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function Use functions Obstacle: local scope requires identifying input and output variables That’s a good thing Interactive debugging / understanding inside a function: %debug in IPython Functions are the basic reusable abstraction G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Modules enable sharing between experiments ⇒ avoid 1000 lines scripts + commented code enable testing Fast experiments as tests ⇒ gives confidence, hence refactorings G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Attentional load makes it impossible to find or understand things Where’s Waldo?G Varoquaux 10
1 How I work progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Why is it hard? Long compute times make us unadventurous Know your tools Refactoring editor Version control G Varoquaux 10
1 Mitigating long compute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management G Varoquaux 11
1 Mitigating long compute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management Memoization / caching g = mem.cache(f) # ”joblib” Python module b = g(a) # computes a using f c = g(a) # retrieves results from store Fits in coding / experimentation loop Black-boxy, persistence only implicit Discourages refactoring (avoid recomputing) G Varoquaux 11
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view Photo-editing software Filters Canvas Tool palettes Typical web application Database Web-pages URLs G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control G Varoquaux 12
1 A design pattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control Decouple steps Goals: Reuse code Mitigate compute time G Varoquaux 12
Adopting software-engineering best practices G Varoquaux 13
1 The ladder of code quality Use static checking in your editor seriously G Varoquaux 14
1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming “Any fool can write code that a computer can understand. Good programmers write code that humans can understand” – Martin Fowler Your future self will thank you G Varoquaux 14
1 The ladder of code quality Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... G Varoquaux 14
1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil G Varoquaux 14
1 The ladder of code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil Over versus under engineering Our goal is scientific discovery Experimentation to develop intuitions ⇒ new ideas As the path becomes clear: consolidation Heavy engineering too early freezes bad ideas G Varoquaux 14
1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library G Varoquaux 15
1 LibrariesIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library Code = Software Well-defined problem = research Functions should work as long as input is valid Targets users, not the authors Made for reuse The workhorse of reproducibility G Varoquaux 15
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Copying scripts scales very poorly: Accumulation of half-dead code ⇒ cognitive overload No consolidation across studies Each study has different variants of different bugs Garden of forking code G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Kirstie Whitaker G Varoquaux 16
1 Libraries enable reproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Frozen food Reusability Apply the approach to a new problem Being able to understand, modify, run in new settings Generalization is the true scientific test Reusable computational science = libraries G Varoquaux 16
2 Making libraries to enable reproducibility G Varoquaux 17
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch G Varoquaux 18
2 Pick the right battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch Your research (PhD) probably does not qualify ⇒ need to cherry-pick contributions G Varoquaux 18
2 Short cycles, limited ambitions Keep coming back to your users Release early, release often G Varoquaux 19
2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity G Varoquaux 20
2 Simplicity Complexity increase superlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity The 80/20 rule 80% of the usecases can be solved with 20% of the lines of code Avoid feature creep G Varoquaux 20
2 Quality Quality is free This is a book, by Philip Crosby G Varoquaux 21
2 You need quality Quality will give you users Bugs give you bad rap Quality will give you developers Contribute to learn and improve Quality avoids scientific errors Do less, do better Goes against the grant-system incentive G Varoquaux 22
2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test G Varoquaux 23
2 Quality: what & how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test Quality enables reuse Beyond mere reproducibility G Varoquaux 23
2 Quality assurance Code review: pull requests Can include newcomers We read each others code Everything is discussed: - Should the algorithm go in? - Are there good defaults? - Are names meaningfull? - Are the numerics stable? - Could it be faster? G Varoquaux 24
2 Quality assurance Unit testing Everything is tested Great for numerics Overall tests enforce on all estimators - consistency with the API - basic invariances - good handling of various inputs G Varoquaux 25
2 Dependencies Beyond installation, a challenge is to ensure package versions play way together: correctness of the code Breakage of backward compability yields irreconcilable dependencies G Varoquaux 26
2 Dependencies and their upgrade It’s a fact: users hate upgrading If it ain’t broken, don’t fix it even if it is, apparently G Varoquaux 27
2 Declaring undependence? Monolythic packages with no dependencies... But: Scaling is hard Complexity grows as square of codebase size [Woodfield 1979] User support grows with userbase size G Varoquaux 28
3 From software back to science G Varoquaux 29
3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge G Varoquaux 30
3 Beyond computational reproducibility Make every computational step reproducible, and good science will emerge Estimating the reproducibility of psychological science [Science 2015] 36% of effects replicate Reasons: Statistical challenges — analysis degrees of freedom Weak insentives — winner’s curse in publication Seldom computational reproducibility G Varoquaux 30
In practice, the best way to improve research is to use the right (conceptual) tools. G Varoquaux 31
3 Managing complexity In practice, the best way to improve research is to use the right (conceptual) tools. The everyday roadblock is cognitive load Machine learning, brain anatomy, psychology R, Python, shell scripts Funding agencies, reviewer 3, the Manuscript G Varoquaux 32
3 Tools that reduce cognitive overload It’s a design problem G Varoquaux 33
3 Tools that reduce cognitive overload Jonathan Ive, an industrial designer, is #4 at Apple Code different. G Varoquaux 34
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 35
3 Some API design principles for the scipy stack Consistency, consistency, consistency np.save(file, obj) pickle.dump(obj, file) fmin(...maxiter=10) lsq linear(...max iter=10) Creates cognitive overload Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 36
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes Objects have hidden states, Objects have no universal interface, entry point, output A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 37
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts How much do usage patterns carry out across the library? Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 38
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Facilitates working with multiple libraries together Easier to get up to speed with a given library Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 39
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Change of behavior depending on input type Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 40
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Interfaces define objects Incompatible behaviors lead to bugs (eg np.matrix) Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 41
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Properties obfuscate the data model of the object Properties can create hidden compute costs Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 42
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Objects are understood by their surface Composition creates cognitive overload Error messages matter Be Pythonic G Varoquaux 43
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Explain the problem Print the offending value Be Pythonic G Varoquaux 44
3 Some API design principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic Avoid syntax hacks G Varoquaux 45
3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) G Varoquaux 46
3 Scikit-learn API Scikit-learn cheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) The estimator is a “contract” (slightly more elaborate than above) It has created an ecosystem of packages Based on duck-typing, not inheritence G Varoquaux 46
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple G Varoquaux 47
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the scikit-learn website: Examples G Varoquaux 47
3 Example-driven development The 3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the nilearn website: Examples G Varoquaux 47
3 Building great documentation Focus on explaining concepts (hint: write plain English) Less is more: prioritize, avoid redundancy Code examples must be short (link to full tutorial examples) Links everywhere: users will land at the wrong place Teach with the docs Plan for maintenance of docs: Continuous integration Check links Runs examples Doctests G Varoquaux 48
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Resource intensive Continuous integration: Data ⇒ Fight for good open data Computation ⇒ Find good algorithms and tradeoffs Forces us to distill the literature (as a review) G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab G Varoquaux 49
3 Reusable science scikit-learn is the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab scipy-lectures: living book for Python in science G Varoquaux 49
Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload G Varoquaux 50
Coding as a scientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload In the industry Reusable Verifiable? Not for silicon valley, but in insurance, healthcare, banking... Moving data-scientist code to production? Software projects going over budget? G Varoquaux 50
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization
@GaelVaroquaux Computational practices for reproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization Reproducible research must borrow from agile development

More Related Content

PDF
Coding for science and innovation
byGael Varoquaux
 
PDF
On the code of data science
byGael Varoquaux
 
PDF
Succeeding in academia despite doing good_software
byGael Varoquaux
 
PDF
Open Source Scientific Software
byGael Varoquaux
 
PDF
Scikit-learn and nilearn: Democratisation of machine learning for brain imaging
byGael Varoquaux
 
PDF
Processing biggish data on commodity hardware: simple Python patterns
byGael Varoquaux
 
PDF
Scikit-learn: the state of the union 2016
byGael Varoquaux
 
PDF
Building a cutting-edge data processing environment on a budget
byGael Varoquaux
 
Coding for science and innovation
byGael Varoquaux
 
On the code of data science
byGael Varoquaux
 
Succeeding in academia despite doing good_software
byGael Varoquaux
 
Open Source Scientific Software
byGael Varoquaux
 
Scikit-learn and nilearn: Democratisation of machine learning for brain imaging
byGael Varoquaux
 
Processing biggish data on commodity hardware: simple Python patterns
byGael Varoquaux
 
Scikit-learn: the state of the union 2016
byGael Varoquaux
 
Building a cutting-edge data processing environment on a budget
byGael Varoquaux
 

What's hot

PPTX
Binary Analysis - Luxembourg
byAbhik Roychoudhury
 
PDF
Python for brain mining: (neuro)science with state of the art machine learnin...
byGael Varoquaux
 
PDF
Deep Learning Cases: Text and Image Processing
byGrigory Sapunov
 
PPTX
NUS PhD e-open day 2020
byAbhik Roychoudhury
 
PDF
Scikit-Learn: Machine Learning in Python
byMicrosoft
 
PPTX
APSEC2020 Keynote
byAbhik Roychoudhury
 
PDF
Data Science Accelerator Program
byGoDataDriven
 
PPTX
Automated Program Repair, Distinguished lecture at MPI-SWS
byAbhik Roychoudhury
 
PPTX
Automated Program Repair Keynote talk
byAbhik Roychoudhury
 
Binary Analysis - Luxembourg
byAbhik Roychoudhury
 
Python for brain mining: (neuro)science with state of the art machine learnin...
byGael Varoquaux
 
Deep Learning Cases: Text and Image Processing
byGrigory Sapunov
 
NUS PhD e-open day 2020
byAbhik Roychoudhury
 
Scikit-Learn: Machine Learning in Python
byMicrosoft
 
APSEC2020 Keynote
byAbhik Roychoudhury
 
Data Science Accelerator Program
byGoDataDriven
 
Automated Program Repair, Distinguished lecture at MPI-SWS
byAbhik Roychoudhury
 
Automated Program Repair Keynote talk
byAbhik Roychoudhury
 

Similar to Computational practices for reproducible science

PDF
2014-10-10-SBC361-Reproducible research
byYannick Wurm
 
PPTX
Writing clean scientific software Murphy cleancoding
bysaber tabatabaee
 
PPTX
2013 ucar best practices
byc.titus.brown
 
PDF
Software maintenance PyConPL 2016
byCesar Cardenas Desales
 
PDF
2013 10-30-sbc361-reproducible designsandsustainablesoftware
byYannick Wurm
 
PPTX
Rapid PowerShell Function Development
byFrank Lesniak
 
PDF
Effective Python 90 specific ways to write better Python Second Edition Brett...
bykoldakeittlj
 
PDF
Effective Python 90 specific ways to write better Python Second Edition Brett...
byhcuscmriqq641
 
PPT
20090918 Agile Computer Control of a Complex Experiment
byJonathan Blakes
 
PDF
Finding Resource Manipulation Bugs in Linux Code
byAndrzej Wasowski
 
PDF
Building a Cutting-Edge Data Process Environment on a Budget by Gael Varoquaux
byPyData
 
PPTX
Capability Building for Cyber Defense: Software Walk through and Screening
byMaven Logix
 
PPTX
Reproducibility: 10 Simple Rules
byAnnika Eriksson
 
PDF
You and your code.pdf
byTony Khánh
 
PPTX
Improving Code Quality Through Effective Review Process
byDr. Syed Hassan Amin
 
PDF
Craftsmanship in Computational Work
byYung-Yu Chen
 
PPTX
20171003 lancaster data conversations Chue-Hong
byLancaster University Library
 
PDF
Euro python 2015 writing quality code
byradek_j
 
PDF
Systems se
byFranco Bressan
 
PDF
Programming practises and project management for professionnal software devel...
byGeeks Anonymes
 
2014-10-10-SBC361-Reproducible research
byYannick Wurm
 
Writing clean scientific software Murphy cleancoding
bysaber tabatabaee
 
2013 ucar best practices
byc.titus.brown
 
Software maintenance PyConPL 2016
byCesar Cardenas Desales
 
2013 10-30-sbc361-reproducible designsandsustainablesoftware
byYannick Wurm
 
Rapid PowerShell Function Development
byFrank Lesniak
 
Effective Python 90 specific ways to write better Python Second Edition Brett...
bykoldakeittlj
 
Effective Python 90 specific ways to write better Python Second Edition Brett...
byhcuscmriqq641
 
20090918 Agile Computer Control of a Complex Experiment
byJonathan Blakes
 
Finding Resource Manipulation Bugs in Linux Code
byAndrzej Wasowski
 
Building a Cutting-Edge Data Process Environment on a Budget by Gael Varoquaux
byPyData
 
Capability Building for Cyber Defense: Software Walk through and Screening
byMaven Logix
 
Reproducibility: 10 Simple Rules
byAnnika Eriksson
 
You and your code.pdf
byTony Khánh
 
Improving Code Quality Through Effective Review Process
byDr. Syed Hassan Amin
 
Craftsmanship in Computational Work
byYung-Yu Chen
 
20171003 lancaster data conversations Chue-Hong
byLancaster University Library
 
Euro python 2015 writing quality code
byradek_j
 
Systems se
byFranco Bressan
 
Programming practises and project management for professionnal software devel...
byGeeks Anonymes
 

More from Gael Varoquaux

PDF
Evaluating machine learning models and their diagnostic value
byGael Varoquaux
 
PDF
Measuring mental health with machine learning and brain imaging
byGael Varoquaux
 
PDF
Machine learning with missing values
byGael Varoquaux
 
PDF
Dirty data science machine learning on non-curated data
byGael Varoquaux
 
PDF
Representation learning in limited-data settings
byGael Varoquaux
 
PDF
Better neuroimaging data processing: driven by evidence, open communities, an...
byGael Varoquaux
 
PDF
Functional-connectome biomarkers to meet clinical needs?
byGael Varoquaux
 
PDF
Atlases of cognition with large-scale human brain mapping
byGael Varoquaux
 
PDF
Similarity encoding for learning on dirty categorical variables
byGael Varoquaux
 
PDF
Machine learning for functional connectomes
byGael Varoquaux
 
PDF
Towards psychoinformatics with machine learning and brain imaging
byGael Varoquaux
 
PDF
Simple representations for learning: factorizations and similarities
byGael Varoquaux
 
PDF
A tutorial on Machine Learning, with illustrations for MR imaging
byGael Varoquaux
 
PDF
Estimating Functional Connectomes: Sparsity’s Strength and Limitations
byGael Varoquaux
 
PDF
Scientist meets web dev: how Python became the language of data
byGael Varoquaux
 
PDF
Machine learning and cognitive neuroimaging: new tools can answer new questions
byGael Varoquaux
 
PDF
Social-sparsity brain decoders: faster spatial sparsity
byGael Varoquaux
 
PDF
Inter-site autism biomarkers from resting state fMRI
byGael Varoquaux
 
PDF
Brain maps from machine learning? Spatial regularizations
byGael Varoquaux
 
PDF
Scikit-learn for easy machine learning: the vision, the tool, and the project
byGael Varoquaux
 
Evaluating machine learning models and their diagnostic value
byGael Varoquaux
 
Measuring mental health with machine learning and brain imaging
byGael Varoquaux
 
Machine learning with missing values
byGael Varoquaux
 
Dirty data science machine learning on non-curated data
byGael Varoquaux
 
Representation learning in limited-data settings
byGael Varoquaux
 
Better neuroimaging data processing: driven by evidence, open communities, an...
byGael Varoquaux
 
Functional-connectome biomarkers to meet clinical needs?
byGael Varoquaux
 
Atlases of cognition with large-scale human brain mapping
byGael Varoquaux
 
Similarity encoding for learning on dirty categorical variables
byGael Varoquaux
 
Machine learning for functional connectomes
byGael Varoquaux
 
Towards psychoinformatics with machine learning and brain imaging
byGael Varoquaux
 
Simple representations for learning: factorizations and similarities
byGael Varoquaux
 
A tutorial on Machine Learning, with illustrations for MR imaging
byGael Varoquaux
 
Estimating Functional Connectomes: Sparsity’s Strength and Limitations
byGael Varoquaux
 
Scientist meets web dev: how Python became the language of data
byGael Varoquaux
 
Machine learning and cognitive neuroimaging: new tools can answer new questions
byGael Varoquaux
 
Social-sparsity brain decoders: faster spatial sparsity
byGael Varoquaux
 
Inter-site autism biomarkers from resting state fMRI
byGael Varoquaux
 
Brain maps from machine learning? Spatial regularizations
byGael Varoquaux
 
Scikit-learn for easy machine learning: the vision, the tool, and the project
byGael Varoquaux
 

Recently uploaded

PDF
Surface Chemistry ( पृष्ठ रसायन ) - Notes PDF - Irfanullah Mehar - JJ Sir Che...
byWorld of Wisdom
 
PPTX
Chimeric Antigen Receptors T-cell (CAR-T) & Bispecific T- cell engager (BiTEs)
byrhysudi12
 
PPTX
PHIVOLCS FAULT FINDER grade - 7 power point
byfelicityguitang
 
PPTX
Antigens, Haptens and epitopes in immunology.pptx
byGurvinder Kaur
 
PDF
Factors affecting Plant Health – Edaphic factors and Biotic Factors
byNistarini College, Purulia (W.B) India
 
PPTX
Grade 9 - Science Fourth Quarter Physics
byModezaPonje
 
PPTX
ANIMAL_CELL_,_TISSUE_AND_ORGAN_CULTURE[1].pptx
byAdrijaKrishna
 
PDF
Control and Coordination - Short Notes (Prashant Kirad) (1).pdf
byAnu kumari
 
PDF
Age of exploration - ppt presentation_EAPT
byericaangelatoledo06
 
PPTX
Muscles_of_mastication_Anatomy of neck.
bybushra9akhtar786
 
PPTX
Digestive system grade 5 Cambridge basic sicence
byVictor Venter
 
PPTX
VIRULENCE FACTOR OF BACTERIA: STRUCTURAL ELEMENT, ENZYMES, TOXINS...
byBIOSPHERE OF KNOWLEDGE
 
PPTX
Mass selection is one of the oldest methods of plant breeding Commonly use_20...
bysashanklakineni
 
PPTX
Cell Transport Powerpoint.pptx for high school students
byraobondalakunta25
 
PPTX
Photogrammetry_Implant_Dentistry_Summary.pptx
byJeldiKusuma
 
PDF
Virtualization 1: Virtual Machine, Hypervisor (Virtual Machine Monitor)
bymrethein98
 
PPTX
Anaerobic respiration (Fermentation, Types, Pasteur effect, Warburg effect, R...
byRashmiMG2
 
PDF
Detection of an NH3 Absorption Band at 2.2μm on Europa
bySérgio Sacani
 
PPTX
Pharmaceutical Calculation Techniques and Terminologies (1).pptx
bymuhammedsajjadhussei
 
PPT
Thermodynamics Chapter-2 by Cengel and Boles
byrasoolurfa
 
Surface Chemistry ( पृष्ठ रसायन ) - Notes PDF - Irfanullah Mehar - JJ Sir Che...
byWorld of Wisdom
 
Chimeric Antigen Receptors T-cell (CAR-T) & Bispecific T- cell engager (BiTEs)
byrhysudi12
 
PHIVOLCS FAULT FINDER grade - 7 power point
byfelicityguitang
 
Antigens, Haptens and epitopes in immunology.pptx
byGurvinder Kaur
 
Factors affecting Plant Health – Edaphic factors and Biotic Factors
byNistarini College, Purulia (W.B) India
 
Grade 9 - Science Fourth Quarter Physics
byModezaPonje
 
ANIMAL_CELL_,_TISSUE_AND_ORGAN_CULTURE[1].pptx
byAdrijaKrishna
 
Control and Coordination - Short Notes (Prashant Kirad) (1).pdf
byAnu kumari
 
Age of exploration - ppt presentation_EAPT
byericaangelatoledo06
 
Muscles_of_mastication_Anatomy of neck.
bybushra9akhtar786
 
Digestive system grade 5 Cambridge basic sicence
byVictor Venter
 
VIRULENCE FACTOR OF BACTERIA: STRUCTURAL ELEMENT, ENZYMES, TOXINS...
byBIOSPHERE OF KNOWLEDGE
 
Mass selection is one of the oldest methods of plant breeding Commonly use_20...
bysashanklakineni
 
Cell Transport Powerpoint.pptx for high school students
byraobondalakunta25
 
Photogrammetry_Implant_Dentistry_Summary.pptx
byJeldiKusuma
 
Virtualization 1: Virtual Machine, Hypervisor (Virtual Machine Monitor)
bymrethein98
 
Anaerobic respiration (Fermentation, Types, Pasteur effect, Warburg effect, R...
byRashmiMG2
 
Detection of an NH3 Absorption Band at 2.2μm on Europa
bySérgio Sacani
 
Pharmaceutical Calculation Techniques and Terminologies (1).pptx
bymuhammedsajjadhussei
 
Thermodynamics Chapter-2 by Cengel and Boles
byrasoolurfa
 

Computational practices for reproducible science

  • 1.
    Computational practices forreproducible science Ga¨el Varoquaux import science science.reproduce()
  • 2.
    Computational practices forreproducible science Ga¨el Varoquaux This talk: High-level advice on code in science Pointers to good software practices
  • 3.
    Computational practices forreproducible science Ga¨el Varoquaux Outline: 1 Coding for science, reproducibly 2 Making libraries 3 From software back to science
  • 4.
    Where I comefrom PhD in physics Computers: Matlab Instrument control Python for Science: Startup guy Mayavi scikit-learn joblib nilearn Applied maths / machine learning for neuroscience G Varoquaux 2
  • 5.
    1 Coding forscience, reproducibly G Varoquaux 3
  • 6.
    1 Computational reproducibility Computationalreproducibility: Automate everything Control the environment G Varoquaux 4
  • 7.
    1 Automate everything Justa simple matter of programming G Varoquaux 5
  • 8.
    1 Automate everything Justa simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment G Varoquaux 6
  • 9.
    1 Automate everything Justa simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility G Varoquaux 7
  • 10.
    1 Automate everything Justa simple matter of programming Scientific workflow: Based on intuition and experimentation ⇒ Iterative Conjecture Experiment needs consolidation keeping flexibility Lost in iterations G Varoquaux 8
  • 11.
    1 Reproducible computationalscience Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is G Varoquaux 9
  • 12.
    1 Reproducible computationalscience Harder than it seemed Impediments Missing steps / files Libraries have changed Non portable code Statistical / numerical instabilities No one knows where the info is Technical Human Code quality matters Manual steps are evil G Varoquaux 9
  • 13.
    1 How Iwork progressive consolidation Start with a script playing to understand the problem G Varoquaux 10
  • 14.
    1 How Iwork progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function Use functions Obstacle: local scope requires identifying input and output variables That’s a good thing Interactive debugging / understanding inside a function: %debug in IPython Functions are the basic reusable abstraction G Varoquaux 10
  • 15.
    1 How Iwork progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Modules enable sharing between experiments ⇒ avoid 1000 lines scripts + commented code enable testing Fast experiments as tests ⇒ gives confidence, hence refactorings G Varoquaux 10
  • 16.
    1 How Iwork progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Attentional load makes it impossible to find or understand things Where’s Waldo?G Varoquaux 10
  • 17.
    1 How Iwork progressive consolidation Start with a script playing to understand the problem Identify blocks/operations ⇒ move to a function As they stabilize, move to a module Clean: delete code & files you have version control Why is it hard? Long compute times make us unadventurous Know your tools Refactoring editor Version control G Varoquaux 10
  • 18.
    1 Mitigating longcompute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management G Varoquaux 11
  • 19.
    1 Mitigating longcompute times by storing intermediate results Makefiles Accessible intermediate results Break control flow & error management Memoization / caching g = mem.cache(f) # ”joblib” Python module b = g(a) # computes a using f c = g(a) # retrieves results from store Fits in coding / experimentation loop Black-boxy, persistence only implicit Discourages refactoring (avoid recomputing) G Varoquaux 11
  • 20.
    1 A designpattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view Photo-editing software Filters Canvas Tool palettes Typical web application Database Web-pages URLs G Varoquaux 12
  • 21.
    1 A designpattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops G Varoquaux 12
  • 22.
    1 A designpattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control G Varoquaux 12
  • 23.
    1 A designpattern in computational experiments MVC pattern (from Wikipedia Model Manages the data and rules of the application View Output represen- tation Possibly several views Controller Accepts input and converts it to commands for model and view For data science: Numerical, data- processing, & ex- perimental logic Results, as files. Data & plots Imperative API Avoid input as files: not expressive Module with functions Post-processing script CSV & data files Script ⇒ for loops A recipe 3 types of files: • modules • command scripts • post-processing scripts CSVs & intermediate data files Separate computation from analysis / plotting Code and text (and data) ⇒ version control Decouple steps Goals: Reuse code Mitigate compute time G Varoquaux 12
  • 24.
    Adopting software-engineering bestpractices G Varoquaux 13
  • 25.
    1 The ladderof code quality Use static checking in your editor seriously G Varoquaux 14
  • 26.
    1 The ladderof code quality Use static checking in your editor seriously Coding convention, good naming “Any fool can write code that a computer can understand. Good programmers write code that humans can understand” – Martin Fowler Your future self will thank you G Varoquaux 14
  • 27.
    1 The ladderof code quality Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... G Varoquaux 14
  • 28.
    1 The ladderof code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil G Varoquaux 14
  • 29.
    1 The ladderof code qualityIncreasingcost ? Use static checking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... Premature software engineering is the root of reproducible science evil Over versus under engineering Our goal is scientific discovery Experimentation to develop intuitions ⇒ new ideas As the path becomes clear: consolidation Heavy engineering too early freezes bad ideas G Varoquaux 14
  • 30.
    1 LibrariesIncreasingcost ? Use staticchecking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library G Varoquaux 15
  • 31.
    1 LibrariesIncreasingcost ? Use staticchecking in your editor seriously Coding convention, good naming Version control Use git + github Code review Unit testing If it’s not tested, it’s broken or soon will be Make a package controlled dependencies and compilation ... A library Code = Software Well-defined problem = research Functions should work as long as input is valid Targets users, not the authors Made for reuse The workhorse of reproducibility G Varoquaux 15
  • 32.
    1 Libraries enablereproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git G Varoquaux 16
  • 33.
    1 Libraries enablereproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Copying scripts scales very poorly: Accumulation of half-dead code ⇒ cognitive overload No consolidation across studies Each study has different variants of different bugs Garden of forking code G Varoquaux 16
  • 34.
    1 Libraries enablereproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Better: a tag in git Kirstie Whitaker G Varoquaux 16
  • 35.
    1 Libraries enablereproducible science Reproducibility Rerun and come to the same conclusion An argument for copying all scripts for each paper Frozen food Reusability Apply the approach to a new problem Being able to understand, modify, run in new settings Generalization is the true scientific test Reusable computational science = libraries G Varoquaux 16
  • 36.
    2 Making libraries toenable reproducibility G Varoquaux 17
  • 37.
    2 Pick theright battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
  • 38.
    2 Pick theright battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting G Varoquaux 18
  • 39.
    2 Pick theright battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch G Varoquaux 18
  • 40.
    2 Pick theright battles: project scope Project idea A software implementing: i) machine learning and ii) neuroimaging and iii) a graphical user interface and iv) 3D plotting Define project scope and vision Break down projects by expertise Don’t solve hard problems Know the software landscape Need a vision = elevator pitch Your research (PhD) probably does not qualify ⇒ need to cherry-pick contributions G Varoquaux 18
  • 41.
    2 Short cycles,limited ambitions Keep coming back to your users Release early, release often G Varoquaux 19
  • 42.
    2 Simplicity Complexity increasesuperlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity G Varoquaux 20
  • 43.
    2 Simplicity Complexity increasesuperlinearly [An Experiment on Unit Increase in Problem Complexity, Woodfield 1979] 25% increase in problem complexity ⇒ 100% increase in code complexity The 80/20 rule 80% of the usecases can be solved with 20% of the lines of code Avoid feature creep G Varoquaux 20
  • 44.
    2 Quality Quality isfree This is a book, by Philip Crosby G Varoquaux 21
  • 45.
    2 You needquality Quality will give you users Bugs give you bad rap Quality will give you developers Contribute to learn and improve Quality avoids scientific errors Do less, do better Goes against the grant-system incentive G Varoquaux 22
  • 46.
    2 Quality: what& how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test G Varoquaux 23
  • 47.
    2 Quality: what& how Great documentation Simplify, but don’t dumb down Focus on what the user is trying to solve Great APIs Example-based development If something is hard to explain, rethink the concepts Limit the number of different concepts and objects Good numerics Write tests based on mathematical properties When a user finds an instability, write a new test Quality enables reuse Beyond mere reproducibility G Varoquaux 23
  • 48.
    2 Quality assurance Codereview: pull requests Can include newcomers We read each others code Everything is discussed: - Should the algorithm go in? - Are there good defaults? - Are names meaningfull? - Are the numerics stable? - Could it be faster? G Varoquaux 24
  • 49.
    2 Quality assurance Unittesting Everything is tested Great for numerics Overall tests enforce on all estimators - consistency with the API - basic invariances - good handling of various inputs G Varoquaux 25
  • 50.
    2 Dependencies Beyond installation,a challenge is to ensure package versions play way together: correctness of the code Breakage of backward compability yields irreconcilable dependencies G Varoquaux 26
  • 51.
    2 Dependencies andtheir upgrade It’s a fact: users hate upgrading If it ain’t broken, don’t fix it even if it is, apparently G Varoquaux 27
  • 52.
    2 Declaring undependence? Monolythicpackages with no dependencies... But: Scaling is hard Complexity grows as square of codebase size [Woodfield 1979] User support grows with userbase size G Varoquaux 28
  • 53.
    3 From softwareback to science G Varoquaux 29
  • 54.
    3 Beyond computationalreproducibility Make every computational step reproducible, and good science will emerge G Varoquaux 30
  • 55.
    3 Beyond computationalreproducibility Make every computational step reproducible, and good science will emerge Estimating the reproducibility of psychological science [Science 2015] 36% of effects replicate Reasons: Statistical challenges — analysis degrees of freedom Weak insentives — winner’s curse in publication Seldom computational reproducibility G Varoquaux 30
  • 56.
    In practice, thebest way to improve research is to use the right (conceptual) tools. G Varoquaux 31
  • 57.
    3 Managing complexity Inpractice, the best way to improve research is to use the right (conceptual) tools. The everyday roadblock is cognitive load Machine learning, brain anatomy, psychology R, Python, shell scripts Funding agencies, reviewer 3, the Manuscript G Varoquaux 32
  • 58.
    3 Tools thatreduce cognitive overload It’s a design problem G Varoquaux 33
  • 59.
    3 Tools thatreduce cognitive overload Jonathan Ive, an industrial designer, is #4 at Apple Code different. G Varoquaux 34
  • 60.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 35
  • 61.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency np.save(file, obj) pickle.dump(obj, file) fmin(...maxiter=10) lsq linear(...max iter=10) Creates cognitive overload Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 36
  • 62.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes Objects have hidden states, Objects have no universal interface, entry point, output A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 37
  • 63.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts How much do usage patterns carry out across the library? Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 38
  • 64.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Facilitates working with multiple libraries together Easier to get up to speed with a given library Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 39
  • 65.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Change of behavior depending on input type Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 40
  • 66.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Interfaces define objects Incompatible behaviors lead to bugs (eg np.matrix) Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 41
  • 67.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Properties obfuscate the data model of the object Properties can create hidden compute costs Shallow is better than deep Error messages matter Be Pythonic G Varoquaux 42
  • 68.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Objects are understood by their surface Composition creates cognitive overload Error messages matter Be Pythonic G Varoquaux 43
  • 69.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Explain the problem Print the offending value Be Pythonic G Varoquaux 44
  • 70.
    3 Some APIdesign principles for the scipy stack Consistency, consistency, consistency Functions are easier to understand than classes A library should hinge on a small number of concepts Common data containers make the ecosystem stronger Each function should have one and only one purpose Code for interfaces, but don’t overdo duck typing Properties are for impedance matching Shallow is better than deep Error messages matter Be Pythonic Avoid syntax hacks G Varoquaux 45
  • 71.
    3 Scikit-learn API Scikit-learncheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) G Varoquaux 46
  • 72.
    3 Scikit-learn API Scikit-learncheat sheet Scikit-learn Fit and predict >>> estimator = Estimator(param1=param1) >>> estimator.fit(X train, y train) >>> y test = estimator.predict(X test) Transform data >>> X red = estimator.transform(X test) The estimator is a “contract” (slightly more elaborate than above) It has created an ecosystem of packages Based on duck-typing, not inheritence G Varoquaux 46
  • 73.
    3 Example-driven development The3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple G Varoquaux 47
  • 74.
    3 Example-driven development The3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the scikit-learn website: Examples G Varoquaux 47
  • 75.
    3 Example-driven development The3-liner as the new cool Teaching others Teaching yourself Write examples that solve end problems Iterate on your API until these are simple User flow on the nilearn website: Examples G Varoquaux 47
  • 76.
    3 Building greatdocumentation Focus on explaining concepts (hint: write plain English) Less is more: prioritize, avoid redundancy Code examples must be short (link to full tutorial examples) Links everywhere: users will land at the wrong place Teach with the docs Plan for maintenance of docs: Continuous integration Check links Runs examples Doctests G Varoquaux 48
  • 77.
    3 Reusable science scikit-learnis the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html G Varoquaux 49
  • 78.
    3 Reusable science scikit-learnis the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Resource intensive Continuous integration: Data ⇒ Fight for good open data Computation ⇒ Find good algorithms and tradeoffs Forces us to distill the literature (as a review) G Varoquaux 49
  • 79.
    3 Reusable science scikit-learnis the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab G Varoquaux 49
  • 80.
    3 Reusable science scikit-learnis the new machine-learning textbook nilearn is the new neuroimaging review article Experiments reproduced at each commit eg: brain reading nilearn.github.io/auto examples/02 decoding/plot miyawaki reconstruction.html Package development consolidates science and moves it outside the lab scipy-lectures: living book for Python in science G Varoquaux 49
  • 81.
    Coding as ascientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload G Varoquaux 50
  • 82.
    Coding as ascientist Final code should be auditable, ideally reusable Tension between interactive computing & automating Main enemy: cognitive overload In the industry Reusable Verifiable? Not for silicon valley, but in insurance, healthcare, banking... Moving data-scientist code to production? Software projects going over budget? G Varoquaux 50
  • 83.
    @GaelVaroquaux Computational practices forreproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive
  • 84.
    @GaelVaroquaux Computational practices forreproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced
  • 85.
    @GaelVaroquaux Computational practices forreproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization
  • 86.
    @GaelVaroquaux Computational practices forreproducible science Software-engineering skills are superpowers “text-book” software engineering (yet never taught) Make you more productive But beware of over-engineering In software development, keep an eye on the end goal Science is iterative, not every step must be reproduced Science is made by humans, not computers Usability, communication, human factors Design for reuse & generalization Reproducible research must borrow from agile development