Recommended
More Related Content
Similar to Data science workflows: from notebooks to production
Similar to Data science workflows: from notebooks to production (20)
Recently uploaded
Recently uploaded (20)
Data science workflows: from notebooks to production
- 1. Data Science Workflows: from notebooks to production Marissa Saunders • 11.01.2019
- 2. Hello
- 3. Thanks to our Sponsors! Partners Premier Logo:
- 4. Overview 1. Data science workflow - an abstraction 2. What is important? 3. Some tools and best practices a. Setting up a workspace b. Managing environments c. Structuring experimental work d. Refining the pipeline e. Annotating data f. Approaches to deployment g. Documentation best practices
- 5. Let’s imagine ... You’re about to start a new project. ● What should you think about before starting? ● How do you organize your workflow? ● Where should your data and code go? ● How do you prioritize experimentation, documentation, clean code, and reproducibility and still have reasonable timelines?
- 6. The workflow, abstracted Data Access Data Processing / Feature Creation Modeling Predictions & Reporting Basic pipeline Exploratory Analysis Experiments Production Process Refinement
- 7. What’s important? Criteria for a good workflow ● Reproducible ● Easy ○ to add / remove features ○ to switch pipeline parts ○ to deploy ○ to come back to 6 months later Data Access Data Processing / Feature Creation Modeling Predictions & Reporting Basic pipeline Exploratory Analysis Experiments Production Process Refinement
- 8. What trade-offs are there? Fast to develop/ iterate Clean code Fast execution Well-documented Scalable
- 9. Setting up a workspace ● Standardized directory structure ● Promotes good development practice ○ Separate exploration,pipeline, and reporting ● Low overhead to get going
- 10. Setting up a workspace cookiecutter-data-science ● http://drivendata.github.io/cookiecutter-data-science ● Based on the cookie cutter package ○ https://cookiecutter.readthedocs.io/en/latest/readme.html ● Standardized directory structure that works well out of the box ● README.md that documents structure ● Standard .gitignore file is set up ● Structure is set up to be pip install-able ● Set-up for Sphinx ● Set-up for tox (standardize testing)
- 11. Setting up a workspace
- 12. Setting up a workspace
- 13. Setting up a workspace
- 14. Managing Environments Build from the environment up ● Package managers ○ virtualenv or conda -> requirements.txt ○ docker or vagrant -> dockerfile / vagrant file ● Keep secrets secret ○ .env and .conf files
- 15. Organizing experimental work Notebooks are for exploration ● Number notebooks for ordering ● Add dates to top of notebooks to help track when changes happen ● For collaboration, add author initials
- 16. Exploration to Experiments to Production ● Natural refinement of the code ○ Notebooks ○ Functions ○ Classes ○ Packages ○ Python scripts ● Other considerations ○ Memory management ■ Data streaming ■ Training in batches ○ Moving between platforms ○ Managing metrics and reporting Exploratory Analysis Experiments Production Process Refinement
- 17. Refining the pipeline Version control, comparing experiments, and reproducibility ● What about git?
- 18. Refining the pipeline Version control and reproducibility ● What about git? ● How about git-LFS?
- 19. Refining the pipeline Version control and reproducibility ● What about git? ● How about git-LFS? ● What about DVC - data version control
- 20. DVC Data science process is a DAG ● DVC keeps track of code, dependencies and output allowing any step to be reproduced if there are upstream changes Separate code storage from data and models ● DVC integrates with git ○ Git stores the code and the dvc files (that store the graph) ○ dvc remotes store the data and the models
- 21. DVC
- 22. Working with DVC Initialize dvc init Configure dvc remote Add files to be tracked by DVC dvc add Store/retrieve data dvc push dvc pull
- 23. Working with DVC Define steps in the DAG dvc run -f sample.dvc -d cmd.py -d input.data -M metric.json -o output.data python cmd.py input.data output.data metrics.json -f: dvc file to store information in -d: any dependencies -M: metric files -o: output files
- 24. DVC file
- 26. Experiments with DVC Make changes with bigrams Set a git checkpoint for the baseline View metrics
- 27. Getting to production So you have a model … and the metrics look good … Now what? ● Human review of results ● Figuring out how to use it in production
- 28. Human review and annotation Viewing, annotating, and storing human reviews can be a challenge Prodigy https://prodi.gy/demo?view_id=ner
- 29. Approaches to deployment Run a model once; Store results in a table One-time ● Simple structure ● Predictions can become dated ● Requires manual updates Run new predictions regularly; Store results in a table Batch ● Loose integration with production ● Little engineering effort required ● Predictions are somewhat up-to-date Run predictions in real- time API ● Realtime predictions ● More engineering and reliability testing required
- 30. Documentation Layers of documentation ● Code comments ● Daily notes / Working notes ● Checkpoints/Summaries ● Code books / how to run ● Project Summary ● Index
- 31. How to put this into practice? ● Checklists ● Routines ● Sticky-note method
- 32. Questions