Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Managing and Versioning Machine Learning Models in Python

5,349 views

Published on

Practical machine learning is becoming messy, and while there are lots of algorithms, there is still a lot of infrastructure needed to manage and organize the models and datasets. Estimators and Django-Estimators are two python packages that can help version data sets and models, for deployment and effective workflow.

Published in: Data & Analytics

Managing and Versioning Machine Learning Models in Python

  1. 1. 10/29/2016 Data Science Camp, Santa Clara Managing and Versioning Machine Learning Models in Python Simon Frid github.com/fridiculous
  2. 2. Session Overview 1. Motivation 1. Image Recognition Use Case 2. Ad Conversion Use Case 3. Fraud Prediction Use Case 2. Strategies and Design Considerations 1. Data Science Workflow 2. What Can We Learn from Software Version Control 3. Python Tools 4. Solutions 1. Estimators and Django-Estimators 2. Demo
  3. 3. Session Overview 1. Motivation 1. Image Recognition Use Case 2. Ad Conversion Use Case 3. Fraud Prediction Use Case 2. Strategies and Design Considerations 1. Data Science Workflow 2. What Can We Learn from Software Version Control 3. Python Tools 4. Solutions 1. Estimators and Django-Estimators 2. Demo
  4. 4. Disclaimer
  5. 5. Use Case 1: Car Rental Marketplace Identifying Cars/Inventory with Image Recognition
  6. 6. How do we Iterate? ✤ Help clarify features. Improve photo attributes e.g. edge detection. ✤ Human in the loop! ✤ Add computational power & GPUs
  7. 7. Use Case: Image Recognition ✤ Lots of models. ✤ Time to Develop. Time to Deploy. ✤ How do we reference these models? Which one do we choose for production?
  8. 8. Use Case 2: Selling Student Loans Predicting Conversion Rate on Ads
  9. 9. Frequent Training ✤ Yearly Seasonality ✤ Irregular Monthly Effects ✤ Current Activity of the User’s Demographics Matters ✤ A/B testing and Multi-Armed Bandits
  10. 10. Selling Student Loans ✤ Lots of Models ✤ Lot of Trained Versions ✤ Lots of Data “Slicing” Options ✤ How do we Reference Models at training time? How do Reference Models for A/B testing?
  11. 11. Use Case 3: Payment Gateway Predicting Fraudulent Transactions
  12. 12. Fraud Patterns Change over Time ✤ A game of Cat and Mouse
  13. 13. Predicting Fraud ✤ Sudden change in Signature Signal ✤ Forensic Analysis of Obsolete Models ✤ Time Relevance of the Features ✤ How do we …
  14. 14. “There are practical little things in housekeeping which no man really understands.”
  15. 15. Session Overview 1. Motivation 1. Image Recognition Use Case 2. Ad Conversion Use Case 3. Fraud Prediction Use Case 2. Strategies and Considerations 1. Data Science Workflow 2. What Can We Learn from Software Version Control 3. Python Tools 4. Solutions 1. Estimators and Django-Estimators 2. Demo
  16. 16. Concept in Software Version Control Definition Technology Needed Repository Versioning Commits, Tags and Labels Push, Pull and Checkout Diff
  17. 17. Concept in Software Version Control Definition Technology Needed Repository The repository is where files' current and historical data are stored, often on a server. Sometimes also called a depot. Persistance & Serialization Versioning Commits, Tags and Labels Push, Pull and Checkout Diff
  18. 18. Concept in Software Version Control Definition Technology Needed Repository The repository is where files' current and historical data are stored, often on a server. Sometimes also called a depot. Persistance & Serialization Versioning The process of assigning either unique version names or unique version numbers to unique states of computer software. Indexing & Hashing Commits, Tags and Labels Push, Pull and Checkout Diff
  19. 19. Concept in Software Version Control Definition Technology Needed Repository The repository is where files' current and historical data are stored, often on a server. Sometimes also called a depot. Persistance & Serialization Versioning The process of assigning either unique version names or unique version numbers to unique states of computer software. Indexing & Hashing Commits, Tags and Labels A tag or label refers to an important snapshot in time, consistent across many files. These files at that point may all be tagged with a user-friendly, meaningful name or revision number. Attributes & Tags Push, Pull and Checkout Diff
  20. 20. Concept in Software Version Control Definition Technology Needed Repository The repository is where files' current and historical data are stored, often on a server. Sometimes also called a depot. Persistance & Serialization Versioning The process of assigning either unique version names or unique version numbers to unique states of computer software. Indexing & Hashing Commits, Tags and Labels A tag or label refers to an important snapshot in time, consistent across many files. These files at that point may all be tagged with a user-friendly, meaningful name or revision number. Attributes & Tags Push, Pull and Checkout To create a working copy from a repository. With respect to pushing and pulling, a push sends a copy of one repository to another repository. To pull retrieves a copy of a target repository. API to persist and retrieve Diff
  21. 21. Concept in Software Version Control Definition Technology Needed Repository The repository is where files' current and historical data are stored, often on a server. Sometimes also called a depot. Persistance & Serialization Versioning The process of assigning either unique version names or unique version numbers to unique states of computer software. Indexing & Hashing Commits, Tags and Labels A tag or label refers to an important snapshot in time, consistent across many files. These files at that point may all be tagged with a user-friendly, meaningful name or revision number. Attributes & Tags Push, Pull and Checkout To create a working copy from a repository. With respect to pushing and pulling, a push sends a copy of one repository to another repository. To pull retrieves a copy of a target repository. API to persist and retrieve Diff represents a specific modification to a document under version control. The granularity of the modification considered a change varies between version control systems. 😃
  22. 22. Session Overview 1. Motivation 1. Image Recognition Use Case 2. Ad Conversion Use Case 3. Fraud Prediction Use Case 2. Strategies and Considerations 1. Data Science Workflow 2. What Can We Learn from Software Version Control 3. Python Tools 4. Solutions 1. Estimators and Django-Estimators 2. Demo
  23. 23. Algorithm Options ✤ scikit-learn ✤ MILK ✤ Statsmodels ✤ pylearn2 ✤ nolearn ✤ nuPIC ✤ Nilearn ✤ gensim ✤ NLTK ✤ spacy ✤ scikit-image ✤ autolearn ✤ TPOT ✤ crab ✤ XGBoost ✤ pydeap ✤ pgmpy ✤ caffe ✤ tensorflow ✤ keras ✤ gym
  24. 24. Persistence Layer Options ✤ s3 - e.g. s3://bucket/project/model.pkl ✤ GitLFS ✤ Elasticsearch and Document-based Stores ✤ Docker ✤ Pachyderm
  25. 25. Serialization Options ✤ cpickle (py2) and pickle (py3) ✤ sklearn.joblib ✤ dill, cloudpickle and picklable-itertools ✤ PMML via jpmml-sklearn ✤ and what about transformer pipelines?
  26. 26. Indexing & Hashing ✤ Hashing the model ✤ Hashing the data ✤ Relational Database Table for Look Up ✤ Key Value Stores like Redis, Dynamo
  27. 27. Labels ✤ Semantic Versioning, Major.Minor.Patch ✤ Tags (django-taggit) ✤ Storing MetaData, create_dates, relationships between models ✤ Notes and learnings (from Human in the Loops)
  28. 28. API… components… ✤ Custom using an ORM/DAL like django and sqlachemy ✤ SaaS & PaaS - Turi, ScienceOps, PredictionIO, Azure ML ✤ Asynchronous Tasks - Airflow, Luigi, Celery ✤ Flows using Docker and Pachyderm
  29. 29. Session Overview 1. Motivation 1. Image Recognition Use Case 2. Ad Conversion Use Case 3. Fraud Prediction Use Case 2. Strategies and Considerations 1. Data Science Workflow 2. What Can We Learn from Software Version Control 3. Python Tools 4. Solutions 1. Estimators and Django-Estimators 2. Demo
  30. 30. Estimators ✤ a standalone client as an API for your ML repo ✤ current focus “to persist upon prediction” ✤ Uses SQLAlchemy and local filesystem (for now) ✤ github.com/fridiculous/estimators ✤ pip install estimators (pre-alpha development version)
  31. 31. Django-Estimators ✤ an django-extension for ML models ✤ current focus “to persist each object” ✤ Uses Django and local filesystem (for now) ✤ github.com/fridiculous/django-estimators ✤ pip install django-estimators (pre-alpha development version)
  32. 32. Demo
  33. 33. Fin.

×