Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RAPIDS 2018 - Machine Learning in Production - headstart.io

100 views

Published on

Headstart takes reproducibility very seriously.

Our system needs to be fully auditable: our “match score” is a crucial element for candidate selection. At any point in time we need to be able to:
- Access the models that were being used in production when the match score was computed;
- Examine their code (including all upstream ETL/preprocessing pipelines);
- Examine the data they were trained on;
- Be able to deserialize the models and run diagnostics/tests on them.

To support our requirements, we developed our own internal model versioning system using Git, Docker, CircleCI, AWS S3 and Pipenv.

This presentation will share the design, implementation and functionalities of our versioning system, with a detailed walkthrough using our skill recommendation engine as a streamlined running example.

Published in: Data & Analytics
  • Be the first to comment

RAPIDS 2018 - Machine Learning in Production - headstart.io

  1. 1. A practical approach to continuous deployment of Machine Learning pipelines Machine Learning in Production Christos Dimitroulas Fullstack developer & DevOps Luca Palmieri Data Scientist & Machine Learning #rapids2018
  2. 2. Prototyping Production #rapids2018
  3. 3. ProductionOperational requirements => Reproducibility! ● Robustness ● Availability ● Monitoring ● Debugging & auditability
  4. 4. ReproducibilityIngredients ● Code ● Environment ● Data Software
  5. 5. ReproducibilityData ● Training ● Input ● Serialized dataset ● Database queries ● Output ● Serialized models ● Serialized transformers ● Inference...
  6. 6. Use caseSkill recommendation ● Training ● Input ● Users’ skills (MongoDB) ● Output ● Serialized SVD model ● Serialized transformers ● Automated retraining ● Automated deployment on changes
  7. 7. AutomationData Science Culture ● Code quality delivers value ● DataOps? ● Data scientist ownership needs to get closer to production
  8. 8. #rapids2018
  9. 9. #rapids2018
  10. 10. Requirements ● Strict dependency management ● Model performance analysis ● Model versioning ● Infrastructure (as code) ● Data versioning ● Continuous integration/delivery #rapids2018
  11. 11. Dependency Management Lockfile! Managed virtualenv! Reproducible environments! #rapids2018
  12. 12. Model Performance Analysis 1. Does the model meet the basic performance requirements? 2. How does the performance compare to the latest model in production? #rapids2018
  13. 13. Model Versioning Two levels of versioning: 1. Which version of the code produced this model? 2. Which data produced this model? #rapids2018
  14. 14. Infrastructure ● Needs to scale (ideally automatically) ● Should be easy to maintain, change or upgrade ● Must be easily reproducible (for other environments e.g. staging) ● Make it easier for other team members to develop infrastructure #rapids2018
  15. 15. Data Versioning Depends largely on how the data input for the model is created. In our use case, input comes from a database query. ● MongoDB automated backups (snapshots) ● Save relevant snapshots alongside models #rapids2018
  16. 16. Continuous Integration tool ● Trigger a pipeline from code changes ● Schedule jobs ● Run tests & scripts #rapids2018
  17. 17. Continuous Delivery serve-skill-recommender-system Build training container skill-recommender-system Private PyPi server Train new models Assess models Build serving API container Upload models #rapids2018
  18. 18. ● S3 is “primitive” - where is git diff for data? ● Painful transition: prototyping is chaos, production is order sancta sanctorum ● Generalize our process to a proper internal tool #rapids2018

×