Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deployment Design Patterns - Deploying Machine Learning and Deep Learning Models into Production

258 views

Published on

Presented at All Things Open 2018
Presented by Dan Zaratsian with Hortonworks
10/22/18 - 3:15 PM - Big Data Track

Published in: Technology
  • Be the first to comment

Deployment Design Patterns - Deploying Machine Learning and Deep Learning Models into Production

  1. 1. Confidential & Proprietary Deployment Design Patterns Deploying Machine Learning Models into Production Dan Zaratsian, Cloud Solutions Engineer @ Google https://github.com/zaratsian October 2018
  2. 2. Confidential & Proprietary Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also using Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  3. 3. Confidential & Proprietary Why is this Important? ● Too much time is lost before deploying. ● Deployments are not scalable. ● Deployments are not easily maintained & updated. ● Deployed models are not monitored. ● Too many models live (and die) on laptops. ● Deployment process is not understood.
  4. 4. Confidential & Proprietary Dan Zaratsian Cloud Solutions Engineer @ Google University of Akron B.S. Electrical Engineering North Carolina State University M.S. Advanced Analytics
  5. 5. Confidential & Proprietary Data Scientist Production Environment End Users (Customers) Value ML Model ML Model Ideal Scenario (Simplified)...
  6. 6. Confidential & Proprietary Data Scientist Production Environment End Users (Customers) No Value ML Model A Typical Scenario (What is commonly done)... Lots of really great code in a Notebook, but may not easily deployed.
  7. 7. Confidential & Proprietary Why is this a challenge? A good model may look something like this... Model Score Code Date Quarter Down YardsToGo PlayType 2018-09-16 1 4 6 Run 2018-09-23 4 1 10 Pass 2018-10-07 4 3 13 Pass 2018-10-14 3 1 10 Run Predicted Yards Gained 6.15 yards -0.90 yards 3.95 yards 1.50 yards Com h iv M Pip e (mo b e t / s o c e)
  8. 8. Confidential & Proprietary Pre c in & Fe t e E g e n Why is this a challenge? It’s common to use dummy variables, standardize values, transform, etc... Model Score Code Date Quarter Down YardsToGo PlayType 2018-09-16 1 4 6 Run 2018-09-23 4 1 10 Pass 2018-10-07 4 3 13 Pass 2018-10-14 3 1 10 Run Predicted Yards Gained 6.15 yards -0.90 yards 3.95 yards 1.50 yards Month Quarter Down YardsToG o PlayType_ Run PlayType_ Run 9 1 4 6 1 0 9 4 1 10 0 1 10 4 3 13 0 1 10 3 1 10 1 0 In o p (or ) M ip e
  9. 9. Confidential & Proprietary Why is this a challenge? Or in many cases, the input is a sparse matrix, heavily feature engineered... Date Quarter Down YardsToGo PlayType 2018-09-16 1 4 6 Run 2018-09-23 4 1 10 Pass 2018-10-07 4 3 13 Pass 2018-10-14 3 1 10 Run Predicted Yards Gained 6.15 yards -0.90 yards 3.95 yards 1.50 yards Pre c in & Fe t e E g e n Model Score Code In o p (or ) M ip e
  10. 10. Confidential & Proprietary A few thoughts... ● Model deployment can mean different things across organizations. ● It’s complex and there’s a need for cross domain knowledge. ● Requirements vary by use case, industry, organization. ● There’s typically no one right way to deploy models.
  11. 11. Confidential & Proprietary Data Scientist Production Environment End Users (Customers) Value ML Model ML Model Deployment Considerations ● What tool was used? ● What programming language? ● What are the dependencies? ● Is the model pipelined generalized? ● Can it easily be deployed across OS, clouds, on-prem, etc? ● Is there a process for promoting new models into production? ● How often are models updated or retrained? ● Where is this server(s) hosted? ● How do we scale? ● How do we secure our apps? ● How do we govern the pipeline? How do end users consume our model?
  12. 12. Confidential & Proprietary ● How will your models be developed (code, drop-and-drag, hybrid)? ● How will your models be deployed? ○ As a batch process ○ Available via API ○ Real-time / Online Stream ○ Recoded (with model coefficients, logic, etc.) ● What scheduler is being used? ● Is your scheduler able to load and deploy your model pipeline? ● How often will the model be retrained? ● Do you have a process in place to monitor model performance? ● Using docker & kubernetes, serverless, ...? Deployment Considerations (Continued)
  13. 13. Confidential & Proprietary Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also using Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  14. 14. Train ML Model Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also available using Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  15. 15. Confidential & Proprietary ML Model Training Process (Simplified) Data Prep & Feature Eng. Model (model.joblib) Model Training Model Evaluation ML Client Environment Data Scientist Database End Users (Customers) Training Data
  16. 16. Batch Deployment Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also available using Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  17. 17. Confidential & Proprietary Reference Architecture: Batch Scoring Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Training DataDatabase
  18. 18. Confidential & Proprietary Reference Architecture: Batch Scoring Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Model (model.joblib) Training DataDatabase Scored Records Database Records Deployed on Server, executed with Scheduler
  19. 19. Deploy as Web App Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also in Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  20. 20. Confidential & Proprietary Reference Architecture: Deploy as Web App Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Training DataDatabase
  21. 21. Confidential & Proprietary Accept and Process Request POST Reference Architecture: Deploy as Web App Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Model (model.joblib) Training DataDatabase Response Database Records Web Framework (Simplified Architecture)
  22. 22. Confidential & Proprietary
  23. 23. Deploy as Web Service (Serverless) Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also in Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  24. 24. Confidential & Proprietary Reference Architecture: Deploy as Web Service (API) Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Training DataDatabase
  25. 25. Confidential & Proprietary Accept and Process Request API Call Reference Architecture: Deploy as Web Service (API) Data Prep & Feature Eng. Model (model.joblib) Data Scientist Model Training Model Evaluation ML Client Environment End Users (Customers) Model (model.joblib) Training DataDatabase Response Web Service (API) (Simplified Architecture)
  26. 26. Confidential & Proprietary
  27. 27. Automated Model Build & Deploy Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also in Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News)
  28. 28. Confidential & ProprietaryConfidential & Proprietary UPDATEDEPLOYEVALUATETUNE ML MODEL PARAMETERS ML MODEL DESIGN DATA PREPROCESSING Cloud AutoML A technology that can automatically creates a Machine Learning Model UPDATEDEPLOYEVALUATETUNE ML MODEL PARAMETERS ML MODEL DESIGN DATA PREPROCESSING
  29. 29. Confidential & Proprietary Why Cloud AutoML? Your own custom models Simple Limited ML expertise needed High quality Confidential & Proprietary
  30. 30. © 2018 Google LLC. All rights reserved. AutoML Training Dataset Train Deploy Serve Generate predictions with a REST API AutoML to the rescue
  31. 31. Confidential & Proprietary Deployment Design Patterns 1. Train ML Model (using sklearn) 2. Deploy as Batch 3. Deploy as Web App (also using Spark) 4. Deploy as Web Service (Serverless) 5. Automated Model Build & Deploy (NLP, News Sources)
  32. 32. Tha s! Dan Zaratsian http://github.com/zaratsian

×