Downloaded 166 times
Uploaded bySeldon
Seldon: Deploying Models at Scale
Seldon provides an open platform for deploying machine learning models at scale. It helps companies bring machine learning to life through its Seldon Core platform, which provides a control plane for managing ML workflows and inference graphs. Seldon Core supports deploying models built with any ML framework or language and integrates with common serving systems. It also includes powerful processing components for routing traffic, transforming data, and monitoring models.
More Related Content
Similar to Seldon: Deploying Models at Scale
Seldon: Deploying Models at Scale
- 1.
- 2. Machine Learning Deployment forEnterprise. O’Reilly AI for IBM: Deploying ML Models at Scale
- 3. We help peopleshape the future by bringing machine learning deployment to life.
- 4. Meet the Team. AlexHousley CEO & Founder Clive Cox PhD CTO Gurminder Sunner VP Engineering Lee Baker Commercial Director Giovanni Vacanti PhD Machine Learning Engineer Inga Veidmane Operations Manager Andrew Turner Sales Manager Janis Klaise PhD Data Scientist Arnaud Van Looveren Data Scientist @seldon_io hello@seldon.io bit.ly/SeldonCoreSlack
- 5. Mapping a globalnetwork of Seldon installs Organisers of the UK’s largest applied meetup focused on their AI/ML tools - now over 2000 members. Supported by Google. Organic traction and a global community. ● Over 120,000 installs (Docker Hub pulls) since Dec ‘17. ● Community across 90 countries and across industries. ● 21% MoM growth in clusters, 138% MoM growth in nodes. ● 98% growth in nodes per cluster is an indicator that people are using it for more demanding workloads. bit.ly/SeldonCoreSlack
- 6. Seldon serves theworld’s largest companies in technology and financial services. Applied data science project followed by managed service. Enterprise Sales - Recommend products and supporting documentation to sales reps. - Increase performance of sales reps and campaigns. 1% increase = £52m revenue. Credit Risk - Reduce credit default amongst UK Card customers. - Support the customer before they miss payments. - Aiming to reduce default rate by 2%. FX Trading - Identify which way currency markets are likely to move within a short timeframe. - 5% uplift over baseline model worth estimate £3m / month. Churn Prediction - Improve customer retention by identifying customers likely to cancel. - Aiming to increase accuracy of churn prediction by 10%. NA Bank
- 7.
- 8. Source: Notes fromthe AI frontier: Applications and value of deep learning - McKinsey April 2018 - https://mck.co/2I5tMld A Small Number of Machine Learning Techniques work across many use cases and industries.
- 9. Source: Notes fromthe AI frontier: Applications and value of deep learning - McKinsey April 2018 - https://mck.co/2I5tMld Deep learning and Reinforcement learning deliver 62% incremental value on average.
- 10. High value MLuse cases “$20.2bn machine learning budget in banking in 2018.” McKinsey 2018 Fraud Detection Compliance & Risk Capital Markets Customer Experience Customer Engagement Identify fraudulent behaviour ● First and third party ● Identify bad actors ● KYC ● Outlier detection Reduce risks and streamline reporting ● Audit trails ● Early warning and response to cyber breach ● Automate risk controls AI in the investment sector ● Fundamental and technical trading strategies ● Market predictions ● FX, equities and derivatives Identifying patterns of behaviour ● Product recommendation ● Customer retention and churn prediction ● Customer journey Customized and intelligent products & services ● Chatbots & virtual agents ● Task management ● CRM ● Personalisation
- 11. Finance + tech nicheskill sets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Regulatory requirements Auditability & provenance 6 Machine Learning Trends Seldon observe
- 12. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Regulatory requirements Auditability & provenance 6 Machine Learning Trends in Financial Services Evolving Standards R Regulatory compliance a group effort. ML without a ‘human in the loop’ will require Explainable AI, or ‘XAI’ Expect to see compliance teams adopt ML Possible driver of M&A within the banking community
- 13. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Regulatory requirements Auditability & provenance 6 Machine Learning Trends in Financial Services Data management ● R Transparency of data governance Regulate the Algo - ‘Play the ball, not the man’ Cloud portability - *Euro bank example of conflict Vital nature of successful Data strategies - "We don’t have better algorithms, we just have more data." Peter Norvig, Google's director of research
- 14. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Auditability & provenance 6 Machine Learning Trends in Financial Services Security ● R As Cloud becomes standard, data & training will become increasingly separate Initiatives like the CNCF* will ally with traditional regulatory bodies Models will be a hybrid of open (Risk, KYC) and proprietary (Trading, Fraud) *Cloud Native Computing Foundation https://www.cncf.io
- 15. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Auditability & provenance 6 Machine Learning Trends in Financial Services Citizen ‘Data Scientist’ $120,000 - Data Scientist annual salary* Best tool for the job - commoditization of ML frameworks ForwardLane seeing portfolios of >200 under management Curation will be the key skillset *Glassdoor Salary Estimates FSI
- 16. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Regulatory requirements Auditability & provenance 6 Machine Learning Trends in Financial Services Speed to market Cost & speed of compute is your new competitive advantage New Revenue models - Can you extract meaning from your data? Model iteration needs to be rapid and optimal
- 17. Finance + tech nicheskillsets Low latency & high data volumes Integration with legacy ecosystem Security & data governance Regulatory requirements Auditability & provenance 6 Machine Learning Trends in Financial Services Legacy tech Key value of ‘Challenger Bank’ community Growth in outsourced banking engine New IT landscape demands ‘plug & play’ architecture Open Source adoption to escalate
- 18. 2018 Market Analysis MachineLearning Deployment ● 25000 banks forecasted to spend $519bn on IT in 2018 (Gartner) ● 26% early adopters of ML spent 15% of their IT budget on ML ● $20.24bn ML budget in banking 2018 ● 52% initial and broad ML implementation ● 71% Kubernetes adoption in orchestration ● $7.47bn current ML budget for Kubernetes users in banking
- 19. Seldon Core Open platformfor deploying machine learning models
- 20. Seldon Focus Seldon Core Building a Model ServingOptimizationMonitoringRoll-out Data Ingestion Data Analysis Data Transform -ation Data Validation Data Splitting Model Validation Trainer Training At Scale Logging
- 21. Goal: Help Businesssucceed with Data Science Data Scientist ● Analyzes the data ● Builds the predictive model ● Optimizes the model DevOps Engineer ● Manages infrastructure ● Monitors the model in production ● First response on issues Business Manager ● Decides the project goals ● Defines business KPIs ● Evaluates ROI ● Provides Approval/Audits
- 22.
- 23. Seldon Deploy UI, Collaboration,Control, Audit MAB (Multi-Arm Bandits) Outlier Detection Explanation Seldon Core (ML Control Plane) 80% of enterprises are hybrid or multi-cloud. Over half use containers, of which 75% use Kubernetes. Seldon Technology Stack Cloud Native Tools : e.g., Ambassador, Argo, Istio Bias Detection, Concept Drift
- 24. Seldon Core :ML Tool Agnostic Use any ML Tool Dockerise REST and gRPC Handle Seldon APIs Deploy! Deploy!
- 25. Seldon Core :ML Tool Agnostic Any Language Current wrappers incl: ● Python ● R ● Java ● NodeJS Any ML Library Current examples incl: ● SKLearn ● TensorFlow ● PyTorch ● H2O Any ML Inference Engine Current examples incl: ● Seldon wrapped inference code ● NVIDIA TensorRT Server ● TensorFlow Serving ● Intel nGraph
- 26. Title Type Here TypeHere Type Here Type Here Seldon Core Wrappers : Source-to-Image https://github.com/openshift/source-to-image from sklearn.externals import joblib class IrisClassifier(object): def __init__(self): self.model = joblib.load('/mnt/model/IrisClassifier.sav') def predict(self,X,features_names): return self.model.predict_proba(X) User Source Code (Github/Local) Seldon Builder Images seldonio/seldon-core-s2i-python2 seldonio/seldon-core-s2i-python3 seldonio/seldon-core-s2i-r seldonio/seldon-core-s2i-java-build seldonio/seldon-core-s2i-nodejs ● ./assemble ● ./run ● ./usage Dependencies S2I User Image Ready to deploy on seldon-core #> s2i build .
- 27. Rich Processing Components ●Routing requests ○ AB Tests ○ Multi-Armed Bandit ● Transformations ○ Feature Normalization ○ Ensembles results ● Metrics ○ Concept drift ○ Outlier detection Seldon Core : Powerful Inference Graphs Flexible ● Custom ○ Build your own ○ 3rd Party ● Dynamic ○ Change graph at inference time ● Pluggable ○ Reuse components in different inference graphs
- 28. Title Type Here TypeHere Type Here Type Here Seldon Core Inference Graphs Seldon/3rd party component Model C Model A Feature Transformation Client component API (REST, gRPC) Model B A/B Test Multi Armed Bandit Direct traffic to the most optimal model Outlier Detection Key features to identify outlier anomalies (Fraud, KYC) Explanation Why is the model doing what it’s doing?
- 29. Title Type Here TypeHere Type Here Type Here Seldon Core Example Inference Graph Seldon/3rd party component Tensorflow model served via NVIDIA TensorRT Server ONNX model served via Intel nGraph Feature Transformation in Java Client component API (REST, gRPC) R Model A/B Test Multi Armed Bandit (e-greedy,python) Outlier Detection Custom Router using business logic
- 30. Model Explanation For example,a machine learning model flags a bank transaction as fraudulent; it becomes necessary to explain what factors contributed to the decision. Explaining complex "black box" models is a key problem. Understanding why a model output a given prediction can…. 1. Provide insight into the model behaviour. Guide decision making processes. 2. Facilitate compliance with regulation. Lime ● Locally interpretable ● Suited for images and text Representation erasure ● Suited for run time explanations ● Return features importance for any “black box” model ● Based on how much the model prediction will change when then input features are changed Lime2 ● Updated version of Lime
- 31. Title Type Here TypeHere Type Here Type Here Multi-Armed Bandits Model C Model AAPI Model B A/B Test Multi Armed Bandit Direct traffic to the most optimal model ● Deploy several models in parallel ● Optimize between models at inference time ● Generalizes A/B testing ● Exploration vs Exploitation ○ ε-greedy ○ Thompson Sampling ● State dependency ○ Contextual bandits
- 32. ● Outlier Detection ○Identify data anomalies ○ Measure the distance of an observation to the center of the distribution ○ Online learning, starts without knowledge of the feature distribution ● Concept Drift ○ Change in the relationship between input and output leads to lower prediction accuracy ○ Detection of drift allows to adjust the predictive models in time ● Bias Detection ○ Human bias in the data like gender bias or machine bias leads to biased predictions ○ Neutralizing bias avoids pitfalls of human decision making Monitoring Machine Learning Models
- 33. Seldon Core :Production Deployment Options Update Inference Graphs ● Rolling ● Canary ● Blue-Green* ● Shadow* Resource Control ● GPUs, CPUs, Memory ● Data Volume Attachment ● Sidecar containers ○ DBs, Other Business logic ● Separate k8s deployments for parts of graph * roadmap
- 34. Example Seldon DeploymentManifest (Kubernetes Custom Resource) Graph Definition Pod Specification Replicas List of predictors
- 35. Title Type Here TypeHere Type Here Type Here 1. Package Create REST or gRPC dockerized microservice 2. Describe Deployment Create/update Kubernetes resource manifest for deployment graph 3. Deploy Manage and analyze the performance of live deployments Seldon Deployment Workflow
- 36. Title Type Here TypeHere Type Here Type Here Get Started with Seldon Core 1. Install a. https://github.com/SeldonIO/seldon-core b. Helm Charts or Ksonnet Registry c. Via GCP MarketPlace d. Via Kubeflow e. Via IBM FfDL 2. Run example charts a. Helm : https://github.com/SeldonIO/seldon-core/blob/master/notebooks/helm _examples.ipynb b. Ksonnet: https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kson net_examples.ipynb c. Provides example graphs - models, AB tests, multi-armed bandits, Outlier detections 3. Customize to your models a. Wrap components using S2i b. Create your own graphs
- 37. Seldon Deploy Machine LearningDeployment for Data Science Teams. Development in progress Pilots Q4 2018, launch early 2019 >500 companies applied for beta ● Modern UX/UI for model management ● Manage multiple Seldon Core clusters ● Team workflows, approvals and audit trails ● Advanced experiments and CI/CD (GitOps) ● Explanations and compliance (GDPR)
- 38. Thank You. Please getin touch with any further questions. @seldon_io hello@seldon.io http://bit.ly/SeldonCoreSlack
- 39. Title Type Here TypeHere Type Here Type Here Seldon Core Architecture Data scientists, engineers and managers Deployment Controller (kubectl, CI/CD, Seldon Deploy) Business Applications Pluggable Authentication REST API or gRPC Kubernetes clusters running Seldon Core Service Orchestrator Kubernetes API Operator 1. N deployment graphs Reverse Proxy (Ambassador) Public docker registry Client docker registry Seldon docker registry