TensorFlow 16: Building a Data Science Platform

1. Building a Data Science Platform
@dcos

2. Why is machine learning taking off?
2

3. CONFIDENTIAL

4. CONFIDENTIAL
2013: $14B fraud losses, 19% YoY growth

5. The case for a data science platform.
5

6. What you want to be doing
6
Data
(clean)
Write awesome ML code
Train
(once)
Deploy
(not me)

7. 7
Sculley, D., Holt, G., Golovin, D. et al. Hidden Technical Debt in Machine Learning Systems
What you’re actually doing

8. © 2018 Mesosphere, Inc. All Rights Reserved.
Deep Learning: The Promise
8

9. © 2018 Mesosphere, Inc. All Rights Reserved.
Deep Learning: The Process
9
Step 1: Training
(In Data Center - Over Hours/Days/Weeks)
Step 2: Inference
(Endpoint or Data Center - Instantaneous)
Dog
Input:
Lots of Labeled
Data
Output:
Trained Model
Deep neural
network model
Trained
Model
Output:
Classification
Trained Model
New Input from
Camera or
Sensor
97% Dog
3%
Panda

10. © 2018 Mesosphere, Inc. All Rights Reserved.
Deep Learning: Some insight
10

11. © 2018 Mesosphere, Inc. All Rights Reserved.
Deep Learning: The Challenges
11

12. Open Source Pipeline
Operationalizing a Machine
Learning model can be super hard.
It is a stage where most enterprise
Machine Learning projects fail. I
cannot tell you how many
companies I've talked to, who have
said their innovation teams had
devised these cool ML projects, but
they were struggling getting the ML
models into production. In this set
of courses, we will talk about how
to train, deploy, and predict with ML
models in a way that their
production ready. And finally, we
delve back into Machine Learning
theory.
Valliappa Lakshmanan.
Tech Lead for Big Data and Machine
Learning Professional Services on Google
Cloud Platform.

13. 1. Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

14. 1. Data Preparation using
Spark
7. Streaming of requests
...
Public Cloud Pipeline
Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

15. 1. Data Preparation using
Spark
7. Kafka stream of
requests
DIY Open Source Pipeline
1. Model
Engineering
2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

16. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

17. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

18. © 2018 Mesosphere, Inc. All Rights Reserved. 18
Challenges
●
● Training/Dev/Test + New Data
● Large amounts
● Quality
● Availability (for cluster)
● Velocity
● Streaming
Solutions
GFS
Input Data Management
Input:
Lots of Labeled
Data
Apache Kafka
Apache Cassandra

19. © 2018 Mesosphere, Inc. All Rights Reserved. 19
Challenges
● Data is typically not ready to be
consumed by ML job*
● Data Cleaning
● Missing/incorrect labels
● Data Preparation
● Same Format
● Same Distribution
Solutions
Data Preparation
* Demo datasets are a fortunate exception :)

20. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

21. © 2018 Mesosphere, Inc. All Rights Reserved. 21
Challenges
● Different Users/Use cases
● Data Analyst/Exploring
● Production Workloads
● Highly Optimized
● How to spawn Environments?
Solutions
Users

22. © 2018 Mesosphere, Inc. All Rights Reserved. 22
Challenges
● Different Users/Use cases
● Data Analyst/Exploring
● Production Workloads
● Highly Optimized
● How to spawn Environments?
Solutions
Users

23. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

24. © 2018 Mesosphere, Inc. All Rights Reserved.
TensorFlow...
24

25. © 2018 Mesosphere, Inc. All Rights Reserved. 25
Data Analytics Ecosystem

26. © 2017 Mesosphere, Inc. All Rights Reserved.
Typical Developer Workflow for TensorFlow (Distributed)
● …
● Provision a set of machines to run your computation
● Install TensorFlow on them
● Write code to map distributed computations to the exact IP address
of the machine where those computations will be performed
● Deploy your code on every machine
● Train your data on the cluster → Output Trained Model
26
Trained
Model
Input
Data Set

27. © 2018 Mesosphere, Inc. All Rights Reserved.
Resource Isolation and Allocation
27

28. © 2017 Mesosphere, Inc. All Rights Reserved. 28
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Tensorflow
Jenkins
Kafka
Spark
Tensorflow

29. © 2018 Mesosphere, Inc. All Rights Reserved.
PHYSICAL
INFRASTRUCTURE
MICROSERVICES, CONTAINERS, & DEV TOOLS
VIRTUAL MACHINES PUBLIC CLOUDS
DATA SERVICES, MACHINE LEARNING, & AI
Security &
Compliance
Application-Aware
Automation Multitenancy
Hybrid Cloud
Management
100+
MORE
DatacenterEdge
Datacenter and Cloud as a Single Computing Resource
Powered by Apache Mesos
20+
MORE

30. Deploy
Scale
Configure
Recover
3 AM
...
Typical Datacenter
siloed, over-provisioned servers,
low utilization
HDFS
Kafka
Kubernetes
Flink
TensorFlow

31. © 2018 Mesosphere, Inc. All Rights Reserved.
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master
31
MESOS ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Flink
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
CDB
Executor
Spark
Task
Spark
Scheduler
Kafka
Scheduler

32. © 2017 Mesosphere, Inc. All Rights Reserved.
Running distributed TensorFlow on DC/OS
32
● Wrapper script to abstract away distributed TensorFlow configuration
○ Separates “deployer” responsibilities from “developer” responsibilities
{
"service": {
"name": "mnist",
"job_url": "...",
"job_context": "..."
},
"gpu_worker": {... },
"worker": {... },
"ps": {... }
}
User
Code
Wrapper
Script

33. © 2017 Mesosphere, Inc. All Rights Reserved.
Running distributed TensorFlow on DC/OS
33
● The dcos-commons SDK cleanly restarts failed tasks and reconnects
them to the cluster

34. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

35. © 2018 Mesosphere, Inc. All Rights Reserved. 35
Challenges
● Many Models
● Different Hyperparameter
● Different Models
● New Training Data
● ...
Solutions
● Persistent Storage + Metadata
Model Management
GFS

36. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

37. © 2018 Mesosphere, Inc. All Rights Reserved. 37
Challenges
● How to Deploy Models?
● Zero Downtime
● Canary
Solutions
● TensorFlow Serving
Model Serving

38. © 2018 Mesosphere, Inc. All Rights Reserved.
Rendezvous Architecture
38
https://mapr.com/ebooks/machine-learning-logistics/

39. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

40. © 2018 Mesosphere, Inc. All Rights Reserved.
● DC/OS TensorFlow Package (currently closed source)
○ https://github.com/mesosphere/dcos-tensorflow
● DC/OS TensorFlow Tools
○ https://github.com/dcos-labs/dcos-tensorflow-tools/
● Tutorial for deploying TensorFlow on DC/OS
○ https://github.com/dcos/examples/tree/master/tensorflow
● Contact:
○ https://groups.google.com/a/mesosphere.io/forum/#!forum/tensorflow-dco
s
○ Slack: chat.dcos.io #tensorflow
Questions and Links
40