Dan Crankshaw
crankshaw@cs.berkeley.edu
http://clipper.ai
https://github.com/ucbrise/clipper
September 7, 2017
A Low-Latency Online Prediction
Serving System
Clipper

Collaborators
Joseph
Gonzalez
Ion
Stoica
Michael
Franklin
Xin Wang
Alexey
Tumanov
Corey Zumar
2
Nishad Singh Yujia Luo Feynman Liang

Big
Data
Training
Learning
Complex Model

Learning Produces a Trained Model
“CAT”
Query Decision
Model

Big
Data
Training
Learning
Application
Decision
Query
?
Serving
Model

Big
Data
Training
Learning
Application
Decision
Query
Model
Prediction-Serving for interactive applications
Timescale: ~10s of milliseconds
Serving

 Challenges of prediction serving
 Clipper overview and architecture
 Deploy and serve models with Clipper
In this talk…

Prediction-Serving
Raises New Challenges

Prediction-Serving Challenges
Support low-latency, high-
throughput serving workloads
???
Create
VWCaffe 10
Large and growing ecosystem
of ML models and frameworks

Models getting more complex
 10s of GFLOPs [1]
Support low-latency, high-throughput serving workloads
Using specialized hardware
for predictions
Deployed on critical path
 Maintain SLOs under heavy load
[1] Deep Residual Learning for Image Recognition. He et al. CVPR

Google
Translate
Serving
82,000 GPUs
running 24/7
[1] https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html
140 billion words a day1
Invented New Hardware!
Tensor Processing Unit
(TPU)

Big Companies Build One-Off Systems
Problems:
 Expensive to build and maintain
 Highly specialized and require ML and
systems expertise
 Tightly-coupled model and application
 Difficult to change or update model
 Only supports single ML framework

Prediction-Serving Challenges
Support low-latency, high-
throughput serving workloads
???
Create
VWCaffe 14
Large and growing ecosystem
of ML models and frameworks

Large and growing ecosystem of ML models and
frameworks
???
Fraud
Detection
15

Large and growing ecosystem of ML models and
frameworks
???
Fraud
Detection
16
Different programming languages:
TensorFlow: Python and C++
Spark: JVM-based
Different hardware:
CPUs and GPUs
Different costs
DNNs in TensorFlow are generally much slower

Large and growing ecosystem of ML models and
frameworks
???
Fraud
Detection
Create VW
Caffe 17
Content
Rec.
Personal
Asst.
Robotic
Control
Machine
Translation

But building new
serving systems for
each framework is
expensive…

Big
Data
Batch Analytics
Model
Training
Use existing systems: Offline Scoring

Big
Data
Model
Training
Batch Analytics
Scoring
X Y
Datastore
Use existing systems: Offline Scoring

Application
Decision
Query
Look up decision in datastore
Low-Latency Serving
X Y
Use existing systems: Offline Scoring

Application
Decision
Query
Look up decision in datastore
Low-Latency Serving
X Y
Problems:
 Requires full set of queries ahead of time
 Small and bounded input domain
 Wasted computation and space
 Can render and store unneeded predictions
 Costly to update
 Re-run batch job
Use existing systems: Offline Scoring

Prediction-Serving Challenges
Support low-latency, high-
throughput serving workloads
???
Create
VWCaffe 24
Large and growing ecosystem
of ML models and frameworks

Prediction-Serving Today
Highly specialized systems for
specific problems
Decision
Query
X Y
Offline scoring with existing
frameworks and systems
Clipper aims to unify these
approaches

What would we like this to look like?
???
Content
Rec.
Fraud
Detection
Personal
Asst.
Robotic
Control
Machine
Translation
Create VW
Caffe 28

Encapsulate each system in isolated
environment
???
Content
Rec.
Fraud
Detection
Personal
Asst.
Robotic
Control
Machine
Translation
Create VW
Caffe 29

Decouple models and applications
Content
Rec.
Fraud
Detection
Personal
Asst.
Robotic
Control
Machine
Translation
30
Caffe
???

Clipper
Predict
MC MC
RPC RPC RPC RPC
Clipper Decouples Applications and Models
Applications
Model Container (MC) MC
Caffe

Clipper Implementation
Clipper
MC MC MC
RPC RPC RPC RPC
Model Abstraction Layer
Caching
Adaptive Batching
Model Container (MC)
Core system: 10,000 lines of C++
Open source system: https://github.com/ucbrise/clipper
Goal: Community-owned platform for model deployment and
serving
Applications
Predict

Clipper
Query Processor
Redis
Clipper
Managemen
t
Applications
Model
Container
Model
Container
Predict
Clipper Implementation
clipper
admin

Driver Program
SparkContext
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Clipper
Clipper Connects Analytics and Serving

Getting Started with Clipper
Docker images available on DockerHub
Clipper admin is distributed as pip package:
pip install clipper_admin
Get up and running without cloning or compiling!

 Containerized frameworks: unified
abstraction and isolation
 Cross-framework caching and batching:
optimize throughput and latency
 Cross-framework model composition:
improved accuracy through ensembles
and bandits
Technologies inside Clipper

Clipper
Predict FeedbackRPC/REST Interface
Caffe
MC MC MC
RPC RPC RPC RPC
Clipper Decouples Applications and Models
Applications
Model Container (MC)

Clipper
Caffe
MC MC MC
RPC RPC RPC RPC
Model Container (MC)

Clipper
Caffe
MC MC MC
RPC RPC RPC RPC
Model Container (MC)
Common Interface  Simplifies Deployment:
 Evaluate models using original code & systems

Container-based Model Deployment
class ModelContainer:
def __init__(model_data)
def predict_batch(inputs)
Implement Model API:

class ModelContainer:
def __init__(model_data)
def predict_batch(inputs)
Implement Model API:
 API support for many programming languages
 Python
 Java
 C/C++
Container-based Model Deployment

Package model implementation and dependencies
Model Container (MC)
Container-based Model Deployment
class ModelContainer:
def __init__(model_data)
def predict_batch(inputs)

Clipper
Caffe
MC MC
RPC RPC RPC RPC
Model Container (MC)
Container-based Model Deployment
MC

Clipper
Caffe
MC MC MC
RPC RPC RPC RPC
Model Container (MC)
Common Interface  Simplifies Deployment:
 Evaluate models using original code & systems
 Models run in separate processes as Docker containers
 Resource isolation: Cutting edge ML frameworks can be buggy

Clipper
Caffe
MC
RPC RPC RPC
Container (MC)
Common Interface  Simplifies Deployment:
 Evaluate models using original code & systems
 Models run in separate processes as Docker containers
 Resource isolation: Cutting edge ML frameworks can be buggy
 Scale-out
MC
RPCRPC RPC
MC MC MC

TensorFlow-
Serving
Predict RPC Interface
Applications
Overhead of decoupled architecture
Clipper
Predict FeedbackRPC/REST Interface
Caffe
MC MC
RPC RPC RPC RPC
Applications
MC MC

Overhead of decoupled architecture
Throughput
(QPS)
Better P99 Latency
(ms)
Better
Model: AlexNet trained on CIFAR-10
*[NSDI 17] Prototype version of Clipper

 Containerized frameworks: unified
abstraction and isolation
 Cross-framework caching and batching:
optimize throughput and latency
 Cross-framework model composition:
improved accuracy through ensembles
and bandits
Technologies inside Clipper

Clipper Architecture
Clipper
Applications
Predict ObserveRPC/REST Interface
MC MC MC
RPC RPC RPC RPC
Model Abstraction Layer
Caching
Latency-Aware Batching
Model Container (MC)

A single
page load
may generate
many queries
Batching to Improve Throughput
 Optimal batch depends on:
 hardware configuration
 model and framework
 system load
 Why batching helps:
Throughput
Throughput-optimized frameworks
Batch size

A single
page load
may generate
many queries
 Optimal batch depends on:
 hardware configuration
 model and framework
 system load
 Why batching helps:
Throughput
Throughput-optimized frameworks
Batch size
Batching to Improve ThroughputLatency-aware
Clipper Solution:
Adaptively tradeoff latency and
throughput…
 Inc. batch size until the latency objective
is exceeded (Additive Increase)
 If latency exceeds SLO cut batch size by

Throughput
(QPS)
Better
Batching to Improve ThroughputLatency-aware

Overhead of decoupled architecture
Clipper
Predict FeedbackRPC/REST Interface
Caffe
MC MC MC
RPC RPC RPC RPC
Applications
MC

 Containerized frameworks: unified
abstraction and isolation
 Cross-framework caching and batching:
optimize throughput and latency
 Cross-framework model composition:
improved accuracy through ensembles
and bandits
Technologies inside Clipper

Clipper Architecture
Clipper
Applications
Predict ObserveRPC/REST Interface
MC MC MC
RPC RPC RPC RPC
Model Abstraction Layer
Model Container (MC)
Model Selection LayerSelection Policy

Clipper
Version 1
Version 2
Version 3
Periodic retraining
Experiment with new
models and frameworks
Model Selection LayerSelection Policy

“CAT
”“CAT
”“CAT
”“CAT
”
“CAT
”
CONFIDE
NT
Selection Policy: Estimate confidence
Policy
Version 2
Version 3

“CAT
”“MAN
”“CAT
”“SPAC
E”
“CAT
”
UNSURE
Selection Policy: Estimate confidence
Policy

Clipper
Model Selection LayerSelection Policy
Exploit multiple models to estimate confidence
 Use multi-armed bandit algorithms to learn
optimal model-selection online
 Online personalization across ML frameworks
*See paper for details [NSDI 2017]

Deploying Models
with Clipper

Driver Program
Predict function
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Clipper
Goal: Connect Analytics and Serving
Analytics
Cluster

You can always implement your own model container…
Model Container (MC)
class ModelContainer:
def __init__(model_data):
def predict_batch(inputs):

Can we do
better?

Driver Program
SparkContext
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Clipper
Deploy Models from Spark to Clipper

Driver Program
SparkContext
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Deploy Models from Spark to Clipper
Clipper

MC
Driver Program
SparkContext
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Deploy Models from Spark to Clipper
Clipper

Problem: Models
don’t run in
isolation
Must extract model plus pre-
and post-processing logic

Clipper provides a library of model deployers
 Deployer automatically and intelligently saves all
prediction code
 Captures both framework-specific models and
arbitrary serializable code
 Replicates training environment and loads prediction
code in a Clipper model container
You will use these in the Clipper exercise

Clipper provides a (growing) library of model deployers
Python
Combine framework specific models with external
featurization, post-processing, business logic
Currently support Scikit-Learn and PySpark
TensorFlow, Caffe2, Keras, coming soon
Scala and Java with Spark:
both MLLib and Pipelines APIs
R support coming in 0.2 (soon!)

Driver Program
Predict function
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Worker Node
Executor
Task Task
Web Server
Database
Cache
Clipper
Goal: Connect Analytics and Serving
Analytics
Cluster

MC
Web Server
Database
Cache
Run on Kubernetes alongside other applications
Clipper
Other
applications
Other
applications
Other
applications
Other
applications

Status of the project
 0.2.0 will be released right after RISE Camp
 Actively working with early users
 Post issues and questions on GitHub and subscribe to our
mailing list clipper-dev@googlegroups.com
http://clipper.ai

What’s next?
 Expand library of model deployers
 Improved model and application reliability and debugging
 Joint work with the Ground project
 Bandit methods, online personalization, and cascaded
predictions
 Studied in research prototypes but need to integrate into system
 Actively looking for contributors:
https://github.com/ucbrise/clipper