Event-Driven Model Serving: Stream Processing vs. RPC with Kafka an...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

1Apache Kafka and Machine Learning – Kai Waehner
Event-Driven Stream Processing and
Model Deployment with Kafka and Tensor...

2Apache Kafka and Machine Learning – Kai Waehner
Key Takeaways
• Model deployment can be completely separated from model t...

3Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mach...

4Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mach...

6Apache Kafka and Machine Learning – Kai Waehner
Machine Learning (ML)
...allows computers to find hidden insights without...

7Apache Kafka and Machine Learning – Kai Waehner
The First Analytic Models
How to deploy the models
in production?
…real-t...

8Apache Kafka and Machine Learning – Kai Waehner
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/...

10Apache Kafka and Machine Learning – Kai Waehner
Scalable, Technology-Agnostic Machine Learning Infrastructures
https://w...

11Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

12Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka - A Distributed Commit Log
Producers
Kafka
Cluster
Consumers

13Apache Kafka and Machine Learning – Kai Waehner
Kafka Streams
Your
app
sinksource
KafkaConnect
KafkaConnect
Kafka Cluste...

14Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 P...

15Apache Kafka and Machine Learning – Kai Waehner
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)...

16Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Source Ecosystem as Infrastructure for ML

17Apache Kafka and Machine Learning – Kai Waehner
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams /
K...

19Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

20Apache Kafka and Machine Learning – Kai Waehner
TensorFlow
TensorFlow is an open source software library for high
perfor...

21Apache Kafka and Machine Learning – Kai Waehner
TensorFlow Ecosystem
+ large community
+ integration with most 3rd party...

23Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

24Apache Kafka and Machine Learning – Kai Waehner
Model Deployment with a Model Server
Open source vs. commercial
Self-man...

25Apache Kafka and Machine Learning – Kai Waehner
TensorFlow Serving
• Serve multiple models or multiple versions
simultan...

26Apache Kafka and Machine Learning – Kai Waehner
Stream Processing with External Model and RPC
Streams
Input Event
Predic...

27Apache Kafka and Machine Learning – Kai Waehner
TensorFlow + Kafka Streams
Filter
Map
2) Configure Kafka Streams Applica...

28Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

29Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Application
Input Event
Prediction

30Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Streams
Input Event
Prediction
Str...

31Apache Kafka and Machine Learning – Kai Waehner
Stream processing with embedded model
Input Event
Prediction
Kafka Clien...

32Apache Kafka and Machine Learning – Kai Waehner
TensorFlow + Kafka Streams
Filter
Map
2) Load TensorFlow Model
3) Config...

33Apache Kafka and Machine Learning – Kai Waehner
What Models should be directly embedded into an App?
Examples
• Python M...

34Apache Kafka and Machine Learning – Kai Waehner
Rebuilding the Features of a Model Server is not that hard…
Application
...

35Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

36Apache Kafka and Machine Learning – Kai Waehner
Stream Processing with Model Server vs. Embedded Model
Why use a model s...

37Apache Kafka and Machine Learning – Kai Waehner
Separation of Model Training and Model Inference
Local
Predictions
Model...

38Apache Kafka and Machine Learning – Kai Waehner
Where to Deploy the Model?
• Public Cloud
• VM / Container
• Serverless
...

39Apache Kafka and Machine Learning – Kai Waehner
Traditional Application Deployment
VM
Application
Input Event
Prediction

40Apache Kafka and Machine Learning – Kai Waehner
Traditional Application Deployment – A Kafka Streams App
1) Java Code (K...

41Apache Kafka and Machine Learning – Kai Waehner
Cloud-Native Deployment leveraging Kubernetes

42Apache Kafka and Machine Learning – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8...

43Apache Kafka and Machine Learning – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8...

44Apache Kafka and Machine Learning – Kai Waehner
Model Deployment at the Edge
Questions to discuss
• Edge Data Center vs....

45Apache Kafka and Machine Learning – Kai Waehner
Example: Edge Computing in Telco Business
• StarlingX (https://www.starl...

46Apache Kafka and Machine Learning – Kai Waehner
Agenda
• Challenges of Machine Learning
• Apache Kafka Ecosystem for Mac...

Data Lake
Batch
Analytics
Event
Streaming
Platform
Batch
Integration
Real Time
Pre-
processing
Machine Sensors
Streaming P...

48Apache Kafka and Machine Learning – Kai Waehner
Time
Model BModel A
Producer
Distributed Commit Log
Direct streaming ing...

49Apache Kafka and Machine Learning – Kai Waehner
Model Deployment with Apache Kafka, KSQL and TensorFlow
“CREATE STREAM A...

Spark
Notebooks
(Jupyter)
Kafka
Cluster
Kafka
Connect
KSQL
Machine Sensors
Kafka Ecosystem
Other Components Real Time
Kafk...

54Apache Kafka and Machine Learning – Kai Waehner
Machine Learning + Apache Kafka à Examples @ Github
https://github.com/k...

55Apache Kafka and Machine Learning – Kai Waehner
Key Takeaways
• Model deployment can be completely separated from model ...

56Apache Kafka and Machine Learning – Kai Waehner
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 49 Ad

View on Slideshare

1 of 49 Ad