Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Streaming all over the World
Real-Life Use Cases & Architectures for Event Streaming
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

Global Scale
Real-time
Persistent Storage
Stream Processing
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)

Agenda
1) Financial Services
2) Insurance
3) Manufacturing
4) Automotive
5) Telecom
6) Retailing / Transportation / Logistics
7) Gaming
8) Healthcare / Pharma / Life Sciences

Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html

AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic

https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/transforming-security-posture-with-innovations-in-data-intelligence-paper.pdf

“… rescue data off of the mainframe, in a cloud native,
microservice-based fashion … [to] … significantly reduce the
reads on the mainframe, saving RBC fixed infrastructure
costs (OPEX). RBC stayed compliant with bank regulations
and business logic, and is now able to create new applications
using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/

Mainframe Offloading
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

Mainframe Replacement
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
MIPS / MSU
MIPS / MSU
MIPS / MSU
Read
No MIPS / MSU

Integration Platform
for legacy and modern technologies
https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf
Integration Platform
for legacy and modern technologies

Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical
operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and ConfluentReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cross-Company Supply Chain Integration
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 2
Supplier
OEM Streaming integration
between companies
API Management
(REST et al) is not
appropriate for
streaming data
Infosec and politics are
your biggest hurdle
Tier 1
Supplier

Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with Kafka, Tiered Storage and TensorFlow IO
https://github.com/tensorflow/io
22
Model X
(at a later time)

Four
Telecom
scenarios
for 2030
https://www2.deloitte.com/content/dam/Deloitte/pl/
Documents/Reports/pl_Deloitte_TMT_Telco_2030.pdf

From Monoliths to decoupled, independent services
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-
media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
(Network Functions Virtualization)

Hotstar
OTT for millions of cricket fans in India
https://www.confluent.io/kafka-summit-san-francisco-2019/scaling-for-indias-cricket-hungry-population/

OSM - Open Source MANO
Interoperability among different
service provider NFV infrastructures and OSS systems
https://osm.etsi.org/news-events/blog/30-open-source-mano-addressing-interoperability-challenge-in-nfv

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent https://osm.etsi.org/

Walmart
Scalable and Reliable Event Streaming
at World’s Largest Retailer
https://kafka-summit.org/sessions/kafka-meets-scaling-reliability-needs-worlds-largest-retailer-walmart-story/

Ride-Sharing
More than just Messaging! Data Correlation in Real-Time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

Track, manage, and locate
tools and other equipment
anytime and anywhere from
the warehouse to the jobsite
https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams

DB Musterfirma | Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile")
33Deutsche Bahn AG | Reisendeninformation
Consistent
real-time information
for travellers
across Germany
RI-Plattform

DB Musterfirma | Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile")
34
Customer timetable
Operational
timetable
Assignments
Railway station
knowledge
Dispositions
Train positions
Matching
Aggregation
Consolidation
Apache
Kafka
Analysis
Railway station
Trains
Mobile Apps
Employees
Deutsche Bahn AG | Reisendeninformation
RI-Plattform

Big Fish Games
Real-time analytics of game telemetry data for live operations
(aka increasing revenue while the player plays the game)
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/
Casual and mid-core games.
2.5 billion games to customers in
150 countries, representing over
450 unique mobile games and
over 3,500 unique PC games.
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/

Unity Ads - Monetization Network
• In 2019, content installed 33 billion times
reaching 3 billion devices worldwide
• Real-time 3D development platform
• Unity is a data-driven company
• Single common data pipeline for analytics, R&D,
monetization, cloud services, etc. for real-time
and batch processing
• One of the largest monetization networks
in the world
• Reward players for watching ads
• Incorporate banner ads
• Incorporate Augmented Reality (AR) ads
• Playable ads
• Cross-Promotions
https://www.confluent.io/blog/how-unity-uses-confluent-for-real-time-event-streaming-at-scale/

William Hill
From a Monolith to a flexible, scalable microservice architecture
• Kafka as central, reliable
streaming infrastructure
• Kafka for messaging,
storage, cache and
processing of data
• Independent decoupled
microservices
• Decoupling and replayability
• Technology independence
• High throughput + low
latency + real time
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-high-throughput-pipelines-with-kafka-from-scratch/

William Hill
The trading platform for millions of bets every day
• Kafka is the heart of all
events and transactions
• “process-to-process”
execution in real time
• Integration with analytic
models for real time
machine learning
• Various data sources and
data sinks (real time,
batch, request-response)
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-
high-throughput-pipelines-with-kafka-from-scratch/

Betting delay and approval in live bets
Synthetic delay to calculate risk, adjust odds, increase margin, reject bet if “too good”
Stateful Correlation of Events
Live Bet
(HTTP from Mobile App)
Time
Betting Engine
(ksqlDB)

Pharma, Life Sciences and Healthcare
Event Streaming to improve traditional and to build new use cases
Streams Processing / AI / ML
Clinical Trials
Patents,
Text etc
Structured &
unstructured
Data
IoT & Business
Applications
Multi-Hybrid-
Cloud

Bayer AG
Connectivity
+
On Premise and
Cloud
+
Hybrid Real Time
Replication at
Scale
Cloud first strategy and started a multi-year transition
to the cloud with a Kafka-based cross-datacenter data hub
https://www.confluent.io/kafka-summit-sf18/bringing-streaming-data-to-the-masses

Recursion Pharmaceutical
Real-time
Event Streaming
+
Machine Learning
Recursion Pharmaceutical
Accelerate drug discovery
https://www.confluent.io/customers/recursion
https://www.confluent.io/kafka-summit-san-francisco-2019/discovering-drugs-with-kafka-streams

Drug Discovery
in manual and slow, bursty batch mode, not scalable

Drug Discovery
in automated, scalable, reliable real-time mode

Digital Image
Processing
(e.g. noise
reduction)
Streaming Analytics for
Drug Discovery in Real Time at Scale
Real Time
Integration
Layer
Batch
Reporting
Platform
BI
DashboardEvent
Streaming
Platform
Real Time
Integration
Layer
Laboratory
Streaming Platform
Other Components
Automated
Drug Analysis
All
Data
Processed
Images
Ingest
Images
Human
Intelligence
Data Processing
(e.g. filtering)
Stateful Workflow
Orchestration

Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer

Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka

Similar to Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka