Event Streaming with Apache Kafka in the Transportation and Logistics.
Track & Trace, Real-time Locating System, Customer 360, Open API, and more…
Examples include Swiss Post, SBB, Deutsche Bahn, Hermes, Migros, Here Technologies, Otonomo, Lyft, Uber, Free Now, Lufthansa, Air France, Singapore Airlines, Amadeus Group, and more.
1. @KaiWaehner - www.kai-waehner.de
Event Streaming with Apache Kafka
in the Transportation and Logistics
Track & Trace, Real-time Locating System, Customer 360, Open API, and more…
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner
3. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
17. @KaiWaehner - www.kai-waehner.de
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers
18. @KaiWaehner - www.kai-waehner.de
Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases
19. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
21. @KaiWaehner - www.kai-waehner.de
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka
22. @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
23. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
24. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
25
26. @KaiWaehner - www.kai-waehner.de
Swiss Post
National postal service of Switzerland
Paradigm shift with modern Kafka-based integration layer
Processing small events and large legacy files + images
https://www.youtube.com/watch?v=yNrtV9_NWgk
27. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Architecture
28. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Updated
Schedules
Event Streaming
at the Edge
29. @KaiWaehner - www.kai-waehner.de
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate with
other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
30. @KaiWaehner - www.kai-waehner.de
SBB
Swiss Federal Railways is the national railway company of Switzerland
IoT challenges: Vehicle tracking, hardware condition, historical sensor information, control train tracks, etc.
Decoupling with persistent storage in Kafka
Kafka native uplink (monitor) and synchronous HTTP downlink (control) messages
Kafka Connect for ingestion into Splunk
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421
31. @KaiWaehner - www.kai-waehner.de
SBB
Swiss Federal Railways is the national railway company of Switzerland
Shared Kafka Connect Platform on Kubernetes
https://www.youtube.com/watch?v=yNrtV9_NWgk
32. @KaiWaehner - www.kai-waehner.de
Hermes
German delivery company
Change Data Capture with HVR + Kafka for real-time delivery and collection service
Predictive delivery planning
Technology and architecture modernization with databases and Kafka
35
TODO Link Kafka Summit Europe
33. @KaiWaehner - www.kai-waehner.de
Migros
Switzerland's largest retail company, largest supermarket
chain and largest employer
Real-time transportation information visualization with MQTT
and Kafka
Supply chain optimization with a single streaming pipeline
(including replay whole day of events)
Forecast the truck arrival time; plan and reschedule truck tours
36
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421
34. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Cross-Company Streaming Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
(REST et al) is not
appropriate for streaming
data
Infosec and politics are your
biggest hurdle
OEM
35. @KaiWaehner - www.kai-waehner.de
Here Technologies
Captures location content such as road networks, buildings, parks and traffic patterns
Sells or licenses mapping content, along with map related navigation and location services to other businesses
https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html
37. @KaiWaehner - www.kai-waehner.de
Postmodern Real-Time Locating / Tracking System (RTLS)
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events
38. @KaiWaehner - www.kai-waehner.de
Ride-Sharing
More than just Messaging! Data correlation in real-time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006
39. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/
https://www.confluent.io/thank-you/stream-processing-kafka-uber/
Trillions of messages and
multiple petabytes of data per day
43
40. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
44
41. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Digital Infrastructure
A Prerequisite for a Smart City
Digital
Infrastructure
Data Centers
Fixed and
Mobile
Connectivity
Network
Infrastructure
Internet of
Things and
Devices
Platforms
and
Applications
45
42. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Event Streaming
The Foundation for a Smart City
Mobility
Services
Citizen
Services
Smart
Buildings
Cybersecurity
Smart
Grid
Weather
Data
Citizen
Data
Traffic
Data
Security
Alerts
Streams of real time events
Personal Data
Weather Service
Connected
Cars
Routing
Service
Traffic Alerts
Surveillance
46
49. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Amadeus IT Group
53
https://kafka.apache.org/powered-by
Major European IT provider for the global travel and tourism industry
Global Distribution System (GDS) provides search, pricing, booking,
ticketing and other processing services in real-time
Customers include airlines, hotels, tour operators, insurers, car
rental and railway companies
51. @KaiWaehner - www.kai-waehner.de
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams
53. @KaiWaehner - www.kai-waehner.de
Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0
54. @KaiWaehner - www.kai-waehner.de
Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
55. @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB
56. @KaiWaehner - www.kai-waehner.de
Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS
57. @KaiWaehner - www.kai-waehner.de
ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDB
DB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”
58. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Events at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/
59. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Event Streaming with Kafka and ksqlDB at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/
62. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(Data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…
63. @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
64. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Multi-Region Cluster
Kafka Clusters
can Stretch over Regions
No Downtime + Zero Data loss
e.g. Stretched over US East + Mid + West
Automated Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without Custom Code
65. @KaiWaehner - www.kai-waehner.de
69
Cluster Linking requires no additional
infrastructure and preserves offsets:
Global Central
Nervous System
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka
66. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Streaming Transportation Data Exchange
MirrorMaker 2 /
Confluent Replicator
Railway
Server: Latest version
Clients: 0.11, 2.0
Streaming replication
between stakeholders
Version compatibility between
different
clients and servers
Mobility Service
Server: AK 2.3
Clients: 2.0, 2.5
Coach Enterprise
Server: CP 54 / AK 2.4
Clients: 0.11, 2.4
Confluent
Cluster Linking
67. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?
68. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
72
69. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
73
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration
70. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design for your Integration Layer
Legacy
Integration
Custom
Application
ESB Connector
Java / Python /
ksqlDB / etc.
Schema Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
71. @KaiWaehner - www.kai-waehner.de
Kafka Cluster
Domain-Driven Design for your Event Steaming Platform
Schema Registry
Event Streaming Platform
Big Data Cluster Big Data Cluster
KSQL
App
KSQL
App
KSQL
App
72. @KaiWaehner - www.kai-waehner.de
Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical
Data
74. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed Commit
Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
78
Model X
(at a later time)
75. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
79
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
77. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
81
78. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
82
79. @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success
80. @KaiWaehner - www.kai-waehner.de
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!