Apache Kafka in the Transportation and Logistics

@KaiWaehner - www.kai-waehner.de
Event Streaming with Apache Kafka
in the Transportation and Logistics
Track & Trace, Real-time Locating System, Customer 360, Open API, and more…
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

The world is changing.

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

Stores become Software

Banks become Software

Taxis become Software
2
min

Cars become Software

Car Production becomes Software

Travel becomes an Experience

This transformation is
happening everywhere

Transportation

Retail

Finance

Evolution of Transport Technology
14
https://transportgeography.org/contents/conclusion/future-transportation-systems/evolution-transport-technology/

Evolution of Logistics Technology
15
www.unity.de/en/

What enables this
transformation?

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

What is Event Streaming?

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
25

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

Swiss Post
National postal service of Switzerland
Paradigm shift with modern Kafka-based integration layer
Processing small events and large legacy files + images
https://www.youtube.com/watch?v=yNrtV9_NWgk

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Hybrid Architecture

Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Updated
Schedules
Event Streaming
at the Edge

Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate with
other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency

SBB
Swiss Federal Railways is the national railway company of Switzerland
IoT challenges: Vehicle tracking, hardware condition, historical sensor information, control train tracks, etc.
Decoupling with persistent storage in Kafka
Kafka native uplink (monitor) and synchronous HTTP downlink (control) messages
Kafka Connect for ingestion into Splunk
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421

SBB
Swiss Federal Railways is the national railway company of Switzerland
Shared Kafka Connect Platform on Kubernetes
https://www.youtube.com/watch?v=yNrtV9_NWgk

Hermes
German delivery company
Change Data Capture with HVR + Kafka for real-time delivery and collection service
Predictive delivery planning
Technology and architecture modernization with databases and Kafka
35
TODO Link Kafka Summit Europe

Migros
Switzerland's largest retail company, largest supermarket
chain and largest employer
Real-time transportation information visualization with MQTT
and Kafka
Supply chain optimization with a single streaming pipeline
(including replay whole day of events)
Forecast the truck arrival time; plan and reschedule truck tours
36
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421

Cross-Company Streaming Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
(REST et al) is not
appropriate for streaming
data
Infosec and politics are your
biggest hurdle
OEM

Here Technologies
Captures location content such as road networks, buildings, parks and traffic patterns
Sells or licenses mapping content, along with map related navigation and location services to other businesses
https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html

Open API Platform
for Car Data

Postmodern Real-Time Locating / Tracking System (RTLS)
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events

Ride-Sharing
More than just Messaging! Data correlation in real-time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/
https://www.confluent.io/thank-you/stream-processing-kafka-uber/
Trillions of messages and
multiple petabytes of data per day
43

FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
44

Digital Infrastructure
A Prerequisite for a Smart City
Digital
Infrastructure
Data Centers
Fixed and
Mobile
Connectivity
Network
Infrastructure
Internet of
Things and
Devices
Platforms
and
Applications
45

Event Streaming
The Foundation for a Smart City
Mobility
Services
Citizen
Services
Smart
Buildings
Cybersecurity
Smart
Grid
Weather
Data
Citizen
Data
Traffic
Data
Security
Alerts
Personal Data
Weather Service
Connected
Cars
Routing
Service
Traffic Alerts
Surveillance
46

Why the “right time” is “real time”…
47

The Smarter City Nervous System
Producers Consumers
• Client Libraries
• Kafka Connectors
• Client Libraries
• Kafka Connectors
X
Consumer Groups 48
Streams

The Smarter City Nervous System
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
Topic (observation-out)
Elasticsearch
Kafka Connect
Elasticsearch Sink
Connector
49

Lufthansa’s KUSCO
(Kafka Unified Streaming Cloud Operations)
Confluent Streaming Event 2020:
https://videos.confluent.io/watch/X4j6nh3e8Kr8EvsRCeSEhJ

Singapore Airlines
Streaming Predictive Maintenance with Kafka and Machine Learning
https://speakerdeck.com/devacto/predictive-maintenance-pipeline-using-kafka-connect-streams-and-ksql

52
https://www.confluent.io/kafka-summit-sf18/hop-airlines-jets-to-real-time/

Amadeus IT Group
53
https://kafka.apache.org/powered-by
Major European IT provider for the global travel and tourism industry
Global Distribution System (GDS) provides search, pricing, booking,
ticketing and other processing services in real-time
Customers include airlines, hotels, tour operators, insurers, car
rental and railway companies

What is Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect

Kafka Loves Your Existing Systems.
✓ Hundreds of ready-to-use
connectors available
✓ Supported by Confluent
or by partners (certified)
✓ Managed Connectors in
Confluent Cloud
$ confluent-hub install mongodb/kafka-connect-mongodb:1.3.0

Confluent Cloud
✓ Infinite Storage
Confluent Platform
✓ Tiered Storage (e.g., AWS S3)
Apache Kafka
• Tiered Storage upcoming
Kafka Stores Your Data Durably.
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Kafka Streams — #1 popular choice
✓ Client library for Java, Scala
✓ Elastic, highly scalable, fault-tolerant
✓ Filter, transform, join, aggregate, window, etc.
✓ Exactly-once processing
✓ Full Kafka security integration
Build Apps, not Processing Clusters
✓ Standard JVM application
✓ Run in container, VMs, K8S, bare-metal, etc.
✓ No separate ‘processing cluster’ needed
✓ For S, M, L, XXL use cases
Kafka Makes Your Business Real-time.
Kafka Cluster
REAL-TIME APPLICATIONS

ksqlDB, the Event Streaming Database
DB
APP
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
TABLES & STATE
ksqlDB
DB One Solution for:
✓ capturing events
✓ stream processing
✓ serving both push
and pull queries
✓ also available in
Confluent Cloud
CREATE SOURCE CONNECTOR postgresSource WITH (...);
CREATE STREAM customer_changes …;
CREATE TABLE customers AS SELECT ... FROM customer_changes;
APP
“The easiest way to get started with Kafka.”

Events at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/

Event Streaming with Kafka and ksqlDB at an Airport
https://www.confluent.io/kafka-summit-sf18/serving-the-real-time-data-needs-of-an-airport/

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Distributed System with Replication and High Availability
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

Distributed System with Replication and High Availability
read,
write
App
(Kafka Streams)
Kafka
(Data)
More Apps
(KSQL, Connect, Python,
REST, “You-name-it”)
Bookings
Team
Fraud
Team
…
Mobile
Team
…

Multi-Region Cluster
Kafka Clusters
can Stretch over Regions
No Downtime + Zero Data loss
e.g. Stretched over US East + Mid + West
Automated Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without Custom Code

69
Cluster Linking requires no additional
infrastructure and preserves offsets:
Global Central
Nervous System
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka

Streaming Transportation Data Exchange
MirrorMaker 2 /
Confluent Replicator
Railway
Server: Latest version
Clients: 0.11, 2.0
Streaming replication
between stakeholders
Version compatibility between
different
clients and servers
Mobility Service
Server: AK 2.3
Clients: 2.0, 2.5
Coach Enterprise
Server: CP 54 / AK 2.4
Clients: 0.11, 2.4
Confluent
Cluster Linking

Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
72

Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
73
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design for your Integration Layer
Legacy
Integration
Custom
Application
ESB Connector
Java / Python /
ksqlDB / etc.
Schema Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!

Kafka Cluster
Domain-Driven Design for your Event Steaming Platform
Schema Registry
Event Streaming Platform
Big Data Cluster Big Data Cluster
KSQL
App
KSQL
App
KSQL
App

Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical
Data

Confluent Tiered Storage for Kafka
77

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed Commit
Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
78
Model X
(at a later time)

Event Streaming Is The Future Of Data
79
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
81

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
82

Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka in the Transportation and Logistics

More Related Content

What's hot

Similar to Apache Kafka in the Transportation and Logistics

More from Kai Wähner

Recently uploaded

Apache Kafka in the Transportation and Logistics