This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes: 1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data. 2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels. 3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms. 4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality. 5) Real-time cybersecurity applications that use streaming data

1. The Top 5 Use Cases and Architectures for Data in Motion in
2022
Kappa Architecture, Omnichannel, Multi-Cloud, Edge Analytics, and Real-time Cybersecurity
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de

2. @KaiWaehner www.kai-waehner.de
https://www.gartner.com/en/information-
technology/insights/top-technology-trends

3. @KaiWaehner www.kai-waehner.de
Cloud
Machine
Learning
Mobile
Data in
Motion
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

4. @KaiWaehner www.kai-waehner.de
Real-time Data in Motion beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

5. @KaiWaehner www.kai-waehner.de
This is a fundamental paradigm shift...
5
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

6. @KaiWaehner www.kai-waehner.de
Apache Kafka is the Platform for Data in
Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6

7. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

8. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

9. @KaiWaehner www.kai-waehner.de
Lambda Architecture
Option 1: Unified serving layer
9
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Serving Layer
Real-Time App
(Data Processing in Motion)
Batch App
(Data Processing at Rest)
ms
min/hr

10. @KaiWaehner www.kai-waehner.de
10
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Real-time Query
Mixed Query
ms
min/hr
Speed
View
Batch
View
Batch Query
Lambda Architecture
Option 2: Separate serving layers

11. @KaiWaehner www.kai-waehner.de
Concerns with the Lambda Architecture
11

12. @KaiWaehner www.kai-waehner.de
12
Data
Source
Real-Time Layer
(Data Processing in Motion)
Real-Time App
(Data Processing in Motion)
Storage
Batch App
(Data Processing at Rest)
Storage
ms
min/hr
Storage
Kappa Architecture
One pipeline for real-time and batch consumers

13. @KaiWaehner www.kai-waehner.de
Tiered Storage for Kafka
13

14. @KaiWaehner www.kai-waehner.de
Kappa @ Uber
14

15. @KaiWaehner www.kai-waehner.de
Kappa @ Shopify
15
Kappa Building Blocks
The Log (Kafka)
Durability with Topic Compaction and Tiered Storage
Consistency via Exactly-Once Semantics (EOS)
Data Integration via Kafka Connect
Elasticity via dynamic Kafka clusters
Streaming Framework (Kafka Streams / Flink)
Reliability and scalability
Fault tolerance
State management
Sinks
Update/Upsert for simplified design:
RDBMS, NoSQL, Compacted Kafka Topics
Append-only: Regular Kafka Topics, Time Series

16. @KaiWaehner www.kai-waehner.de
Kappa @ Disney
16
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning without a Data Lake

17. @KaiWaehner www.kai-waehner.de
Benefits of the Kappa Architecture
The Kappa architecture leverages a single source of truth with a focus on simplicity in the
enterprise architecture
• Improve streaming to handle all the cases
• One codebase that is always in synch
• One set of infrastructure and technology
• The heart of the infrastructure is real-time, scalable, and reliable
• Improved data quality with guaranteed ordering and no mismatches
• No need to re-architect for new use cases, just connect new consumers (real-time, near
real-time, batch, RPC)
• Kappa is NOT a free lunch – know the trade-offs and best practices
17

18. @KaiWaehner www.kai-waehner.de
Kappa is NOT a free lunch
18

19. @KaiWaehner www.kai-waehner.de
Kappa Concerns Solved
• Data availability / retention
 Compacted Topics, Tiered Storage
• Data consistency and fault-tolerance
 Exactly-once semantics, Multi-Region Clusters, Cluster Linking
• Handling late-arriving data
 State management in the streaming application, proper data sinks, replay with
guaranteed ordering and timestamps
• Data reprocessing and backfill
 Dynamic clusters, stateful applications (Kafka Streams, ksqlDB, external stream
processing framework like Apache Flink)
• Data integration
 Kafka Connect for sources and sinks, clients for any language, REST Proxy (real-time
but also batch and RPC
19

20. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

21. @KaiWaehner www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for growth
“Good enough” to run on
yesterday’s data

22. @KaiWaehner www.kai-waehner.de
Real-time automation of customer interactions
Improved Shipping and Delivery
Methods
Customer-Driven
In-Store Experiences
Hybrid model
Shopping
Social Influencers / Virtual Reality Shopping:
Journey-focused innovation
General Trends:
● Highly competitive market, work to thin margins
● Moving from High Street (brick & mortar) to Online (Omni-Channel)
● Personalized Customer Experience - optimal buyer journey
Customer
Experience
(CX) Operational
Efficiencies
New
Business
Models
Disruptive
Trends
in
Retail
Warehouse logistics teams aligned with
real-time, in-store demands
Automating the supply chain and core
business processes
Data-Driven
Business Decisions and
Personalized Promotions

23. @KaiWaehner www.kai-waehner.de
“Walmart is a $500 billion in revenue
company, so every second is worth millions of
dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud

24. @KaiWaehner www.kai-waehner.de
Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
● Investment in Kafka and Confluent has helped topline company
growth
● 8,500 nodes processing 11 billion events per day
● Deliver an omnichannel experience so every customer can
shop the way they want to

25. @KaiWaehner www.kai-waehner.de
Context-specific Customer 360
25
Electrical retailer
Hyper-personalized online retail
experience, turning each
customer visit into a one-on-one
marketing opportunity
Correlation of historical customer
data with real-time digital signals
Maximize customer satisfaction
and revenue growth, increased
customer conversions
https://www.confluent.io/customers/ao/

26. @KaiWaehner www.kai-waehner.de
Dick’s Sporting Goods
26
America’s largest sporting goods retail company
Focused on helping athletes achieve their personal best
Reshape the way athletes gain access to context-specific product
information in real time for a more seamless purchasing
experience online and in stores
Handle pricing and promotions, marketing, and athlete services in
real time to ensure a consistent omnichannel experience and
positive athlete service interaction
Fully-managed multi-cloud strategy with Confluent Cloud for
improved time-to-market and reduced operations cost.
confluent.io/customers/dicks-sporting-goods

27. @KaiWaehner www.kai-waehner.de
Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

28. @KaiWaehner www.kai-waehner.de
Live commerce with real-time data correlation
including integration of CRM, loyalty, inventory, chatbots, location-based services, etc.
28

29. @KaiWaehner www.kai-waehner.de
Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

30. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

31. @KaiWaehner www.kai-waehner.de
Spaghetti: Data architectures are often complex
31

32. @KaiWaehner www.kai-waehner.de
Kafka provides a solution:
An immutable stream of facts with the freedom to act, adapt, and change
32
Kafka

33. @KaiWaehner www.kai-waehner.de
Domain
33
Data
Product
Data Mesh: A new technology-agnostic decentralized implementation
pattern
Data Mesh
Data ownership
by domain
Data as
a product
Data available anywhere,
self serve
Data governed
wherever it is

34. @KaiWaehner www.kai-waehner.de
34
Mesh is one logical cluster. Data product has another.
Data
Product
Data Product has its own
cluster for internal use

35. @KaiWaehner www.kai-waehner.de
35
With stream processing the real-time applications are decentralized
Data
Product
STREAM
PROCESSOR
ksqlDB
Query is the interface to
the mesh
Events are the interface
to the mesh

36. @KaiWaehner www.kai-waehner.de
36
Operational and Data Product Streaming Planes with
Cluster Linking

37. @KaiWaehner www.kai-waehner.de
Data Mesh Example: Hybrid Multi-Cloud Architecture
37
Data Engineers
Data Scientists
Data Architects Operators
Architects
SMEs
Data Governance
Shared Services
Application team
Generalist Eng
Generalists Eng
Specialized / Legacy Engineers

38. @KaiWaehner www.kai-waehner.de
Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!

39. @KaiWaehner www.kai-waehner.de
Data Governance: Tracking data lineage with Streams in real-time
39
• Lineage must work across domains and data products—and systems, clouds, data centers.
• Event streaming is a foundational technology for this.
On-premise

40. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

41. @KaiWaehner www.kai-waehner.de
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration

42. @KaiWaehner www.kai-waehner.de
Edge Use Cases with Low Latency Requirements
https://www.youtube.com/watch?v=A9DDe0alvGo

43. @KaiWaehner www.kai-waehner.de
Low Latency 5G Use Cases for Edge and Hybrid
Cloud
with AWS Wavelength (based on AWS Outposts) and Confluent

44. @KaiWaehner www.kai-waehner.de
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Retail Architecture

45. @KaiWaehner www.kai-waehner.de
Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability

46. @KaiWaehner www.kai-waehner.de
Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)

47. @KaiWaehner www.kai-waehner.de
Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

48. @KaiWaehner www.kai-waehner.de
Devon Energy Corporation
Oil & Gas Industry
Improve drilling and well completion operations
Edge stream processing/analytics + closed-loop control ready
Replication to the cloud in real-time at scale
Vendor agnostic (pumping, wireline, coil, offset wells, drilling
operations, producing wells
Cloud agnostic (AWS, GCP, Azure)

49. @KaiWaehner www.kai-waehner.de
The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

50. @KaiWaehner www.kai-waehner.de
What is Cybersecurity?
Protection of computer systems and networks from information disclosure and theft
Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors
50

51. @KaiWaehner www.kai-waehner.de
Supply Chain Attack
Targeting less-secure elements in the supply chain
51
https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains
https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002

52. @KaiWaehner www.kai-waehner.de
Real-time Data in Motion beats Slow Data.
Security
Access control and encryption
Regulatory compliance
Rules engine
Security monitoring
Surveillance
Cybersecurity
Risk classification
Threat detection
Intrusion detection
Incident response
Fraud detection

53. @KaiWaehner www.kai-waehner.de
Data in Motion
The Backbone for Cybersecurity
Industria
l OT
Enterpris
e IT
Consumer
IoT
Logs Personal
Sensors Security
Streams of real time
events
53
Connected
Vehicles
Cyber
Security
Continuous
Data Correlation
Monitoring
Alerting
Proactive Actions

54. @KaiWaehner www.kai-waehner.de
End-to-End Cybersecurity
with the Kafka Ecosystem
Personel
Crew, Cargo
Vessel
Fuel Consumption, Speed,
Planned Maintenance
Tracking
Position, Course, Weather, Draft
Drone or Satellite Relay
COMMs Resilient Kafka
Edge Analytics
Data
Integration
Streaming Analytics
Machine Doing
On-Prem Systems
Bi-Directional Hybrid Cloud Replication
ON SHORE
ON PREM
Staging, Filtering
Shore Edge Analytics

55. @KaiWaehner www.kai-waehner.de
SIEM / SOAR
Situational Awareness
Operational Awareness
Intrusion Detection
Signals and Noise
Signature Detection
Incident Response
Threat Hunting & Intelligence
Vulnerability Management
Digital Forensics
…
was not built for cybersecurity!

56. @KaiWaehner www.kai-waehner.de
Integrate with all legacy and modern interfaces
Record, filter, curate a broad set of traffic streams
Let analytic sinks consume just the right amount of data
Drastically reduce the complexity of the enterprise architectures
Drastically reduce the cost of SIEM / SOAR deployments
Add new analytics engines
Add stream-speed detection and response at scale in real-time
Add mission-critical (non-) security-related applications
…
is the backbone for cybersecurity!

57. @KaiWaehner www.kai-waehner.de
Confluent Sigma
Sigma Stream Processors
Zeek Data and
Detections Viewer
Sigma Rule Editor
sigma rules topic
DNS
dns
detections
topic
dns topic
rule parsing,
filtering,
aggregation,
windowing
sigma
rules
cache
CONN
DHCP
HTTP
SSL
x509
Zeek Data

58. @KaiWaehner www.kai-waehner.de
Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

59. @KaiWaehner www.kai-waehner.de
How does
Confluent
help?

60. @KaiWaehner www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
60

61. @KaiWaehner www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness
/
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver
1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched,
Hybrid, Multi-
Region)
Build Contextual
Event-Driven Apps
(Stretched,
Hybrid, Multi-
Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
61

62. @KaiWaehner www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native.
Everywhere.

63. @KaiWaehner www.kai-waehner.de
Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for Business
Success

64. Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions?
Feedback?
Let’s connect!