SlideShare a Scribd company logo
1 of 71
The Rise of Data in Motion in the Public Sector
Citizen Services, National Security, Smart City, and more
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
The world is changing.
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Normal Life becomes Software
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
National Security becomes Software
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Lighting becomes Software
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Parking becomes Software
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Public Sector
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Collaboration is Key for Success
Car Maker
Supplier
Government
Telco
Mobility Services
Cloud Providers
Software Providers
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion across the Public and Private Sector
Real-time data
without context!
Possible with
simple
messaging
solutions.
Real-time data
with context!
What does the
ride cost?
How long does
the driver need?
Data is correlated
in real time.
https://www.confluent.io/events/kafka-summit-europe-2021/development-of-dynamic-pricing-for-tours-using-real-time-data-feeds/
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
This is a fundamental paradigm shift...
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
(private or public)
datacenter
Future of data
Cloud-
native
Event
Streaming
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Real-time Data beats Slow Data.
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Real-time Data beats Slow Data.
Transportation
Traffic planning
Parking System
Public transportation
Citizen Services
ID application process
Covid tracking
Utilities billing
Law Enforcement
Border control
Security monitoring
Surveillance
Cybersecurity
Risk classification
Intrusion detection
Incident response
Fraud detection
Data in Motion in the Public Sector
Your Processes as Streams of Events, powered by Apache Kafka
Intelligent
Parking
System
Emergency
Service
Traffic
Research
Team
Traffic
Planning
System
Car
Sensor
Data
Smart City
Example
Public Sector – Examples for Data in Motion
Citizen Services
• Health services, e.g. hospital modernization, track & trace - Covid distance control
• Public administration - reduce bureaucracy, data democratization
• eGovernment - Efficient and digital citizen engagement, e.g. personal ID application process
Smart City
• Smart driving, parking, buildings, environment
• Waste management
• Open exchange – e.g. mobility services (1st
and 3rd
party)
Energy
• Smart grid and utilities infrastructure (energy distribution,
smart home, smart meters, smart water, etc.)
National Security
• Law enforcement, surveillance, police/interior security data exchange
• Defense and military (border control, intelligent solider)
• Cybersecurity for situational awareness and threat intelligence
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion is the Underpinning
of an Event-driven Architecture
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Mainframe
CSV File
STREAM
PROCESSING
CONNECTORS
Example Architecture for Data in Motion
ksqlDB
KStreams
Bi-directional real-time integration between legacy and modern worlds
Dashboard
Oracle
DB
ORACLE CDC
SOURCE
CONNECTOR
Real-time
App
Digital Infrastructure
A Prerequisite for Data in Motion in the Public Sector
Digital
Infrastructure
Data Centers
Fixed and
Mobile
Connectivity
Network
Infrastructure
Internet of
Things and
Devices
Platforms
and
Applications
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion
The Foundation for a Smart Life
Mobility
Services
Citizen
Services
Smart
Buildings
Cybersecurity
Smart
Grid
Weather
Data
Citizen
Data
Traffic
Data
Security
Alerts
Streams of real time events
Personal Data
Weather Service
Connected
Cars
Routing
Service
Traffic Alerts
Surveillance
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Smart City
Digital Buildings, Connected Vehicles and Things, Citizens
Real-time Operations, Logistics, Predictive Maintenance, Security
Traffic Data
Vehicles, Accidents,
Traffic Lights, etc.
Surveillance Data
Buildings, People,
Distance Measuring, etc.
Mobility Services
Multi-modal integration,
E-car Charging Stations,
Parking Lots, etc.
Drone Data
Deliveries,
Survey/Inspection
of Traffic, People,
Construction Areas etc.
Edge Analytics Bidirectional Edge to Cloud Integration
Data Ingestion
Stream
Processing
Data
Integration
Logistics
Track & Trace
Routing
Monitoring
Alerting
Command & Control
Batch Analytics
Reporting
Machine Learning
Backend Systems
Oracle, SAP,
OSIsoft PI, etc.
X = Event Streaming
X = Other Technologies
Bi-Directional Hybrid Cloud
Replication
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Wavelength
Public 5G
Campus #1 5G
Wavelength
Campus #2 5G
Connected Services
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Why the “right time” is “real time”…
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
The Smarter City Nervous System
Producers Consumers
• Client Libraries
• Kafka Connectors
• Client Libraries
• Kafka Connectors
X
Consumer Groups
Streams
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
The Smarter City Nervous System
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
Topic (observation-out)
Elasticsearch
Kafka Connect
Elasticsearch Sink
Connector
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Ohio Department of Transportation
Hub for all things autonomous, connected, and smart mobility initiatives in Ohio
Intelligent Transportation System to help reduce traffic and fatalities on the road in real-time
Confluent Cloud + Confluent Operator for applications
(Kafka Connect, KSQL, Kafka Streams, Custom Code) for low operations efforts and high flexibility
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Ohio Department of Transportation
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Customer
timetable
Operational
timetable
Assignments
Railway station
knowledge
Dispositions
Train positions
Matching
Aggregation
Consolidation
Apache
Kafka
Analysis
Railway station
Trains
Mobile Apps
Employees
Deutsche Bahn AG | Reisendeninformation
RI-Plattform
RI-Platform @ Deutsche Bahn (German Railway)
Traveler Information with Single Point of Truth
3rd Party Integration with Google Maps
Real-time train schedule updates
Reach new people and expand business
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
https://railway-news.com/db-partners-with-google-maps-on-journey-planning/
FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
NAV (Norwegian Work and Welfare Department)
23,000 employees
USD 50Bn disbursements per year
Assist people through all phases of life
Unemployment benefits, health insurance, social security, pensions, parental benefits
Kafka Summit: https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/
Webinar: https://videos.confluent.io/watch/qekArwtPMKE7f6hKRcfTdX
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
From the vision to the project
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
From the vision to the project
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Life as a Stream of Events
(by leveraging event streaming technology)
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Reducing bureaucracy
(by avoiding spaghetti architectures)
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Reducing bureaucracy
(by domain-driven design)
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Diverging forces
(by integration, but true decoupling)
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
VA (Veterans Affairs)
The United States Department of Veterans Affairs (VA)
is a department of the federal government charged with
integrating life-long healthcare services to
eligible military veterans
Improving veteran benefit services for ratings, awards,
claims through data streaming
Assess, route, and verify the service request in real-time.
Improve the experience for veterans and their families.
Reduce management and operations complexity.
Examples: Change in status of a claim, requests for
additional documentation, context when calling into the
call center, due benefits when checking in at a hospital
Confluent for true decoupling, real-time integration,
omnichannel communication. Components include Kafka
Streams, ksqlDB, Oracle CDC, Tiered Storage, etc.
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
University of California, San Diego
https://www.confluent.io/customers/uc-san-diego/
Transition to build out their online learning platform due to the
pandemic; plus new #1 priority: Student testing process
Confluent as Integration-platform-as-a-service (iPaaS)
middleware layer to set data in motion for 90 million records a day
Increased time to market with agile development and decoupled
applications
Opening up new revenue streams for other UC campuses -
including the UC Office of President, and tracing student health
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Regulated vs. Deregulated Markets
Example: US Regulated vs. Deregulated Electricity Market
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Source: https://www.electricchoice.com/map-deregulated-energy-markets/
The Energy Sector
is Changing…
Flexible, scalable, secure
smart grids
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
The Energy Sector is Changing…
https://electrek.co/2021/08/27/tesla-plans-enter-texas-deregulated-energy-market-massive-battery/
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/
CRM
3rd party
payment
provider
Real-Time
Asset Management
Customer data
Payment processing and
fraud detection as a service
Manager
Outage Management
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Energy Production and Distribution
with a Hybrid Architecture
Wavelength
AWS Cloud VPC
Carrier #1 5G
Wavelength
Carrier #2 5G
Smart
Meters
Smart
Buildings
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Smart
Grid
Upstream
Operations Management
Well Completion
Operations
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Real-Time
Supply Chain Management
Event Streaming for Energy Production
(Upstream + Midstream)
at the Edge with a 5G Campus Network
Wavelength
Cloud VPC
Public 5G
Campus 5G
Smart
Home
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Energy Production at the
Disconnected Edge (Upstream)
Time
P
C1
C2
C3
Predictive Analytics
Human
Machine
Interface
Predictive
Maintenance
Sensors
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Edge Processing at the
Intelligent Gas Station
(Downstream)
Time
P
C3 C1
C2
Upsell in the store
Context-specific coupon
(e.g. carwash not in use à 30% off))
Real-time inventory
and intelligent pricing
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Leipziger Stadtwerke
Kunbus Edge Devices (PI with custom Linux)
OTA (over the air update) with Mender
Kafka for IT Integration, Data Processing, Innovative Digital Services
Kafka Digital Customer Interface to connect public utilities, grid
operator, housing industry, end-consumer, industrial customer
https://m.youtube.com/watch?v=5tAkpxvvHA4
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Cybersecurity – The Threat is Real!
https://www.mckinsey.com/business-functions/risk-and-resilience/our-insights/the-energy-sector-threat-how-to-address-cybersecurity-vulnerabilities
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion in the Public Sector
1) The World is Changing
2) Life is a Stream of Events
3) Smart City
4) Citizen Services
5) Energy and Utilities
6) National Security
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Digital
Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Data in Motion
The Foundation for Cybersecurity
Industrial
OT
Enterprise
IT
Consumer
IoT
Logs Personal
Sensors Securit
y
Streams of real time events
Connected
Vehicles
Cyber
Security
Continuous
Data Correlation
Monitoring
Alerting
Proactive Actions
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Firewalls & Network Devices
Antivirus
Access Logs
Intrusion Detection
Audit Logs
Text Files
Binary Files
Databases
APIs
Network Flows
Syslog
The Data
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Legacy SIEM needs to evolve
Forwarder
Network traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
• Proprietary forwarders that can only send
data to single source
• Data locked from being shared
• Difficult to scale with growing data volumes
• Prohibitively high indexing costs
• Unable to filter out noisy data
• Slow batch processing
HTTP proxy logs
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Forensic
Archive AI/ML
Build A Real-time SIEM / SOAR Pipeline
Filter,
transform
aggregate
APP SIEM
Index
Search
Curated
streams
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Machine Data
spooldir (files), SNMP Traps,
Databases, Sftp, MQs
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Confluent + Splunk SIEM Reference Architecture
(assuming that Splunk UFs already exist – otherwise integrate directly via Kafka to Splunk)
Splunk
Universal
Forwarders
(UFs)
Windows
Event Logs
SNMP
Syslog
Watchlist
Zeek IDS
Splunk
Heavy
Forwarders
Machine
Learning
Splunk S2S
Connector
Splunk
HEC
Splunk
Indexers
Splunk
Search
Head
Real-time stream
processing with
ksqlDB
...
3rd party apps /
ecosystems
Moving log data
from Splunk UFs
to your
destination of
choice
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
AI/ML
Border Protection with Kafka for Security and Surveillance
Image Recognition, Data Correlation, Threat Detection and Alerting in Real-Time
Filter,
transform
aggregate
APP SIEM
Index
Search
Curated
streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
Database
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Video Streams
TensorFlow
H2O.ai
Datarobot
Customer/Site 1 Customer/Site 2 (Air-Gapped) Customer/Site N
Central Data Center / Public Cloud
Event Streaming in Air-Gapped and Zero Trust Environments
Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure
Confluent Data Diode as Unidirectional Gateway for Zero Trust
Sensor A Sensor B Sensor X
MQTT
Confluent Platform (Single Broker)
Confluent Platform deployed on a Raspberry Pi and
leveraging Cluster Linking to publish sensor data to
Command Post in a DDIL environment.
Command Post running Confluent
Platform aggregating information from
Squires and other sensor data
Weather
Personnel
Logistics
Targets Sensor data published to
Command Post when
connected to network
Enhanced Situational
Awareness
Internet of Battle Things (IoBT) at the Edge
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Internet of Battle Things (IoBT) and Smart Soldiers at the Edge
Live Demo
NASA enables real-time data from Mars
• Real-time data extends into the far
frontiers via its Deep Space
Network (DSN)
• Data grows exponentially from
spacecraft and other systems
• Responsive citizen engagement,
real-time situational awareness,
anomaly detection, event-driven
missions and security operations
• Global fabric of Kafka clusters allows
for real-time sharing, event
streaming, and other combinations
of both real-time and historical data
https://federalnewsnetwork.com/big-data/2021/09/nasa-adopting-apache-kafka-to-enable-real-time-data-from-mars/
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Why
Confluent?
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.
kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryKai Wähner
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...Kai Wähner
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaKai Wähner
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureDatabricks
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesApache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesKai Wähner
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureDatabricks
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner
 
Event Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaEvent Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaKai Wähner
 
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeApache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
 
Capital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream ProcessingCapital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 

What's hot (20)

Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance Industry
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesApache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and Architectures
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
 
Event Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache KafkaEvent Streaming in Retail with Apache Kafka
Event Streaming in Retail with Apache Kafka
 
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeApache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
 
Capital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream ProcessingCapital One Delivers Risk Insights in Real Time with Stream Processing
Capital One Delivers Risk Insights in Real Time with Stream Processing
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 

Similar to Apache Kafka in the Public Sector (Government, National Security, Citizen Services)

Big data and public transport
Big data and public transportBig data and public transport
Big data and public transportTristan Wiggill
 
Mr. Paul Chang's presentation at QITCOM 2011
Mr. Paul Chang's presentation at QITCOM 2011Mr. Paul Chang's presentation at QITCOM 2011
Mr. Paul Chang's presentation at QITCOM 2011QITCOM
 
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...Kai Wähner
 
Realtime Big Data Analytics for Event Detection in Highways
Realtime Big Data Analytics for Event Detection in HighwaysRealtime Big Data Analytics for Event Detection in Highways
Realtime Big Data Analytics for Event Detection in HighwaysYork University
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityKai Wähner
 
Big data in freight transport
Big data in freight transportBig data in freight transport
Big data in freight transportPer Olof Arnäs
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionKai Wähner
 
Mit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenMit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenconfluent
 
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Paolo Nesi
 
Real-time data integration to the cloud
Real-time data integration to the cloudReal-time data integration to the cloud
Real-time data integration to the cloudSankar Nagarajan
 
AWS Smart Cities Webinar - April 2018
AWS Smart Cities Webinar - April 2018AWS Smart Cities Webinar - April 2018
AWS Smart Cities Webinar - April 2018Amazon Web Services
 
Transforming City with Internet of Things
Transforming City with Internet of ThingsTransforming City with Internet of Things
Transforming City with Internet of ThingsRofiqi Setiawan
 
Locations big data its
Locations big data itsLocations big data its
Locations big data itsGeoMedeelel
 
A Full End-to-End Platform as a Service for Smart City Applications
A Full End-to-End Platform as a Service for SmartCity ApplicationsA Full End-to-End Platform as a Service for SmartCity Applications
A Full End-to-End Platform as a Service for Smart City ApplicationsCharalampos Doukas
 
Open Data Hub
Open Data HubOpen Data Hub
Open Data HubFabMob
 
Analytics in IoT
Analytics in IoTAnalytics in IoT
Analytics in IoTwesley Dias
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertconfluent
 
Paul Davidson – Opening up public data to improve transparancy and efficiency
Paul Davidson – Opening up public data to improve transparancy and efficiencyPaul Davidson – Opening up public data to improve transparancy and efficiency
Paul Davidson – Opening up public data to improve transparancy and efficiencyCorvé Open Government Preconference 2010
 

Similar to Apache Kafka in the Public Sector (Government, National Security, Citizen Services) (20)

Big data and public transport
Big data and public transportBig data and public transport
Big data and public transport
 
Mr. Paul Chang's presentation at QITCOM 2011
Mr. Paul Chang's presentation at QITCOM 2011Mr. Paul Chang's presentation at QITCOM 2011
Mr. Paul Chang's presentation at QITCOM 2011
 
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...
Enabling Smarter Cities and Connected Vehicles with an Event Streaming Platfo...
 
Realtime Big Data Analytics for Event Detection in Highways
Realtime Big Data Analytics for Event Detection in HighwaysRealtime Big Data Analytics for Event Detection in Highways
Realtime Big Data Analytics for Event Detection in Highways
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart City
 
Big data in freight transport
Big data in freight transportBig data in freight transport
Big data in freight transport
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy Production
 
Mit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauenMit Streaming die Brücken zum Erfolg bauen
Mit Streaming die Brücken zum Erfolg bauen
 
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
 
Real-time data integration to the cloud
Real-time data integration to the cloudReal-time data integration to the cloud
Real-time data integration to the cloud
 
AWS Smart Cities Webinar - April 2018
AWS Smart Cities Webinar - April 2018AWS Smart Cities Webinar - April 2018
AWS Smart Cities Webinar - April 2018
 
Transforming City with Internet of Things
Transforming City with Internet of ThingsTransforming City with Internet of Things
Transforming City with Internet of Things
 
Locations big data its
Locations big data itsLocations big data its
Locations big data its
 
A Full End-to-End Platform as a Service for Smart City Applications
A Full End-to-End Platform as a Service for SmartCity ApplicationsA Full End-to-End Platform as a Service for SmartCity Applications
A Full End-to-End Platform as a Service for Smart City Applications
 
Open Data Hub
Open Data HubOpen Data Hub
Open Data Hub
 
Analytics in IoT
Analytics in IoTAnalytics in IoT
Analytics in IoT
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
 
Paul Davidson – Opening up public data to improve transparancy and efficiency
Paul Davidson – Opening up public data to improve transparancy and efficiencyPaul Davidson – Opening up public data to improve transparancy and efficiency
Paul Davidson – Opening up public data to improve transparancy and efficiency
 
Web Mapping
Web MappingWeb Mapping
Web Mapping
 
Smart city case studies in the USA
Smart city case studies in the USASmart city case studies in the USA
Smart city case studies in the USA
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaKai Wähner
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 

More from Kai Wähner (13)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 

Recently uploaded

User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeKaylee Miller
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdfOffsiteNOC
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfMind IT Systems
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridMathew Thomas
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...jackiepotts6
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.Ritesh Kanjee
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houstonjennysmithusa549
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...MyFAA
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurPriyadarshini T
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developersmichealwillson701
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsUniversity of Antwerp
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptxAGATSoftware
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easymichealwillson701
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energyjeyasrig
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJpolinaucc
 

Recently uploaded (20)

User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller Resume
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM Grid
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houston
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developers
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data Mesh
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easy
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energy
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJ
 

Apache Kafka in the Public Sector (Government, National Security, Citizen Services)

  • 1. The Rise of Data in Motion in the Public Sector Citizen Services, National Security, Smart City, and more Kai Waehner Field CTO contact@kai-waehner.de linkedin.com/in/kaiwaehner @KaiWaehner www.confluent.io www.kai-waehner.de kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 2. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 3. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 4. The world is changing. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 5. Normal Life becomes Software kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 6. National Security becomes Software kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 7. Lighting becomes Software kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 8. Parking becomes Software kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 9. Public Sector kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 10. Collaboration is Key for Success Car Maker Supplier Government Telco Mobility Services Cloud Providers Software Providers kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 11. Data in Motion across the Public and Private Sector Real-time data without context! Possible with simple messaging solutions. Real-time data with context! What does the ride cost? How long does the driver need? Data is correlated in real time. https://www.confluent.io/events/kafka-summit-europe-2021/development-of-dynamic-pricing-for-tours-using-real-time-data-feeds/
  • 12. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 13. This is a fundamental paradigm shift... Infrastructure as code Data in motion as continuous streams of events Future of the (private or public) datacenter Future of data Cloud- native Event Streaming kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 14. Real-time Data beats Slow Data. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 15. Real-time Data beats Slow Data. Transportation Traffic planning Parking System Public transportation Citizen Services ID application process Covid tracking Utilities billing Law Enforcement Border control Security monitoring Surveillance Cybersecurity Risk classification Intrusion detection Incident response Fraud detection
  • 16. Data in Motion in the Public Sector Your Processes as Streams of Events, powered by Apache Kafka Intelligent Parking System Emergency Service Traffic Research Team Traffic Planning System Car Sensor Data Smart City Example
  • 17. Public Sector – Examples for Data in Motion Citizen Services • Health services, e.g. hospital modernization, track & trace - Covid distance control • Public administration - reduce bureaucracy, data democratization • eGovernment - Efficient and digital citizen engagement, e.g. personal ID application process Smart City • Smart driving, parking, buildings, environment • Waste management • Open exchange – e.g. mobility services (1st and 3rd party) Energy • Smart grid and utilities infrastructure (energy distribution, smart home, smart meters, smart water, etc.) National Security • Law enforcement, surveillance, police/interior security data exchange • Defense and military (border control, intelligent solider) • Cybersecurity for situational awareness and threat intelligence kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 18. Data in Motion is the Underpinning of an Event-driven Architecture MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 19. Mainframe CSV File STREAM PROCESSING CONNECTORS Example Architecture for Data in Motion ksqlDB KStreams Bi-directional real-time integration between legacy and modern worlds Dashboard Oracle DB ORACLE CDC SOURCE CONNECTOR Real-time App
  • 20. Digital Infrastructure A Prerequisite for Data in Motion in the Public Sector Digital Infrastructure Data Centers Fixed and Mobile Connectivity Network Infrastructure Internet of Things and Devices Platforms and Applications kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 21. Data in Motion The Foundation for a Smart Life Mobility Services Citizen Services Smart Buildings Cybersecurity Smart Grid Weather Data Citizen Data Traffic Data Security Alerts Streams of real time events Personal Data Weather Service Connected Cars Routing Service Traffic Alerts Surveillance kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 22. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 23. Smart City Digital Buildings, Connected Vehicles and Things, Citizens Real-time Operations, Logistics, Predictive Maintenance, Security Traffic Data Vehicles, Accidents, Traffic Lights, etc. Surveillance Data Buildings, People, Distance Measuring, etc. Mobility Services Multi-modal integration, E-car Charging Stations, Parking Lots, etc. Drone Data Deliveries, Survey/Inspection of Traffic, People, Construction Areas etc. Edge Analytics Bidirectional Edge to Cloud Integration Data Ingestion Stream Processing Data Integration Logistics Track & Trace Routing Monitoring Alerting Command & Control Batch Analytics Reporting Machine Learning Backend Systems Oracle, SAP, OSIsoft PI, etc. X = Event Streaming X = Other Technologies Bi-Directional Hybrid Cloud Replication kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 24. CRM Real-Time Location System (RTLS) for Asset Tracking Customer data Advanced Planning and Scheduling (APS) Manager Get report API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Wavelength Public 5G Campus #1 5G Wavelength Campus #2 5G Connected Services kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 25. Why the “right time” is “real time”… kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 26. The Smarter City Nervous System Producers Consumers • Client Libraries • Kafka Connectors • Client Libraries • Kafka Connectors X Consumer Groups Streams kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 27. The Smarter City Nervous System Topic (observation-raw) Metadata and Geo (lat/long) Enrichment Topic (observation-meta) Streams Health Score & Incident Detection Topic (observation-out) Elasticsearch Kafka Connect Elasticsearch Sink Connector kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 28. Ohio Department of Transportation Hub for all things autonomous, connected, and smart mobility initiatives in Ohio Intelligent Transportation System to help reduce traffic and fatalities on the road in real-time Confluent Cloud + Confluent Operator for applications (Kafka Connect, KSQL, Kafka Streams, Custom Code) for low operations efforts and high flexibility kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 29. Ohio Department of Transportation kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 30. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector Customer timetable Operational timetable Assignments Railway station knowledge Dispositions Train positions Matching Aggregation Consolidation Apache Kafka Analysis Railway station Trains Mobile Apps Employees Deutsche Bahn AG | Reisendeninformation RI-Plattform
  • 31. RI-Platform @ Deutsche Bahn (German Railway) Traveler Information with Single Point of Truth 3rd Party Integration with Google Maps Real-time train schedule updates Reach new people and expand business kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector https://railway-news.com/db-partners-with-google-maps-on-journey-planning/
  • 32. FREE NOW Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc. Various information about the trip, location and business performance kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 33. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 34. NAV (Norwegian Work and Welfare Department) 23,000 employees USD 50Bn disbursements per year Assist people through all phases of life Unemployment benefits, health insurance, social security, pensions, parental benefits Kafka Summit: https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/ Webinar: https://videos.confluent.io/watch/qekArwtPMKE7f6hKRcfTdX kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 35. From the vision to the project kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 36. From the vision to the project kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 37. Life as a Stream of Events (by leveraging event streaming technology) kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 38. Reducing bureaucracy (by avoiding spaghetti architectures) kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 39. Reducing bureaucracy (by domain-driven design) kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 40. Diverging forces (by integration, but true decoupling) kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 41. VA (Veterans Affairs) The United States Department of Veterans Affairs (VA) is a department of the federal government charged with integrating life-long healthcare services to eligible military veterans Improving veteran benefit services for ratings, awards, claims through data streaming Assess, route, and verify the service request in real-time. Improve the experience for veterans and their families. Reduce management and operations complexity. Examples: Change in status of a claim, requests for additional documentation, context when calling into the call center, due benefits when checking in at a hospital Confluent for true decoupling, real-time integration, omnichannel communication. Components include Kafka Streams, ksqlDB, Oracle CDC, Tiered Storage, etc. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 42. University of California, San Diego https://www.confluent.io/customers/uc-san-diego/ Transition to build out their online learning platform due to the pandemic; plus new #1 priority: Student testing process Confluent as Integration-platform-as-a-service (iPaaS) middleware layer to set data in motion for 90 million records a day Increased time to market with agile development and decoupled applications Opening up new revenue streams for other UC campuses - including the UC Office of President, and tracing student health kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 43. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 44. Regulated vs. Deregulated Markets Example: US Regulated vs. Deregulated Electricity Market kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector Source: https://www.electricchoice.com/map-deregulated-energy-markets/
  • 45. The Energy Sector is Changing… Flexible, scalable, secure smart grids kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 46. The Energy Sector is Changing… https://electrek.co/2021/08/27/tesla-plans-enter-texas-deregulated-energy-market-massive-battery/ kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 47. Tesla Trillions of messages per day for IoT use cases https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/ https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/
  • 48. CRM 3rd party payment provider Real-Time Asset Management Customer data Payment processing and fraud detection as a service Manager Outage Management API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Energy Production and Distribution with a Hybrid Architecture Wavelength AWS Cloud VPC Carrier #1 5G Wavelength Carrier #2 5G Smart Meters Smart Buildings kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 49. Smart Grid Upstream Operations Management Well Completion Operations Customer data Truck schedule Payment data Route details Streams of real time events Real-Time Supply Chain Management Event Streaming for Energy Production (Upstream + Midstream) at the Edge with a 5G Campus Network Wavelength Cloud VPC Public 5G Campus 5G Smart Home kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 50. Energy Production at the Disconnected Edge (Upstream) Time P C1 C2 C3 Predictive Analytics Human Machine Interface Predictive Maintenance Sensors Always on (even “offline”) Replayability Reduced traffic cost Better latency kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 51. Edge Processing at the Intelligent Gas Station (Downstream) Time P C3 C1 C2 Upsell in the store Context-specific coupon (e.g. carwash not in use à 30% off)) Real-time inventory and intelligent pricing kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 52. Leipziger Stadtwerke Kunbus Edge Devices (PI with custom Linux) OTA (over the air update) with Mender Kafka for IT Integration, Data Processing, Innovative Digital Services Kafka Digital Customer Interface to connect public utilities, grid operator, housing industry, end-consumer, industrial customer https://m.youtube.com/watch?v=5tAkpxvvHA4 kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 53. Cybersecurity – The Threat is Real! https://www.mckinsey.com/business-functions/risk-and-resilience/our-insights/the-energy-sector-threat-how-to-address-cybersecurity-vulnerabilities kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 54. Data in Motion in the Public Sector 1) The World is Changing 2) Life is a Stream of Events 3) Smart City 4) Citizen Services 5) Energy and Utilities 6) National Security kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 55. Cybersecurity The threat is real! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on” kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 56. Data in Motion The Foundation for Cybersecurity Industrial OT Enterprise IT Consumer IoT Logs Personal Sensors Securit y Streams of real time events Connected Vehicles Cyber Security Continuous Data Correlation Monitoring Alerting Proactive Actions kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 57. Firewalls & Network Devices Antivirus Access Logs Intrusion Detection Audit Logs Text Files Binary Files Databases APIs Network Flows Syslog The Data kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 58. Legacy SIEM needs to evolve Forwarder Network traffic Firewall logs RDBMS Application logs Adaptors Beats Sensor Data Challenges: • Proprietary forwarders that can only send data to single source • Data locked from being shared • Difficult to scale with growing data volumes • Prohibitively high indexing costs • Unable to filter out noisy data • Slow batch processing HTTP proxy logs kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 59. Forensic Archive AI/ML Build A Real-time SIEM / SOAR Pipeline Filter, transform aggregate APP SIEM Index Search Curated streams HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs HTTP proxy logs QRadar Arcsight Splunk Elastic Machine Data spooldir (files), SNMP Traps, Databases, Sftp, MQs kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 60. Confluent + Splunk SIEM Reference Architecture (assuming that Splunk UFs already exist – otherwise integrate directly via Kafka to Splunk) Splunk Universal Forwarders (UFs) Windows Event Logs SNMP Syslog Watchlist Zeek IDS Splunk Heavy Forwarders Machine Learning Splunk S2S Connector Splunk HEC Splunk Indexers Splunk Search Head Real-time stream processing with ksqlDB ... 3rd party apps / ecosystems Moving log data from Splunk UFs to your destination of choice kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 61. Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 62. AI/ML Border Protection with Kafka for Security and Surveillance Image Recognition, Data Correlation, Threat Detection and Alerting in Real-Time Filter, transform aggregate APP SIEM Index Search Curated streams Forensic Archive HDFS S3 Big Query CDC Syslog Network traffic Firewall logs Database Application logs HTTP proxy logs QRadar Arcsight Splunk Elastic Video Streams TensorFlow H2O.ai Datarobot
  • 63. Customer/Site 1 Customer/Site 2 (Air-Gapped) Customer/Site N Central Data Center / Public Cloud Event Streaming in Air-Gapped and Zero Trust Environments Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure Confluent Data Diode as Unidirectional Gateway for Zero Trust
  • 64. Sensor A Sensor B Sensor X MQTT Confluent Platform (Single Broker) Confluent Platform deployed on a Raspberry Pi and leveraging Cluster Linking to publish sensor data to Command Post in a DDIL environment. Command Post running Confluent Platform aggregating information from Squires and other sensor data Weather Personnel Logistics Targets Sensor data published to Command Post when connected to network Enhanced Situational Awareness Internet of Battle Things (IoBT) at the Edge kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 65. Internet of Battle Things (IoBT) and Smart Soldiers at the Edge Live Demo
  • 66. NASA enables real-time data from Mars • Real-time data extends into the far frontiers via its Deep Space Network (DSN) • Data grows exponentially from spacecraft and other systems • Responsive citizen engagement, real-time situational awareness, anomaly detection, event-driven missions and security operations • Global fabric of Kafka clusters allows for real-time sharing, event streaming, and other combinations of both real-time and historical data https://federalnewsnetwork.com/big-data/2021/09/nasa-adopting-apache-kafka-to-enable-real-time-data-from-mars/ kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 68. The Rise of Data in Motion 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 69. I N V E S T M E N T & T I M E V A L U E 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management... kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector
  • 70. Car Engine Car Self-driving Car Confluent completes Apache Kafka. Cloud-native. Everywhere. kai-waehner.de - @KaiWaehner - Data in Motion in the Public Sector