Apache Kafka for Cybersecurity and SIEM / SOAR Modernization

Cybersecurity and SIEM / SOAR Modernization
Data in Motion for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka
Agenda
1) Cybersecurity in 202X
2) Data in Motion as Cybersecurity Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization

What is Cybersecurity?
Protection of computer systems and networks from information disclosure, theft,
Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors
4

Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, SolarWinds …
Damage: Billions of dollars
”Supply chain attack”
Digital Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
Billions of devices

Supply Chain Attack
Targeting less-secure elements in the supply chain
6
https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains
https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002

7
“It takes 20 years to build a
reputation and few minutes
of cyber-incident to ruin it.”
Stephane Nappo

Security Attacks are Exploding with Significant Costs
$3.86 MILLION
Average cost of a data
breach 1
280 DAYS
Average time to identify
and contain a breach 1
+$17.5 BILLION
SIEM and IT monitoring
spend forecast in 2025 2
1 Report on Cost of Data Breach
2 Report on IT and Security Spend

Security Attacks are Exploding with Significant Costs
https://www.bankinfosecurity.com/tracking-darkside-ransomware-gangs-profits-a-16682

SECURITY Landscape
Security SIEM
Access Control
(RBAC, Audit Logs, …)
Real-time
Monitoring
(Logging, SiteOps, …)
Encryption
OT Security
Hardware-
based Security
Cybersecurity
CYBERSECURITY is a key piece of the security strategy
SIEM and SOAR are a (key) piece of the cybersecurity strategy
SOAR

SECURITY
Security SIEM
Encryption
OT Security
Hardware-
based Security
Cybersecurity
How would you have a holistic view and understanding of all the events and potential abuses that are taking place within your organization?
Collect and correlate the different activities happening on critical networks
CYBERSECURITY is a key piece of the security strategy
SIEM and SOAR a (key) piece of the cybersecurity strategy
Sometimes
not needed
(in DMZ /
air gapped env)
Complex and
error prone
No help
against insiders
Continuous
real-time
data correlation
required
SOAR
Avoid risk (change operations) +
Transfer some risk (buy insurance)
Real-time
Monitoring
(Logging, SiteOps, …)
Access Control
(RBAC, Audit Logs, …)

Cybersecurity
Act in real-time to threats!
Faster detection
and response
ultimately leads to
better prevention
Reduce
Mean Time to Detect (MTTD)
and
Mean Time to Respond
(MTTR)

13

The Four Stages of an Adaptive Security Architecture
https://www.gartner.com/smarterwithgartner/build-adaptive-security-architecture-into-your-organization/

Key Challenge: Find the Needle(s) in the Haystack
Detect true positives in real-time
• Threat detection
• Intrusion prevention
• Anomaly detection
• Compliance auditing
• Proactive response
Reduce false positives
• Automation
• Process big volumes of data in real-time
• Integration of all sources
• No ‘ignore’ on certain events
• Creation of filters and correlated event rules
• Improve signal-to-noise ratio (SNR)
• Correlate “collection of needles” in “signature needle”

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

This is a fundamental paradigm shift...
18
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Real-time Data in Motion beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Compliance
Trading
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

Real-time Data in Motion beats Slow Data.
Security
Access control and encryption
Regulatory compliance
Rules engine
Security monitoring
Surveillance
Cybersecurity
Risk classification
Threat detection
Intrusion detection
Incident response
Fraud detection

Data in Motion
The Backbone for Cybersecurity
Industrial
OT
Enterprise
IT
Consumer
IoT
Logs Personal
Sensors Security
Streams of real time events
21
Connected
Vehicles
Cyber
Security
Continuous
Data Correlation
Monitoring
Alerting
Proactive Actions

Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
22

Low-Latency Performance At High Throughput
Designed for high
performance at massive
scale to support your
security needs
few ms
<10ms
end-to-end latency at
massive throughput
(i.e. GBs / sec)
Synchronize data across your
organization in real-time
Take action on insights from your
data immediately
Remove data silos by moving
from batch to event streaming
Read more about our internal performance benchmarking

Enrich And Transform With Stream Processing
{
"timestamp": "2020-02-
02T18:36:40.325Z",
"event": {
"host": "netsec1",
"type": “suricata_eve",
"subtype": "http"
},
"node": {
"ipaddr": "192.168.9.9",
"subtype": "ubuntu",
"hostname": "netsec1"
},
"conn": {
"ip_protocol": "TCP",
"src_addr": “192.168.1.245",
"src_port": 49445,
"dst_addr": "12.187.9.10",
"dst_port": 443
}
}
Let’s resolve these to hostnames
Who owns this public IP and where is
it located? What is its reputation?
Translate this to a user friendly
service name
Was the connection successfully
established?

End-to-End Cybersecurity
with the Kafka Ecosystem
Personel
Crew, Cargo
Vessel
Fuel Consumption, Speed,
Planned Maintenance
Tracking
Position, Course, Weather, Draft
Drone or Satellite Relay
COMMs Resilient Kafka
Edge Analytics
Bidirectional Ship Edge to Cloud, Shore Edge to Cloud
Relay Ingestion
Data
Integration
Streaming Analytics
Machine Doing
On-Prem Systems
Bi-Directional Hybrid Cloud Replication
ON SHORE
ON PREM
Staging, Filtering
Shore Edge Analytics

Shipping Industry
Cybersecurity, Situational Awareness, Threat Intelligence
Disconnected and Air-gapped Environments, SIEM/SOAR
Personal Data
Crew, Cargo
Vessel Data
Fuel Consumption, Speed,
Planned Maintenance
Automatic Identification System (AIS)
Unique Identification,
Position, Course, Weather, Draft
Drone Data
Deliveries,
Survey/Inspection
of Assets such as Oil Rigs,
Pipelines, Offshore Turbines
Edge Analytics
Bidirectional Edge to Cloud Integration
Data Ingestion
Stream
Processing
Data
Integration
Logistics
Track&Trace
Routing
Monitoring
Alerting
Command&Control
Batch Analytics
Reporting
Machine Learning
Backend Systems
X = Event Streaming
X = Other Technologies
Bi-Directional Hybrid Cloud
Replication

SIEM / SOAR
Situational Awareness
Operational Awareness
Intrusion Detection
Signals and Noise
Signature Detection
Incident Response
Threat Hunting & Intelligence
Vulnerability Management
Digital Forensics
…
was not built for cybersecurity!

Integrate with all legacy and modern interfaces
Record, filter, curate a broad set of traffic streams
Let analytic sinks consume just the right amount of data
Drastically reduce the complexity of the enterprise architectures
Drastically reduce the cost of SIEM / SOAR deployments
Add new analytics engines
Add stream-speed detection and response at scale in real-time
Add mission-critical (non-) security-related applications
…
is the backbone for cybersecurity!

Every enterprise is different…
Flexibility is key for your cybersecurity initiative!
Confluent is an independent foundation.
29

Kafka Connect
Confluent
Various Data
Producers
Flexible Scalable Real-Time Backplane for the Cybersecurity Platform
Splunk TensorFlow
Kafka Forwarder
TensorFlow +
Kafka plugin
Event Streaming Platform
OT Domain SIEM Domain Analytics Domain
30
Huge volumes of
real-time data from
various Kafka topics
Backpressure handling
and a low velocity
Kafka topic
High velocity, raw
Kafka topic for
forensics and ML

The Confluent Curation Fabric for Cybersecurity

Deliver Contextually Rich Data
To Reduce False Positives
Application Logs
Network Logs
Database logs
OS Logs
Collect all data sources
into Confluent Platform
Filter events streams
and only send priority
events to SIEM
Shorten SIEM retention
window
Offload fast query
and search
Send high
priority data
to SIEM
Send all data
to S3/HDFS for
cold storage
Open up data access
to new use cases

Situational Awareness is a State of Knowledge
34
“Situation awareness is the perception of the elements in the environment within a volume of
time and space, the comprehension of their meaning, and the projection of their status in the
near future.”
Endsley, M. R. SAGAT: A methodology for the measurement of situation awareness (NOR DOC
87-83). Hawthorne, CA: Northrop Corp.
X
X X
O X

Cyber Situational Awareness
is the subset of all situation awareness necessary to support taking actions in cyber
35
Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 1995, 37(1), 32-64

Situational Awareness includes all Environments
36
Beyond network: All environments, including application data, logs, people, processes
Not just view the dashboard, but understand what’s going on in real-time
Find relevant data to create critical (rare) alerts
Three segments:
• Perception of the elements in the environment
• Comprehension of the situation
• Projection of future status
X
X X
O X

Human – Computer Interface for Decision Making
37
https://www.youtube.com/watch?v=mPJdzzm67sg

Human – Computer Interface for Decision Making
38
https://www.youtube.com/watch?v=mPJdzzm67sg

Firewalls & Network Devices
Antivirus
Access Logs
Intrusion Detection
Audit Logs
Text Files
Binary Files
Databases
APIs
Network Flows
Syslog
The Data

Transactions
Low Velocity, Low Volume
Netflow / PCAP
High Velocity, High / Ridiculous Volume
Ingested via Network Analyzer Gateway
Logs
Low Velocity, Moderate Volume
Store PCAP headers
in Tiered Storage
or
3rd
Party like Corelight
as intermediary
Data Producers

Streams
Streams
Streams
Streams
Event type-specific
parsing and
normalization
logs-conn-shared
logs-resolve-names
logs-geoip-asn-iprep
Streams
Streams
logs-index
established connection
and client/server
detection
DNS name resolution
GeoIP, IP Reputation
and Autonomous
System lookup
Data Normalization and Enrichment
à Improve the signal + filter to lower the noise

SIEM Forwarder
Threat Detection
Near real-time
Data Consumers
No constraints on integration flows
Data curation on the fly
Flexible choice of (multiple) consumers
Sink to Data Lake
Analytical Workloads
Batch
Native Kafka App
Transactional Workloads
Real-time

Sigma
43
• Open-source framework
• Domain specific language (DSL)
• Specify patterns in cyber data
https://github.com/SigmaHQ/sigma

Confluent Sigma
44
Sigma Rules
(YAML)
Source
Data
Sigma
Processor
SIEM
Applications
Enriched
Dashboards
Filtered
Detections
Filter,
Transform,
Aggregate
Anomalies detected in the Stream Processing layer and not in the SIEM tools

Sigma Rule
Detections
• List of detections for each condition
• Single or list of values
• Individual values or regex
• Detection names can also include
operators (ex. name|endswith,
name|contains, name|greater_than)
• Aggregations and windowing
Conditions
• Nested conditions based on defined
detections
Detection Names
• Generic Sigma names defined
• Translated during parsing to meet end
SIEM tool using field mapping file

Sigma Rule with Aggregation
Windowing
• timeframe value for aggregations
• seconds, minutes, hours, …
Aggregation
• count, min, max, sum, average
• various operations (<, >, =, …)
Additional Streams
• A sigma rule with an aggregation
condition will spawn an additional stream
• Additional stream needed to perform
aggregation
• Intermediate topic and table produced
• Output will be published to defined topic

Detection Parser
• Parses all defined detections
• Common operators implemented
Condition Parser
• Flat conditions (not nested)
Sigma Parsing
Aggregation Parser
• Spawns a new stream
• Supports count – easy to add others

Confluent Sigma
Sigma Stream Processors
Zeek Data and
Detections Viewer
Sigma Rule Editor
sigma rules topic
CONN
DHCP
HTTP
SSL
DNS
x509
Zeek Data
zeek topics
sigma rules topic
dns topic
dns
detections
topic
dns
detections
topic

Confluent Sigma
Sigma Stream Processors
Zeek Data and
Detections Viewer
Sigma Rule Editor
sigma rules topic
DNS
dns
detections
topic
dns topic
rule parsing,
filtering,
aggregation,
windowing
sigma
rules
cache
CONN
DHCP
HTTP
SSL
x509
Zeek Data

Cisco ThousandEyes Endpoint Agents
Gain visibility from any employee to any application over any network
Proactive and real-time monitoring of application experience and network connectivity
Kafka Streams for stateful network tests and interactive queries for fetching results
Kafka Streams for windowed aggregations for alerting use cases
Kafka Connect for integration with backend systems such as MySQL, Elastic, MongoDB
50
https://www.thousandeyes.com/blog/kafka-streams-in-the-endpoint-agent

MetaRouter
Secure, scalable and flexible clickstream data routing
Seamless platform to process and route one billion events per day
Utmost security for high-volume customers
51
https://blog.metarouter.io/why-we-love-kafkas-open-source-data-pipelines
https://blog.metarouter.io/how-we-process-one-billion-events-per-day-with-kafka

Threat Intelligence
53
Mitigate harmful events in cyberspace
Proactive cybersecurity posture that is predictive, not just reactive
Bolster overall risk management policies
Improved detection of threats
Better decision-making during and following the detection of a cyber intrusion
See the whole board, more quickly.
See around corners.
See the enemy before they see you.

Transactions vs. Analytics
54
Threat intelligence =
awareness-in-motion
The PATTERN is
valuable, not the data.

Streams
logs-index
Authorized access
using RBAC
Machine Learning
Predictions via UDFs
PII Anonymization
logs-alerts
logs-index-gdpr
Analytics and Actionable Insights in Motion
Make sense of the signal and the noise of the data
Continuous signature processing
Prevent, contain, and neutralize threats proactively
Access for
data science teams

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
“Ingests tens of terabytes of data each day and
transforms it into actionable insights through streams
processing, context-smart applications, and advanced
analytics techniques. Kafka serves as a massive data
pipeline within the platform. It provides us the ability to
operate on data in-stream, enabling us to reduce Mean
Time to Detect (MTTD) and Mean Time to Respond
(MTTR). Faster detection and response ultimately leads
to better prevention.”

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

Crowdstrike
Cybersecurity cloud solution for endpoint security, threat intelligence, and cyberattack response services
Ingest ~5 trillion events per week into the cloud platform
Very important that this platform is available, operational, reliable and maintainable
Four critical roles for operating the streaming data infrastructure :
Observability, Availability, Operability, Data Quality
58
https://www.crowdstrike.com/blog/

Digital Forensics
60
• Application of science to criminal and civil laws, mainly during criminal investigation
• Forensic scientists collect, preserve, and analyze scientific evidence during the course of
an investigating digital media in a forensically sound manner
• Identify, preserve, recover, analyze and present facts and opinions about the digital
information

Distributed Digital Forensics at Scale with Kafka and Spark
61
• Digital Forensics Compute Cluster (DFORC2)
• High Speed Distributed Computing Capability for Digital Forensics
• Extended the digital forensics platform Autopsy with Kafka and Spark to add distributed
compute power for data processing
https://publications.waset.org/10007817/digital-forensics-compute-cluster-a-high-speed-distributed-computing-capability-for-digital-forensics

Forensics on Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• Capture the complete attack vector
• Playback of an attack for the
training of humans or machines
• Create threat surface simulations
• Compliance / regulatory processing
Real-time Consumer for
an automated actuation
Consumer of Historical Data

Old New
Commit Log
Real-time processing
Syslog WEC Zeek
Near real-time ingestion
for visualization
Near real-time ingestion
for batch analytics
Batch ingestion
of consolidated data
One-time batch load
for model training
Kafka’s Distributed Commit Log Captures the Running History of Signals
Enables real decoupling and domain-driven design
Absorbs velocity and volume to protect and stabilize slow consumers
Organic truncation (retention time)

Confluent Tiered Storage for Kafka for Forensics of Historical Data
64
(Only available in Confluent Platform)
Store data forever
Hot and cold storage
Cheap object store
Easy scale up/down
No changes in clients

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
The Role of AI and Machine Learning for Forensics
Model Training with Kafka and TensorFlow I/O
https://github.com/tensorflow/io
65
Model X
(at a later time)

“CREATE STREAM AnomalyDetection AS
SELECT facility_code, detectAnomaly(syslog_values)
WHERE severity_level = ’Warning’
FROM syslog_source_topic;“
User Defined Function (UDF)
66
The Role of AI and Machine Learning for Forensics
Model Deployment with ksqlDB and TensorFlow

A Kafka cluster
powered by
Tiered Storage is an
affordable solution
for both real-time
analytics and digital
forensics at scale!
Digital Forensics

Zero Trust
69
• EVERYTHING needs protecting, not just firewalls and computing assets
• It is not Cyber Network Security, but Threat Intelligence that includes HUMAN intelligence
• Safe IT/OT integration at industrial sites
• There is no such thing as a “unidirectional firewall”
• Hardware and / or software-based
• Replica servers instead of direct access
• Surveillance for Safety and Theft Protection

Authenication and Authorization are important
70
• RBAC
• AD/LDAP
• Audit Logs
• Encryption
• BYOK
• …
… but just one piece of the puzzle!

IT/OT Architecture Layers
71

Customer/Site 1 Customer/Site 2 Customer/Site N
Central Data Center / Public Cloud
Data in Motion in Air-Gapped and Zero Trust Environments
Secure intermediary (on Linux) between the existing (Windows) hardware and modern (Linux) infrastructure
Handle operational awareness and situational awareness at the same time

Safety and Theft Prevention
Surveillance is Awareness, for either Ops or Cyber
73
Business need:
• Reduce cost for security patrols and avoid time traveling to remote locations
• Reduce security and safety incidents
• Avoid illegal tapping, vandalism, leakage, and emissions
Current State:
• Security personnel perform regular patrols
• Depending on location of operations, roads can be challenging with rough terrain
Desired State:
• Utilize thermal imaging cameras to monitor tank levels, spot leaks in tanks, pipelines, and facilities
• Minimize emissions with less risk of business interruption and regulatory compliance
• Leverage drones to patrol sites to prevent vandalism and theft
• Monitor pipeline condition and integrity, perform routine inspection assessments with autonomous
drones
• Improve emergency response to spills and incidents 19

AI/ML
Border Protection with Kafka for Security and Surveillance
Image Recognition, Data Correlation, Threat Detection and Alerting in Real-Time
Filter,
transform
aggregate
APP SIEM
Index
Search
Curated
streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
Database
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Video Streams
TensorFlow
H2O.ai
Datarobot

Hybrid Architecture – Initiation from On-Prem
Replicator / Cluster Linking always initiate communication from the on prem site

Unidirectional Networks
When a Firewall is NOT Enough!
76
• Secure OT – IT bridge
• Hardware based data diode or unidirectional gateway
• Real time monitoring of safety-critical networks
• Secure cloud connectivity of critical OT networks
• Database replication and file transfer
• Transferring application and operating system updates
• Vendors use different terms: Unidirectional network =
Unidirectional Gateway = Data Diode

Unidirectional Security Gateway
77
Source: Waterfall

Firewall vs. Unidirectional Gateway
78
https://waterfall-security.com/wp-content/uploads/2021/06/firewalls-vs-unidirectional-gateways-v1.1.pdf

Unidirectional Networks
Various architecture options, including database replication, file transfer, device emulation data
sniffing, remote diagnostics and maintenance, cloud integration, and scheduled updates
79

Confluent Data Diode
https://docs.confluent.io/kafka-connect-data-diode
Software-based Unidirectional Gateway for Zero Trust Security Architectures
Streaming from Industrial Networks to Enterprise Networks
UDP-based Source and Sink Kafka Connectors for High Volume and Open Architecture
Run over a one-way/UDF hardware interface (Ethernet cable, OWL Cyber, etc.)
Optionally/eventually do filtering, anomaly detection, analytics, receive upstream traffic, etc.
Site
Site
Site
Work
Center
Apache
PLC4x
Work
Center
Cloud
Streams processing
Data Lake
Data Diode
UDP Sink
Data Diode
UDP Source
Kafka
Cluster
NUC Pair
Kafka
Instance

Site
Site
Site
Work
Center
Apache
PLC4x
Work
Center
Cloud
Streams processing
Data Lake
Data Diode
UDP Sink
Data Diode
UDP Source
Kafka
Cluster
NUC Pair
Kafka
Instance

Site
Site
Site
Work
Center
Apache
PLC4x
Work
Center
Azure
Cloud provider(s) and number of clusters TBD
Streams processing
Azure data Lake
(or similar)
The “clean” Kafka instance can
optionally/eventually do filtering,
anomaly detection, analytics,
receive upstream traffic, etc
Hardware DD or Private
UDP fabric (1M CAT6)
Policy Dependent
Streams processing in
cloud for view across
sites
Several methods to move
data due to site COMMs
Rep, MRC, CLink
Sink Connectors
Isolate dirty networks from
clean networks
One-way ONLY
2 sets for Duplex Ops
NOT guaranteed (udp)
64KB datagram limit (udp)
Confluent Cloud
Kafka
Instance
Data Diode
UDP Sink
Air-Gapped Edges
Data Diode
UDP Source
Kafka
Cluster
NUC Pair

The Challenge with SIEM / SOAR Platforms
Forwarder
Network traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Machine Data
HTTP proxy logs
Splunk
ArcSight
Elastic
Proprietary forwarders can only
send data to single tool
Data is locked from being shared
Difficult to scale with growing
data volumes
High indexing costs of proprietary
tools hinder wide adoption
Filtering out noisy data is complex
and slows response
No one tool can support all
security and SIEM requirements

Cybersecurity Requirements
Real-time data access to all your
security experts
Historical and contextual data
access for forensic reporting
Rapid detection of vulnerabilities
and malicious behavior
Predictive modeling of security
incidents using newer capabilities
like ML/AI
Scalable platform that grows with
your data needs
Ingest diverse, voluminous, and
high velocity data at scale
Reduce indexing costs and OPEX
managing legacy SIEM
Enable data portability to any SIEM
tool or downstream application
Deliver the right insights, to the
right people, at the right time
Enable an open, real-time and
portable data architecture

Forensic
Archive AI/ML
Build A Real-time SIEM / SOAR Pipeline
Filter,
transform
aggregate
APP SIEM
Index
Search
Curated
streams
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Machine Data
spooldir (files), SNMP Traps,
Databases, Sftp, MQs

Datastores
Web / Mobile
PoS Systems
SaaS
Applications
IoT Sensors
Legacy Apps
and Systems
Machine data
ML Engines
Ingest and
aggregate events
from everywhere
Join, enrich, transform and
analyze data in real-time
Store and persist events in a
highly available and scalable
platform
BI Tools
Standardize schemas to
ensure data compatibility
to all downstream apps
SIEM and
Observability
tools
Data lakes &
warehouses
Real-time alerts
and dashboards
Convert raw events into clean, usable data Unlock value
Bring all your data together
in real-time
Integrate with 100+
pre-built connectors
Applications
How Confluent Works

Old New
Scan
Scan
Scan
Each SIEM has its own position (offset)
Raw-Big-Data-Topic
Small-Data-Topic
Preprocess
and
consolidate

Confluent + Splunk SIEM Reference Architecture
Splunk
Universal
Forwarders
(UFs)
Windows
Event Logs
SNMP
Syslog
Watchlist
Zeek IDS
Splunk
Heavy
Forwarders
Machine
Learning
Splunk S2S
Connector
Splunk
HEC
Splunk
Indexers
Splunk
Search
Head
Real-time stream
processing with
ksqlDB ...
3rd party apps
/ ecosystems
Moving log data
from Splunk UFs
to your
destination of
choice

SYSLOG
MAINFRAMES
DATABASE
CONNECTORS
CONNECTORS
ELASTICSEARCH CONNECTOR
RAW STREAMS
PROCESSED STREAMS & NORMALISED RECORDS
ksqlDB - CONTINUOUS QUERIES
TRANSFORMATION, JOINS, FILTERING & EVENT DETECTION
SEARCH
VISUALISE /
OBSERVE
SECURITY AIOPS
EVENTS GOVERNANCE - SCHEMA REGISTRY
ELASTICSEARCH INDEXES
MATERIALISED VIEWS
YOUR API
Confluent + Elastic SIEM Reference Architecture

Leveraging the Kafka Platform for Migration and Backpressure Handling
91
https://medium.com/lets-xplore/how-we-replaced-splunk-at-100tb-scale-in-120-days-e5a59db63f6

Splunk S2S Source
Connector
Allows customers to cost-effectively &
reliably read data from Splunk
Universal Forwarders to Confluent
Platform / Kafka
Enables users to forward data from
universal forwarders into a Kafka topic to
to unlock analytical capabilities of
data.
Cost Savings by avoiding use of Heavy
Forwarders and directly sending data to
Kafka via Universal Forwarders
Unlock data from multiple UFs across
Windows and Linux Servers to build real-
time applications.
Ready solution to integrate with Splunk,
helps $ savings of ~12-24 engineering
months on average to design, build, test,
and maintain highly complex Splunk
integrations & connectivity.

Splunk S2S Connector
93
CONFIDENTIAL

Palo Alto Networks SOAR
94
Cortex Data Lake collects, transforms and integrates
enterprise’s security data to enable Palo Alto Networks
solution
Billions of messages pass through the Kafka clusters
Leverages various Confluent components
Multiple Kafka clusters in production, size from 10 to just
under a 100 brokers each
Design principles:
• Cloud agnostic infrastructure
• Massively scalable
• Aggressive ETA on integrations
• Schema versioning support
• Microservices architecture
• Operational efficiency
https://medium.com/engineering-at-palo-alto-networks

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
96

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
97

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

Confluent Platform
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Hybrid and Global Event Streaming
Streaming Replication for Transactional and Analytics Workloads across Hybrid and Multi-Cloud
Integrate with all Data Sources, Data Lake, SIEM/SOAR, and any Applications
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
100

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka for Cybersecurity and SIEM / SOAR Modernization

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Kafka for Cybersecurity and SIEM / SOAR Modernization

Similar to Apache Kafka for Cybersecurity and SIEM / SOAR Modernization (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka for Cybersecurity and SIEM / SOAR Modernization