Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry

Data Streaming in the Defence Industry
Data in Motion for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones
Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
confluent.io
kai-waehner.de
linkedin.com/in/kaiwaehner

@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka
Agenda
1) Defence and Cybersecurity in 202X
2) Data in Motion as Defence Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization

Modern Warfare
Information technology and real-time information as game changer for defence
4

Cybersecurity
Protection of computer systems and networks from information disclosure and theft
Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors
5

The need for secure and scalable defence
Defence requires real-time information everywhere!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, SolarWinds …
Damage: Billions of dollars
”Supply chain attack”
Digital Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
Billions of devices

Supply Chain Attack
Targeting less-secure elements in the supply chain
7
https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains
https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002

SECURITY
Security SIEM
Encryption
OT Security
Hardware-
based Security
Cybersecurity
How would you have a holistic view and understanding of all the events and potential abuses that are taking place within your organization?
Collect and correlate the different activities happening on critical networks
CYBERSECURITY is a key piece of the security strategy
SIEM and SOAR a (key) piece of the cybersecurity strategy
Sometimes
not needed
(in DMZ /
air gapped env)
Complex and
error prone
No help
against insiders
Continuous
real-time
data correlation
required
SOAR
Avoid risk (change operations) +
Transfer some risk (buy insurance)
Real-time
Monitoring
(Logging, SiteOps, …)
Access Control
(RBAC, Audit Logs, …)

Key Challenge: Find the Needle(s) in the Haystack
Detect true positives in real-time
• Threat detection
• Intrusion prevention
• Anomaly detection
• Compliance auditing
• Proactive response
Reduce false positives
• Automation
• Process big volumes of data in real-time
• Integration of all sources
• No ‘ignore’ on certain events
• Creation of filters and correlated event rules
• Improve signal-to-noise ratio (SNR)
• Correlate “collection of needles” in “signature needle”

This is a fundamental paradigm shift...
20
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Real-time Data beats Slow Data.
Cybersecurity
Risk classification
Threat detection
Intrusion detection
Incident response
Command Post
Intelligent Navigation
Vehicle Inspection
Location-based
Services
Logistics
Supply Chain
Inventory
management
Fleet Management
Military
Security monitoring
Surveillance
Command and
Control
Military Intelligence

Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
22

Data in Motion
The Backbone for Defence
Command
Post
Enterprise
IT
Strategic
Planning
Logs Personal
Sensors Security
Streams of real time events
23
Connected
Vehicles
Cyber
Security
Continuous
Data Correlation
Monitoring
Alerting
Proactive Actions

End-to-End Cyber Defence
with the Kafka Ecosystem
Personel
Crew, Cargo
Vessel
Fuel Consumption, Speed,
Planned Maintenance
Tracking
Position, Course, Weather, Draft
Drone or Satellite Relay
COMMs Resilient Kafka
Edge Analytics
Bidirectional Ship Edge to Cloud, Shore Edge to Cloud
Relay Ingestion
Data
Integration
Streaming Analytics
Machine Doing
On-Prem Systems
Bi-Directional Hybrid Cloud Replication
ON SHORE
ON PREM
Staging, Filtering
Shore Edge Analytics

Integrate with all legacy and modern interfaces
Record, filter, curate a broad set of traffic streams
Let analytic sinks consume just the right amount of data
Drastically reduce the complexity of the enterprise architectures
Drastically reduce the cost of SIEM / SOAR deployments
Add new analytics engines
Add stream-speed detection and response at scale in real-time
Add mission-critical (non-) security-related applications
…
is the backbone for data streams in defence!

Every enterprise is different…
Flexibility is key for your cybersecurity initiative!
Confluent is an independent foundation.
30

Kafka Connect
Confluent
Various Data
Producers
Flexible Scalable Real-Time Backplane for the Defence Platform
Splunk TensorFlow
Kafka Forwarder
TensorFlow +
Kafka plugin
Event Streaming Platform
OT Domain SIEM Domain Analytics Domain
31
Huge volumes of
real-time data from
various Kafka topics
Backpressure handling
and a low velocity
Kafka topic
High velocity, raw
Kafka topic for
forensics and ML

Cyber Situational Awareness
is the subset of all situation awareness necessary to support taking actions in cyber
36
Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 1995, 37(1), 32-64

Human – Computer Interface for Decision Making
38
https://www.youtube.com/watch?v=mPJdzzm67sg

Human – Computer Interface for Decision Making
39
https://www.youtube.com/watch?v=mPJdzzm67sg

Firewalls & Network Devices
Antivirus
Access Logs
Intrusion Detection
Audit Logs
Text Files
Binary Files
Databases
APIs
Network Flows
Syslog
The Data

Transactions
Low Velocity, Low Volume
Netflow / PCAP
High Velocity, High / Ridiculous Volume
Ingested via Network Analyzer Gateway
Logs
Low Velocity, Moderate Volume
Store PCAP headers
in Tiered Storage
or
3rd
Party like Corelight
as intermediary
Data Producers

Streams
Streams
Streams
Streams
Event type-specific
parsing and
normalization
logs-conn-shared
logs-resolve-names
logs-geoip-asn-iprep
Streams
Streams
logs-index
Established connection
and client/server
detection
DNS name resolution
GeoIP, IP Reputation
and Autonomous
System lookup
Data Normalization and Enrichment
à Improve the signal + filter to lower the noise

SIEM Forwarder
Threat Detection
Near real-time
Data Consumers
No constraints on integration flows
Data curation on the fly
Flexible choice of (multiple) consumers
Sink to Data Lake
Analytical Workloads
Batch
Native Kafka App
Transactional Workloads
Real-time

Sigma
44
• Open-source framework
• Domain specific language (DSL)
• Specify patterns in cyber data
https://github.com/SigmaHQ/sigma

Sigma Rule
Detections
• List of detections for each condition
• Single or list of values
• Individual values or regex
• Detection names can also include
operators (ex. name|endswith,
name|contains, name|greater_than)
• Aggregations and windowing
Conditions
• Nested conditions based on defined
detections
Detection Names
• Generic Sigma names defined
• Translated during parsing to meet end
SIEM tool using field mapping file

Confluent Sigma
Sigma Stream Processors
Zeek Data and
Detections Viewer
Sigma Rule Editor
sigma rules topic
DNS
dns
detections
topic
dns topic
rule parsing,
filtering,
aggregation,
windowing
sigma
rules
cache
CONN
DHCP
HTTP
SSL
x509
Zeek Data
https://github.com/confluentinc/cyber/tree/master/confluent-sigma

Threat Intelligence
54
Mitigate harmful events in cyberspace
Proactive cybersecurity posture that is predictive, not just reactive
Bolster overall risk management policies
Improved detection of threats
Better decision-making during and following the detection of a cyber intrusion
See the whole board, more quickly.
See around corners.
See the enemy before they see you.

Transactions vs. Analytics
55
Threat intelligence =
awareness-in-motion
The PATTERN is
valuable, not the data.

Streams
logs-index
Authorized access
using RBAC
Machine Learning
Predictions via UDFs
PII Anonymization
logs-alerts
logs-index-gdpr
Analytics and Actionable Insights in Motion
Make sense of the signal and the noise of the data
Continuous signature processing
Prevent, contain, and neutralize threats proactively
Access for
data science teams

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

Digital Forensics
61
• Application of science to criminal and civil laws, mainly during criminal investigation
• Forensic scientists collect, preserve, and analyze scientific evidence during the course of
an investigating digital media in a forensically sound manner
• Identify, preserve, recover, analyze and present facts and opinions about the digital
information

Distributed Digital Forensics at Scale with Kafka and Spark
62
• Digital Forensics Compute Cluster (DFORC2)
• High Speed Distributed Computing Capability for Digital Forensics
• Extended the digital forensics platform Autopsy with Kafka and Spark to add distributed
compute power for data processing
https://publications.waset.org/10007817/digital-forensics-compute-cluster-a-high-speed-distributed-computing-capability-for-digital-forensics

Forensics on Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• Capture the complete attack vector
• Playback of an attack for the
training of humans or machines
• Create threat surface simulations
• Compliance / regulatory processing
Real-time Consumer for
an automated actuation
Consumer of Historical Data

Confluent Tiered Storage for Kafka for Forensics of Historical Data
65
(Only available in Confluent Platform)
Store data forever
Hot and cold storage
Cheap object store
Easy scale up/down
No changes in clients

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
The Role of AI and Machine Learning for Forensics
Model Training with Kafka and TensorFlow I/O
https://github.com/tensorflow/io
66
Model X
(at a later time)

“CREATE STREAM AnomalyDetection AS
SELECT facility_code, detectAnomaly(syslog_values)
WHERE severity_level = ’Warning’
FROM syslog_source_topic;“
User Defined Function (UDF)
67
The Role of AI and Machine Learning for Forensics
Model Deployment with ksqlDB and TensorFlow

Zero Trust
70
• EVERYTHING needs protecting, not just firewalls and computing assets
• It is not cyber network security, but threat intelligence that includes human intelligence
• Safe IT/OT integration at industrial sites
• There is no such thing as a “unidirectional firewall”
• Hardware and / or software-based
• Replica servers instead of direct access
• Surveillance for Safety and Theft Protection

Unidirectional Networks for Air-gapped
Environments
When a Firewall is NOT Enough!
77
• Secure OT – IT bridge
• Hardware based data diode or unidirectional gateway
• Real time monitoring of safety-critical networks
• Secure cloud connectivity of critical OT networks
• Database replication and file transfer
• Transferring application and operating system updates
• Vendors use different terms: Unidirectional network =
Unidirectional Gateway = Data Diode

Confluent Data Diode
https://docs.confluent.io/kafka-connect-data-diode
Software-based Unidirectional Gateway for Zero Trust Security Architectures
Streaming from Industrial Networks to Enterprise Networks
UDP-based Source and Sink Kafka Connectors for High Volume and Open Architecture
Run over a one-way/UDF hardware interface (Ethernet cable, OWL Cyber, Waterfall, etc.)
Optionally/eventually do filtering, anomaly detection, analytics, receive upstream traffic, etc.
Site
Site
Site
Work
Center
Apache
PLC4x
Work
Center
Cloud
Streams processing
Data Lake
Data Diode
UDP Sink
Data Diode
UDP Source
Kafka
Cluster
NUC Pair
Kafka
Instance

Real-time
Anomaly
Detection
Machine
Learning
Real-time
Situational
Awareness
Event
Driven
Mission
Distributed
Command
and Control
Security
Operations
Adversarial
Threats
Assets, Weapons,
Sensors, etc.
Zero Trust Edge Architecture
OBSERVE
ORIENT
DECIDE
ACT
Collect, analyze, and share data in real-time. Provide a fuller picture of the operating environment.

85
Sensor A Sensor B Sensor X
MQTT
Confluent Platform (Single Broker)
Confluent Platform deployed on a small computer and
leveraging Cluster Linking to publish sensor data to
Command Post in a DDIL environment.
Command Post running Confluent
Platform aggregating information from
Squires and other sensor data
Weather
Personnel
Logistics
Targets Sensor data published to
Command Post when
connected to network
Enhanced
Situational Awareness
Smart Soldiers at the Edge

The Challenge with SIEM / SOAR Platforms
Forwarder
Network traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Machine Data
HTTP proxy logs
Splunk
ArcSight
Elastic
Proprietary forwarders can only
send data to single tool
Data is locked from being shared
Difficult to scale with growing
data volumes
High indexing costs of proprietary
tools hinder wide adoption
Filtering out noisy data is complex
and slows response
No one tool can support all
security and SIEM requirements

Forensic
Archive AI/ML
Build A Real-time SIEM / SOAR Pipeline
Filter,
transform
aggregate
APP SIEM
Index
Search
Curated
streams
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Machine Data
spooldir (files), SNMP Traps,
Databases, Sftp, MQs

Old New
Scan
Scan
Scan
Each SIEM has its own position (offset)
Raw-Big-Data-Topic
Small-Data-Topic
Preprocess
and
consolidate

Confluent + Splunk SIEM Reference Architecture
(assuming that Splunk UFs already exist – otherwise integrate directly via Kafka to Splunk)
Splunk
Universal
Forwarders
(UFs)
Windows
Event Logs
SNMP
Syslog
Watchlist
Zeek IDS
Splunk
Heavy
Forwarders
Machine
Learning
Splunk S2S
Connector
Splunk
HEC
Splunk
Indexers
Splunk
Search
Head
Real-time stream
processing with
ksqlDB
...
3rd party apps /
ecosystems
Moving log data
from Splunk UFs
to your
destination of
choice

Palo Alto Networks SOAR
97
Cortex Data Lake collects, transforms and integrates
enterprise’s security data to enable Palo Alto Networks
solution
Billions of messages pass through the Kafka clusters
Multiple Kafka clusters in production, size from 10 to just
under a 100 brokers each
Leverages various Confluent components
Design principles:
• Cloud agnostic infrastructure
• Massively scalable
• Aggressive ETA on integrations
• Schema versioning support
• Microservices architecture
• Operational efficiency
https://medium.com/engineering-at-palo-alto-networks

Why Confluent?

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka. Cloud-native. Everywhere.

Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
confluent.io
kai-waehner.de
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry

Similar to Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry (20)

More from Kai Wähner

More from Kai Wähner (14)

Recently uploaded

Recently uploaded (20)

Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry