This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.

1. Building the High Speed Cyber Security
Data Pipeline Using Apache NiFi
Praveen Kanumarlapudi

2. Cyber Security

3. 60% of Small
Businesses Fold Within
6 Months of a Cyber
Attack.

4. How to make it
success ?

5. Global Security Key Stake Holders
Security Operations Center Data Scientists Data Analysts Executives
An information security
operations center
("ISOC" or "SOC") is a
facility where
enterprise information
systems (websites,
applications, databases,
data centers and
servers, networks,
desktops and other
endpoints) are
monitored, assessed,
and defended.
Technology : SIEM
Security data scientists
have the skills to
understand complex
algorithms and build
advanced models for
threat and anomaly
detection and applying
these concepts to real
security data sets in
single or clustered
environments.
Technology : Python, R,
Big Data, Spark/Scala or
MATLAB…
Map and trace the data
from system to system
for solving a given
business or incident
problem.
Design and create data
reports using various
reporting tools that
help business executive
to make better
decisions.
Implements new
metrics for business
(KPIs)
Technology : SQL, SIEM,
Big Data, Reporting
tools
CSO’s,
CISO’s

6. Cyber Security ‘BIG data’ challenges
• Speed , Volume and Variety
 Data Ingestion
 Cleansing
 Transformation
• data reliance
 Executives – KPI Metrics
 Data scientists
 SOC
 Data Analysts
• Real-Time context

7. A couple of years Ago !
Network logs
Web logs
AD Logs
Infrastructure
logs
Application
Logs
Threat Intel
3rd Party RG
RDBMS
unstructured(semi)structured
Syslog
servers
SIEM APP
Sqoop
PySpark
SIEM Tool
Data Source Ingestion Integration Delivery
Flume
UBA Tools
SOCDataScienceKPI/Reporting

8. Challenges
• Complexity of Architecture
• Debugging
• Data Source Dependencies
• Lack of Centralized logging
• Multiple Data Copies
• Stress on Network
• Transformations with respect to destination

9. Solution Framework
 Single Data entry point – avoids network traffic and
duplicate data flowing around
 Transformations according destination – reduces the
reliance on source
 Should be capable of handling different formats and
different sources
Ingest Clean/Route
Transform for
1
Transform for
2
Route to 1
Route to 2
Archive

10. Deployment Models
Data Sources

11. Challenges
 Good architectural understanding of all
systems
 Good amount of coding effort
 Long development hours
 Maintenance overheads
 Maintain the sync between the systems
 Provenance

12. • Guaranteed delivery
• Processors that supports multiple
formats
• Ease to develop the flows and
deploy in minutes
• Open Source and rich community

13. The Data Gateway
Network logs
Web logs
AD Logs
Infrastructure
logs
Application
Logs
Threat Intel
3rd Party RG
RDBMS
unstructured(semi)structured
Data Source Data Gateway Delivery
SOCDataScienceKPI/Reporting
SOC

14. Sample Flow

15. To Azure

16. To Splunk

17. Grafana dashboard – Last 7 days

18. Yesterday

19. In the middle of the day

20. Metrics
 100+ production flows
 ~ 20 Billion events
 1000+ Transformations

21. Next ?
 MiNiFi
 Stateless NiFi
 Registry
 SAM
 Real-Time Model training
 CI/CD, NiFi API’s