With the rise of IoT and complexity of applications, clouds, networks and infrastructure, it is becoming more difficult to protect data and infrastructure from attackers. When groups of bad actors collaborate, share information, provide unauthorized access, and do botnet as a service, attacks in terabit units also start easily. On the other hand, it is also difficult to find enough security analysts to deal with and defend against such attacks. Here is the emergence of community cooperation like Apache Metron and efforts to open source. Metron provides a comprehensive framework for applications, networks and security built on Apache Hadoop and open source streaming analysis (eg Apache Nifi, Apache Kafka) tools in scalable data management and processing stacks. Extensions such as profiling, machine learning, and visualization work and real-time streaming detection make SOC analysts more efficient, while intrinsic scalability of open source gives data scientists security insight from data laboratories So that it can be quickly incorporated into production. This section explains how real-world businesses and managed service providers use Apache Metron, identify and resolve security threats on a large scale, and explain methods and ideas for adapting the platform to your security architecture · I will demonstrate.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved.
Big Traffic, Big Trouble: Big Data
How Cyber Security became about scale

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved.
Simon Elliston Ball
• Product Manager (Hortonworks)
• Data Scientist
• Elephant herder
• @sireb

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved.
Big Business
• $tn market
• Sharing networks
Sources: BT and KPMG Report, Taking the Offensive

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved.
Criminals as a Service
DDoS
attacks
attackers $5
per hour
defenders
~$40k
Stolen
Accounts
$1.6bn fine
(Facebook)
100m
Linkedin
accounts: 5

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved.
Too Much Data

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved.
Not Enough People
! 300k
unfilled job
openings " 3,000,000
professionals
needed
(IBM)
# 350,000
shortage by 2022
Sources: https://www.cyberseek.org/heatmap.html
https://www.business-standard.com/article/companies/india-needs-3-million-cyber-security-professionals-right-now-ibm-118051300153_1.html
https://www.ey.com/sg/en/newsroom/news-releases/news-ey-singapore-companies-confident-of-predicting-and-resisting-cyber-attacks
https://www.computerweekly.com/news/450420193/Europe-faces-shortage-of-350000-cyber-security-professionals-by-2022
https://researchcenter.paloaltonetworks.com/2017/05/cso-japan-aiming-close-cybersecurity-skills-gap-tokyo-2020/
$ 80% of
companies report
insufficient skills
%132,060
shortage and
growing

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved.
Scaling for Quantity Scaling for Speed
t = 1 t = 2 t = 3 t = n
Sketch
Period: 0<t<1
Sketch
Period: 1<t<2
Sketch
Period: 2<t<3
Sketch
Period: n-1<t<n
Behavior profiler

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Metron: Open Source, Open Platform, Extensible
for consistent cyber security data
REAL-TIME PROCESSING CYBER SECURITY ENGINE
Cyber Security Stream Processing Pipeline
Telemetry Data
Sources
Telemetry Data
Collectors
Telemetry
Parsers Enrichment
Threat
Intel Profiler
Alert
Triage
Indexers
and
Writers
SecurityEndPoint
Devices
(Fireye,PaloAlto,
BlueCoat,etc.)
Machine
GeneratedLogs
(AD,App/Web
Server,firewall,
VPN,etc.)
IDS
(Suricata,Snort,
etc.)
NetworkData
PCAP,Netflow,Bro,
etc.)
ThreatIntelligence
Feeds
(Soltra,OpenTaxi
third-partyfeeds)
Performance
NetworkIngest
Probes
Real-Time
Enrich/Threat
IntelStreams
/Other…
DataVault
Real-TimeSearch
EvidentiaryStore
ThreatIntelligence
Platform
ModelasaService
CommunityModels
DataScience
Workbench
PCAPForensics
Modules
Data Services
& Integration
Layer
Telemetry
Ingest Buffer

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved.
IoT: the cloud
for bot nets

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved.
Scaling the people: humans + data science
Collect Visualize Hypothesis Test Automate

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved.
Botnet Demo Dashboard

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved.
Catching a Hacker
• Authentication data from a Red
Team exercise at Los Alamos
National Labs
• Beyond the visualization
• Automation
• Instant response
That one!

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved.
Visualization Demo

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved.
Turning Insight into Profiles
t = 1 t = 2 t = 3 t = n
Sketch
Period: 0<t<1
Sketch
Period: 1<t<2
Sketch
Period: 2<t<3
Sketch
Period: n-1<t<n

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved.
Alerts UI Demo

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved.
Packaged Insights: Community and Ecosystem
Data Pipeline
and Analytics
in Motion ü Visualization and Exploration
ü Reporting and Compliance
ü Turnkey Hardware
&
&
ü Real-time Security Data Enrichment,
Analytics, and Management

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved.
Thank you.

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
Any questions?
• Apache Metron: http://metron.apache.org
• Twitter: @sireb