Dave Russell - Presented at Big Data Vilnius 27-29th November 2018 https://www.bigdataconference.lt/

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
27-29 November, Vilnius
Apache Metron in
the Real World
Dave Russell - Hortonworks
www.roaringelephant.org

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Who am I?

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Why Apache Metron?

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
Months until
breach
noticed
Avg.
months log
retention
9 6
VS
3
Months
missing

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
28 Months
Police
One/Berkut
Yahoo/FSB FB/Cambridge
Analytica
35 Months 48 Months
Time until breach actually noticed

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
“Sometime in the next few years
we're going to have our first
category-one cyber-incident; one
that will need a national response.”
Ian Levy
Technical Director
National Cyber Security Centre

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Andhra Pradesh Police, India
Aristotle University of Thessaloniki,
Greece
Automobile Dacia, Romania
Cambrian College, Canada
Chinese public security bureau
CJ CGV
Dalian Maritime University
Deutsche Bahn
Dharmais Hospital, Indonesia
Faculty Hospital, Nitra, Slovakia
FedEx
Garena Blade and Soul
Guilin University Of Aerospace
Technology
Guilin University Of Electronic
Technology
Harapan Kita
Hospital[disambiguation needed],
IndonesiaHezhou University
Sandvik
São Paulo Court of Justice
Saudi Telecom Company
Sberbank
Shandong University
State Governments of India
Government of Gujarat
Government of Kerala
Government of Maharashtra
Government of West Bengal
Suzhou Vehicle Administration
Sun Yat-sen University, China
Telefónica
Telenor Hungary, Hungary
Telkom (South Africa)
Timrå Municipality, Sweden
Universitas Jember, Indonesia
University of Milano-Bicocca, Italy
University of Montreal, Canada
Vivo, Brazil
Hitachi
Honda
Instituto Nacional de Salud,
Colombia
Lakeridge Health
LAKS
LATAM Airlines Group
MegaFon
Ministry of Internal Affairs of the
Russian Federation
Ministry of Foreign Affairs (Romania)
National Health Service (England)
NHS Scotland
Nissan Motor Manufacturing UK
O2, Germany
Petrobrás
PetroChina
Portugal Telecom
Pulse FM
Q-Park
Renault
Russian Railways

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
2018 so far...
340M Records
150M Records
92M Records
And many, many, many more..
https://en.wikipedia.org/wiki/List_of_data_
breaches

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
What Does Apache
Metron Look Like?

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Security telemetry source: authentication logsSecurity telemetry source: authentication logs

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
What is Apache Metron?

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Built on top on proven open source big data technology

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
An architecture for real-time cybersecurity analytics

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Source

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Collectors

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Profiling by time
t = 1 t = 2 t = 3 t = n
Wide range of algorithms
including:
à HyperLogLogPlus
à Bloom filters
à T-digests
à Statistical Baselining
à Hashing functions
à Outlier detection
à GeoHashing over time
à Locality Sensitive HashingApprox. Data
Sketch
Approx. Data
Sketch
Approx. Data
Sketch
Approx. Data
Sketch
Combined
Baseline
Statistic

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron Modules

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache
Metron (Part 1)

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved
The Wider Apache
Metron Ecosystem

29. 29 © Hortonworks Inc. 2011–2018. All rights reserved

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache
Metron (Part 2)

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved

33. 33 © Hortonworks Inc. 2011–2018. All rights reserved

34. 34 © Hortonworks Inc. 2011–2018. All rights reserved

35. 35 © Hortonworks Inc. 2011–2018. All rights reserved
Deploying Apache Metron

36. 36 © Hortonworks Inc. 2011–2018. All rights reserved
AD/AssetDB/HR/Threat
HDF
HDFS
NiFi - Ingest
HDP
Phase 0 – Current State
ADP Event
Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets
3
1
2
4
5

37. 37 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich
/ GeoIP / Index
Solr
Investigator UI
HDP
Phase 1 - Ingest and Archive
ADP Event
Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
10
6
8
7
9
11
12
13
Banana

38. 38 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich
/ GeoIP / Index
Solr
Enrichment Data
Investigator UI
HDP
Phase 2 – Enrich and Threat Intel
ADP Event
Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
14
Banana / Kibana / ZoomData

39. 39 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich
/ GeoIP / Index
Solr
Enrichment Data
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 3 – NiFi Data Ingestion + Analytics / UEBA Profiling
ADP Event
Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
Source Data (via NiFi)
15
16
Banana

40. 40 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Source Data (via NiFi)
Zeppelin
HDFS
Spark
Historical Analysis
Model as a Service
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich
/ GeoIP / Index
Automated Response
Solr
Enrichment Data
Netflow / PCAP /
Snort (Kafka direct)
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 4 – ArcSight Logger Migration + New Data Sources
ADP Event
Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets
Banana
AD/AssetDB/HR/Threat
17
18
19
20

41. 41 © Hortonworks Inc. 2011–2018. All rights reserved
Considerations for
Sizing Apache Metron

42. 42 © Hortonworks Inc. 2011–2018. All rights reserved
• Events per second (average and peak)
• Retention time for Hot / Warm / Cold
zones
• Enrichments
• Node sizing
• I/O Considerations
• PCAP?
Sizing an HCP deployment

43. 43 © Hortonworks Inc. 2011–2018. All rights reserved
3 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~3 months

44. 44 © Hortonworks Inc. 2011–2018. All rights reserved
12 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months

45. 45 © Hortonworks Inc. 2011–2018. All rights reserved
Hot
Warm Cold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months
24 Months

46. 46 © Hortonworks Inc. 2011–2018. All rights reserved
Cold
Beyond 24 months
Hot
Warm ColdColdColdCold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months

47. 47 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?

48. 48 © Hortonworks Inc. 2011–2018. All rights reserved

49. 49 © Hortonworks Inc. 2011–2018. All rights reserved
Appendix