Tracing your security telemetry with Apache Metron

1. Tracing Your Security Telemetry
With Apache Metron
Justin Leet
Systems Architect
June 29, 2016

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Metron?

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What Apache Metron Does?
“Apache Metron provides a scalable advanced
security analytics framework built with the Hadoop
Community evolving from the Cisco OpenSOC
Project.
A cyber security application framework that provides
organizations the ability to detect cyber anomalies
and enable organizations to rapidly respond to
identified anomalies.”

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Metron Timeline
Sep 2014 •OpenSOC Beta
June 2015
•OpenSOC Community Edition
Dec 2015
•Metron enters Apache Incubator
April 2016
•Apache Metron 0.1
Now
•Working towards 0.2 release

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who is Metron for?

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Capabilities

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streaming Parsing and Enrichment

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Metron’s parsing bolt can be configured two ways
– And outputs JSON
 Grok Parser
– Less work to implement
– Regex-like syntax
– Good for lower volumes of data
 Java Parser
– More work to implement
– Good for higher volumes of data
Parsing

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment / Threat Intel

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Add additional information to raw source during streaming
 Adding it during streaming allows ML models to score in real time instead of
batch
 Primarily stored in HBase
 Several enrichments
– GeoIP
– Host
– Threat Intelligence
Enrichment

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Occurs in the same Storm topology as enrichment
 Very similar process and flow
 Use a threat feed aggregator!
– Soltra adapter is provided to read feed and stream into HBase
– Flat File loader and Stix bulk loader available without threat feed aggregator
Threat Intel

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Field Description
ip_src_addr Octet source IP
ip_dest_addr Octet destination IP
ip_src_port Integer source port
ip_dest_port Integer destination port
protocol String protocol (e.g. TCP)
timestamp Sensor epoch timestamp
source.type yaf, snort, etc.
start_time Metron epoch timestamp
end_time Metron epoch timestamp
Metron JSON

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Standalone Storm topology
 Reads from Kafka
 Writes packets to HDFS
 Kibana panel forwards request to REST PCAP service
– MR Job launched
– Delivers results back to Kibana
PCAP

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
PCAP

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing a Source Through Metron

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Sensor to Parser

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Caching proxy
– Mostly useful as a source of easy to get and easily readable logs
Squid
1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
Time Elapsed Remote Host Code/Statu
s
Bytes Metho
d
URL rfc931 Peer Status/ Peer Host Type
1467125585.752 5288 127.0.0.1 TCP_MISS/2
00
32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid - Grok
Time Elapsed Remote Host Code/Statu
s
Bytes Metho
d
URL rfc931 Peer Status/ Peer Host Type
1467125585.752 5288 127.0.0.1 TCP_MISS/2
00
32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED}
%{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code}
%{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} -
%{WORD:UNWANTED}/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}/%{WORD:UNWANTED}

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Definition
{ "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "pars
erConfig":
{ "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "tim
estampField": "timestamp" },
"fieldTransformations" : [
{
"transformation" : "MTL" ,"output" : [ "full_hostname",
"domain_without_subdomains" ] ,"config" : { "full_hostname" :
"URL_TO_HOST(url)" ,"domain_without_subdomains" :
"DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ] }

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Result

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment Topology

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Loading some WHOIS derived data.
– Not directly making WHOIS query, just using a CSV containing a few rows of data.
Squid – Enrichment Definition
{
"zkQuorum" : ”localhost:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" : [ "whois" ]
}
}
}
}

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Enrichment Result

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment Topology

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Loading a list of malicious domains
– ZeuS tracker
Squid – Enrichment Definition
{
"zkQuorum": "localhost:2181",
"sensorToFieldList": {
"squid": {
"type": "THREAT_INTEL",
"fieldToEnrichmentTypes": {
"url": ["zeusList”]
}
}
}
}

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Threat Intel Result

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Justin Leet
Systems Architect
jleet@hortonworks.com
justinjleet@gmail.com