Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Tracing Your Security Telemetry
With Apache Metron
Justin Leet
Systems Architect
June 29, 2016
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Metron?
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What Apache Metron Does?
“Apache Metron provides a scalable advanced...
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Metron Timeline
Sep 2014 •OpenSOC Beta
June 2015
•OpenSOC Com...
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who is Metron for?
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Capabilities
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streaming Parsing and Enrichment
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Metron’s parsing bolt can be configured two ways
– And outputs JSO...
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment / Threat Intel
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Add additional information to raw source during streaming
 Addin...
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Occurs in the same Storm topology as enrichment
 Very similar pr...
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Field Description
ip_src_addr Octet source IP
ip_dest_addr Octet de...
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Standalone Storm topology
 Reads from Kafka
 Writes packets to ...
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
PCAP
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing a Source Through Metron
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Sensor to Parser
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Caching proxy
– Mostly useful as a source of easy to get and easi...
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid - Grok
Time Elapsed Remote Host Code/Statu
s
Bytes Metho
d
UR...
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Definition
{ "parserClassName": "org.apache.metron...
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Result
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment Topology
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Loading some WHOIS derived data.
– Not directly making WHOIS quer...
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Enrichment Result
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment Topology
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Loading a list of malicious domains
– ZeuS tracker
Squid – Enrich...
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Threat Intel Result
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Justin Leet
Systems Architect
jleet@hortonworks.com
just...
Upcoming SlideShare
Loading in …5
×

Tracing your security telemetry with Apache Metron

3,704 views

Published on

Tracing your security telemetry with Apache Metron

Published in: Technology

Tracing your security telemetry with Apache Metron

  1. 1. Tracing Your Security Telemetry With Apache Metron Justin Leet Systems Architect June 29, 2016
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Apache Metron?
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Apache Metron Does? “Apache Metron provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project. A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.”
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Metron Timeline Sep 2014 •OpenSOC Beta June 2015 •OpenSOC Community Edition Dec 2015 •Metron enters Apache Incubator April 2016 •Apache Metron 0.1 Now •Working towards 0.2 release
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Who is Metron for?
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Core Capabilities
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streaming Parsing and Enrichment
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Metron’s parsing bolt can be configured two ways – And outputs JSON  Grok Parser – Less work to implement – Regex-like syntax – Good for lower volumes of data  Java Parser – More work to implement – Good for higher volumes of data Parsing
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment / Threat Intel
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Add additional information to raw source during streaming  Adding it during streaming allows ML models to score in real time instead of batch  Primarily stored in HBase  Several enrichments – GeoIP – Host – Threat Intelligence Enrichment
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Occurs in the same Storm topology as enrichment  Very similar process and flow  Use a threat feed aggregator! – Soltra adapter is provided to read feed and stream into HBase – Flat File loader and Stix bulk loader available without threat feed aggregator Threat Intel
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Field Description ip_src_addr Octet source IP ip_dest_addr Octet destination IP ip_src_port Integer source port ip_dest_port Integer destination port protocol String protocol (e.g. TCP) timestamp Sensor epoch timestamp source.type yaf, snort, etc. start_time Metron epoch timestamp end_time Metron epoch timestamp Metron JSON
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Standalone Storm topology  Reads from Kafka  Writes packets to HDFS  Kibana panel forwards request to REST PCAP service – MR Job launched – Delivers results back to Kibana PCAP
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved PCAP
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing a Source Through Metron
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Sensor to Parser
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Caching proxy – Mostly useful as a source of easy to get and easily readable logs Squid 1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html Time Elapsed Remote Host Code/Statu s Bytes Metho d URL rfc931 Peer Status/ Peer Host Type 1467125585.752 5288 127.0.0.1 TCP_MISS/2 00 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid - Grok Time Elapsed Remote Host Code/Statu s Bytes Metho d URL rfc931 Peer Status/ Peer Host Type 1467125585.752 5288 127.0.0.1 TCP_MISS/2 00 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED} %{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{WORD:UNWANTED}/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}/%{WORD:UNWANTED}
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Topology Definition { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "pars erConfig": { "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "tim estampField": "timestamp" }, "fieldTransformations" : [ { "transformation" : "MTL" ,"output" : [ "full_hostname", "domain_without_subdomains" ] ,"config" : { "full_hostname" : "URL_TO_HOST(url)" ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ] }
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Topology Result
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment Topology
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Loading some WHOIS derived data. – Not directly making WHOIS query, just using a CSV containing a few rows of data. Squid – Enrichment Definition { "zkQuorum" : ”localhost:2181" ,"sensorToFieldList" : { "squid" : { "type" : "ENRICHMENT" ,"fieldToEnrichmentTypes" : { "domain_without_subdomains" : [ "whois" ] } } } }
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Enrichment Result
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment Topology
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Loading a list of malicious domains – ZeuS tracker Squid – Enrichment Definition { "zkQuorum": "localhost:2181", "sensorToFieldList": { "squid": { "type": "THREAT_INTEL", "fieldToEnrichmentTypes": { "url": ["zeusList”] } } } }
  27. 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Threat Intel Result
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? Justin Leet Systems Architect jleet@hortonworks.com justinjleet@gmail.com

×