Successfully reported this slideshow.

A streaming architecture for Cyber Security - Apache Metron

3

Share

Loading in …3
×
1 of 13
1 of 13

A streaming architecture for Cyber Security - Apache Metron

3

Share

Download to read offline

Description

Apache Metron, introduced at Big Data Week London, 2017, along with a few of the reasons for a data science driven approach to cyber security

Transcript

  1. 1. A streaming architecture for Cyber Security with NiFi, Hadoop, Storm and Metron
  2. 2. Simon Elliston Ball • Product Manager • Data Scientist • Elephant herder • @sireb
  3. 3. IoT: Mirai Reports of 1.2 Tbps 500,000 devices at peak DDoS attacks on Dyn DNS services
  4. 4. Drowning in Data
  5. 5. The value of real time Data in Motion: why wait until it’s at rest? Correct context: the world moved on
  6. 6. Better data = analyst efficiency Fully enriched data Real context Consistency = faster triage and better coverage
  7. 7. Network Level Taps
  8. 8. Data Sources and Aggregation Open standards for data models = more productive data scientists + shareable models Business level data sources link security to real business risk.
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataServicesandIntegrationLayer ModulesReal-time Processing Cyber Security Engine Telemetry Parsers Apache Metron: a framework for Big Data Driven cyber security TelemetryIngestBuffer Telemetry Data Collectors Real-time Enrich / Threat Intel Streams Performance Network Ingest Probes / OtherMachine Generated Logs (AD, App / Web Server, firewall, VPN, etc.) Security Endpoint Devices (Fireye, Palo Alto, BlueCoat, etc.) Network Data (PCAP, Netflow, Bro, etc.) IDS (Suricata, Snort, etc.) Threat Intelligence Feeds (Soltra, OpenTaxi, third-party feeds) Telemetry Data Sources Data Vault Real-Time Search Evidentiary Store Threat Intelligence Platform Model as a Service Community Models Data Science Workbench PCAP Forensics Threat IntelligenceEnrichment Indexers and WriterProfiler Alert Triage Cyber Security Stream Processing Pipeline
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment is the key to context Human Resources Database Metron Data Metron Data App Logs Active Directory Network Traffic Logs IoT Asset Database Geo, Threat, Traditional Security data sources Business Risk Data Metron Data Standard, Consistent Data Format Streaming enrichment Batch enrichment Fully Enriched data ready for analysis Wide variety of real- time and batch sources
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved But time is context too… profiling by time t = 1 t = 2 t = 3 t = n Wide range of algorithms including:  HyperLogLogPlus  Bloom filters  T-digests  Statistical Baselining  Hashing functions  Outlier detection  GeoHashing over time  Locality Sensitive Hashing Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Combined Baseline Statistic
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Stellar: Excel functions for Cyber security { "profile": "auth_distribution", "foreach": "'global'", "onlyif": "profile == 'attempts_by_user'", "init": { "s": "STATS_INIT()" }, "update": { "s": "STATS_ADD(s, total_count)" }, "result": "s" } Building a Profile Using a Profile window := PROFILE_WINDOW('...') profile := PROFILE_GET('attempts_by_user', user, window) distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile)) distribution_profile := PROFILE_GET('auth_distribution', 'global', window) stats := STATS_MERGE(distribution_profile) distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5) distinct_auth_attempts_stddev := STATS_SD(stats) • Simple • Expression based • Function composition • Boolean operators • In-stream
  13. 13. Thank you! Apache Metron: http://metron.apache.org Twitter: @sireb

Editor's Notes

  • Simon -> Hand over to James for next
    TALK TRACK
    The project is called Apache Metron. It’s an incubating Apache project and we would love for anyone interested in be more involved with it.
    It’s designed to be a comprehensive via of all cybersecurity data, all accessed through a single pane of glass.

    The data from multiple sources – security endpoints such as Fireeye, Palo Alto, Bluecoat are part of the picture – these companies are doing amazing well, but from a contextual threat perspective they are part of the story.
    There are also machine logs, network data, threat intelligence feeds – all together this is collected and then processed through a real-time cyber security engine.

    On the other side – the far right hand size, you see some of the results that enabled by a full contextual view with real-time stream processing – search and dashboarding portal – a single pane of glass as mentioned, shared community analytics models.

    This allows everyone, the community as a whole to work together to combat cybersecurity threats that are becoming increasing sophisticated and difficult to counter these days.
  • Description

    Apache Metron, introduced at Big Data Week London, 2017, along with a few of the reasons for a data science driven approach to cyber security

    Transcript

    1. 1. A streaming architecture for Cyber Security with NiFi, Hadoop, Storm and Metron
    2. 2. Simon Elliston Ball • Product Manager • Data Scientist • Elephant herder • @sireb
    3. 3. IoT: Mirai Reports of 1.2 Tbps 500,000 devices at peak DDoS attacks on Dyn DNS services
    4. 4. Drowning in Data
    5. 5. The value of real time Data in Motion: why wait until it’s at rest? Correct context: the world moved on
    6. 6. Better data = analyst efficiency Fully enriched data Real context Consistency = faster triage and better coverage
    7. 7. Network Level Taps
    8. 8. Data Sources and Aggregation Open standards for data models = more productive data scientists + shareable models Business level data sources link security to real business risk.
    9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataServicesandIntegrationLayer ModulesReal-time Processing Cyber Security Engine Telemetry Parsers Apache Metron: a framework for Big Data Driven cyber security TelemetryIngestBuffer Telemetry Data Collectors Real-time Enrich / Threat Intel Streams Performance Network Ingest Probes / OtherMachine Generated Logs (AD, App / Web Server, firewall, VPN, etc.) Security Endpoint Devices (Fireye, Palo Alto, BlueCoat, etc.) Network Data (PCAP, Netflow, Bro, etc.) IDS (Suricata, Snort, etc.) Threat Intelligence Feeds (Soltra, OpenTaxi, third-party feeds) Telemetry Data Sources Data Vault Real-Time Search Evidentiary Store Threat Intelligence Platform Model as a Service Community Models Data Science Workbench PCAP Forensics Threat IntelligenceEnrichment Indexers and WriterProfiler Alert Triage Cyber Security Stream Processing Pipeline
    10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment is the key to context Human Resources Database Metron Data Metron Data App Logs Active Directory Network Traffic Logs IoT Asset Database Geo, Threat, Traditional Security data sources Business Risk Data Metron Data Standard, Consistent Data Format Streaming enrichment Batch enrichment Fully Enriched data ready for analysis Wide variety of real- time and batch sources
    11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved But time is context too… profiling by time t = 1 t = 2 t = 3 t = n Wide range of algorithms including:  HyperLogLogPlus  Bloom filters  T-digests  Statistical Baselining  Hashing functions  Outlier detection  GeoHashing over time  Locality Sensitive Hashing Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Combined Baseline Statistic
    12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Stellar: Excel functions for Cyber security { "profile": "auth_distribution", "foreach": "'global'", "onlyif": "profile == 'attempts_by_user'", "init": { "s": "STATS_INIT()" }, "update": { "s": "STATS_ADD(s, total_count)" }, "result": "s" } Building a Profile Using a Profile window := PROFILE_WINDOW('...') profile := PROFILE_GET('attempts_by_user', user, window) distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile)) distribution_profile := PROFILE_GET('auth_distribution', 'global', window) stats := STATS_MERGE(distribution_profile) distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5) distinct_auth_attempts_stddev := STATS_SD(stats) • Simple • Expression based • Function composition • Boolean operators • In-stream
    13. 13. Thank you! Apache Metron: http://metron.apache.org Twitter: @sireb

    Editor's Notes

  • Simon -> Hand over to James for next
    TALK TRACK
    The project is called Apache Metron. It’s an incubating Apache project and we would love for anyone interested in be more involved with it.
    It’s designed to be a comprehensive via of all cybersecurity data, all accessed through a single pane of glass.

    The data from multiple sources – security endpoints such as Fireeye, Palo Alto, Bluecoat are part of the picture – these companies are doing amazing well, but from a contextual threat perspective they are part of the story.
    There are also machine logs, network data, threat intelligence feeds – all together this is collected and then processed through a real-time cyber security engine.

    On the other side – the far right hand size, you see some of the results that enabled by a full contextual view with real-time stream processing – search and dashboarding portal – a single pane of glass as mentioned, shared community analytics models.

    This allows everyone, the community as a whole to work together to combat cybersecurity threats that are becoming increasing sophisticated and difficult to counter these days.
  • More Related Content

    Related Books

    Free with a 30 day trial from Scribd

    See all

    Related Audiobooks

    Free with a 30 day trial from Scribd

    See all

    ×