A streaming architecture for Cyber Security - Apache Metron
•Download as PPTX, PDF•
3 likes•5,098 views
Apache Metron, introduced at Big Data Week London, 2017, along with a few of the reasons for a data science driven approach to cyber security
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A streaming architecture for Cyber Security - Apache Metron
Similar to A streaming architecture for Cyber Security - Apache Metron (20)
More from Simon Elliston Ball
More from Simon Elliston Ball (11)
Recently uploaded
Recently uploaded (20)
A streaming architecture for Cyber Security - Apache Metron
- 1. A streaming architecture for Cyber Security with NiFi, Hadoop, Storm and Metron
- 2. Simon Elliston Ball • Product Manager • Data Scientist • Elephant herder • @sireb
- 3. IoT: Mirai Reports of 1.2 Tbps 500,000 devices at peak DDoS attacks on Dyn DNS services
- 5. The value of real time Data in Motion: why wait until it’s at rest? Correct context: the world moved on
- 6. Better data = analyst efficiency Fully enriched data Real context Consistency = faster triage and better coverage
- 8. Data Sources and Aggregation Open standards for data models = more productive data scientists + shareable models Business level data sources link security to real business risk.
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataServicesandIntegrationLayer ModulesReal-time Processing Cyber Security Engine Telemetry Parsers Apache Metron: a framework for Big Data Driven cyber security TelemetryIngestBuffer Telemetry Data Collectors Real-time Enrich / Threat Intel Streams Performance Network Ingest Probes / OtherMachine Generated Logs (AD, App / Web Server, firewall, VPN, etc.) Security Endpoint Devices (Fireye, Palo Alto, BlueCoat, etc.) Network Data (PCAP, Netflow, Bro, etc.) IDS (Suricata, Snort, etc.) Threat Intelligence Feeds (Soltra, OpenTaxi, third-party feeds) Telemetry Data Sources Data Vault Real-Time Search Evidentiary Store Threat Intelligence Platform Model as a Service Community Models Data Science Workbench PCAP Forensics Threat IntelligenceEnrichment Indexers and WriterProfiler Alert Triage Cyber Security Stream Processing Pipeline
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment is the key to context Human Resources Database Metron Data Metron Data App Logs Active Directory Network Traffic Logs IoT Asset Database Geo, Threat, Traditional Security data sources Business Risk Data Metron Data Standard, Consistent Data Format Streaming enrichment Batch enrichment Fully Enriched data ready for analysis Wide variety of real- time and batch sources
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved But time is context too… profiling by time t = 1 t = 2 t = 3 t = n Wide range of algorithms including: HyperLogLogPlus Bloom filters T-digests Statistical Baselining Hashing functions Outlier detection GeoHashing over time Locality Sensitive Hashing Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Combined Baseline Statistic
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Stellar: Excel functions for Cyber security { "profile": "auth_distribution", "foreach": "'global'", "onlyif": "profile == 'attempts_by_user'", "init": { "s": "STATS_INIT()" }, "update": { "s": "STATS_ADD(s, total_count)" }, "result": "s" } Building a Profile Using a Profile window := PROFILE_WINDOW('...') profile := PROFILE_GET('attempts_by_user', user, window) distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile)) distribution_profile := PROFILE_GET('auth_distribution', 'global', window) stats := STATS_MERGE(distribution_profile) distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5) distinct_auth_attempts_stddev := STATS_SD(stats) • Simple • Expression based • Function composition • Boolean operators • In-stream
- 13. Thank you! Apache Metron: http://metron.apache.org Twitter: @sireb
Editor's Notes
- Simon -> Hand over to James for next TALK TRACK The project is called Apache Metron. It’s an incubating Apache project and we would love for anyone interested in be more involved with it. It’s designed to be a comprehensive via of all cybersecurity data, all accessed through a single pane of glass. The data from multiple sources – security endpoints such as Fireeye, Palo Alto, Bluecoat are part of the picture – these companies are doing amazing well, but from a contextual threat perspective they are part of the story. There are also machine logs, network data, threat intelligence feeds – all together this is collected and then processed through a real-time cyber security engine. On the other side – the far right hand size, you see some of the results that enabled by a full contextual view with real-time stream processing – search and dashboarding portal – a single pane of glass as mentioned, shared community analytics models. This allows everyone, the community as a whole to work together to combat cybersecurity threats that are becoming increasing sophisticated and difficult to counter these days.