Threat Detection and Response at Scale with Dominique Brezinski Slide 1

Threat Detection and Response at Scale with Dominique Brezinski Slide 2

Threat Detection and Response at Scale with Dominique Brezinski Slide 3

Next SlideShares

Upcoming SlideShare

What to Upload to SlideShare

What to Upload to SlideShare

Loading in …3

×

Threat Detection and Response at Scale with Dominique Brezinski

Jun. 12, 2018

• •

Download to read offline

Data & Analytics

Security monitoring and threat response has diverse processing demands on large volumes of log and telemetry data. Processing requirements span from low-latency stream processing to interactive queries over months of data. To make things more challenging, we must keep the data accessible for a retention window measured in years. Having tackled this problem before in a massive-scale environment using Apache Spark, when it came time to do it again, there were a few things I knew worked and a few wrongs I wanted to right.

We approached Databricks with a set of challenges to collaborate on: provide a stable and optimized platform for Unified Analytics that allows our team to focus on value delivery using streaming, SQL, graph, and ML; leverage decoupled storage and compute while delivering high performance over a broad set of workloads; use S3 notifications instead of list operations; remove Hive Metastore from the write path; and approach indexed response times for our more common search cases, without hard-to-scale index maintenance, over our entire retention window. This is about the fruit of that collaboration.

Business Analysis Debra Paul

Learn to Write DAX: A practical guide to learning Power Pivot for Excel and Power BI Matt Allington

Python Data Science Essentials - Second Edition Boschetti Alberto

Power Pivot and Power BI: The Excel User's Guide to DAX, Power Query, Power BI & Power Pivot in Excel 2010-2016 Rob Collie

Supercharge Excel: When you learn to Write DAX for Power Pivot Matt Allington

Probability, Markov Chains, Queues, and Simulation: The Mathematical Basis of Performance Modeling William J. Stewart

Numerical Methods for Stochastic Computations: A Spectral Method Approach Dongbin Xiu

Outnumbered: From Facebook and Google to Fake News and Filter-bubbles – The Algorithms That Control Our Lives David Sumpter

Data Model Patterns: A Metadata Map David C. Hay

Data Visualization: a successful design process Andy Kirk

Agent-Based and Individual-Based Modeling: A Practical Introduction, Second Edition Steven F. Railsback

Dynamic Models in Biology Stephen P. Ellner

Learning Python Design Patterns Gennadiy Zlobin

Guerrilla Data Analysis Using Microsoft Excel: 2nd Edition Covering Excel 2010/2013 Oz du Soleil

Python Machine Learning Sebastian Raschka

Learning Python Design Patterns - Second Edition Giridhar Chetan

Threat Detection and Response at Scale with Dominique Brezinski

1. Dominique Brezinski — Apple Information Security • Threat Detection and Response • at Scale
2. This is about the data platform aspect, not the specific analytics
3. Agenda • Use Cases, Scale, • and Challenges/Solutions
4. Enabling Detection and Analytics
5. Diverse threats require diverse data sets
6. Streams (left joined) with context and filtered or (inner joined) with indicators
7. Large time window, multi-dataset graphs
8. Enabling Triage and Containment
9. Search and Query
10. WHERE date > current_date() - 30 days
11. Scale
12. >100TB new data a day
13. >300 billion events per day
14. Most queried table: 504,761,911,529,518 bytes, 11,149,012,553,409 rows Yeah, trillions!
15. Streaming Ingestion Architecture
16. WHERE src_ip = x AND dst_ip = y Total data size: 504 terabytes, 11,149,387,374,965 rows Scanned data size: 36.5 terabytes, 722,630,063,648 rows Additional reduction thanks to data skipping (bytes): 92.4% Additional reduction thanks to data skipping (rows): 93.2%
17. Simple. Unified.