Scaling Security Threat Detection with Apache Spark and Databricks
•
2 likes•626 views
Apple must detect a wide variety of security threats, and rises to the challenge using Apache Spark across a diverse pool of telemetry. This talk covers some of the home-grown solutions we’ve built to address complications of scale
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Scaling Security Threat Detection with Apache Spark and Databricks
Similar to Scaling Security Threat Detection with Apache Spark and Databricks (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Scaling Security Threat Detection with Apache Spark and Databricks
- 2. Scaling Security Threat Detection with Spark and Databricks Josh Gillner Apple Detection Engineering
- 3. ▪ Protecting Apple’s Systems ▪ Finding & responding to security threats using log data ▪ Threat research and hunting ^^^ Looking for this guy Who are we? - Apple Detection Engineering
- 5. What is a detection?
- 6. Detection === Code That Finds Bad Stuff ▪ Get an input dataset(s) ▪ Apply logic ▪ Output ▪ 1 notebook -> 1 job -> 1 detection
- 7. What Happens Next? Analyst Review Suggestion, Enrichment & Automated Containment Alert Orchestration System Contain Issue+ This needs to be fast
- 9. Problem #1 — Development Overhead ▪ Average time to write, test, and deploy a basic detection === 1 week ▪ New ideas/week > deployed jobs/week (unsustainable) ▪ Writing scalatests, preserving test samples…testing is too cumbersome ▪ > 60% of new code is boilerplate (!!)
- 10. Problem #2 — Mo’ Detections, Mo’ Problems Want to add a cool new feature to all detections? Refactor many different notebooks Config all over the place in disparate notebooks Want to configure multiple detections at once? Ongoing tuning and maintenance? One-off tuning doesn’t scale to hundreds of detections
- 11. Problem #3 — No Support for Common Patterns ▪ Common enrichments or exclusions ▪ Creating and using statistical baselines ▪ Write detection test using scalatest Things People Often Do (but must write code for) …everyone implements in a different way …fixes/updates must be applied in 10 places
- 12. DetectionKit Auto-Tuning Alerts Modular Postprocessing Automated Enrichments Complex Exclusions Notebook-Based CI Centralized Configuration Test Generation Alert Standardization Future-Looking Abstraction Multi-Stage Alerting Modular Investigation Templates Signal-Based Detections Rate Limit Failsafes Preprocessing Transformations Automated Tagging Statistics Tables Entity-Based Deduplication Asset Attribution
- 13. Components ▪ Input ▪ Detection and Alert abstractions ▪ Emitters ▪ Configuration ▪ Tuning ▪ Modular Pre/Postprocessing ▪ Functional Testing ▪ Complex Exclusions ▪ Templatized Investigations
- 14. Input ▪ All detection begins with input loading ▪ Pass in inputs through config object ▪ External control through config ▪ decide spark.read vs .readStream ▪ path, schema, format ▪ no hardcoding -> dynamic input behavior ▪ Abstracts away details of getting data ^^^ This should not change if someDataset is a production table or test sample file
- 15. Detection and Alert Abstraction ▪ Logic is described in form of Spark DataFrame ▪ Supports additional post-processing transformation ▪ Basic interface for consumption by other code Detection val alerts: Map[String, Alert] = Alert val modules: ArrayBuffer[Transformer] = def PostProcessor(input: DataFrame): DataFrame = ??? def df: DataFrame = /* alert logic here */ val config: DetectionConfig Input and other runtime configs Test generation
- 16. Emitter ▪ Takes output from Alert and send them elsewhere ▪ Also schedules the job in Spark cluster Alert MemoryEmitter FileEmitter KinesisEmitter DBFS on AWS S3 In-memory Table AWS Kinesis
- 17. Config Inference ▪ If things can (and should) be changed, move it outside of code ▪ eg. detection name, description, input dataset, emitter ▪ Where possible, supply a sane default or infer them val checkpointLocation: String = "dbfs:/mnt/defaultbucket/chk/detection/ / / .chk/" name = "CodeRed: Something Has Happened" alertName = "JoshsCoolDetection" version = "1" DetectionConfigInfer
- 18. Config Inheritance ▪ Fine-grained configurability ▪ Could be multiple Alerts in same Detection ▪ Individually configurable, otherwise inherit parent config Detection Alert val config: DetectionConfig Alert Alert
- 19. Modular Pre/PostProcessing ▪ DataFrame -> DataFrame transform applied to input dataset ▪ Supplied in config ▪ Useful for things like date filtering without changing detection Preprocessing Postprocessing ▪ Mutable Seq of transform functions inside Detection ▪ Applied sequentially to output foreachBatch Transformers ▪ Some operations not stream-safe ▪ Where the crazy stuff happens
- 20. Manual Tuning Lifecycle ▪ Tuning overhead scales with number of detections ▪ Feedback loop can take days while analysts suffer :( ▪ This need to be faster… ideally automated and self- service The data/ environment changes DE tweaks detection False positive alerts Analyst requests tuning pain
- 21. Self-Tuning Alerts Detection Analyst Review Alert Labels (FP, TP, etc.) Analyst Consensus! Modify Behavior Alert Orchestration System
- 22. Complex Exclusions ▪ Arbitrary SQL expressions applied on all results in forEachBatch ▪ Stored in rev-controlled TSV ▪ Integrated into Detection Test CI…malformed or over-selective items will fail tests ▪ Preservation of excluded alerts in a separate table Eventually, detections look like this >>> So….
- 23. Repetitive Investigations…What Happens? • Analysts run queries in notebooks to investigate • Most of these queries look the same, just different filter Analyst Review Alert Orchestration System
- 24. Automated Investigation Templates ▪ Find corresponding template notebook ▪ Fill it out ▪ Attach to cluster ▪ Execute Alert Orchestration System Workspace API
- 25. This lets us automate useful things like… Interactive Process Trees in D3 Baselines of Typical Activity
- 26. Automated Containment Machines can find, investigate, and contain issues without humans Automated Investigation Alert Orchestration System ODBC API • Run substantiating queries via ODBC • Render verdict Contain Issue
- 27. Detection Testing Why is it so painful? ▪ Preserving/exporting JSON samples ▪ Local SparkSession isn’t a real cluster ▪ Development happens in notebooks, testing happens in IDE ▪ Brittle to even small changes to schema, etc
- 28. Detection Functional Tests ▪ 85% reduction in test LoC ▪ write and run tests in notebooks! ▪ use Delta sample files in dbfs, no more exporting JSON ▪ scalatest generation using config and convention Trait: DetectionTest ^^ this is a complete test ^^
- 29. Detection Test CI Git PR CI System Test Notebooks Workspace API /Alerts/Test/PRs/<Git PR number>_<Git commit hash> Jobs API Build Scripts pass/fail “Testing has never been this fun!!” — detection engineers, probably
- 30. Jobs CI — Why? ▪ Managing hundreds of jobs in Databricks UI ▪ Each job has associated notebook, config, dbfs files ▪ No inventory of which jobs should be running, where ▪ We need job linting >>>
- 32. Deploy/Reconfigure Jobs with Single PR CI System Config Linter Stacks CLI Jobs Helper Deploy Job/ Notebooks/Files Kickstart/Restart Set Permissions
- 33. Cool Things with Jobs CI! ▪ Deploy or reconfigure many jobs concurrently ▪ Auto job restarts on notebook/ config change ▪ Standardization of retries, timeout, permissions ▪ Automate alarm creation for new jobs ^^^ No one likes manually crafting Stacks JSON — so we generate it
- 34. Saving Time with Automated Historical Insight
- 35. Problem #1 — Cyclical Investigations ▪ Alert comes in, analysts spend hours looking into it ▪ But the same thing happened 3 months ago and was determined to be benign ▪ Lots of wasted cycles on duplicative investigations
- 36. Problem #2 — Disparate Context ▪ Want to find historical incident data? ▪ look in many different places ▪ many search UIs, syntaxes ▪ Manual, slow & painful ▪ New analysts won’t have historical knowledge
- 37. Problem #3 — Finding Patterns Which incidents relate to other incidents? Do we see common infrastructure, actors? How much work is repeated? Case #55557 Case #44447 Case #33337 }(Some IP Address)
- 38. Solution: Document Recommendations ▪ Collect all incident-related tickets, correspondence, and investigations ▪ Normalize them into a Delta table ▪ Automate suggestion of related knowledge using our own corpus of documents Emails Tickets Alerts Notebooks Detection Code Wikis
- 39. “Has This Happened Before?” -> Automated Includes analyst comments and verdicts displayHTML suggestions, clickable links to original document
- 40. Automated Suggestions alert_hash 112233445 serial = C12345678ABC src_ip = 88.88.88.123 mime_type = [“text/html"] dt = 2020-03-15 C12345678ABC 88.88.88.123 Entities C12345678ABC 88.88.88.123 00:de:ad:00:be:ef 01:8b:ad:00:f0:0d joshsaccount joshshostname Enriched Entities { Emails Tickets Alerts Notebooks Detections Wikis Document Search Suggestion
- 41. Anatomy of an Alert These are not valuable for search! (too common) These are good indicators of document relevance
- 42. Entity Tokenization and Enrichment IP Address Regex Domain Hashes Accounts Serials UDIDs File Path Emails MAC Addresses Alert Payload VPN Sessions Enrichments DHCP Sessions Asset Data Account Data
- 43. Suggestion Algorithm ▪ Gather match statistics for each entity: ▪ historical rarity ▪ document count rarity ▪ doc type distribution ▪ Compute entity weight based on average ranked percentiles of those features ▪ More common terms == less valuable ▪ Return the best n hits by confidence ▪ Not That Expensive™
- 44. Q & A ?
- 45. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.