Download as PDF, PPTX
Uploaded byEvans Ye
Using the SDACK Architecture to Build a Big Data Product
The document outlines the development of a threat analytic big data product using the SDACK architecture, which integrates Spark, Docker, Akka, Cassandra, and Kafka for efficient data processing. It addresses challenges faced by infosec teams in managing large volumes of logs and detecting anomalies, utilizing various techniques for both streaming and batch analytics. The presentation highlights the advantages of using specific technologies for scalability, stability, and accuracy in data processing and anomaly detection.
More Related Content
Similar to Using the SDACK Architecture to Build a Big Data Product
Using the SDACK Architecture to Build a Big Data Product
- 1. Yu-hsin Yeh (EvansYe) Apache Big Data NA 2016 Vancouver Using the SDACK Architecture to Build a Big Data Product
- 2. Outline • A ThreatAnalytic Big Data product • The SDACK Architecture • Akka Streams and data pipelines • Cassandra data model for time series data
- 3. Who am I •Apache Bigtop PMC member • Software Engineer @ Trend Micro • Develop big data apps & infra
- 4. A Threat Analytic Big Data Product
- 5. Target Scenario InfoSec Team Security informationand event management (SIEM)
- 6. Problems • Too manylog to investigate • Lack of actionable, prioritized recommendations
- 7.
- 8.
- 9. On-Premises InfoSec Team Security information andevent Anomaly Detection Expert Rule Filtering Prioritization
- 10.
- 11. On the cloud? InfoSec Team AD WindowsEvent Log DNS …
- 12. There’re too manyPII data in the log
- 13. System Requirement • Identifyanomaly activities based on data • Lookup extra info such as ldap, active directory, hostname, etc • Correlate with other data • EX: C&C connection followed by an anomaly login • Support both streaming and batch analytic workloads • Run streaming detections to identify anomaly in timely manner • Run batch detections with arbitrary start time and end time • Scalable • Stable
- 14. No problem. I havea solution. How exactly are we going to build this product? PM RD
- 15. SMACK Stack Toolbox forwide variety of data processing scenarios
- 16. SMACK Stack • Spark:fast and general purpose engine for large-scale data processing scenarios • Mesos: cluster resource management system • Akka: toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications • Cassandra: distributed, highly available database designed to handle large amounts of data across datacenters • Kafka: high-throughput, low-latency distributed pub-sub messaging system for real-time data feeds Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka
- 17.
- 18. No problem. I havea solution. Can’t we simplify it? PM RD
- 19.
- 20. SDACK Stack • Spark:fast and general purpose engine for large-scale data processing scenarios • Docker: deployment and resource management • Akka: toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications • Cassandra: distributed, highly available database designed to handle large amounts of data across datacenters • Kafka: high-throughput, low-latency distributed pub-sub messaging system for real-time data feeds Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka
- 21. • Three keytechniques in Docker • Use Linux Kernel’s Namespaces to create isolated resources • Use Linux Kernel’s cgroups to constrain resource usage • Union file system that supports git-like image creation Docker - How it works
- 22. • Lightweight • Fastcreation • Efficient to ship • Easy to deploy • Portability • runs on *any* Linux environment The nice things about it
- 23.
- 24.
- 25.
- 26.
- 27. DB a d d d i w e b Actor Actor Actor … … Log Web PortalDetection ResultAPI Server
- 28.
- 29. Docker Compose kafka: build: . ports: -“9092:9092” spark: image: spark port: - “8080:8080” …… Web Portal API Server
- 30. Akka Streams and data pipelines
- 31.
- 32.
- 33.
- 34.
- 35. • Each streamingjob occupies at least 1 core • There’re so many type of log • AD, Windows Event, DNS, Proxy, Web Application, etc • Kafka offset loss when code changed • No back-pressure (back then) • Estimated back-pressure available in Spark 1.5 • Integration with Akka based info lookup services Why not Spark Streaming?
- 36. Akka • High performanceconcurrency framework for Java and Scala • Actor model for message-driven processing • Asynchronous by design to achieve high throughput • Each message is handled in a single threaded context (no lock, synchronous needed) • Let-it-crash model for fault tolerance and auto-healing • Clustering mechanism available The Road to Akka Cluster, and Beyond
- 37. Akka Streams • AkkaStreams is a DSL library for doing streaming computation on Akka • Materializer to transform each step into Actor • Back-pressure enabled by default Source Flow Sink The Reactive Manifesto
- 38. Reactive Kafka • AkkaStreams wrapper for Kafka • Commit processed offset back into Kafka • Provide at-least-once message delivery guarantee https://github.com/akka/reactive-kafka
- 39. Message Delivery Sematics •Actor Model: at-most-once • Akka Persistence: at-least-once • Persist log to external storage (like WAL) • Reactive Kafka: at-least-once + back-pressure • Write offset back into Kafka • At-least-once + Idempotent writes = exactly-once
- 40.
- 41.
- 42. Scale Out • Scaleout by creating more docker containers with same Kafka consumer group
- 43. Be careful using Reactive Kafka Manual Commit
- 44. A bug thatperiodically reset the committed offset kafka source manual commit #partition offset partition 0 50 partition 1 90 partition 2 70 0 1 2
- 45. A bug thatperiodically reset committed offset kafka source manual commit kafka source manual commit #partition offset partition 0 300 partition 1 250 partition 2 70 0 1 2 #partition offset partition 2 350
- 46.
- 47.
- 48. PR merged in0.8 branch Apply patch if version <= 0.8.7 https://github.com/akka/reactive-kafka/pull/144/commits
- 49. Akka Cluster • ConnectsAkka runtimes into a cluster • No-SPOF • Gossip Protocol • ClusterSingleton library available
- 50. Lookup Service Integration AkkaStreams Akka Streams Akka Streams ClusterSingleton Ldap Lookup Hostname Lookup Akka Cluster
- 51. Lookup Service Integration AkkaStreams Akka Streams Akka Streams ClusterSingleton Ldap Lookup Hostname Lookup Akka Cluster
- 52. Lookup Service Integration AkkaStreams Akka Streams Akka Streams ClusterSingleton Ldap LookupHostname Lookup Akka Cluster Service Discovery
- 53. Async Lookup Process AkkaStreams ClusterSingleton Hostname Lookup Cache Synchronous Asynchronous
- 54. Async Lookup Process AkkaStreams ClusterSingleton Hostname Lookup Cache Synchronous Asynchronous
- 55. Akka Streams Cache Akka Streams Cache AsyncLookup Process Akka Streams ClusterSingleton Hostname Lookup Cache Synchronous Asynchronous
- 56. Cassandra Data Model forTime Series Data
- 57.
- 58. Problem • We needto do analytics based on event time instead of processing time • Example: modelling user behaviour • Log may arrive late • network slow, machine busy, service restart, etc • Log arrived out-of-order • multiple sources, different network routing, etc
- 59. Problem • Spark streamingcan’t handle event time ts 12:03 ts 11:50 ts 12:04 ts 12:01 ts 12:02
- 60. • Store timeseries data into Cassandra • Query Cassandra based on event time • Get most accurate result upon query executed • Late tolerance time is flexible • Simple Solution
- 61. Cassandra Data Model •Retrieve a entire row is fast • Retrieve a range of columns is fast • Read continuous blocks on disk (not random seek)
- 62. Data Model forAD log use shardId to distribute the log into different partitions
- 63. Data Model forAD log partition key to support frequent queries for a specific hour
- 64. Data Model forAD log clustering key for range queries
- 65.
- 66. • Spark: streamingand batch analytics • Docker: ship, deploy and resource management • Akka: resilient, scalable, flexible data pipelines • Cassandra: batch queries based on event data • Kafka: durable buffer, streaming processing Recap: SDACK Stack
- 67.