Building an Event-oriented Data Platform with Kafka, Eric Sammer

2,842 views

Published on

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality.
Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

Published in: Engineering
  • Be the first to comment

Building an Event-oriented Data Platform with Kafka, Eric Sammer

  1. 1. building a system for machine and event-oriented data e. sammer | @esammer kafka summit 2016
  2. 2. © 2015 Rocana, Inc. All Rights Reserved. context
  3. 3. © 2015 Rocana, Inc. All Rights Reserved. what we do 3 • we build a system for the operation of modern data centers • triage and diagnostics, exploration, trends, advanced analytics of complex systems • our data: logs, metrics, human activity, anything that occurs in the data center • “enterprise software” (i.e. we build for others.) • today: how we built what we built
  4. 4. © 2015 Rocana, Inc. All Rights Reserved. our typical customer use cases 4 • >100K events / sec (8.6B events / day), sub-second end to end latency, full fidelity retention, critical use cases • quality of service - “are credit card transactions happening fast enough?” • fraud detection - “detect, investigate, prosecute, and learn from fraud.” • forensic diagnostics - “what really caused the outage last friday?” • security - “who’s doing what, where, when, why, and how, and is that ok?” • user behavior - ”capture and correlate user behavior with system performance, then feed it to downstream systems in realtime.”
  5. 5. © 2015 Rocana, Inc. All Rights Reserved. depth: 3 meters
  6. 6. © 2015 Rocana, Inc. All Rights Reserved. high level architecture – sources 6
  7. 7. © 2015 Rocana, Inc. All Rights Reserved. high level architecture – sinks 7
  8. 8. © 2015 Rocana, Inc. All Rights Reserved. guarantees 8 • no single point of failure exists • all components scale horizontally[1] • data retention and latency is a function of cost, not tech[1] • every event is delivered provided no more than N - 1 failures occur (where N is the kafka replication level) • all operations, including upgrade, are online[2] • every event is (or appears to be) delivered exactly once[3] [1] we’re positive there’s a limit, but thus far it has been cost. [2] from the user’s perspective, at a system level. [3] when queried via our UI. lots of details here.
  9. 9. © 2015 Rocana, Inc. All Rights Reserved. events
  10. 10. © 2015 Rocana, Inc. All Rights Reserved. modeling our world 10 • everything is an event • each event contains a timestamp, type, location, host, service, body, and type- specific attributes (k/v pairs) • build specialized aggregates as necessary - just optimized views of the data
  11. 11. © 2015 Rocana, Inc. All Rights Reserved. event schema 11 { id: string, ts: long, event_type_id: int, location: string, host: string, service: string, body: [ null, string ], attributes: map<string> }
  12. 12. © 2015 Rocana, Inc. All Rights Reserved. event types 12 • some event types are standard – syslog, http, log4j, generic text record, … • users define custom event types • producers populate event type • transformations can turn an event of type A into B • event type metadata tells downstream systems how to interpret body and attributes
  13. 13. © 2015 Rocana, Inc. All Rights Reserved. ex: generic syslog event 13 event_type_id: 100, // rfc3164, rfc5424 (syslog) body: … // raw syslog message bytes attributes: { // extracted fields from body syslog_message: “DHCPACK from 10.10.0.1 (xid=0x45b63bdc)”, syslog_severity: “6”, // info severity syslog_facility: “3”, // daemon facility syslog_process: “dhclient”, syslog_pid: “668”, … }
  14. 14. © 2015 Rocana, Inc. All Rights Reserved. ex: generic http event 14 event_type_id: 102, // generic http event body: … // raw http log message bytes attributes: { http_req_method: “GET”, http_req_vhost: “w2a-demo-02”, http_req_path: “/api/v1/search?q=service%3Asshd&p=1&s=200”, http_req_query: “q=service%3Asshd&p=1&s=200”, http_resp_code: “200”, … }
  15. 15. © 2015 Rocana, Inc. All Rights Reserved. stream processing
  16. 16. © 2015 Rocana, Inc. All Rights Reserved. a reminder… 16
  17. 17. © 2015 Rocana, Inc. All Rights Reserved. data processing 17 • each processing job gets a full stream of the fire hose, decides what it wants to consider or operate on • output of “non-terminal” jobs always just events • result: all processing jobs are composable • many jobs take user rules or configuration from our ui
  18. 18. © 2015 Rocana, Inc. All Rights Reserved. the jobs 18 • transformation engine: configuration-based data transformation • metric aggregation: olap cube construction of time series data (e.g. host 17 user cpu time) • model build/eval: train/evaluate various kinds of models (e.g. anomaly detection) • trigger engine: detect complex patterns in the stream, emit events on match (e.g. complex event processing, automated workflow, alerting) • action engine: perform some action upon receiving a specific event type (e.g. email notification, 3rd party api invocation) • storage: write all the things to hdfs
  19. 19. © 2015 Rocana, Inc. All Rights Reserved. transformation use cases 19
  20. 20. © 2015 Rocana, Inc. All Rights Reserved. event feedback loops 20
  21. 21. © 2015 Rocana, Inc. All Rights Reserved. metrics and time series
  22. 22. © 2015 Rocana, Inc. All Rights Reserved. aggregation 22 • mostly for time series metrics • two halves: on write and on query • data model: (dimensions) => (aggregates) • on write – reduce(a: A, b: A) => A over window – store “base” aggregates, all associative and commutative • on query – perform same aggregate or derivative aggregates – group by the same dimensions – we use SQL (Impala)
  23. 23. © 2015 Rocana, Inc. All Rights Reserved. aside: late arriving data (it’s a thing) 23 • never trust a (wall) clock • producer determines event time, rest of the system uses this always • data that shows up late always processed according to event time • apache beam describes these issues perfectly • this is real and you must deal with it
  24. 24. © 2015 Rocana, Inc. All Rights Reserved. ex: service event volume by host and minute 24 • dimensions: ts, window, location, host, service, metric • on write, aggregates: count, sum, min, max, last • epoch, 60000, us-west-2a, w2a-demo-1, sshd, event_volume => 17, 42, 1, 10, 8 • on query: – SELECT floor(ts / 60000) as bin, host, service, metric, sum(value_sum) FROM events WHERE ts BETWEEN x AND y AND metric = ”event_volume” GROUP BY bin, host, service, metric • if late arriving data existed in events, the same dimensions would repeat with a another set of aggregates and would be rolled up as a result of the group by • tl;dr: normal window aggregation operations
  25. 25. © 2015 Rocana, Inc. All Rights Reserved. extension, pain, and advice
  26. 26. © 2015 Rocana, Inc. All Rights Reserved. extending the system 26 • custom producers • custom consumers • event types • parser / transformation plugins • custom metric definition and aggregate functions • custom processing jobs on landed data
  27. 27. © 2015 Rocana, Inc. All Rights Reserved. pain (aka: the struggle is real) 27 • lots of tradeoffs when picking a stream processing solution – samza: right features, but low level programming model, not supported by vendors. missing security features. – storm: too rigid, too slow. not supported by all Hadoop vendors. – flink: relatively new, fledgling community. growing. – spark streaming: tons of issues initially, but lots of community energy. improving. • stack complexity, (relative im)maturity • beam-style retractions required for correct, timely, efficient aggregates of complex metrics (non-assoc/commutative)
  28. 28. © 2015 Rocana, Inc. All Rights Reserved. if you’re going to try this… 28 • read all the literature on stream processing[1] • treat it like the distributed systems problem it is • understand, make, and make good on guarantees • find the right abstractions • never trust the hand waving or “hello worlds” • fully evaluate the projects/products in this space • understand it’s not just about search [1] wait, like all of it? yea, like all of it.
  29. 29. © 2015 Rocana, Inc. All Rights Reserved. things I didn’t talk about 29 • reprocessing data when bad code / transformations are detected • dealing with data quality issues (“the struggle is real” part 2) • the user interface and all the fancy analytics – data visualization and exploration – event search – anomalous trend and event detection – metric, source, and event correlation – motif finding – noise reduction and dithering • event delivery semantics (e.g. at least once, exactly once, etc.) • alerting, notification, and other subsystems
  30. 30. © 2015 Rocana, Inc. All Rights Reserved. questions? thank you. @esammer | esammer@rocana.com

×