Streaming, Database & Distributed Systems Bridging the Divide

3,478 views

Published on

There is something very interesting about stream processing. It builds upon messaging, rather than using a file system, as a more typical database does. But stream processing engines themselves are really a type of database. A database designed specifically to blend streams and tables so you can query continuous results. As such they span an architectural no-mans-land that sits between Database and Distributed Systems fields.

This talk will look at Stateful Stream Processing. Can a streaming engine provide the guarantees of a database? When is a streaming engine best? How do they work, under the covers?

Published in: Technology

Streaming, Database & Distributed Systems Bridging the Divide

  1. 1. Streaming, Database & Distributed Systems: Bridging the Divide Ben Stopford (@benstopford) Codemesh 2016
  2. 2. Event Driven Systems Most stateful systems have to pull from these three worlds
  3. 3. Today we have 2 goals 1.  Understand Stateful Stream Processing (now & near future) 2.  Case for SSP as a general framework for building data-centric systems.
  4. 4. Data systems come in different forms •  Database (OLTP) •  Analytics Database (OLAP/Hadoop) •  Messaging •  Distributed log •  Stream Processing •  Stateful Stream Processing
  5. 5. Database (OLTP) Focuses on providing a consistent view that supports updates and queries on individual tuples.
  6. 6. Analytics Database (OLAP/Hadoop) 1.  Focuses on aggregations via table scans. 2.  Executes as distributed system
  7. 7. Messaging Focuses on asynchronous information transfer with limited state
  8. 8. Distributed Log 1.  Similar to messaging, but data can be retained 2.  Executes as distributed system (scale + fault tolerance)
  9. 9. Stream Processing Manipulate concurrent streams of events Comes from CEP background (ephemeral)
  10. 10. Stateful Stream Processing Moves stream processing to be a more general framework for building data-centric systems.
  11. 11. What is stream processing? Data Index Query Engine Query Engine vs Database Finite source Stream Processor Infinite source
  12. 12. Infinite streams need windows How many items will we bring into the machine at one time?
  13. 13. Windows bound a computation How many items will we bring into the machine at one time?
  14. 14. Buffering allows us to handle late events How many items will we bring into the machine at one time?
  15. 15. Some query Over some time window Emitting at some frequency Continually executing query Stream(s) Stream Processing Engine Derived Stream
  16. 16. Avg(p.time – o.time) From orders, payment Group by payment.region over 1 day window emitting every second Stream Processing orders! payments! Completion time, by region!
  17. 17. Avg(o.time – p.time) From orders, payment Group by payment.region over 1 day window emitting every second Materialised View (DB ) Query orders! payments! Completion time, by region!
  18. 18. Avg(o.time – p.time) From orders, payment, user Group by user.region over 1 day window emitting every second Stateful Stream Processing Streams Stream Processing Engine Derived Stream Query Derived “Table” Table “View” is output as table or stream
  19. 19. Table == Stream + Window0 n == 0 N Table is a stream with an infinite window (i.e. buffer from 0 -> now) window !
  20. 20. SSP is about creating materialised views. Materialised as a table, or materialised as a stream
  21. 21. Features: similar to database query engine Join Filter Aggr- egate View Windowed Streams
  22. 22. Can distribute over many machines in two dimensions Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Scale Out Scale Forward
  23. 23. Stateful Stream Processing engines typically use Kafka (a distributed commit log) Join Filter Aggr- egate View Kafka (a distributed log)
  24. 24. A log is very simple idea Messages are added at the end of the log Just think of the log as a file Old New
  25. 25. Readers have a position & scan Sally is here George is here Fred is here Old New Scan Scan Scan
  26. 26. Can “Rewind & Replay” the log Rewind & Replay
  27. 27. Compacted Log (Tabular View) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1 Version 2 Version 3 Version 5 STEAM (All versions) COMPACTED STREAM (Latest Key only)
  28. 28. The log is a Distributed System For scalability and fault tolerance
  29. 29. Shard on the way in Producers Kafka Consumers
  30. 30. Each shard is a queue Producers Kafka Consumers
  31. 31. Producers Kafka Many consumers share partitions in one topic Consumers share consumption of a single topic
  32. 32. The Log reassigns data on failure Producers Kafka Many consumers share partitions in one topic
  33. 33. Kafka supplies two levels of leader election Replicas in Kafka have an elected leader Consumers in Kafka have an elected leader
  34. 34. The log is important for SSP Maintains History: Acts like a “push based” distributed file system
  35. 35. The log is important: Two Primitives Stream Compacted Stream (‘table’)
  36. 36. The Log is, to a streaming engine, what HDFS is to Hadoop
  37. 37. But it’s a bit more than a HDFS replacement: Processors inherit the idea of “membership” from the log
  38. 38. So stateful Stream Processors use the Log Join Filter Aggr- egate View Kafka (Distributed Log)
  39. 39. They also use local storage Join Filter Aggr- egate View (1) a Kafka (2) Local KV Store
  40. 40. Local KV store has a few uses (1)  It caches streams on disk (2) It caches “tables” on disk Join Filter Aggr- egate View This makes join operations fast as they’re entirely local Streams just cache recent messages to help with joins Tables are fully “realised” locally
  41. 41. Stateful Stream Processing stream Compacted stream Join Stream data Stream-Tabular Data Infinite Stream Locally Cached Table (disk resident) KafkaKafka Streams
  42. 42. e.g. Useful for Enrichment stream Compacted stream Join Orders Customers KafkaKafka Streams Local DB
  43. 43. Aggregates need intermediary state stream Compacted stream Join Orders Customers KafkaSum(orders) group by region Persist current value, in case we fail
  44. 44. State store inherits durability from the log State store flushes back to the log Join Filter Aggr- egate View
  45. 45. Separate Data, Processing & View View OrdersPayments View View Storage Layer (a Kafka) Processing & View Query
  46. 46. You can query the views from anywhere View OrdersPayments View View Storage Layer (a Kafka) Processing & View Query
  47. 47. So what happens on failure? View OrdersPayments View View Storage Layer (a Kafka) Processing & View
  48. 48. Clustering Reroutes Data to surviving node View OrdersPayments View View Storage Layer (Kafka) Ownership of partitions is re-routed from dead node Processing & View
  49. 49. But what about state? View OrdersPayments View View Storage Layer (Kafka) “Cold” replica of state takes over Processing & View
  50. 50. Primitives for sharding & replication Stock OrdersPayments Stock Stock Redundant copies are cached on other nodes Sharding spread data over processors
  51. 51. So processors inherit much from the log Clustering comes from the log You just write the functional bit
  52. 52. General framework for distributed, realtime data computation Protection from broker failure Protection from engine failure Join tables & streams (in process) Event Driven Create views which can be queried Query
  53. 53. But stream processing has a problem
  54. 54. Correctness Guarantees in multi layer topologies Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View
  55. 55. Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Duplicates are a side effect of all at-least-once delivery mechanisms Data is rerouted, on failure, which can cause duplicates
  56. 56. Idempotance isn’t enough Join Filter Aggr- egate View Join Filter Aggr- egate View Filter Join Filter Aggr- egate View Join Filter Aggr- egate View
  57. 57. Distributed Snapshots* (transactions) Join Filter Aggr- egate View Join Filter Aggr- egate View Join Filter Aggr- egate View Transaction markers: [Begin], [Prepare], [Commit], [Abort] Buffer Chandy, Lamport - Distributed Snapshots: Determining Global States of Distributed Systems *In development in Kafka
  58. 58. So why use these tools?
  59. 59. (1) Streaming is a superset of batch
  60. 60. Databases look backwards
  61. 61. Batch == Streaming from offset 0 Query Query Query Distributed File System (HDFS) Query Query QueryDistributed Log (Kafka) MPP Batch System MPP Streaming System
  62. 62. Streaming is the superset of batch Streaming Batch Database Global, Linearisible consistency model
  63. 63. (2) Separates store & view
  64. 64. “Engine” part is lightweight but stateful Storage Just a java process which uses a library Log handles fault tolerance of both layers
  65. 65. Separates Concerns of Model & View – Think MVC Storage View & Controller Model
  66. 66. Physically Separates Read & Write – Think CQRS Storage View & Controller Model
  67. 67. Database vs SSP Data Index Query Engine Query Engine vs Database Stateful Stream Processor Query Query View Index Data
  68. 68. (3) Decentralised approaches are more general
  69. 69. Rather than pushing processing into an “appliance” (code -> data) Centralised Processing App
  70. 70. Data Decentric Architecture Distributed Log Decentralised Processing over many user-specific views
  71. 71. This more general than than just analytics use cases
  72. 72. It’s more than taking a database and adding push notifications
  73. 73. Whether you’re building a hulking, multistage, analytic platform Query Final View Intermediary View (2) Intermediary View (1)
  74. 74. Or a simple microservice that needs to run hot-hot & scale Business Logic Manage local state Join various streams Hot secondary instance
  75. 75. Composable Primatives Declarative Function Traditional DB Work Distribution Replication Sharding Query Engine Distributed DB Distributed Systems Membership Global Consistency
  76. 76. General framework for distributed, event- driven data computation Protection from broker failure Protection from engine failure Join tables & streams (in process) Event Driven Create views which can be queried Query
  77. 77. Stateful Stream Processing Framework for building a streaming data systems, just for you “~)
  78. 78. Find out more: •  http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/ •  https://martin.kleppmann.com/2015/02/11/database-inside-out-at-salesforce.html •  http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf •  https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cidr07p42.pdf •  http://highscalability.com/blog/2015/5/4/elements-of-scale-composing-and-scaling-data- platforms.html •  https://speakerdeck.com/bobbycalderwood/commander-decoupled-immutable-rest-apis-with- kafka-streams •  https://timothyrenner.github.io/engineering/2016/08/11/kafka-streams-not-looking-at-facebook.html •  https://www.madewithtea.com/processing-tweets-with-kafka-streams.html •  http://www.infolace.com/blog/2016/07/14/simple-spatial-windowing-with-kafka-streams/ •  http://www.slideshare.net/zacharycox/updating-materialized-views-and-caches-using-kafka
  79. 79. The end @benstopford http://benstopford.com

×