Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stateful Distributed Stream Processing

8,597 views

Published on

More complex streaming applications generally need to store some state of the running computations in a fault-tolerant manner. This talk discusses the concept of operator state and compares state management in current stream processing frameworks such as Apache Flink Streaming, Apache Spark Streaming, Apache Storm and Apache Samza.

We will go over the recent changes in Flink streaming that introduce a unique set of tools to manage state in a scalable, fault-tolerant way backed by a lightweight asynchronous checkpointing algorithm.

Talk presented in the Apache Flink Bay Area Meetup group on 08/26/15

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Stateful Distributed Stream Processing

  1. 1. Stateful distributed stream processing Gyula Fóra gyfora@apache.org @GyulaFora
  2. 2. This talk § Stateful processing by example § Definition and challenges § State in current open-source systems § State in Apache Flink § Closing 2Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  3. 3. Stateful processing by example § Window aggregations • Total number of customers in the last 10 minutes • State: Current aggregate § Machine learning • Fitting trends to the evolving stream • State: Model 3Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  4. 4. Stateful processing by example § Pattern recognition • Detect suspicious financial activity • State: Matched prefix § Stream-stream joins • Match ad views and impressions • State: Elements in the window 4Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  5. 5. Stateful operators § All these examples use a common processing pattern § Stateful operator (in essence): 𝒇:   𝒊𝒏, 𝒔𝒕𝒂𝒕𝒆 ⟶ 𝒐𝒖𝒕, 𝒔𝒕𝒂𝒕𝒆. § State hangs around and can be read and modified as the stream evolves § Goal: Get as close as possible while maintaining scalability and fault-tolerance 5Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  6. 6. State-of-the-art systems § Most systems allow developers to implement stateful programs § Trick is to limit the scope of 𝒇 (state access) while maintaining expressivity § Issues to tackle: • Expressivity • Exactly-once semantics • Scalability to large inputs • Scalability to large states 6Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  7. 7. § States available only in Trident API § Dedicated operators for state updates and queries § State access methods • stateQuery(…) • partitionPersist(…) • persistentAggregate(…) § It’s very difficult to implement transactional states Exactly-­‐‑once  guarantee 7Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  8. 8. Storm Word Count 8Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  9. 9. § Stateless runtime by design • No continuous operators • UDFs are assumed to be stateless § State can be generated as a stream of RDDs: updateStateByKey(…) 𝒇:   𝑺𝒆𝒒[𝒊𝒏 𝒌], 𝒔𝒕𝒂𝒕𝒆 𝒌 ⟶ 𝒔𝒕𝒂𝒕𝒆. 𝒌 § 𝒇 is scoped to a specific key § Exactly-once semantics 9Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  10. 10. val stateDstream = wordDstream.updateStateByKey[Int]( newUpdateFunc, new HashPartitioner(ssc.sparkContext.defaultParallelism), true, initialRDD) val updateFunc = (values: Seq[Int], state: Option[Int]) => { val currentCount = values.sum val previousCount = state.getOrElse(0) Some(currentCount + previousCount) } Spark Streaming Word Count 10Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  11. 11. § Stateful dataflow operators (Any task can hold state) § State changes are stored as a log by Kafka § Custom storage engines can be plugged in to the log § 𝒇 is scoped to a specific task § At-least-once processing semantics 11Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  12. 12. Samza Word Count public class WordCounter implements StreamTask, InitableTask { //Some omitted details… private KeyValueStore<String, Integer> store; public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { //Get the current count String word = (String) envelope.getKey(); Integer count = store.get(word); if (count == null) count = 0; //Increment, store and send count += 1; store.put(word, count); collector.send( new OutgoingMessageEnvelope(OUTPUT_STREAM, word ,count)); } } 12Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  13. 13. What can we say so far? § Trident + Consistent state accessible from outside – Only works well with idempotent states – States are not part of the operators § Spark + Integrates well with the system guarantees – Limited expressivity – Immutability increases update complexity § Samza + Efficient log based state updates + States are well integrated with the operators – Lack of exactly-once semantics – State access is not fully transparent 13Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  14. 14. § Take what’s good, make it work + add some more § Clean and powerful abstractions • Local (Task) state • Partitioned (Key) state § Proper API integration • Java: OperatorState interface • Scala: mapWithState, flatMapWithState… § Exactly-once semantics by checkpointing 14Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  15. 15. Flink Word Count words.keyBy(x => x).mapWithState { (word, count: Option[Int]) => { val newCount = count.getOrElse(0) + 1 val output = (word, newCount) (output, Some(newCount)) } } 15Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  16. 16. Local State § Task scoped state access § Can be used to implement custom access patterns § Typical usage: • Source operators (offset) • Machine learning models • Use cyclic flows to simulate global state access 16Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  17. 17. Local State Example (Java) public class MySource extends RichParallelSourceFunction { // Omitted details private OperatorState<Long> offset; @Override public void run(SourceContext ctx) { Object checkpointLock = ctx.getCheckpointLock(); isRunning = true; while (isRunning) { synchronized (checkpointLock) { offset.update(offset.value() + 1); // ctx.collect(next); } } } } 17Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  18. 18. Partitioned State § Key scoped state access § Highly scalable § Allows for incremental backup/restore § Typical usage: • Any per-key operation • Grouped aggregations • Window buffers 18Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  19. 19. Partitioned State Example (Scala) // Compute the current average of each city's temperature temps.keyBy("city").mapWithState { (in: Temp, state: Option[(Double, Long)]) => { val current = state.getOrElse((0.0, 0L)) val updated = (current._1 + in.temp, current._2 + 1) val avg = Temp(in.city, updated._1 / updated._2) (avg, Some(updated)) } } case class Temp(city: String, temp: Double) 19Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  20. 20. Exactly-once semantics § Based on consistent global snapshots § Algorithm designed for stateful dataflows 20Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27 Detailed  mechanism
  21. 21. Exactly-once semantics § Low runtime overhead § Checkpointing logic is separated from application logic 21Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27 Blogpost  on  streaming  fault-­‐‑tolerance
  22. 22. Summary § State is essential to many applications § Fault-tolerant streaming state is a hard problem § There is a trade-off between expressivity vs scalability/fault-tolerance § Flink tries to hit the sweet spot with… • Providing very flexible abstractions • Keeping good scalability and exactly-once semantics 22Apache  Flink  Meetup  @  MapR2015-­‐‑08-­‐‑27
  23. 23. Thank you!

×