Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Zurich Flink Meetup
Similar to Zurich Flink Meetup (20)
Recently uploaded
Recently uploaded (20)
Zurich Flink Meetup
- 1. 1 Kostas Kloudas @kkloudas Flink Meetup @ Zurich JUNE 22, 2017 Stateful Stream Processing with Apache Flink®
- 2. 2 Original creators of Apache Flink® Providers of the dA Platform, a supported Flink distribution
- 3. 3 questions and some history ▪ What is stateful stream processing? ▪ Why care about it? ▪ How does Flink do it? ▪ The evolution of Flink. 3
- 5. 5 Batch jobs t 2017-06-14 01:00am 2017-06-14 00:00am 2017-06-13 11:00pm 2017-06-13 10:00pm ... ● Continuously ingesting data ● Time-bounded batch files ● Periodic batch jobs The ol’ traditional batch way
- 6. 6 intermediate state t 2017-06-14 01:00am 2017-06-14 00:00am 2017-06-13 11:00pm 2017-06-13 10:00pm ... ● Compute a counter: #(A) per hour / 2 min ● What if: ● interval crosses batch boundaries? → carry intermediate results to next batch ● events out of order? → ??? The ol’ traditional batch way
- 7. ▪ So, for a simple counting program: • Custom logic for handling state • Custom logic for handling time • Custom logic for fault tolerance 7 The ol’ traditional batch way
- 8. ▪ So, for a simple counting program: • Custom logic for handling state • Custom logic for handling time • Custom logic for fault tolerance 8 The ol’ traditional batch way Difficult and has nothing to do with your program.
- 9. Why should we care? ▪...this is just for continuous data, right? 9
- 10. Why should we care? ▪...this is just for continuous data, right? 10 Most datasets are continuously arriving streams.
- 11. Stream Processing 11 Computation over an endless stream of data Your Code ...
- 12. Distributed Stream Processing 12 Your Code ... ... ... Your Code Your Code ● Partitions input by some key ● Distributes computation across multiple instances ● Each instance is responsible for some keys
- 13. qwe Stateful Stream Processing 13 ... ... Your Code Your Code update local variables/structures var x = … if (condition(x)) { … }
- 14. Stateful Stream Processing 14 ... ... Your Code Your Code qwe update local variables/structures var x = … if (condition(x)) { … } ● Embedded local state ● State co-partitioned with the input stream by key
- 15. A practical stream processor 15 state ●Fault-tolerance ●Scalability ●Efficiency ●Event-time ●Allows you to work in event-time time
- 16. 16 Stateful Stream Processor that handles consistently, robustly, and efficiently Large Distributed State Time / Order / Completeness ● Stateful stream processing as a new paradigm to continuously process continuously arriving data ● Produce accurate results ● Real-time is only a natural consequence of the model A practical stream processor
- 17. This is where Flink shines... ▪ Supports out-of-order streams ▪ Manages state transparently • exactly-once processing ▪ Offers high throughput and low latency ▪ Scales to large deployments • https://data-artisans.com/blog/blink-flink-alibaba-search • https://data-artisans.com/blog/rbea-scalable-real-time-analytics-at-king 17
- 18. Apache Flink® 18
- 19. About time ... 19 ... ... Your Code Your Code When are my results complete?
- 20. 20 ... ... Your Code Your Code When are my results complete? Processing Time drawbacks: • Incorrect results • Irreproducible results About time ...
- 22. Event Time: Watermarks 22 ● Special markers, called Watermarks ● Flow with elements ● A watermark of timestamp t means that no records with timestamp < t should be expected
- 25. Fault tolerance ▪ How to ensure exactly-once semantics? 25
- 26. Fault tolerance simple case 26 event log single process main memoryperiodically take a Snapshot of the memory
- 27. 27 event log single process main memoryRecovery restore snapshot and replay events since snapshot persists events (temporarily) Fault tolerance simple case
- 28. Fault tolerance distributed ▪ How to create consistent snapshots of distributed state? ▪ How to do it efficiently? 28
- 29. Distributed Snapshots 29 Coordination via markers, injected into the streams
- 30. 30 State index (Hash Table or RocksDB) Events flow without replication or synchronous writes stateful operation source Distributed Snapshots
- 31. 31 Trigger checkpoint Inject checkpoint barrier stateful operation source Distributed Snapshots
- 32. 32 stateful operation source Take state snapshot Trigger state copy-on-write Distributed Snapshots
- 33. 33 stateful operation source DFS Durably persist snapshots asynchronously Processing pipeline continues Distributed Snapshots
- 34. 34 ... Your Code Your Code Your Code State State State Your Code State ● Consistent snapshotting: Fault tolerance
- 35. 35 ... Your Code Your Code Your Code State State State Your Code State checkpointed state checkpointed state checkpointed state File System Checkpoint ● Consistent snapshotting: Fault tolerance
- 36. 36 ... Your Code Your Code Your Code State State State Your Code State checkpointed state checkpointed state checkpointed state File System Restore ● Recover all embedded state ● Reset position in input stream Fault tolerance
- 38. State Management: misc. 38 ▪ Savepoints ▪ Rescaling ▪ Queryable State
- 39. Apache Flink Ecosystem 39 Integration POSIX Java/Scala Collections POSIX
- 40. Apache Flink Stack 40 DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow Libraries Streaming and batch as first class citizens.
- 41. Levels of abstraction 41 Process Function (events, state, time) DataStream API (streams, windows) Table API (dynamic tables) Stream SQL low-level (stateful stream processing) stream processing & analytics declarative DSL high-level langauge
- 42. API and Execution 42 Source DataStream<String> lines = env.addSource(new FlinkKafkaConsumer010(…)); DataStream<Event> events = lines.map(line -> parse(line)); DataStream<Statistic> stats = stream .keyBy("id") .timeWindow(Time.seconds(5)) .sum(new MyAggregationFunction()); stats.addSink(new BucketingSink(path)); map() [1] keyBy()/ window()/ apply() [1] Transformation Transformation Sink Streaming DataflowkeyBy()/ window()/ apply() [2] map() [1] map() [2] Source [1] Source [2] Sink [1]
- 46. Conclusion 46
- 47. TL;DR ▪ Stateful stream processing as a paradigm for continuous data processing ▪ Flink is a sophisticated and battle-tested stateful stream processor ▪ Efficiency, management, and operational issues for state are taken very seriously 47
- 49. 49 Stream Processing and Apache Flink®'s approach to it @StephanEwen Apache Flink PMC CTO @ data ArtisansFLINK FORWARD IS COMING BACK TO BERLIN SEPTEMBER 11-13, 2017 BERLIN.FLINK-FORWARD.ORG -
- 52. Current setup 52 Hazelcast & Custom code ZMQ Cassandra Ingest Dispatch Analysis Storage
- 53. Another approach 53 ZMQ ... Apache Kafka RabbitMQ Kinesis .... Cassandra ... Elasticsearch HDFS Apache Kafka Redis... Ingest Dispatch Analysis Storage
- 54. Hazelcast Pain Points ▪Explicit load-balancing, repartitioning ▪Time handling: •sync/reordering ▪State handling: •fault tolerance, shuffling 54
- 55. Hazelcast Pain Points ▪Explicit load-balancing, repartitioning ▪Time handling: •sync/reordering ▪State handling: •fault tolerance, shuffling 55 Flink handles all this for you transparently ...focus on your application logic
- 56. Flink Bonus Points ▪ Programming model/APIs • SQL/CEP, Datastream API, ProcessFunction ▪ Rich open-source ecosystem ▪ Production Readiness • Monitoring, logging, web interface ▪ Runtime Flexibility • Rescaling, AB testing 56
- 57. Additional Considerations ▪DevOps: • Stitching together many systems = maintenance pain ▪ Vibrant and Passionate Community ▪ Enterprise Support 57