Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Robert Metzger - Connecting Apache Flink to the World - Reviewing the streaming connectors

1,350 views

Published on

http://flink-forward.org/kb_sessions/connecting-apache-flink-with-the-world-reviewing-the-streaming-connectors/

Getting data in and out of Flink in a reliable fashion is one of the most important tasks of a stream processor. This talk will review the most important and frequently used connectors in Flink. Apache Kafka and Amazon Kinesis Streams both fall into the same category of distributed, high-throughput and durable publish-subscribe messaging systems. The talk will explain how the connectors in Flink for these systems are implemented. In particular we’ll focus on how we ensure exactly-once semantics while consuming data and how offsets/sequence numbers are handled. We will also review two generic tools in Flink for connectors: A message acknowledging source for classical message queues (like those implementing AMQP) and a generic write ahead log sink, using Flink’s state backend abstraction. The objective of the talk is to explain the internals of the streaming connectors, so that people can understand their behavior, configure them properly and implement their own connectors.

Published in: Data & Analytics
  • Be the first to comment

Robert Metzger - Connecting Apache Flink to the World - Reviewing the streaming connectors

  1. 1. Robert Metzger, Aljoscha Krettek @rmetzger_ Connecting Apache Flink® to the World: Reviewing the streaming connectors
  2. 2. What to expect from this talk  Overview of all available connectors  Kafka connector internals  End-to-end exactly-once  Apache Bahir and the future of connectors  [Bonus] Message Queues and the Message Acknowledging Source 2
  3. 3. Connectors in Apache Flink® “Hello World, let’s connect” 3
  4. 4. Connectors in Flink 1.1 Connector Source Sink Notes Streaming files Both source and sink are exactly-once Apache Kafka Consumers (sources) exactly-once Amazon Kinesis Consumers (sources) exactly-once RabbitMQ / AMQP Consumers (sources) exactly-once Elasticsearch No guarantees Apache Cassandra Exactly-once with idempotent updates Apache Nifi No guarantees Redis No guarantees 4 There is also a Twitter Source and an ActiveMQ connector in Apache Bahir
  5. 5. Streaming connectors by activity Streaming connectors ordered by number of threads/mentions on the user@flink list:  Apache Kafka (250+) (since 0.7)  Apache Cassandra (38) (since 1.1)  ElasticSearch (34) (since 0.10)  File sources (~30) (since 0.10)  Redis (27) (since 1.0)  RabbitMQ (11) (since 0.7)  Kinesis (10) (since 1.1)  Apache Nifi (5) (since 0.10) 5Date of evaluation 5.9.2016
  6. 6. The Apache Kafka Connector 6
  7. 7. Apache Kafka connector: Intro “Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.” 7This page contains material copied from http://kafka.apache.org/documentation.html#introduction
  8. 8. Apache Kafka connector: Consumer  Flink has two main Kafka consumer implementations • For Kafka 0.8 an implementation against the “SimpleConsumer” API of Kafka • For Kafka 0.9+ we are using the new Kafka consumer (KAFKA-1326)  The producers are basically the same 8
  9. 9. Kafka 0.8 Consumer 9 Fetcher Thread Fetcher Thread Kafka Broker topicB:1 topicB:3 Kafka Broker topicA:3 topicB:6 topicB:5 Kafka Broker topicB:4 topicB:2 topicA:1 Kafka Broker topicA:2 topicB:0 topicA:0 Consumer Thread Fetcher Thread Fetcher Thread Consumer Thread topicA:2 topicB:0 topicA:0 topicB:1 topicB:3topicB:4 topicB:2 topicA:1 topicA:3 topicB:6 topicB:5 KafkaClusterFlinkCluster Each TaskManager has one Consumer Thread, coordinating Fetcher Threads for each Kafka broker TaskManagerTaskManager
  10. 10. Kafka 0.8 Broker rebalance 10 Fetcher Thread Fetcher Thread Kafka Broker topicB:1 topicB:3 Kafka Broker topicA:3 topicB:6 topicB:5 Kafka Broker topicB:4 topicB:2 topicA:1 Kafka Broker topicA:2 topicB:0 topicA:0 Consumer Thread Fetcher Thread Fetcher Thread Consumer Thread topicA:2 topicB:0 topicA:0 topicB:1 topicB:3topicB:4 topicB:2 topicA:1 topicA:3 topicB:6 topicB:5 KafkaClusterFlinkCluster The consumer is able to handle broker failures 1 Broker fails 2 Thread returns partitions
  11. 11. Kafka 0.8 Broker rebalance 11 Fetcher Thread Fetcher Thread Kafka Broker topicB:1 topicB:3 Kafka Broker topicA:3 topicB:6 topicB:5 Kafka Broker topicB:4 topicB:2 topicA:1 Kafka Broker topicA:2 topicB:0 topicA:0 Consumer Thread Fetcher Thread Fetcher Thread Consumer Thread topicA:2 topicB:0 topicA:0 topicB:1 topicB:3topicB:4 topicB:2 topicA:1 topicA:3 topicB:6 topicB:5 KafkaClusterFlinkCluster On a failure, the Consumer Thread re-assigns partitions and spawns new threads as needed 1 Broker fails 2 Thread returns partitions topicB:4 topicB:2 topicA:1
  12. 12. Kafka 0.8 Broker rebalance 12 Fetcher Thread Fetcher Thread Kafka Broker topicB:1 topicB:3 Kafka Broker topicA:3 topicB:6 topicB:5 Kafka Broker topicB:4 topicB:2 topicA:1 Kafka Broker topicA:2 topicB:0 topicA:0 Consumer Thread Fetcher Thread Fetcher Thread Consumer Thread topicA:2 topicB:0 topicA:0 topicB:1 topicB:3 topicA:3 topicB:6 topicB:5 KafkaClusterFlinkCluster On a failure, the Consumer Thread re-assigns partitions and spawns new threads as needed 3 Kafka reassigns partitions topicB:4 topicB:2 topicA:1 topicB:2 topicB:4 topicA:1 topicB:2 Fetcher Thread topicB:4 topicA:1 topicB:4 topicB:2 topicA:1 4 Flink assigns partitions to existing or new threads
  13. 13. Kafka 0.9+ Consumer 13 Kafka Broker topicB:1 topicB:3 Kafka Broker topicA:3 topicB:6 topicB:5 Kafka Broker topicB:4 topicB:2 topicA:1 Kafka Broker topicA:2 topicB:0 topicA:0 Consumer Thread Consumer Thread KafkaClusterFlinkCluster New Kafka Consumer Magic TaskManager TaskManager Since Kafka 0.9, the new Consumer API handles broker failures/rebalancing, offset committing, topic querying, …
  14. 14. Exactly-once for Kafka consumers  Mechanism is the same for all connector versions  Offsets to Zookeeper / Broker for group.id restart and external tools (at-least-once)  Offsets checkpointed for exactly-once with Flink state 14
  15. 15. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 0 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 0, 0 This toy example is reading from a Kafka topic with two partitions, each containing “a”, “b”, “c”, … as messages. The offset is set to 0 for both partitions, a counter is initialized to 0.
  16. 16. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator a counter = 0 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 1, 0 The Kafka consumer starts reading messages from partition 0. Message “a” is in-flight, the offset for the first consumer has been set to 1.
  17. 17. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator a counter = 1 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 2, 1 a b Trigger Checkpoint at source Message “a” arrives at the counter, it is set to 1. The consumers both read the next records (“b” and “a”). The offsets are set accordingly. In parallel, the checkpoint coordinator decides to trigger a checkpoint at the source …
  18. 18. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator a counter = 2 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 3, 1 a b offsets = 2, 1 c The source has created a snapshot of its state (“offset=2,1”), which is now stored in the checkpoint coordinator. The sources emitted a checkpoint barrier after messages “a” and “b”.
  19. 19. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 3 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 3, 2 a b offsets = 2, 1 counter = 3 c b The map operator has received checkpoint barriers from both sources. It checkpoints its state (counter=3) in the coordinator. At the same time, the consumers are further reading more data from the Kafka partitions.
  20. 20. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 4 Zookeeper/Broker offset partition 0: 0 offset partition 1: 0 Flink Checkpoint Coordinator Pending: Completed: offsets = 3, 2 a offsets = 2, 1 counter = 3 c b Notify checkpoint complete The checkpoint coordinator informs the Kafka consumer that the checkpoint has been completed. It commits the checkpoints offsets into Zookeeper. Note that Flink is not relying on the Kafka offsets in ZK for restoring from failures
  21. 21. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 4 Zookeeper/Broker offset partition 0: 2 offset partition 1: 1 Flink Checkpoint Coordinator Pending: Completed: offsets = 3, 2 a offsets = 2, 1 counter = 3 c b Checkpoint in Zookeeper/ Broker The checkpoint is now persisted in Zookeeper. External tools such as the Kafka Offset Checker can see the lag of the consumer group.
  22. 22. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 5 Zookeeper/Broker offset partition 0: 2 offset partition 1: 1 Flink Checkpoint Coordinator Pending: Completed: offsets = 4, 2 offsets = 2, 1 counter = 3 c b d The processing further advances
  23. 23. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 5 Zookeeper/Broker offset partition 0: 2 offset partition 1: 1 Flink Checkpoint Coordinator Pending: Completed: offsets = 4, 2 offsets = 2, 1 counter = 3 c b d Failure Some failure has happened (such as worker failure)
  24. 24. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 3 Zookeeper/Broker offset partition 0: 2 offset partition 1: 1 Flink Checkpoint Coordinator Pending: Completed: offsets = 2, 1 offsets = 2, 1 counter = 3 Reset all operators to last completed checkpoint The checkpoint coordinator restores the state at all the operators participating at the checkpointing. The Kafka sources start from offset 2 and 1, the counter’s value is 3.
  25. 25. a b c d e a b c d e Flink Kafka Consumer Flink Kafka Consumer Flink Map Operator counter = 3 Zookeeper/Broker offset partition 0: 2 offset partition 1: 1 Flink Checkpoint Coordinator Pending: Completed: offsets = 3, 1 offsets = 2, 1 counter = 3 Continue processing … c The system continues with the processing, the counter’s value is consistent across a worker failure.
  26. 26. End-to-End exactly once 26
  27. 27. Consistently move and process data 27 Process Transform Analyze Exactly-once: • Apache Kafka • Kinesis • RabbitMQ / ActiveMQ • File monitoring Exactly-once: • Rolling file sink • With idempotent updates • Apache Cassandra • Elasticsearch • Redis At-least-once (duplicates): • Apache Kafka  Flink allows to move data between systems, keeping consistency
  28. 28. Continuous File Monitoring 28 Some FileSystem Monitoring task Periodic Querying Parallel file reader Parallel file reader Parallel file reader • File Path • Offset Records  The monitoring task checkpoints the last “modification time”  The file readers checkpoint the current file + offset and the list of pending files to read
  29. 29. Rolling / Bucketing File Sink  System time bucketing  Bucketing based on record data 29 Bucketing Operator 11:00 10:00 9:00 9 5 1 1 8 4 2 4 6 2 3 4 Bucketing Operator 8:00 0-4 5-9
  30. 30. Bucketing File Sink exactly-once  On Hadoop 2.7+, we call truncate() to remove invalid data on restore  On earlier versions, we’ll write a metadata file with valid offsets  Downstream consumers must take valid offset metadata into account 30
  31. 31. Kafka Producer: Avoid data loss  Apache Kafka does currently not provide the infrastructure to produce in an exactly-once fashion  By avoiding data-loss, we can guarantee at-least-once. 31 Flink Kafka Producer Kafka broker Kafka partition unacknowledged=7 On checkpoint, Flink calls flush() and waits for unack == 0  Guarantee that data has been written ACK ACK ACK ACK
  32. 32. Apache Bahir and the future of connectors What’s next 32
  33. 33. Future of Connectors in Flink  Kafka 0.10 support, with timestamps  Dynamic scaling support for Kafka and other connectors  Refactor Kafka connector API 33
  34. 34. Apache Bahir™  Bahir is a community specialized in connectors, allowing faster releases independent of engine releases.  Apache Bahir™ has been created for providing community- contributed connectors a platform, following Apache governance.  The Flink community decided to move some of our connectors there. Kafka, Kinesis, streaming files, … will stay in Flink!  Flink connectors in Bahir: ActiveMQ, Redis, Flume sink, RethinkDB (incoming), streaming Hbase (incoming).  New connector contributions are welcome! 34 Disclaimer: The description of the Bahir community is my personal view. I am not a representative of the project.
  35. 35. Time for questions… 35
  36. 36. Connectors in Apache Flink  Ask me now!  Follow me on Twitter: @rmetzger_  Ask the Flink community on user@flink.apache.org  Ask me privately on rmetzger@apache.org 36
  37. 37. Message Queues Exactly-once for 37
  38. 38. Message Queues supported by Flink 38  Traditional message queues have different semantics than Kafka, Kinesis, etc.  RabbitMQ • Advanced Message Queuing Protocol (AMQP) • Available in Apache Flink  ActiveMQ • Java Message Service (JMS) • Available in Apache Bahir (no release yet) Image source: http://www.instructables.com/id/Spark-Core-Photon-and-CloudMQTT/step1/What-is-Message-Queuing/
  39. 39. Message Queue Semantics 39 Flink RabbitMQ Source Offset Flink Kafka Consumer  In MQs, messages are removed once they are consumed  Replay not possible
  40. 40. Message Acknowledging  Once a checkpoint has been completed by all operators, the messages in the queue are acknowledged, leading to their removal from the queue. 40 id=8 id=7 id=6 id=5 id=4 id=3 id=2 id=1 Flink RabbitMQ Source Checkpoint 1: id=1 id=2 id=3 Checkpoint 2: id=4 id=5 id=6 Checkpoint 1 completed id=8 id=7 id=6 id=5 id=4 Flink RabbitMQ Source Checkpoint 1: id=1 id=2 id=3 Checkpoint 2: id=4 id=5 id=6 Message queue ACK id=1 ACK id=2 ACK id=3
  41. 41. Message Acknowledging  In case of a failure, all the unacknowledged messages are consumed again 41 id=8 id=7 id=6 id=5 id=4 id=3 id=2 id=1 Flink RabbitMQ Source Checkpoint 1: id=1 id=2 id=3 Checkpoint 2: id=4 id=5 id=6 System failure Flink RabbitMQ Source Checkpoint 1: id=1 id=2 id=3 Message queue id=8 id=7 id=6 id=5 id=4 id=3 id=2 id=1 Message are not lost and send again after recovery
  42. 42. Message Acknowledging  What happens if the system fails after a checkpoint is completed, but before all messages have been acknowledged? 42 Checkpoint 1 completed id=8 id=7 id=6 id=5 id=4 Flink RabbitMQ Source Checkpoint 1: id=1 id=2 id=3 Checkpoint 2: id=4 id=5 id=6ACK id=1 ACK id=2 ACK id=3 FAIL  Flink stores a correlation ID of each (un-acked) message to de-duplicate on restore id=3

×