Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

No data loss pipeline with apache kafka

How to build a no data loss pipeline using Apache kafka

  • Login to see the comments

No data loss pipeline with apache kafka

  1. 1. No Data Loss Pipeline with Apache Kafka Jiangjie (Becket) Qin @ LinkedIn
  2. 2. ● Data loss o producer.send(record) is called but record did not end up in consumer as expected ● Message reordering o send(record1) is called before send(record2) o record2 shows in broker before record1 does o matters in cases like DB replication Data loss and message reordering
  3. 3. Kafka based data pipeline Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker Today’s Agenda: ● No data loss ● No message reordering ● Mirror maker enhancement ○ Customized consumer rebalance listener ○ Message handler
  4. 4. Synchronous send is safe but slow... producer.send(record).get() Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  5. 5. Using asynchronous send with callback can be tricky producer.send(record,callback) Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  6. 6. Producer can cause data loss when ● block.on.buffer.full=false ● retries are exhausted ● sending message without using acks=all Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  7. 7. Is this good enough? producer.send(record,callback) ● block.on.buffer.full=TRUE ● retries=Long.MAX_VALUE ● acks=all ● resend in callback when message send failed Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  8. 8. Message reordering might happen if: ● max.in.flight.requests.per.connection > 1, AND ● retries are enabled Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker Kafka BrokerProducer message 0 message 1 message 0 failed retry message 0 Timeline
  9. 9. Message reordering might also happen if: ● producer is closed carelessly o close producer in user thread, or o close without using close(0) Producer Record Accumulator Sender Thread Kafka Broker Timeline 1.msg 0 2.callback(msg 0) ack expt. User Thread close prod. 3.msg 1 notify
  10. 10. ● close producer in the callback on error ● close producer with close(0) to prevent further sending after previous message send failed Producer Record Accumulator Sender Thread Kafka Broker Timeline 1.msg 0 2.callback(msg 0) ack expt. User Thread close(0) notify
  11. 11. To prevent data loss: ● block.on.buffer.full=TRUE ● retries=Long.MAX_VALUE (for some use cases) ● acks=all To prevent reordering: ● max.in.flight.requests.per.connection=1 ● close producer in callback with close(0) on send failure Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  12. 12. Not a perfect solution: ● Producer needs to be closed to guarantee message order. E.g. In mirror maker, one message send failure to a topic should not affect the whole pipeline. ● When producer is down, message in buffer will still be lost Producer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  13. 13. Correct producer setting is not enough ● acks=all still can lose data when unclean leader election happens. ● Two replicas are needed at any time to guarantee data persistence. Kafka Brokers Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  14. 14. ● replication factor >= 3 ● min.isr = 2 ● Replication factor > min.isr o If replication factor = min.isr, partition will be offline when one replica is down Kafka Brokers Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  15. 15. Settings we use: ● replication factor = 3 ● min.isr = 2 ● unclean leader election disabled Kafka Brokers Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  16. 16. ● Consumer might lose message when offsets are committed carelessly. E.g. commit offsets before processing messages completely o Disable auto.offset.commit o Commit offsets only after the messages are processed Consumer Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  17. 17. Kafka based data pipeline Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker Today’s Agenda: ● No data loss ● No message reordering ● Mirror maker enhancement ○ Customized consumer rebalance listener ○ Message handler
  18. 18. ● Consume-then-produce pattern ● Only commit consumer offsets of messages acked by target cluster ● Default to no-data-loss and no- reordering settings Mirror Maker Enhancement Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  19. 19. ● Customized Consumer Rebalance Listener o Can be used to propagate topic change from source cluster to target cluster. E.g. partition number change, new topic creation. Mirror Maker Enhancement Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  20. 20. ● Customized Message Handler, useful for o partition-to-partition mirror o filtering out messages o message format conversion o other simple message processing Mirror Maker Enhancement Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  21. 21. ● startup/shutdown acceleration o parallelized startup and shutdown o 26 nodes cluster with 4 consumer each takes about 1 min to startup and shutdown Mirror Maker Enhancement Kafka Cluster (Colo 1) Producer Kafka Cluster (Colo 2) ConsumerMirror Maker
  22. 22. Q&A

×