Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Kafka 0.11 の Exactly Once Semantics

3,224 views

Published on

Apache Kafka Meetup Japan #3 https://kafka-apache-jp.connpass.com/event/58619/ LT発表(突発)資料

Published in: Technology

Apache Kafka 0.11 の Exactly Once Semantics

  1. 1. Apache Kafka 0.11 Exactly Once Semantics / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka • Exactly Once : • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) 3
  4. 4. Kafka
  5. 5. Kafka 5
  6. 6. Kafka ACK → ACK → 6
  7. 7. Kafka Broker → 7
  8. 8. Kafka → → 8
  9. 9. Kafka At Least Once At Most Once 9
  10. 10. Kafka At Least Once At Most Once Exactly Once 9
  11. 11. Exactly Once :
  12. 12. Kafka 0.11.0.0 • 2017/06/28 • KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams • https://issues.apache.org/jira/browse/KAFKA-4923 • Confluent Blog: Exactly-once Semantics are Possible: Here’s How Kafka Does it • https://www.confluent.io/blog/exactly-once-semantics-are-possible- heres-how-apache-kafka-does-it/ • “In the following posts in this series, we will go into more details …” • Exactly Once Delivery and Transactional Messaging in Kafka • https://docs.google.com/document/d/ 11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/ • KIP-98 → KAFKA-5059: Implement Transactional Coordinator 11
  13. 13. Kafka 13
  14. 14. Kafka : Exactly Once 14
  15. 15. Producer Exactly Once 15
  16. 16. Producer Exactly Once 1. 2. 3. x N 4. → 16
  17. 17. Producer Exactly Once 1. 2. 3. x N 4. → → → 1 17
  18. 18. Producer Exactly Once 5.Broker Commit Marker 
 COMMITTED ABORTED ↓ ↓ 18
  19. 19. Consumer Exactly Once 19
  20. 20. Consumer Exactly Once 6. 
 ↓ READ_COMMITTED ↓ READ_UNCOMMITTED 20
  21. 21. Consumer Exactly Once 7. 8. 
 - COMMITTED 
 - ABORTED Transaction List 
 21
  22. 22. • Kafka API RecordBatch • Kafka < 0.11 Message MessageSet API • Kafka 0.11 MessageSet/Message RecordBatch/Record • • Producer Broker Consumer ID PID Producer ID 22
  23. 23. Exactly Once • : Exactly Once • Producer • Broker • Commit Marker • Consumer • → OSS
  24. 24. • Exactly Once • Producer Broker • Consumer • 3% • • Confluent • OSS Kafka • Exactly Once 24

×