This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include: - Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication. - Producers can write in a transactional fashion and receive acknowledgments of committed writes from brokers. - Brokers store commit markers to track the progress of transactions and ensure no data loss during failures. - Consumers can read from brokers in a transactional mode and receive data only from committed transactions, guaranteeing no duplication of records. - This allows reliable message delivery semantics between producers and consumers with Kafka acting as

1. Apache Kafka 0.11
Exactly Once Semantics
/ @laclefyoshi / ysaeki@r.recruit.co.jp

2. •
• Kafka
• Exactly Once :
•
2

3. • 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
3

4. Kafka

5. Kafka
5

6. Kafka
ACK
→
ACK
→
6

7. Kafka
Broker
→
7

8. Kafka
→
→
8

9. Kafka
At Least Once At Most Once
9

10. Kafka
At Least Once At Most Once Exactly Once
9

11. Exactly Once :

12. Kafka 0.11.0.0
• 2017/06/28
• KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams
• https://issues.apache.org/jira/browse/KAFKA-4923
• Confluent Blog: Exactly-once Semantics are Possible: Here’s How
Kafka Does it
• https://www.confluent.io/blog/exactly-once-semantics-are-possible-
heres-how-apache-kafka-does-it/
• “In the following posts in this series, we will go into more details …”
• Exactly Once Delivery and Transactional Messaging in Kafka
• https://docs.google.com/document/d/
11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/
• KIP-98 → KAFKA-5059: Implement Transactional Coordinator
11

14. Kafka
13

15. Kafka : Exactly Once
14

16. Producer Exactly Once
15

17. Producer Exactly Once
1.
2.
3. x N
4. →
16

18. Producer Exactly Once
1.
2.
3. x N
4.
→
→
→ 1
17

19. Producer Exactly Once
5.Broker Commit Marker
COMMITTED ABORTED
↓ ↓
18

20. Consumer Exactly Once
19

21. Consumer Exactly Once
6.
↓
READ_COMMITTED
↓
READ_UNCOMMITTED
20

22. Consumer Exactly Once
7.
8.
- COMMITTED
- ABORTED Transaction List
21

23. • Kafka API RecordBatch
• Kafka < 0.11 Message MessageSet
API
• Kafka 0.11 MessageSet/Message RecordBatch/Record
•
• Producer Broker Consumer
ID PID Producer ID
22

24. Exactly Once
• : Exactly Once
• Producer
• Broker
• Commit Marker
• Consumer
•
→ OSS

25. • Exactly Once
• Producer Broker
• Consumer
• 3%
•
• Confluent
• OSS
Kafka
• Exactly Once
24