​Apache Kafka’s rise in popularity as a streaming platform has demanded a revisit of its traditional at least once message delivery semantics. In this talk, we present the recent additions to Apache Kafka to achieve exactly once semantics. We shall discuss the newly introduced transactional APIs and use Kafka Streams as an example to show how these APIs are leveraged for streams tasks.

1. 1Confidential
Introducing ExactlyOnce
Semantics in Apache® KafkaTM
Jason Gustafson, Apurva Mehta, Guozhang Wang, and Sriram
Subramaniam
Matthias J. Sax | Software Engineer
matthias@confluent.io
@MatthiasJSax

2. 2Confidential
Outline
• Kafka’sexisting delivery semantics.
• What’s new?
• How do you use it?
• Summary.

3. 3Confidential
What Kafka offers today
• At-least-once,in-order delivery per partition.
• Producer retries can introduce duplicates.

4. 4Confidential
Stream Processing with Apache Spark
Read-process-write pattern:

5. 5Confidential
Example: duplicate write
Producer Broker
Topic Partition

6. 6Confidential
Example: duplicate write
Producer Broker
Topic Partition
send (k,v)

7. 7Confidential
Example: duplicate write
Producer Broker (k,v)
Topic Partition
append (k,v)

8. 8Confidential
Example: duplicate write
Producer Broker (k,v)
Topic Partition
ack

9. 9Confidential
Example: duplicate write
Producer Broker (k,v)
Topic Partition
send (k,v)

10. 10Confidential
Example: duplicate write
Producer Broker (k,v) (k,v)
Topic Partition
append (k,v)

11. 11Confidential
Example: duplicate write
Producer Broker (k,v) (k,v)
Topic Partition
ack

12. 12Confidential
Why improve?
• Stream processing is becoming a bigger part of the data landscape.
• Apache Kafka is the foundation for such stream processing.
• Strengthening Kafka’ssemantics expands the universe of
streaming applications.

13. 13Confidential
What’s new?

14. 14Confidential
What’s new
• Idempotent producer:exactly-once writes.
• Transactionalproducer:Atomic writes across multiple partitions.
• Exactly-once stream processing: read-process-write.

15. 15Confidential
What’s new
Idempotent Producer

16. 16Confidential
Example: Idempotent Producer
Producer Broker
Topic Partition

17. 17Confidential
Example: Idempotent Producer
Producer Broker
send (k,v)
seq = 0
pid = 73 Topic Partition

18. 18Confidential
Example: Idempotent Producer
Producer Broker
append (k,v)
seq = 0
pid = 73 Topic Partition
(k,v)
seq = 0
pid = 73

19. 19Confidential
Example: Idempotent Producer
Producer Broker
ack
Topic Partition
(k,v)
seq = 0
pid = 73

20. 20Confidential
Example: Idempotent Producer
Producer Broker
send (k,v)
seq = 0
pid = 73 Topic Partition
(k,v)
seq = 0
pid = 73

21. 21Confidential
Example: Idempotent Producer
Producer Broker
ack (dup)
Topic Partition
(k,v)
seq = 0
pid = 73

22. 22Confidential
What’s new
Atomic Multi-Partition Writes
(aka “transactions”)

23. 23Confidential
Atomic Multi-Partition Writes
Producer
Topic A, Partition 0
Topic B, Partition 0
Topic B, Partition 1

24. 24Confidential
Atomic Multi-Partition Writes
Producer
Topic A, Partition 0 m1 m5
m3 m4 m6Topic B, Partition 0
m2Topic B, Partition 1

25. 25Confidential
Atomic Multi-Partition Writes
Producer
Topic A, Partition 0 m1 m5 C
m3 m4 m6Topic B, Partition 0
m2Topic B, Partition 1
C
C
atomic commit

26. 26Confidential
Atomic Multi-Partition Writes
Consumer
Topic A, Partition 0 m1 m5 C
m3 m4 m6Topic B, Partition 0
m2Topic B, Partition 1
C
C
read committed

27. 27Confidential
TransactionalAPI
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(record0);
producer.send(record1);
producer.sendOffsetsToTxn(…);
producer.commitTransaction();
} catch (ProducerFencedException e) {
producer.close();
} catch (KafkaException e) {
producer.abortTransaction();
}

28. 28Confidential
How to use exactly-once capabilities:
• Streams API (the easiest way to use exactly-once semantics)
• Config parameter processing.mode = “exactly_once”
• Idempotent Producer
• Config parameter enable.idempotence = true
• Transactional Producer
• Config parameter transactional.id = “my-unique-tid”
• And Transactional API (hard to use!)
• Transactional Consumer
• Config parameter isolation.level = “read_committed”
(default: “read_uncommitted”)

29. 29Confidential
Stream Processing with Kafka’s Streams API
Transactional read-process-write-commit pattern:

30. 30Confidential
Stream Processing with Apache Spark
Transactional read-process-write-commit pattern:

31. 31Confidential
When to use this?
Available in Kafka 0.11, June 2017. Try it out!

32. 32Confidential
Putting it together
• We understood Kafka’sexisting delivery semantics.
• Learned how these have been strengthened.
• Learned how the new semantics work.
• Saw, it’s easy to use with higher levelAPIs like Kafka Streams or
Apache Spark.

33. 33Confidential
Thank You
We are hiring!