In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them. I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol. Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings. Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.

1. natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
Exactly Once Delivery
is a harsh mistress
Natan Silnitsky
Backend Infra Developer, Wix.com

2. A Scala/Java high-level SDK for Apache Kafka.
Greyhound

3. As it happens,
we have a distributed system at Wix.
~1400 micro-services
@NSilnitsky

4. As it happens,
we have a distributed system at Wix.
Edit MySite
~1400 micro-services
~1015M messages every day
@NSilnitsky

5. PurchaseCompleted
Classic ecommerce flow
UpdateInventory(ItemN)
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
@NSilnitsky
Payments Checkout Inventory

6. Classic ecommerce flow - hard to achieve
PurchaseCompleted
UpdateInventory(ItemN)
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
😳 But... how do we update all items exactly once? on failures/restarts too...
Payments Checkout Inventory

7. PurchaseCompleted
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
Classic ecommerce flow - hard to achieve
Inventory:
Item1 9 → 8
Item2 5 → 4
@NSilnitsky

8. PurchaseCompleted
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
Classic ecommerce flow - hard to achieve
Inventory:
Item1 9 → 7
Item2 5 → 3
Payments
* use DB

9. Achieving Exactly-Once delivery in
distributed systems is NOT easy.

10. Message delivery over the network
is unreliable.
@NSilnitsky

11. Message delivery over the network
is unreliable.
Message
Consumer
Message
Producer
Broker
@NSilnitsky

12. Kafka Broker
Topic-1
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
Topic-2
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
(append-only logs)
Partitions
@NSilnitsky

13. Kafka
Consumer
Kafka
Producer
Kafka Broker
Topic-1
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
@NSilnitsky

14. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

15. Kafka Broker
Kafka
Producer
0 1 2 3 4 5
😕 Services need to address message duplicates
at-most-onceat-least-once exactly-once
Producer retries on
every failure

16. Kafka Broker
0 1 2 3 4 5
😕 Services need to handle messages exactly once
consumerRecords = Consumer.poll
process(consumerRecords)
consumer.commit
at-most-onceat-least-once exactly-once
Consumer retries
on every failure
Kafka
Consumer

17. Kafka Broker
0 1 2 3 4 5
😕 Messages may be lost
Consumer commits
before processing
(AutoCommit - Default)
consumerRecords = Consumer.poll
Consumer.commit
process(consumerRecords)
at-most-onceat-least-once exactly-once
Kafka
Consumer

18. Kafka Broker
0 1 2 3 4 5
Kafka
Consumer
😕 Really really really hard to do
Messages are read and processed
exactly once by the consumer
?
at-least-once exactly-onceat-most-once

19. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

20. The “Two Generals Problem"
thought experiment
Alice Bob

21. Tomorrow
7 AM
The “Two Generals Problem"
thought experiment
Alice Bob

22. The “Two Generals Problem"
thought experiment
Alice Bob

23. The “Two Generals Problem"
thought experiment
Alice Bob
Tomorrow
7 AM

24. OK!
Tomorrow
7 AM
The “Two Generals Problem"
thought experiment
Alice Bob

25. ?
The “Two Generals Problem"
thought experiment
Alice Bob
OK!

26. The “Two Generals Problem"
thought experiment
AliceService BobService

27. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

28. Kafka Broker
Processor Observer
Consumer
-Producer
Consumer
Topic A Topic B
@NSilnitsky

29. Kafka Broker
Processor Observer
Consumer
-Producer
Consumer
Topic A Topic B
@NSilnitsky

30. Kafka Broker
Kafka
Idempotent
Producer
0 1 2 3 4 5
0 1
1
Attaches offset to
message
Enable.idempotence = true
@NSilnitsky

31. Kafka Broker
Kafka
Idempotent
Producer
0 1 2 3 4 5
0 1
1
Attaches offset to
message
Enable.idempotence = true
duplicate!
@NSilnitsky

32. Kafka Broker
Topic A Topic B
Processor Observer
Consumer
-Producer
Consumer
@NSilnitsky

33. Kafka Broker
consumer.poll
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
AB
01
Topic A Topic B
@NSilnitsky
isolation.level =
"read_committed"

34. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
01
Topic A Topic B
@NSilnitsky

35. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
012
Topic A Topic B
@NSilnitsky

36. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
producer.commitTransaction
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
012
Topic A Topic B
@NSilnitsky

37. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
01
Topic A Topic B
@NSilnitsky

38. Kafka Broker
Throughput impact is between 3% to 25% worse.
Processor Observer
@NSilnitsky

39. Kafka Broker
Throughput impact is between 2% to 25% worse.
The bigger the batch the better the throughput and longer the latency.
Processor Observer
@NSilnitsky

40. Kafka Broker
Processor Observer
No end-to-end exactly-once guarantees
out of the box.
@NSilnitsky

41. Kafka Broker
Deduplicate using offsets from Kafka Transaction
@NSilnitsky
ABC
012
Topic B
Observer
DB Table
Partition Offset Value
0 2 C
... ... ...

42. In reality, this is more complex
than how I describe it.
Out of scope:
▪ Two-phase-commit with transaction coordinator
▪ Transaction log
▪ Additional “fencing” data
▪ 1 processor-producer - 1 partition (up to Kafka 2.4.x)
@NSilnitsky

43. Exactly Once in Kafka Streams
StreamsConfig:
processing.guarantee = "exactly_once"
consumers will be configured with:
● isolation.level = "read_committed"
producers will be configured with:
● enable.idempotence = true
@NSilnitsky

44. Purchase
Completed
UpdateInventory(Item1)
Exactly Once (Classic ecommerce) flow at Wix
UpdateInventory(Item2)
UpdateInventory(ItemN)
ABCDEF
1
iNB
1
I3I8
I1I2I5I5
01
updates
@NSilnitsky

45. Kafka Broker
Exactly Once (Classic ecommerce) flow at Wix
- Dedup
@NSilnitsky
Topic B
Observer
Inventory
Partition Offset Item Value
0 0 I1 9->8
0 1 I2 5->401
I1I2
UPDATE offset 0, Item1 -1
UPDATE offset 1 Item2 - 1

46. Kafka Broker
Exactly Once (Classic ecommerce) flow at Wix
- Dedup
@NSilnitsky
Topic B
Observer
Inventory
Partition Offset Item Value
0 0 I1 8
0 1 I2 401
I1I2
UPDATE offset 0, Item1 -1
UPDATE offset 1 Item2 - 1
UPDATE offset 1 Item2 - 1

47. Exactly-Once delivery is...
like the holy grail of message delivery over the network.
It’s a tough nut to crack.

48. Exactly-Once delivery is...
complex to implement, to use, and it requires fine-tuning.

49. Exactly-Once delivery is...
crucial for achieving atomic actions in distributed systems.

50. Thank You
natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil

51. Resources
EoS in Kafka talk by Jason Gustafson
Exactly-once Semantics are Possible by Neha Narkhede (Performance)
Transactions in Apache Kafka
Revisiting Exactly One Semantics by Jason Gustafson, Confluent
Proposal to improve EOS Produce scalability - Kafka 2.5
How akka works with exactly once message delivery by Hugh McKee

52. A Scala/Java high-level SDK for Apache Kafka.
github.com/wix/greyhound

53. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky
natansil.com