1) The document discusses Kafka transactions and exactly-once processing in Kafka. 2) It describes the current approach Kafka uses to achieve exactly-once semantics, including idempotent writes within a partition and transactional writes across partitions. 3) It also discusses challenges with the current approach, such as lack of scalability due to the need to create a producer for each input partition, and proposes solutions in KIP-447 to address these challenges.

1. 1
Exactly-Once Made Fast
Boyang Chen: Engineer@Confluent
Guozhang Wang: Engineer@Confluent

2. 2
- Recap: exactly-once semantics (EOS) for Kafka
- What cost are we paying for EOS today
- Closing the gaps: usability and scalability
Overview

3. 3
Back in 2017..

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9
Commit

10. 10
Commit

11. 11
Commit

12. 12
Commit
Atomic

13. 13
The Kafka Approach for Exactly Once
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions
[KIP-98, KIP-129]

14. 14
The Kafka Approach for Exactly Once
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions
[KIP-98, KIP-129]

15. 15
Kafka Transactions

16. 16

17. 17
txn-status: non-exist, partitions: {}

18. 18
txn-status: non-exist, partitions: {}
producer.initTxn();

19. 19
txn-status: empty, partitions: {}
producer.initTxn();
non-exist

20. 20
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
}
} catch (KafkaException e) {
}txn-status: empty, partitions: {}
producer.initTxn();

21. 21
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: empty, partitions: {}
producer.initTxn();

22. 22
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();
non-exist {}

23. 23
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

24. 24
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

25. 25
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

26. 26
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

27. 27
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();
{output-0}

28. 28
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

29. 29
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();
on-going

30. 30
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();

31. 31
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();

32. 32
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: committed, partitions: {output-0, offset-0}
producer.initTxn();
prep-commit

33. 33
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

34. 34
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

35. 35
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();
on-going

36. 36
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();

37. 37
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();

38. 38
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: aborted, partitions: {output-0, offset-0}
producer.initTxn();
prep-abort

39. 39
The Kafka Approach for Exactly Once:
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions

40. 40

41. 41

42. 42
At a given time, an input partition should only be processed by a single client

43. 43
At a given time, an input partition should only be processed by a single client
Consumer Group

44. 44
At a given time, an input partition should only be processed by a single client
Consumer Group

45. 45
At a given time, an input partition should only be processed by a single client
Consumer Group

46. 46
The “Single Writer” Fencing Problem

47. 47
When Taking Over the Partition:
1) The previous txn must have completed commit or abort so there are no
concurrent transactions.
2) Other clients will be fenced write processing results for those input
partitions, a.k.a we have a “single writer”.

48. 48
Transactional ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

49. 49
Transactional ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

50. 50
Transactional ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

51. 51
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 1

52. 52
Consumer Group
txn.Id = A, epoch = 1

53. 53
Consumer Group
txn.Id = A, epoch = 1

54. 54
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, initializing...

55. 55
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 2

56. 56
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 2
Num. producer transaction IDs ~= num. input partitions
Producers need to be dynamically created when rebalance

57. 57
EOS Scalability

58. 58
Number of Input
Partitions
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00

59. 59
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00

60. 60
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once

61. 61
KIP-447

62. 62
What problems
are KIP-447
solving ?

63. 63
What problems
are KIP-447
solving ?
● Make one producer per process model work

64. 64
What problems
are KIP-447
solving ?
● Make one producer per process model work
● Unblock technical challenges
○ Offset commit fencing
○ Concurrent transaction

65. 65
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

66. 66
● We are fencing on the transactional producer side,
which assumes a static partition assignment
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

67. 67
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

68. 68

69. 69

70. 70

71. 71

72. 72

73. 73

74. 74
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

75. 75
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

76. 76

77. 77

78. 78

79. 79

80. 80

81. 81

82. 82

83. 83
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
● Add new APIs
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

84. 84
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(offsets);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output, offset}

85. 85
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(offsets,
consumer.groupMetadata());
producer.commitTxn();
} catch (KafkaException e) {
}
txn-status: on-going, partitions: {output, offset}

86. 86
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
● Add new APIs
○ Expose group generation through
consumer#groupMetadata()
○ Commit transaction with consumer metadata through
producer#sendOffsetsToTransaction(offsets,
groupMetadata)
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

87. 87
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction

88. 88
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction
● Only one open transaction allowed for each input
partition

89. 89
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section

90. 90
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Broker uses pending offsets to indicate
other ongoing transaction

91. 91
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Broker uses pending offsets to indicate
other ongoing transaction
○ Observed: consumer always needs to fetch offset after
rebalance

92. 92
What problems
are KIP-447
solving ?
● Offset commit fencing
● Concurrent transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Broker uses pending offsets to indicate
other ongoing transaction
○ Observed: consumer always needs to fetch offset after
rebalance
○ Action: OffsetFetchRequest will back-off until pending
offsets are cleared, either by previous transaction
complete or timeout

93. 93

94. 94

95. 95

96. 96

97. 97

98. 98

99. 99
447 Summary

100. 100
447 Summary
● Resolve the semantic mismatch between producer
and consumer
○ Offset commit fencing
○ Concurrent transaction

101. 101
447 Summary
● Resolve the semantic mismatch between producer
and consumer
○ Offset commit fencing
○ Concurrent transaction
● Make the one producer per processing unit possible

102. 102
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once

103. 103
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once
Exactly Once After 447

104. 104
Scale Testing

105. 105
Prove the 447 scalability improvement to
break the limit
- At_least_once
- Exactly_once
- Exactly_once_beta (post KIP-447 EOS)
Scale Testing

106. 106
Scale Testing

107. 107
Scale Testing
● Num.brokers = 3
● Num.input.partitions = 200
● Num.output.partitions = 100
● Test.interval.ms = 4 min
● Num.threads = 3
● Num.records.second = 1000
● Commit.interval.ms = 1 second
● Num.instances = 10, 20, 30...

108. 108
Scale Testing
● Num.brokers = 3
● Num.input.partitions = 200
● Num.output.partitions = 100
● Test.interval.ms = 4 min
● Num.threads = 3
● Num.records.second = 1000
● Commit.interval.ms = 1 second
● Num.instances = 10, 20, 30...

109. 109
Scale Testing
● At_least_once and
exactly_once_beta perform
steadily
● Exactly_once (pre KIP-447)
throughput degrades
significantly around 20-25
applications

110. 110
Opt-in on Kafka Streams

111. 111
Upgrade Procedure

112. 112
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5

113. 113
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5
● Upgrade the stream application binary and keep the
PROCESSING_GUARATNEE setting at "exactly_once". Do the first rolling
bounce, and make sure the group is stable with every instance on 2.6 binary.

114. 114
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5
● Upgrade the stream application binary and keep the
PROCESSING_GUARATNEE setting at "exactly_once". Do the first rolling
bounce, and make sure the group is stable with every instance on 2.6 binary.
● Upgrade the PROCESSING_GUARANTEE setting to "exactly_once_beta" and do
a second rolling bounce to start using new thread producer for EOS.

115. 115
Conclusion

116. 116
1. Walkthrough Kafka transaction model

117. 117
1. Walkthrough Kafka transaction model
2. The usability and scalability issues with “single writer”

118. 118
1. Walkthrough Kafka transaction model
2. The usability and scalability issues with “single writer”
3. How KIP-447 solves the challenges

119. 119
1. Walkthrough Kafka transaction model
2. The usability and scalability issues with “single writer”
3. How KIP-447 solves the challenges
4. How to adopt KIP-447 in Kafka Streams

120. 120
Thank you!

121. 121
- KIP-98 - Exactly Once Delivery and Transactional
Messaging - Apache Kafka
- KIP-447: Producer scalability for exactly once
semantics - Apache Kafka
Resources