Exactly Once Semantics Revisited (Jason Gustafson, Confluent) Kafka Summit NYC 2019
•
2 likes•1,996 views
Two years ago, we helped to contribute a framework for exactly once semantics (or EOS) to Apache Kafka. This much-needed feature brought transactional guarantees to stream processing engines such as Kafka Streams. In this talk, we will recount the journey since then and the lessons we have learned as usage has gradually picked up steam. What did we get right and what did we get wrong? Most importantly, we will discuss how the work is continuing to evolve in order to provide more reliability and better performance. This talk assumes basic familiarity with Kafka and the log abstraction. What you will get out of it is a deeper understanding of the underlying architecture of the EOS framework in Kafka, what its limitations are, and how you can use it to solve problems.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Exactly Once Semantics Revisited (Jason Gustafson, Confluent) Kafka Summit NYC 2019
Similar to Exactly Once Semantics Revisited (Jason Gustafson, Confluent) Kafka Summit NYC 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Exactly Once Semantics Revisited (Jason Gustafson, Confluent) Kafka Summit NYC 2019
- 1. 1 Revisiting Exactly Once Semantics (EOS) Jason Gustafson: Engineer@Confluent
- 2. 2 - Review exactly once mechanics - Identify the remaining gaps - Discuss how they are being addressed Overview
- 3. 3 Kafka EOS
- 4. 4 A B C D E Read Process Write Input Output
- 5. 5 A B C D E Read Process Write Input Output
- 6. 6 A B C D E A’ Read Process Write Input Output
- 7. 7 A B C D E A’ Read Process Write Input Output
- 8. 8 A B C D E A’ B’ C’ Read Process Write Input Output
- 9. 9 A B C D E A’ B’ C’ Read Process Write Input Output
- 10. 10 A B C D E A’ B’ C’ D’ E’ Read Process Write Input Output
- 11. 11 1 1 1 1 1 1 1 1 Read Process Write Input Output
- 12. 12 1 1 1 1 1 1 1 1 Read Process Write Input Output
- 13. 13 1 1 1 1 1 1 1 1 2 Read Process Write Input Output
- 14. 14 1 1 1 1 1 1 1 1 2 Read Process Write Input Output
- 15. 15 1 1 1 1 1 1 1 1 2 4 Read Process Write Input Output
- 16. 16 1 1 1 1 1 1 1 1 2 4 Read Process Write Input Output
- 17. 17 1 1 1 1 1 1 1 1 2 4 2 Read Process Write Input Output
- 18. 18 A B C D E Read Process Write Input Output
- 19. 19 A B C D E Read Process Write Input Output
- 20. 20 A B C D E B’ Read Process Write Input Output A’
- 21. 21 A B C D E B’ Read Process Write Input Output A’
- 22. 22 A B C D E B’ D’ Read Process Write Input Output A’ C’ E’
- 23. 23 A B C D E A’ B’ Read Process Write Input Output
- 24. 24 A B C D E A’ B’ Read Process Write Input Output 2 Position
- 25. 25 A B C D E A’ B’ Read Process Write Input Output 2 Position ReadPosition
- 26. 26 A B C D E A’ B’ Read Process Write Input Output 2 Position ReadPosition WritePosition
- 27. 27 A B C D E A’ B’ Read Process Write Input Output 2 Position ReadPosition WritePosition
- 28. 28 A B C D E A’ B’ C’ D’ Read Process Write Input Output 2 Position ReadPosition WritePosition
- 29. 29 A B C D E A’ B’ C’ D’ Read Process Write Input Output 2 4 Position ReadPosition WritePosition
- 30. 30 A B C D E A’ B’ C’ D’ Read Process Write Input Output 2 4 Position ReadPosition WritePosition
- 31. 31 A B C D E A’ B’ C’ D’ E’ Read Process Write Input Output 2 4 Position ReadPosition WritePosition
- 32. 32 A B C D E A’ B’ C’ D’ E’ Read Process Write Input Output 2 4 5 Position ReadPosition WritePosition
- 33. 33 A B C D E A’ B’ C’ D’ E’ Read Process Write Input Output 2 4 5 Position ReadPosition WritePosition Atomic
- 34. 34 Read Process Write ReadPosition WritePosition Atomic The Kafka Approach: 1) Transactional writes across multiple partitions 2) Protocol to guarantee single writer
- 35. 35 Read Process Write ReadPosition WritePosition Atomic The Kafka Approach: 1) Transactional writes across multiple partitions 2) Protocol to guarantee single writer
- 39. 39 Output (O) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 40. 40 Output (O) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 41. 41 Output (O) ongoing (O) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 42. 42 Output (O) A’ B’ ongoing (O) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 43. 43 Output (O) A’ B’ ongoing ongoing (O) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 44. 44 Output (O) 2 A’ B’ ongoing ongoing (O) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 45. 45 Output (O) 2 A’ B’ ongoing ongoing prepare commit (O) (O, P) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 46. 46 Output (O) 2 A’ B’ ongoing ongoing prepare commit (O) (O, P) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 47. 47 Output (O) 2 A’ B’ ongoing ongoing prepare commit (O) (O, P) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 48. 48 Output (O) 2 A’ B’ ongoing ongoing prepare commit done commit (O) (O, P) (O, P) () Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 49. 49 Output (O) 2 A’ B’ done commit () Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 50. 50 Output (O) 2 A’ B’ done commit () Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 51. 51 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing () (O) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn
- 52. 52 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing () (O) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn Timeout AbortTxn
- 53. 53 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing prepare abort () (O) (O, P) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn Timeout AbortTxn
- 54. 54 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing prepare abort () (O) (O, P) (O, P) Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn Timeout AbortTxn
- 55. 55 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing prepare abort finish abort () (O) (O, P) (O, P) () Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn Timeout AbortTxn
- 56. 56 Output (O) 2 5 A’ B’ C’ D’ E’ done commit ongoing ongoing prepare abort finish abort () (O) (O, P) (O, P) () Status Partitions A B C D E Input Position (P) Transaction Log AddPartition Write BeginTxn CommitTxn Timeout AbortTxn
- 57. 57 Read Process Write ReadPosition WritePosition Atomic The Kafka Approach: 1) Transactional writes across multiple partitions 2) Protocol to guarantee single writer
- 58. 58 Single Writer (WTF is a transactional ID?)
- 59. 59 A B C D E Input Output Processor
- 60. 60 A B C Input Output Processor 1 D E F G H I Processor 2 Processor 3
- 61. 61 OutputA B C Input D E F G H I Processor 1 Processor 2 Processor 3
- 62. 62 OutputA B C Input D E F G H I “Single writer” does not mean that an output partition has only one writer. Processor 1 Processor 2 Processor 3
- 63. 63 OutputA B C Input D E F G H I The guarantee we need is that there is a single writer tied to each input partition. Processor 1 Processor 2 Processor 3
- 64. 64 Consumer Group OutputA B C Input D E F G H I Processor 1 Processor 2 Processor 3
- 65. 65 Consumer Group OutputA B C Input D E F G H I Processor 1 Processor 2 Processor 3
- 66. 66 Processor 1 Consumer Group OutputA B C Input D E F G H I Processor 2 Processor 3
- 68. 68 Processor 2 A B C D E F A’ Input Output 1 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition
- 69. 69 Processor 2 A B C D E F A’ B’ C’ Input Output 1 3 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition
- 70. 70 Processor 2 A B C D E F A’ B’ C’ Input Output 1 3 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Transaction committing but not complete
- 71. 71 Processor 2 A B C D E F A’ B’ C’ Input Output 1 3 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Transaction committing but not complete
- 72. 72 Processor 2 A B C D E F A’ B’ C’ Input Output 1 3 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Input partition reassigned to processor 2
- 73. 73 Processor 2 A B C D E F A’ B’ C’ B’ Input Output 1 3 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Processor 2 reads latest committed position of 1
- 74. 74 Processor 2 A B C D E F A’ B’ C’ B’ Input Output 1 3 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Transaction from processor 1 completes
- 76. 76 Processor 1’ A B C D E F A’ B’ C’ Input Output 1 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Processor 1 has an ongoing transaction
- 77. 77 Processor 1’ A B C D E F A’ B’ C’ Input Output 1 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Processor 1 is partitioned from the cluster
- 78. 78 Processor 1’ A B C D E F A’ B’ C’ Input Output 1 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Partition reassigned to processor 2
- 79. 79 Processor 1’ A B C D E F A’ B’ C’ Input Output 1 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Transaction is aborted
- 80. 80 Processor 1’ A B C D E F A’ B’ C’ B’ Input Output 1 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition
- 81. 81 Processor 1’ A B C D E F A’ B’ C’ B’ Input Output 1 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition Processor 1 is able to communicate again
- 82. 82 Processor 1’ A B C D E F A’ B’ C’ B’ D’ Input Output 1 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition
- 83. 83 Processor 1’ A B C D E F A’ B’ C’ B’ D’ Input Output 1 2 Position Read Process Write ReadPosition WritePosition Processor 1 Read Process Write ReadPosition WritePosition
- 84. 84 ● Configured by the producer `transactional.id` property ● Defines a single writer scope ● Enforced by a monotonic epoch ● Initialization protocol to await pending transaction completion Transactional ID
- 85. 85 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 86. 86 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 87. 87 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 88. 88 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 89. 89 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 90. 90 Consumer Group A B C Input Processor 1 D E F H I J Processor 2 Processor 3 txnl.id=A epoch=1 txnl.id=B epoch=1 txnl.id=C epoch=1
- 91. 91 Processor 1 Consumer Group A B C Input D E F H I J txnl.id=A epoch=1 txnl.id=B epoch=1 txnl.id=C epoch=1 Processor 2 Processor 3
- 92. 92 Processor 1 Consumer Group A B C Input D E F H I J txnl.id=A epoch=2 txnl.id=B epoch=1 txnl.id=C epoch=1 Processor 2 Processor 3
- 93. 93 Read Process Write ReadPosition WritePosition The Kafka Approach: 1) Transactional writes across multiple partitions 2) Protocol to guarantee single writer Atomic
- 94. 94 Read Process Write ReadPosition WritePosition Atomic Bump transactional ID epoch and await pending transaction completion
- 95. 95 Read Process Write ReadPosition WritePosition Atomic Multi-partition transactional write protected by epoch Bump transactional ID epoch and await pending transaction completion
- 96. 96 EOS In Code
- 97. 979797 EOS In Code consumer.assign(partitions) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 98. 989898 EOS In Code consumer.assign(partitions) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 99. 999999 EOS In Code consumer.assign(partitions) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 101. 101 Consumer Group OutputA B C Input Processor D E F G H I Processor Processor
- 102. 102 Kafka Streams Application OutputA B C Input Thread D E F G H I Thread Thread
- 103. 103 Kafka Streams Application OutputA B C Input Thread D E F G H I Thread Thread
- 104. 104 Streams Thread Consumer Producer state state state Input Partition Assignment Output Partitions
- 105. 105 Streams Thread Consumer Producer state state state Input Partition Assignment Output Partitions Task
- 106. 106 Streams Thread Consumer Producer state state state Input Partition Assignment Output Partitions For EOS, how do we guarantee single writer scope for each input partition?
- 107. 107 0 1 2 3 4 5t=0 At t=0, we have one thread which is assigned all partitions
- 108. 108 0 1 2 3 4 5t=0 At t=1, the group rebalances and we have two threads 0 1t=1 2 3 4 5
- 109. 109 0 1 2 3 4 5t=0 We must track the transactional state dependence across rebalances! 0 1t=1 2 3 4 5
- 110. 110 0 1 2 3 4 5t=0 We must track the transactional state dependence across rebalances! 0 1t=1 2 3 4 5 t=2 0 1 2 3 4 5
- 111. 111 0 1 2 3 4 5t=0 0 1t=1 2 3 4 5 t=2 0 1 2 3 4 5 0 1 2 3 4 5t=3 We must track the transactional state dependence across rebalances!
- 112. 112 Streams Thread Consumer Producer state state state Input Partition Assignment Output Partitions
- 114. 114 0 1 2 3 4 5t=0 transactional.id = input partition
- 115. 115 t=0 transactional.id = input partition0 1t=1 2 3 4 5 0 1 2 3 4 5
- 116. 116 t=0 transactional.id = input partitiont=1 t=2 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5
- 117. 117 t=0 t=1 t=2 t=3 0 1 2 3 4 5 transactional.id = input partition 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5
- 119. 119 Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool Txn Manager
- 120. 120 What is the problem?
- 121. 121 ● The transactional producer assumes a static assignment of input partitions ● Consumer group partition assignments are dynamic What is the problem?
- 122. 122122122 What is the problem? consumer.assign(partitions) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 123. 123123123 What is the problem? consumer.assign(partitions) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 124. 124124124 What is the problem? consumer.subscribe(topics) producer.initTransactions() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 125. 125 How to fix this
- 126. 126 1. Allow the producer to multiplex many transactional IDs Options
- 127. 127 Txn Manager Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool
- 128. 128 Txn Manager Txn Manager Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool Txn Manager
- 129. 129 Record Accumulator Record Accumulator Txn Manager Txn Manager Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool Txn Manager
- 130. 130 1. Allow the producer to multiplex many transactional IDs Options
- 131. 131 1. Allow the producer to multiplex many transactional IDs 2. Producer pooling for better resource sharing Options
- 132. 132 1. Allow the producer to multiplex many transactional IDs 2. Producer pooling for better resource sharing 3. Address the assignment dependency problem Options
- 133. 133 1. Allow the producer to multiplex many transactional IDs 2. Producer pooling for better resource sharing 3. Address the assignment dependency problem Options
- 134. 134 KIP-447
- 135. 135 Txn Coordinator 1 Txn Coordinator 2 Txn Coordinator 3 Kafka Streams Application Thread Thread Thread
- 136. 136 Txn Coordinator 1 Txn Coordinator 2 Txn Coordinator 3 Kafka Streams Application Thread Thread Thread txnl.id=A
- 137. 137 Txn Coordinator 1 Txn Coordinator 2 Txn Coordinator 3 Kafka Streams Application Thread Thread Thread txnl.id=A txnl.id=B
- 138. 138 Txn Coordinator 1 Txn Coordinator 2 Txn Coordinator 3 Kafka Streams Application Thread Thread Thread txnl.id=A txnl.id=B txnl.id=C txnl.id=D txnl.id=E
- 139. 139 1. Use the shared group id to find the transaction coordinator KIP-447 Recipe
- 140. 140 Txn Coordinator 1 Txn Coordinator 2 Txn Coordinator 3 Kafka Streams Application Thread Thread Thread
- 141. 141 1. Use the shared group id to find the transaction coordinator 2. Make the transaction coordinator aware of group partition assignments KIP-447 Recipe
- 142. 142 0 1 2 3 4 5t=0 0 1t=1 2 3 4 5 t=2 0 1 2 3 4 5 0 1 2 3 4 5t=3
- 143. 143 0 1 2 3 4 5generation=0 0 1generation=1 2 3 4 5 generation=2 0 1 2 3 4 5 0 1 2 3 4 5generation=3
- 144. 144 1. Use the shared group id to find the transaction coordinator 2. Make the transaction coordinator aware of group partition assignments 3. Add logic to initialize and fence with consideration of assignment KIP-447 Recipe
- 145. 145 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transactional Id Initialization
- 146. 146 Bump Epoch Is Transaction In Progress? Begin Abort Is Transaction Completing? Await Completion Yes Yes No Return New Epoch No Transaction Assignment Initialization For all assigned partitions:
- 147. 147 1. Use the shared group id to find the transaction coordinator 2. Make the transaction coordinator aware of group partition assignments 3. Add logic to initialize and fence with consideration of assignment 4. Expose all this in a nice API KIP-447 Recipe
- 148. 148148148 KIP-447 In Code consumer.subscribe(topics) producer.initTxn() while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 149. 149149149 KIP-447 In Code consumer.subscribe(topics, onAssignment(partitions, gen) { producer.initTxn(partitions, gen) }) while (true) { input, offsets = consumer.poll() output = process(input) producer.beginTransaction() producer.send(output) producer.sendOffsets(offsets) producer.commitTransaction() }
- 150. 150 Streams Thread Consumer Producer state state state Input Partition Assignment Output Partitions
- 151. 151 Error Resiliency
- 152. 152 kafka-producer-network-thread | producer-1] ERROR o.a.k.clients.producer.internals.Sender - [Producer clientId=producer-1] The broker returned org.apache.kafka.common.errors.UnknownProducerIdException: This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception. for topic-partition foo-0 at offset -1. This indicates data loss on the broker, and should be investigated.
- 153. 153 Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool Txn Manager
- 154. 154 Record Accumulator Anatomy of the Kafka Producer IO Thread Network Layer Buffer Pool Txn Manager
- 155. 155 Partition Sequence Number foo-0 5 bar-1 23 baz-0 16 State Machine Anatomy of the Transaction Manager state: InTransaction
- 156. 156 Offset ProducerId Epoch Sequence Log Broker Partition State 0 1 2 3 4 3 0 0 1 0 2 1 1 1 2 1 1 2 1 1
- 157. 157 0 1 2 3 4 3 0 0 1 0 2 1 1 1 2 1 1 2 1 1 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 1 1 1 1 3 2 1 Producer State Cache
- 158. 158 0 1 2 3 4 5 3 0 0 1 0 1 2 1 1 1 2 1 1 1 2 1 1 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 1 1 1 2 3 2 1 Producer State Cache
- 159. 159 0 1 2 3 4 5 6 3 0 0 1 0 1 0 2 1 1 1 2 1 2 1 1 2 1 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache
- 160. 160 0 1 2 3 4 5 6 3 0 0 1 0 1 0 2 1 1 1 2 1 2 1 1 2 1 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache Records deleted after hitting retention limit
- 161. 161 4 5 6 0 1 0 2 1 2 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache
- 162. 162 4 5 6 0 1 0 2 1 2 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 Producer State Cache
- 163. 163 Well..
- 164. 164 - Used to change the partition key in Kafka Streams - Once repartitioned data has been consumed, it is no longer needed - Proactively actively delete unneeded data! Repartition Topics
- 165. 165 0 1 2 3 4 5 6 3 0 0 1 0 1 0 2 1 1 1 2 1 2 1 1 2 1 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache
- 166. 166 0 1 2 3 4 5 6 3 0 0 1 0 1 0 2 1 1 1 2 1 2 1 1 2 1 1 2 2 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache
- 167. 167 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence 0 2 2 1 1 2 3 2 1 Producer State Cache
- 168. 168 Offset ProducerId Epoch Sequence Log Broker Partition State ProducerId Epoch Sequence Producer State Cache
- 169. 169 How to fix this
- 170. 170 Log
- 171. 171 Log
- 172. 172 m Log ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 173. 173 m Log ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 174. 174 m Log ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot n ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 175. 175 m Log ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot n ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 176. 176 m Log ProducerId Epoch Sequence 0 3 5 n.snapshot n ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 177. 177 Maybe just don’t do that?
- 178. 178 m Log ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot n ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 179. 179 Don’t forget about replication
- 180. 180 m n replica 1 ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot
- 181. 181 m n replica 1 ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot replica 2
- 182. 182 m n replica 1 ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot replica 2
- 183. 183 m n replica 1 ProducerId Epoch Sequence 0 3 5 1 1 1 n.snapshot ProducerId Epoch Sequence 0 2 1 1 1 1 m.snapshot replica 2 ProducerId Epoch Sequence 0 3 5
- 184. 184 ● Log record deletion ● Log start offset inconsistency ● Unclean leader election Sequence Conflicts
- 185. 185 KIP-360
- 186. 186 replica 1 ProducerId Epoch Sequence 0 3 5 1 1 1 State Cache replica 2 ProducerId Epoch Sequence 0 3 5 State Cache
- 187. 187 replica 1 (leader) ProducerId Epoch Sequence 0 3 5 1 1 1 State Cache replica 2 (follower) ProducerId Epoch Sequence 0 3 5 State Cache
- 188. 188 replica 1 (leader) ProducerId Epoch Sequence 0 3 5 1 1 3 replica 2 (follower) ProducerId Epoch Sequence 0 3 5 State Cache State Cache
- 189. 189 replica 1 (leader) ProducerId Epoch Sequence 0 3 5 1 1 3 replica 2 (follower) ProducerId Epoch Sequence 0 3 5 1 1 3 State Cache State Cache
- 190. 190 replica 1 (follower) ProducerId Epoch Sequence 0 3 5 1 1 1 replica 2 (leader) ProducerId Epoch Sequence 0 3 5 State Cache State Cache
- 191. 191 replica 1 (follower) ProducerId Epoch Sequence 0 3 5 1 1 1 replica 2 (leader) ProducerId Epoch Sequence 0 3 5 1 2 2 State Cache State Cache
- 192. 192 replica 1 (follower) ProducerId Epoch Sequence 0 3 5 1 2 2 replica 2 (leader) ProducerId Epoch Sequence 0 3 5 1 2 2 State Cache State Cache
- 193. 193 Error Current Impact Future Recovery OutOfOrderSequence FATAL Abort transaction, bump epoch and continue UnknownProducerId FATAL Abort transaction, bump epoch and continue KIP-360 Error Handling
- 194. 194 Error Current Impact Future Recovery OutOfOrderSequence FATAL Abort transaction, bump epoch and continue UnknownProducerId FATAL Abort transaction, bump epoch and continue ProducerFenced FATAL Rejoin consumer group, reinitialize transaction state KIP-360/KIP-447 Error Handling
- 195. 195 Conclusion
- 196. 196 Kafka has an elegant transaction model which has held up well. To reach full maturity: ● Address the producer/consumer semantic mismatch ● Improve producer resilience to errors
- 197. 197 Thank you!
- 198. 198 - KIP-447: https://cwiki.apache.org/confluence/display/KAFK A/KIP-447%3A+Producer+scalability+for+exactly+o nce+semantics - KIP-360: https://cwiki.apache.org/confluence/display/KAFK A/KIP-360%3A+Improve+handling+of+unknown+pro ducer - Contribute to Apache Kafka: https://kafka.apache.org/contributing - Join Confluent: https://www.confluent.io/careers/ Resources