1) Apache Kafka is a distributed streaming platform that allows for continuous stream processing. It provides APIs for producers and consumers to read and write streams, as well as connectors and stream processing capabilities. 2) Kafka faces two main problems: duplicate writes and ensuring exactly-once processing. It solves duplicate writes through an idempotent producer that assigns unique IDs to each message. 3) Kafka solves exactly-once processing through transactions. Producers can initiate transactions to send multiple records atomically, and consumers only see committed transactions, ensuring exactly-once semantics for stream processing applications.

1. 1
Exactly Once Semantics in
Apache Kafka

2. 2
Apache Kafka: A Distributed Streaming Platform
Consumers
Producers
Connectors
Processing

3. 3
A Distributed What???
A Streaming Platform is a little like…
• A messaging system
• Except it scales horizontally, stores the streams persistently,
and allows continuous stream processing
• Hadoop
• But not batch oriented

4. 4
Logs: A data structure for continuous streams

5. 5
APIs
1. Producer and Consumer: Read and write streams
2. Connect: Managed Connectors that connect existing
systems
3. Streams: Transformations of streams

6. 6
Producer & Consumer API

7. 7
Consumers Scale With Groups

8. 8
Connect API

9. 9
Kafka Connect Does The Hard Parts
1. Scale out
2. Fault Tolerance
3. Central Management
4. Schemas

10. 10

11. 11

12. 12

13. 13

14. 14
Streams API
• Full power of a modern stream processing framework
• Distributed and fault-tolerant
• Natively uses event-time
• Stateful processing: joins, aggregations, etc
• Integrates tables and streams
• Easy re-processing
• Just a library

15. 15
Wordcount Example

16. 16
Wordcount Example

17. 17
Deploy as you wish

18. 24
Not Limited To Java
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

19. 25
The Semantics of Working With Streams

20. 26
Two Problems
1. Duplicate writes
2. Exactly-once processing

21. 27
Problem #1:
Duplicate Writes

22. 28
Duplicate Writes

23. 29
Duplicate Writes

24. 30
Duplicate Writes

25. 31
Duplicate Writes

26. 32
Duplicate Writes

27. 33
Duplicate Writes

28. 34
Duplicate Writes

29. 35
Duplicate Writes

30. 36
Duplicate Writes

31. 37
Problem #2:
Duplicate Processing

32. 38
Read From Offset=0

33. 39
Process and Update State

34. 40
Commit offset 0 as processed

35. 41
Read from offset=1

36. 42
Process and Update State

37. 43
App crashes!

38. 44
Restore from offset=0, resume processing

39. 45
Chose your undesirable semantics
1. Update state, then save offset => At-Least-Once Delivery
2. Save offset, then update state => At-Most-Once Delivery

40. 46
Two workarounds
1. Make processing idempotent
• Much harder than it sounds in practice
2. Store offset in the application DB and update transitionally
• Not all stores support transactions
• Must handle zombies

41. 47
Solving These Problems

42. 48
Solving Problem #1:
Avoiding Duplicate Writes with an Idempotent
Producer

43. 49
Basic idea
1. Unique ID for each message
2. Server deduplicates

44. 50
Basic idea has problems
1. Random access database of all message ids?
2. Message IDs would be bulky
3. Must handle server fail-over

45. 51
Better idea: Do it like TCP
1. Unique producer id for each producer (PID)
2. Each producer assigns a sequential number to each
message it sends
3. The unique identifier is the PID + sequence number
4. Sequence number and PID both stored in the log

46. 52
The idempotent producer

47. 53
The idempotent producer

48. 54
The idempotent producer

49. 55
The idempotent producer

50. 56
The idempotent producer

51. 57
The idempotent producer

52. 58
The idempotent producer

53. 59
The idempotent producer

54. 60
Idempotent Producer
• Works transparently – no API changes.
• Fast enough you don’t need to worry about it
• Will be on by default in the future

55. 61
Solving Problem #2:
Avoiding Duplicate Processing with Transactions

56. 62

57. 63
It’s More Complex Than I’ve Let On
• Multiple partitions
• Multiple input streams
• Non-determinism
• Diverse data stores
• Zombies

58. 64
Transactions in Kafka

59. 65
Introducing transactions
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(record0);
producer.send(record1);
producer.commitTransaction();
} catch (KafkaException e) {
producer.abortTransaction();
}

60. 66
Introducing ‘transactions’

61. 67
Initializing ‘transactions’

62. 68
Transactional sends – part 1

63. 69
Transactional sends – part 2

64. 70
Commit – phase 1

65. 71
Commit – phase 2

66. 72
Commit – phase 2

67. 73
Success!

68. 74
Consumer returns only committed messages

69. 75
Transactions => Stream Processing

70. 76
Factor problem into two parts
1. Transforming input streams to output streams (Streams)
2. Connecting output streams to data systems (Connect)

71. 77
Stream processing with Kafka
1. Read from input streams
2. Process and update state
3. Produce to output streams
4. Save offsets

72. 78
Stream processing as a sequence of transactions
BEGIN
1. Read from input streams
2. Process and update state
3. Produce to output streams
4. Save Offsets
COMMIT

73. 79
The Theory
• Two Generals
• Atomic Broadcast
• Consensus

74. 80
In Practice

75. 81
Performance
• Up to +20% producer throughput
• Up to +50% consumer throughput
• Up to -20% disk utilization
• Savings start when you batch
• Details: https://bit.ly/kafka-eos-perf

76. 82
Cool!
But how do I use this?

77. 83
Producer Configs
• enable.idempotence = true
• acks = “all”
• retries > 1 (preferably MAX_INT)
• transactional.id = ‘some unique id’

78. 84
Consumer configs
• isolation.level:
• “read_committed”, or
• “read_uncommitted”

79. 85
Streams config
• processing.mode = “exactly_once”

80. 86
Confluent
• Founded by the original creators of Apache Kafka
• Headquarters based in Palo Alto, CA
KSQL: Streaming SQL for Apache Kafka
Developer Preview
(https://github.com/confluentinc/ksql)

81. 87
Thank You!