Presentation by Guozhang Wang, Confluent at the Bay Area Apache Kafka July Meetup

1. 1
Exactly-once Data Processing
with Kafka Streams
Guozhang Wang
Kafka Meetup SF, July 27, 2017

2. 2
Outline
• What is exactly-once for stream processing?
• How to achieve exactly-once with Kafka?
• Kafka Streams: exactly-once made easy

3. 3

4. 4
Stream Processing
Source SinkProcess
State
Source Sink

5. 5
Stream Processing with Kafka
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D

6. 6
Stream Processing with Kafka
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

7. 7
Exactly-Once
• An application property for stream processing,
• .. that for each received record,
• .. it will be processed exactly once,
• .. even under failures

8. 8
Stream Processing with Kafka
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

9. 9
Error Scenario #1: Duplicate Write
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack

10. 10
Error Scenario #1: Duplicate Write
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack

11. 11
Error Scenario #2: Re-process
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
commit
ack
ack

12. 12
Error Scenario #2: Re-process
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D

13. 13
Error Scenario #2: Re-process
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D

14. 14
Error Scenario #3: Data loss
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

15. 15
Error Scenario #3: Data loss
State
Process
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

16. 16
Error Scenario #3: Data loss
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack

17. 17
Error Scenario #3: Data loss
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack

18. 18
Exactly-Once does NOT mean..
• Two Generals problem can now be solved
• .. or FLP result is proved wrong
• .. or TCP at transport level is “perfect”
• .. or you can get distributed consensus in any settings

19. 19
What can cause incorrect results?
• Unbounded network partition (algorithmical proof)
• A long GC or hard crash
• A bad config in your system
• A human operating error
• A bug in your code

20. 20
What can cause incorrect results?
• Unbounded network partition (algorithmical proof)
• A long GC or hard crash
• A bad config in your system
• A human operating error
• A bug in your code
99.9%
0.01%

21. 21
What can cause incorrect results?
• Unbounded network partition (algorithmical proof)
• A long GC or hard crash
• A bad config in your system
• A human operating error
• A bug in your code
99.9%
0.01%
Can we do better for the 99.99% ?

22. 22
So how to achieve Exactly-Once?

23. 23
Option #1: “Just give up”
Streaming
Source Sink
Batch
State
State

24. 24
Option #2: At-least-once + Dedup
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

25. 25
Option #2: At-least-once + Dedup
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D

26. 26
Option #2: At-least-once + Dedup
Process
State
KafkaTopic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit

27. 27
Option #2: At-least-once + Dedup
2
2
3
3
4
4
Dedup

28. 28
Option #3: The Kafka Way!(0.11+)
• Idempotent producer: send exactly-once per partition
• Transactional messaging: multiple-sends atomically

29. 29
Idempotent Producer
Producer
Kafka Topic Cack
pid = 1
pid = 1
seq = 28
pid = 1
seq = 28

30. 30
Idempotent Producer
Producer
Kafka Topic Cack
pid = 1
pid = 1
seq = 28
pid = 1
seq = 28
config: enable.idempotence = true

31. 31
Atomic Multi-Sends (aka. “transactions”)
Producer
Kafka Topic C
Kafka Topic D
producer.beginTxn();
producer.send(rec1); // topic C
producer.send(rec2); // topic D
producer.sendOffsetsToTxn(A, 10);
KafkaTopic A
producer.commitTxn();
try {
} catch (KafkaException e) {
}
Atomic
Commit

32. 32
Atomic Multi-Sends (aka. “transactions”)
Producer
Kafka Topic C
Kafka Topic D
producer.beginTxn();
producer.send(rec1); // topic C
producer.send(rec2); // topic D
producer.sendOffsetsToTxn(A, 10);
KafkaTopic A
producer.commitTxn();
try {
} catch (KafkaException e) {
}
Atomic
Commit
producer.abortTxn();

33. 33
Atomic Multi-Sends (aka. “transactions”)
Consumer
Kafka Topic C
Kafka Topic D
Read
Committed
consumer.subscribe(C, D);
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
}
config: isolation.level = read_committed (default = read_uncommitted)

34. 34
Exactly-Once Processing with Kafka
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
KafkaTopic A
Kafka Topic B
commit

35. 35
Exactly-Once Processing with Kafka
• Offset commit for source topics
• Value update on processor state
• Acked produce to sink topics
All or Nothing

36. 36
Kafka Streams (0.10+)
• New client library besides producer and consumer
• Powerful yet easy-to-use
• Event-at-a-time, Stateful
• Windowing with out-of-order handling
• Highly scalable, distributed, fault tolerant
• and more..

37. 37
Anywhere, anytime
Ok. Ok. Ok. Ok.

38. 38
Anywhere, anytime
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.0.0</version>
</dependency>

39. 39
Kafka Streams DSL
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

40. 40
Kafka Streams DSL
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

41. 41
Kafka Streams DSL
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

42. 42
Kafka Streams DSL
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

43. 43
Kafka Streams DSL
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

44. 44
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);

45. 45
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);

46. 46
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);

47. 47
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);

48. 48
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);

49. 49
Processor Topology
KStream<..> stream1 = builder.stream(”topic3”);
KStream<..> stream2 = builder.stream(”topic3”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.aggregateByKey(...);
aggregated.to(”topic3”);
State

50. 50
Processing in Kafka Streams
Kafka Topic B Kafka Topic A

51. 51
Processing in Kafka Streams
Kafka Topic B Kafka Topic A
Processor Topology
P1
P2
P1
P2

52. 52
Processing in Kafka Streams
Kafka Topic AKafka Topic B

53. 53
Processing in Kafka Streams
Kafka Topic B Kafka Topic A
MyApp.1 MyApp.2
Task2Task1

54. 54
States in Stream Processing
MyApp.2MyApp.1
Kafka Topic B
Task2Task1
Kafka Topic A
State State

55. 55
Fault Tolerance in Streams
StateProcess
StateProcess
StateProcess
Kafka
Kafka Streams
Kafka
Kafka Changelog

56. 56
• All or Nothing for the following:
• Offset commit for source topics
• Value update on processor state
• Acked produce to sink topics

57. 57
Exactly-Once with Kafka Streams (0.11+)
• Process data in transactions of:
• A batch of input records from source topics
• A batch of output records to changelog topics
• A batch of output records to sink topics
config: processing.mode = exactly-once (default = at-least-once)

58. 58
Exactly-Once with Failures
State
Process
StateProcess
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

59. 59
Exactly-Once with Failures
State
Process
StateProcess
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

60. 60
Exactly-Once with Failures
StateProcess
StateProcess
StateProcess
Kafka Streams
Kafka
Kafka Changelog
Kafka

61. 61
Exactly-Once with Failures
StateProcess
StateProcess
StateProcess
Kafka Streams
Kafka
Kafka Changelog
Kafka

62. 62
Exactly-Once life is goooood~

63. 63
What if not all my data is in Kafka?

64. 64

65. 65
Connectors
• 40+ since first release this
Feb (0.9+)
• 13 from &
partners

66. 66
End-to-End Exactly-Once

67. 67
Take-aways
• Exactly-once: important property for stream processing
• Kafka Streams: exactly-once made easy
Join Kafka Summit 2017 SF (discount code available!)
Additional resources:
http://www.confluent.io/resources
Guozhang Wang | guozhang@confluent.io | @guozhangwang

68. 68
Thank You!
Guozhang Wang
Kafka Meetup SF, July 27, 2017