"Apache Flink and Kafka Streams are the dominant stream processing technologies available today. But which one do you choose to build an event streaming application? Both Kafka Streams and Flink offer similar features despite being distinctly different distributed processing frameworks. So how does one go about selecting the correct one? In this talk, attendees will learn the information needed to match their event streaming requirements and objectives with the correct streaming framework. Specifically, I'll cover the following topics: 1. Architecture 2. Deploying applications 3. Work/Task Assignment and Distribution 4. API comparison 5. Stateful operations and state durability You'll leave with the knowledge of both Kafka Streams and Flink's strengths and weaknesses and enough information to determine which framework best suits your event streaming needs."

1. Comparing Kafka Streams and Flink
Bill Bejeck, Staff DevX Engineer
Apache Kafka committer and PMC member
bill@confluent.io | @bbejeck

2. Agenda
2
@bbejeck
• Architecture Overview
• Deployment
• Task / Work Assignment
• API
• Time Semantics
• State / Fault Tolerance

3. Agenda
3
@bbejeck

4. Architecture Overview

5. Architecture Overview – Kafka Streams
5
@bbejeck

6. Architecture Overview – Kafka Streams
6
@bbejeck

7. Architecture Overview – Kafka Streams
7
@bbejeck

8. Architecture Overview - Flink
8
@bbejeck

9. Architecture Overview – Kafka Streams
9
@bbejeck

10. Deployment

11. 11
@bbejeck
Deployment – Flink

12. 12
@bbejeck
Deployment – Flink

13. 13
@bbejeck
Deployment – Flink

14. Deployment – Kafka Streams
14
@bbejeck

15. Deployment – Kafka Streams
15
@bbejeck

16. Task / Work Assignment

17. 17
@bbejeck
Shuffling/Repartitioning – Kafka Streams
//Details omitted for clarity
KStream<String, StockTrade> stream = builder.stream("stock-trade”)
stream.filter((key, trade) -> trade.numShares > 1000)
.groupBy((k, v) -> v.tickerSymbol)
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.aggregate(TradeAgg::new,
(symbol, trade, agg) -> agg.update(trade))
.toStream().map((wKey, value) -> KeyValue.pair(wKey.key(), value))
.to("trade-aggregates”);

18. 18
@bbejeck
Shuffling/Repartitioning – Kafka Streams
//Details omitted for clarity
KStream<String, StockTrade> stream = builder.stream("stock-trade”)
stream.filter((key, trade) -> trade.numShares > 1000)
.groupBy((k, v) -> v.tickerSymbol)
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.aggregate(TradeAgg::new,
(symbol, trade, agg) -> agg.update(trade))
.toStream().map((wKey, value) -> KeyValue.pair(wKey.key(), value))
.to("trade-aggregates”);

19. 19
@bbejeck
Task Distribution - Repartitioning – Kafka Streams

20. Task / Work Assignment – Kafka Streams
20
@bbejeck

21. Task / Work Assignment – Kafka Streams
21
@bbejeck

22. Task / Work Assignment – Kafka Streams
22
@bbejeck

23. Task Distribution & Assignment – Kafka Streams
23
@bbejeck

24. Task Distribution & Assignment – Kafka Streams
24
@bbejeck

25. Task Distribution & Assignment – Kafka Streams
25
@bbejeck

26. 26
@bbejeck
Task / Work Assignment - Kafka Streams
//Details omitted for clarity
KStream<String, StockTrade> stream = builder.stream("stock-trade”)
stream.filter((tickerSymbol, trade) -> trade.numShares > 1000)
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.aggregate(TradeAgg::new,
(symbol, trade, agg) -> agg.update(trade))
.toStream().map((wKey, value) -> KeyValue.pair(wKey.key(), value))
.to("trade-aggregates”);

27. 27
@bbejeck
Task / Work Assignment - Kafka Streams

28. 28
@bbejeck
Task Distribution & Assignment – Flink
// default parallelism set to 2 in configuration
KafkaSource<StockTrade> kafkaSource=KafkaSource.<StockTrade>builder()...build()
KafkaSink<StockTrade> kafkaSink=KafkaSink.<StockTrade>builder()...build()
DataStream<StockTrade> stream = env.fromSource(kafkaSource);
stream.filter(trade -> trade.numShares > 1000)
.keyBy(StockTrade::ticker)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.aggregate(new TradeAgg())
.map(...)
.sinkTo(kafkaSink);

29. 29
@bbejeck
Task Distribution & Assignment – Flink
// default parallelism set to 2 in configuration
KafkaSource<StockTrade> kafkaSource=KafkaSource.<StockTrade>builder()...build()
KafkaSink<StockTrade> kafkaSink=KafkaSink.<StockTrade>builder()...build()
DataStream<StockTrade> stream = env.fromSource(kafkaSource);
stream.filter(trade -> trade.numShares > 1000)
.keyBy(StockTrade::ticker)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.aggregate(new TradeAgg())
.map(...)
.sinkTo(kafkaSink);

30. 30
@bbejeck
Task Distribution & Assignment – Flink

31. 31
@bbejeck
Task Distribution & Assignment – Flink

32. 32
@bbejeck
Task Distribution & Assignment – Flink

33. 33
@bbejeck
Task Distribution & Assignment – Flink

34. 34
@bbejeck
Task Distribution & Assignment – Flink

35. 35
@bbejeck
Task Distribution & Assignment – Flink
• Flink doesn’t work in key-value pairs
• Keys are Virtual – defined by functions
• Individual keys are allocated into KeyGroups for distribution

36. 36
@bbejeck
Task Distribution & Assignment – Flink
Key Groups

37. 37
@bbejeck
Task Distribution & Assignment – Flink
Max Parallelism
• Determines the number of Key Groups per operator
• Limits the number of parallel tasks keyed state can scale to
• Default level for cluster, jobs can set individually

38. 38
@bbejeck
Task Distribution & Assignment – Flink
Max Parallelism

39. 39
@bbejeck
Task Distribution & Assignment - Kafka Streams
//Details omitted for clarity
KStream<String, StockTrade> stream = builder.stream("stock-trade”)
stream.filter((tickerSymbol, trade) -> trade.numShares > 1000)
.repartition(Repartitioned.numberOfPartitions(10))
.groupByKey()
.windowedBy(...)
.aggregate(TradeAgg::new,
(symbol, trade, agg) -> agg.update(trade))
.toStream().map((wKey, value) -> KeyValue.pair(wKey.key(), value))
.to("trade-aggregates”);

40. 40
@bbejeck
Task Distribution & Assignment – Flink
// default parallelism set to 2 in configuration
KafkaSource<StockTrade> kafkaSource=KafkaSource.<StockTrade>builder()...build()
KafkaSink<StockTrade> kafkaSink=KafkaSink.<StockTrade>builder()...build()
DataStream<StockTrade> stream = env.fromSource(kafkaSource);
stream.filter(trade -> trade.numShares > 1000)
.keyBy(StockTrade::ticker)
.window(TumblingEventTimeWindows.of(Time.minutes(1)))
.aggregate(new TradeAgg())
.setParallelism(4)
.map(...)
.sinkTo(kafkaSink);

41. API

42. 42
@bbejeck
API - Comparison
Kafka Streams
KStream<K, V> filter(final Predicate<? super K, ? super V> predicate)
Flink
SingleOutputStreamOperator<T> filter(FilterFunction<T> filter)

43. 43
@bbejeck
API - Comparison
SingleOutputStreamOperator<T> filter(FilterFunction<T> filter)
T = SomeObject {
String id, // Key still present encapsulated in T
Long value
OR
StockTrade trade
}

44. 44
@bbejeck
API – Flink
// default parallelism set to 2 in configuration
KafkaSource<StockTrade> kafkaStockTradeSource =
KafkaSource.<StockTrade>builder()
.setTopics("stock-trades")
.setBootstrapServers("bootstrap-servers")
.setProperties(properties)
.setStartingOffsets(OffsetsInitializer.earliest())
.setGroupId("flink-stock-trade")
.setDeserializer(new StockTradeDeserializationSchema())
.build();

45. 45
@bbejeck
API - Flink
StockTradeDeserializationSchema implements KafkaRecordDeserializationSchema<StockTrade> {
//Details omitted for clarity
@Override
public void deserialize(ConsumerRecord<byte[], byte[]> record,
Collector<StockTrade> out) throws IOException {
// deserialize key and value
// create object with key and value
out.collect(trade);
}
}

46. 46
@bbejeck
API - Flink
KafkaSink<TradeAgg> kafkaSink =
KafkaSink.<StockTrade>builder()
.setKafkaProducerConfig(properties)
.setBootstrapServers("boostrap-servers")
.setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
.setRecordSerializer(new StockTradeSerializationSchema())
.build();

47. 47
@bbejeck
API - Flink
//Details omitted for clarity
StockTradeSerializationSchema implements KafkaRecordSerializationSchema<TradeAgg> {
final String topic; // constructor param
public ProducerRecord<byte[], byte[]> serialize(TradeAgg aggregate,
KafkaSinkContext context,
Long timestamp) {
// serialize key and value from aggregate obj
return new ProducerRecord(...);
}
}

48. Time Semantics

49. 49
@bbejeck
Time Semantics – Stream Processing

50. 50
@bbejeck
Time Semantics – Kafka Streams

51. 51
@bbejeck
Time Semantics – Kafka Streams

52. 52
@bbejeck
Time Semantics – Kafka Streams

53. 53
@bbejeck
Time Semantics – Kafka Streams

54. 54
@bbejeck
Time Semantics – Flink

55. 55
@bbejeck
Time Semantics – Flink

56. State / Fault Tolerance

57. 57
@bbejeck
State / Fault Tolerance - Flink

58. 58
@bbejeck
State / Fault Tolerance - Flink

59. 59
@bbejeck
State / Fault Tolerance – Kafka Streams

60. Task Distribution & Assignment – Kafka Streams
60
@bbejeck

61. Task Distribution & Assignment – Kafka Streams
61
@bbejeck

62. 62
@bbejeck
State / Fault Tolerance – Kafka Streams

63. 63
@bbejeck
State / Fault Tolerance – Kafka Streams

64. 64
@bbejeck
Summary
• Kafka Streams offers flexible deployment
• High Availability for processing
• Standby Tasks mitigate time to restore stateful processing
• Flink highly optimized
• Good for large stateful operations
• Snapshots offer quick recovery for moderate state

65. @bbejeck
Resources
• Kafka Streams 101 - https://developer.confluent.io/learn-kafka/kafka-
streams/get-started/
• Apache Flink 101 - https://developer.confluent.io/courses/apache-flink/intro/
• Building Flink apps in Java - https://developer.confluent.io/courses/flink-
java/overview/
• Kafka Streams in Action - 2nd edition Summer/Fall!
• https://www.manning.com/books/kafka-streams-in-action-second-edition
65

66. Thank you!
@bbejeck
bill@confluent.io
cnfl.io/meetups cnfl.io/slack
cnfl.io/blog