With Apache Kafka, it's typical to place different events in their own topic. But different event types can be related. Consider customer interactions with an online retailer. The customer searches through the site and clicks on various items before deciding on a final purchase. But businesses can gain insight by processing these events in sequence. Using the event type per topic leaves a lot of work for developers. Is there a better way? Fortunately, there is. Schema Registry now supports having multiple event types in the same topic. By placing various event types in a single topic, you can now handle different related events in-order. In this presentation, I'll introduce Schema Registry then we'll dive into how it handles multiple event types in a single topic, including examples. You will learn how and when to apply the multiple event types per topic pattern. Additionally, you'll learn how schema references work in Schema Registry.

1. Multiple Events in a single
topic
Bill Bejeck, Integration Architect

2. @bbejeck
Nice to meet you!
• Integration Architect on DevX team
• Prior to DevX ~3 years as engineer on Kafka Streams team
• Apache Kafka® Committer
• Author of “Kafka Streams in Action” - 2nd edition underway!
2

3. Event objects form a contract

4. @bbejeck
Kafka brokers work with bytes only
4

5. @bbejeck
Kafka brokers work with bytes only
5

6. @bbejeck
Potentially make unexpected changes
6

7. @bbejeck
Potentially make unexpected changes
7

8. @bbejeck
Use Schemas FTW!
8

9. @bbejeck
Use Schemas FTW!
9

10. @bbejeck
Use Schemas FTW!
10

11. @bbejeck
Registration details
11

12. @bbejeck
Registration details
12

13. @bbejeck
Registration details
13

14. @bbejeck
Registration details
14

15. @bbejeck
Registration details
15

16. Understanding subjects

17. @bbejeck
What is a Subject ?
• Defines a namespace for a schema
• Compatibility checks are per subject
• When a schema evolves, the subject remains, but the schema gets a new ID and
version
• Different approaches – subject name strategies
17

18. @bbejeck
Subjects - TopicNameStrategy
18

19. @bbejeck
Subjects - RecordNameStrategy
19

20. @bbejeck
Subjects - TopicRecordNameStrategy
20

21. Mapping events to topics

22. @bbejeck
Mapping events to topics
• Conventional wisdom – topic per distinct event
22
Purchase Customer Interaction

23. @bbejeck
Topic naming
23

24. @bbejeck
Topic naming
24

25. @bbejeck
Mapping events to topics
• For related events the order is important – one topic
• But we want to make sure we can limit the different record types
• What are our options?
25

26. Leverage schema references

27. @bbejeck
Schema references
• A field in a schema has an object represented by a registered schema
• Allows for schema re-use and modularity
• Enhances multi-event topics by enabling TopicNameStrategy (restricts the number of
types)
27

28. @bbejeck
Schema Refs
28

29. @bbejeck
Schema Refs
29

30. @bbejeck
Schema Refs
30

31. @bbejeck
Schema Refs
31

32. @bbejeck
Schema Refs
32

33. @bbejeck
Schema Refs redux - TopicNameStrategy
33

34. How to handle ?

35. @bbejeck
Avro (Consumer)
35
records = consumer.poll(Duration.ofSeconds(5));
records.forEach(record -> {
final SpecificRecord avroRecord = record.value();
if (avroRecord instanceof PageView) {
PageView pageView = (PageView) action;
.....
} else if (avroRecord instanceof Purchase) {
Purchase purchase = (Purchase) action;
.....
}
});

36. @bbejeck
Protobuf (Consumer)
36
records = protoConsumer.poll(Duration.ofSeconds(5));
records.forEach(record -> {
final CustomerEventProto.CustomerEvent customerEvent =
record.value();
ActionCase actionCase = customerEvent.getActionCase();
switch (actionCase) {
case PURCHASE -> ...;
case PAGE_VIEW -> ...;
case ACTION_NOT_SET -> ...;
}
});

37. @bbejeck
Protobuf (Consumer)
37
records = protoConsumer.poll(Duration.ofSeconds(5));
records.forEach(record -> {
final CustomerEventProto.CustomerEvent customerEvent =
record.value();
customerEvent = record.value();
if (customerEvent.hasPageView()) {
...
}
else if (customerEvent.hasPurchase()) {
...
}
});

38. @bbejeck
Kafka Streams
38
public CustomerInfo transform(String readOnlyKey,
SpecificRecord value){
CustomerInfo customerInfo = store.get(readOnlyKey);
if (customerInfo == null) {
customerInfo = CustomerInfo.newBuilder()
.setCustomerId(readOnlyKey)
.build();
}

39. @bbejeck
Kafka Streams
39
if(value instanceof PageView) {
PageView pageView = (PageView) value;
customerInfo.getPageViews().add(pageView.getUrl());
} else if (value instanceof Purchase) {
Purchase purchase = (Purchase) value;
customerInfo.getItems().add(purchase.getItem());
}
//Details left out

40. @bbejeck
Resources
• Martin Kleppmann - https://www.confluent.io/blog/put-several-event-types-kafka-topic/
• Robert Yokota - https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-
topic/
• Documentation - https://docs.confluent.io/platform/current/schema-
registry/index.html#sr-overview
• Examples - https://github.com/bbejeck/multiple-events-kafka-summit-europe-2021
40

41. Thank you!
@bbejeck
bill@confluent.io
cnfl.io/meetups cnfl.io/slack
cnfl.io/blog