Events stored in Kafka are just bytes, this is one of the reasons Kafka is so flexible. But when developing a producer or consumer you want objects, not bytes. Documenting and defining events provides a common way to discuss and agree on an approach to using Kafka. It also informs developers how to consume events without needing access to the developers responsible for producing events. This talk will introduce the most popular formats for documenting events that flow through Kafka, such as AsyncAPI, Avro, CloudEvents, JSON schemas, and Protobuf. It will discuss the differences between the approaches and how to decide on the documentation strategy for you. Alongside the formats, this session will also look at the tooling available for the different approaches. Tools for testing and code generation can make a big difference to your day-to-day developer experience. If you aren't already documenting your events or want to see other approaches, then this is the talk for you.

1. From Bytes to Objects
Apache Kafka
Kate Stanley
Dale Lane
Describing your events

2. Acme Air
© 2021 IBM Corporation

3. Acme Air
© 2021 IBM Corporation

4. Acme Air
© 2021 IBM Corporation

5. Data is stored in bytes
© 2021 IBM Corporation
Producer Consumer
01 0F 11 01 01 0F 11 01

6. Data is stored in bytes
© 2021 IBM Corporation
Producer Consumer
String Serializer
Integer Serializer
String Deserializer
Integer Deserializer
01 0F 11 01 01 0F 11 01

7. Adding structure
with schemas

8. Adding structure with
Schemas
© 2021 IBM Corporation

9. Where to store the schema
© 2021 IBM Corporation
Producer Consumer
? ?

10. Where to store the schema
© 2021 IBM Corporation
Producer Consumer

11. Where to store the schema
© 2021 IBM Corporation
Producer Consumer

12. Where to store the schema
© 2021 IBM Corporation
Producer Consumer

13. Schema in the message
© 2021 IBM Corporation
Producer Consumer
{
“schema”: {
“properties”: {
“flight_number”: {“type”: “string”},
“passengers”: {“type”: ”number”},
…
},
“flight_number”: “1234”,
“passengers”: 45,
“aircraft”: ”Airbus”
}

14. Schema Registry

15. Schema Registry
© 2021 IBM Corporation
Producer Consumer
Schema registry

16. Schema Registry
© 2021 IBM Corporation
Producer Consumer
Schema registry

17. Schema formats

18. Schema Formats
JSON Schema
Google Protocol Buffer
Apache Avro
© 2021 IBM Corporation

19. JSON Schema
A vocabulary that allows you
to annotate and validate JSON documents
Describes your existing data format(s)
Provides clear human- and machine-
readable documentation
Code and schema generators available in
Java
https://json-schema.org
© 2021 IBM Corporation
{
”type”: “object”,
“properties”: {
“flight_number”: {“type”: “string”},
“passengers”: {“type”: “integer”},
“aircraft”: {
“type”: “string”,
”enum”: [“Airbus”, “Boeing”]
}
},
“required”: [“flight_number”, “aircraft”]
}

20. Protobuf
Mechanism for serializing structured data
Created by Google
Language neutral
Supports generated code in Java, Python,
Objective-C and C++
https://developers.google.com/protocol-buffers/
© 2021 IBM Corporation
message Flight {
required string flight_number = 1;
required int32 passengers = 2;
enum Aircraft {
AIRBUS = 0;
BOEING = 1;
}
optional Aircraft marshmallows = 4;
}
Flight flight = Flight.newBuilder()
.setFlightNumber(“1234”)
.setPassengers(45)
.setAircraft(“AIRBUS”)
.build();
.proto

21. Apache Avro
A data serialization system that provides:
Rich data structures
A compact, fast, binary data format
Simple integration with dynamic languages
Java, Python and C support
© 2021 IBM Corporation
.avsc
{
"type": "record",
"name": ”Flight",
"fields": [
{"name": ”passengers", "type": ”int"},
{"name": ”flight_number", "type": ”string", "logicalType": "uuid"},
{"name": ”aircraft", "type": "enum", "symbols" : [”Airbus", ”Boeing"]}
]
}

22. Apache Avro
A data serialization system that provides:
Rich data structures
A compact, fast, binary data format
Simple integration with dynamic languages
Java, Python and C support
© 2021 IBM Corporation
.avsc
{
"type": "record",
"name": ”Flight",
"fields": [
{"name": ”passengers", "type": ”int"},
{"name": ”flight_number", "type": ”string", "logicalType": "uuid"},
{"name": ”aircraft", "type": "enum", "symbols" : [”Airbus", ”Boeing"]}
]
}

23. Adding metadata

24. CloudEvents
“A specification for describing event data
in a common way”
© 2021 IBM Corporation

25. CloudEvents
“A specification for describing event data
in a common way”
© 2021 IBM Corporation
------------------ headers -------------------
ce_specversion: "1.0"
ce_type: "com.example.someevent"
ce_source: "/mycontext/subcontext"
ce_id: "1234-1234-1234"
ce_time: "2018-04-05T03:56:24Z"
content-type: application/avro
.... further attributes ...
------------------- value --------------------
... application data encoded in Avro ...
-----------------------------------------------
binary

26. CloudEvents
“A specification for describing event data
in a common way”
© 2021 IBM Corporation
------------------ headers -------------------
ce_specversion: "1.0"
ce_type: "com.example.someevent"
ce_source: "/mycontext/subcontext"
ce_id: "1234-1234-1234"
ce_time: "2018-04-05T03:56:24Z"
content-type: application/avro
.... further attributes ...
------------------- value --------------------
... application data encoded in Avro ...
-----------------------------------------------
-------------------------- headers ---------------------------
content-type: application/cloudevents+json; charset=UTF-8
--------------------------- value ----------------------------
{
"specversion" : "1.0",
"type" : "com.example.someevent",
"source" : "/mycontext/subcontext",
"id" : "1234-1234-1234",
"time" : "2018-04-05T03:56:24Z",
"datacontenttype" : "application/json",
... further attributes omitted ...
"data" : {
... application data encoded in JSON ...
}
}
---------------------------------------------------------------
binary structured

27. Example Schema
Registries

28. Example Schema Registries
© 2021 IBM Corporation
Apicurio Registry
Format options: Apache Avro, Google protocol buffers, JSON Schema
Storage options: Kafka, PostgreSQL
API support: Apicurio, IBM, Confluent
License: Apache 2.0
https://github.com/Apicurio/apicurio-registry
Confluent Schema Registry
Format options: Apache Avro, Google protocol buffers, JSON Schema
Storage options: Kafka
API support: Confluent
License: Confluent community license
https://docs.confluent.io/platform/current/schema-registry/index.html#

29. Leveraging the
events

30. Acme Air
© 2021 IBM Corporation

31. © 2021 IBM Corporation
</>
I have a new project for you…
…

32. © 2021 IBM Corporation
</>
Connect

33. © 2021 IBM Corporation
</>
Connect

34. © 2021 IBM Corporation
</>
We’ve already got a topic with
events from the flights!
Connect

35. © 2021 IBM Corporation
</>
We’ve already got a topic with
events from the flights!
Connect

36. © 2021 IBM Corporation
</>

37. © 2021 IBM Corporation
</>

38. © 2021 IBM Corporation
</>
?

39. © 2021 IBM Corporation
</>
How can I know what Kafka clusters and topics
are available?
When I find topics I want to use, how can I
understand the data that is on them?
How can I start using these topics as efficiently
as possible?
</>

40. AsyncAPI

41. AsyncAPI
© 2021 IBM Corporation

42. AsyncAPI
© 2021 IBM Corporation
info
channels
components
servers

43. Describe the overall spec
© 2021 IBM Corporation
info
channels
components
servers

44. Describe the overall spec
© 2021 IBM Corporation
info
channels
components
servers

45. Explain how people connect
to your Kafka cluster
© 2021 IBM Corporation
info
channels
components
servers

46. Explain how people connect
to your Kafka cluster
© 2021 IBM Corporation
info
channels
components
servers

47. Describe your topics and the
messages that are on them
© 2021 IBM Corporation
info
channels
components
servers

48. Describe your topics and the
messages that are on them
© 2021 IBM Corporation
info
channels
components
servers
Describe your topics and the
messages that are on them

49. © 2021 IBM Corporation
info
channels
components
servers
Reusable definitions

50. Reusable definitions
© 2021 IBM Corporation
info
channels
components
servers
Reusable definitions

51. Reusable definitions
© 2021 IBM Corporation
info
channels
components
servers
Reusable definitions

52. AsyncAPI
© 2021 IBM Corporation
info
channels
components
servers

53. Demos

54. Generating
documentation

56. Generating mock
data

58. Rapid prototyping

60. Generating code

62. Socializing Kafka
topic information

65. AsyncAPI
Discover topics available to use
Understand how to connect and consume
data
Improve developer efficiency
© 2021 IBM Corporation

66. Summary
Use schemas to get structure
Schema registry provides a centralized store
Documentation tools like AsyncAPI provide
additional metadata
© 2021 IBM Corporation

67. Thank you
Kate Stanley | @katestanley91
Dale Lane | @dalelane
© 2021 IBM Corporation
Kafka with IBM:
https://developer.ibm.com/components/kafka/
https://github.com/ibm-messaging/kafka-java-vertx-starter
http://ibm.biz/what-is-event-endpoint-management
https://community.ibm.com/community/user/integration/blogs/dale-lane1/2021/06/27/a-demo-of-
event-endpoint-management
Schema registries:
https://www.apicur.io/registry/
https://docs.confluent.io/platform/current/schema-registry/index.html
AsyncAPI:
https://www.asyncapi.com