Erik Tank gives a presentation on adding Avro to Kafka streams. He discusses Avro schemas, the Avro workflow with the Schema Registry, best practices for schema design and compatibility, and main takeaways around carefully choosing topic names and schemas and using .avdl files. The presentation provides an overview of using Avro with Kafka to enable serialization, evolution and flexibility for messaging needs.

1. Adding Avro to your Kafka
streams to meet your
messaging needs
https://github.com/skeletonkey/Avro
erik_tank

2. HELLO!
I am Erik Tank
Lead Engineer at Ticketmaster
You can find me at @erik_tank
erik_tank

3. I am Erik
HATEOAS
@erik_tank

4. Let’s Talk Schema
Schemata - Plural of schema
Schema
“representation of a plan or theory in the
form of an outline or model” (Oxford)
@erik_tank

5. Let’s Talk Schema
Pros
❑Contract
❏Clarity
❏Compatibility
❏Decoupling
@erik_tank

6. Let’s Talk Schema
Cons
❑No mechanism for evolution
❑No contract enforcement
Avro
Schem
a
Registry
@erik_tank

7. History of Avro
(Aircraft)
1956
Avro Vulcan B. 1
High-altitude strategic bomber
Operated by the Royal Air Force
(RAF) from 1956 until 1984
1963
Merged into Hawker
Siddeley Aviation
However, the Avro name has
been used for some aircraft
since then
1909
Roe I Triplane
First aircraft build by the Roes.
First flight on June 5th
1909
1910
Founded by Alliott &
Humphrey Verdon Roe
One of the world's first
aircraft builders.
@erik_tank

8. History of Avro (Aircraft)
2006
Hadoop
Search removed from Nutch and with
only the data processing/sharing parts
Hadoop is born.
2009
Apache Avro
A specification based design to
server the data exchange
needs to a world with multiple
*everything*.
2002
Nutch
Doug Cutting set out to
build an open source full
scale search engine
2003
Writable/SequenceFile
Improving the ability to
MapReduce and other
parallel data computations
@erik_tank

9. Avro
❑Language neutral serialization system
❑Rich data structure
❑Compact, fast, binary data format
❑Integration to dynamic languages
❑Code generation (Java)
❑RPC
❑Schema evolution
@erik_tank

10. Why Use
❑ Data compression
❑ Language support
❑ Schema evolution
❑ Flexibility
@erik_tank

11. Why Not
❑ Initial Lift
❑ Binary format
❑ Not turn key in every language
❑ Prototyping
❑ You’re afraid?
@erik_tank

12. Avro Workflow
Source: https://docs.confluent.io/current/schema-registry/index.html
@erik_tank

13. Avro Workflow
Source: https://docs.confluent.io/current/schema-registry/index.html (modified)
server client
@erik_tank

14. Short Kafka Detour
❑Topic contains messages
❑message is a key-value pair
❑subject – Schema Registry
topic_name-key
topic_name-value
@erik_tank

15. Picking a Topic Name
❑Avoid names that change over time
❑Settle on a template for your topics
❑ <message type>.<dataset name>.<data name>
❑Topic should reflect its purpose
Source: https://riccomini.name/how-paint-bike-shed-kafka-topic-naming-conventions
@erik_tank

16. Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

17. Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

18. Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

19. Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

20. Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

21. Best Practices (Do’s)
❑Default Value
❑Document!!!
❑Carefully consider topic name
❑Carefully consider changes to schema
❑Deal with Avro as a DB connection
@erik_tank

22. Best Practices (Dont’s)
❑Change the purpose of the topic
❑Change field types
❑Delete or Rename fields
@erik_tank

23. What’s in an extension?
❑.avsc – JSON schema file
❑.avdl – IDL file
❑.avpr – JSON protocol file
@erik_tank

24. .avsc{
"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [
{
"name": ”channel",
"type": {
"type": "enum",
"name": "Channel",
"symbols": [ "INTERNET", "IVR", "PHONE" ],
"default": "INTERNET",
"doc": "Category that the client is allows to transact on"
}
},
{
"name": "Market",
"type": {
"type": "enum",
"name": "Market",
"symbols": [ "UNITEDSTATES", "AUSTRALIA", "CANADA", "GERMANY", "GREATBRITAIN", "IRELAND", "MEXICO", "NEWZEALAND", "NORTHIRELAND", "SPAIN" ],
"default": "UNITEDSTATES",
"doc": "Market which client is allowed to operate in"
}
},
{
"name": "feature_set",
"type": [ "null", "string" ],
"default": null,
"doc": "The human readable client name/id"
},
{
"name": "RuleSet",
"type": {
"name": "RuleSet",
"type": "record",
"fields": [
{
"name": "enable_insurance",
"type": "boolean",
"default": false,
"doc": "Flag indicating if client is allowed to offer insurance"
},
{
"name": "require_delivery_method_to_complete_order",
"type": "boolean",
"default": false,
"doc": "Flag indicating if client is required to provided a delivery method to complete an order"
},
{
"name": "shopping_cart_ttl",
"type": "int",
"default": 600,
"doc": "Max time that a cart is allowed to exists."
},
{
"name": "configurable_customer_info_obj_fields",
"type": ["null", "string"],
"default": null,
"doc": "What customer information fields are required for customer data to be considered 'compelte'"
}
]
}
}
]
}
Full Specs: https://avro.apache.org/docs/1.9.0/spec.html
{
"name": ”channel",
"type": {
"type": "enum",
"name": "Channel",
"symbols": [ "INTERNET", "IVR", "PHONE" ],
“default": "INTERNET",
"doc": "Category that client can transact on”
}
}
{
"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [
{
{
"name": "feature_set",
"type": [ "null", "string" ],
"default": null,
"doc": "The human readable client name/id”
},
{
"name": "enable_insurance",
"type": "boolean",
"default": false,
"doc": "Flag indicating if …"
},
@erik_tank

25. .avsc vs .avdl
@namespace("com.jundy.client")
protocol Rules {
enum Channel { "INTERNET", "IVR", "PHONE" };
record Rules {
/**
Category that the client is allows to transact
*/
Channel channel = “INTERNET”;
/**
The human readable client name/id
*/
union { null, string } featureSet = null;
}
}
{"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [{
"name": "channel",
"type": { "type": "enum",
"name": "Channel",
"symbols": [ "INTERNET", "IVR", "PHONE" ],
"default": "INTERNET",
"doc": "Category that the client is allows to transact on"
}},{"name": "feature_set",
"type": [ "null", "string" ],
"default": null,
"doc": "The human readable client name/id"
}]}
"name": "channel",
"type": { "type": "enum",
"name": "Channel",
"symbols": [ "INTERNET", "IVR", "PHONE" ],
"default": "INTERNET",
"doc": "Category that the …"
enum Channel { "INTERNET", "IVR", "PHONE" };
record Rules {
/**
Category that the client is allows to transact
*/
Channel channel = “INTERNET”;
"name": "feature_set",
"type": [ "null", "string" ],
"default": null,
"doc": "The human readable client name/id"
/**
The human readable client name/id
*/
union { null, string } featureSet = null;

26. Main Takeaways
❑Carefully consider topic name
❑Schema changes with caution
❑Use .avdl
❑Prototypes end up in production!!!!
@erik_tank

27. THANKS!Any questions?
You can find me at:
❑@erik-tank
You can find this talk at:
❑https://github.com/skeletonkey/Avro

28. THANKS!Any questions?
You can find me at:
❑@erik-tank
You can find this talk at:
❑https://github.com/skeletonkey/Avro

29. THANKS!Any questions?
You can find me at:
❑@erik-tank
You can find this talk at:
❑https://github.com/skeletonkey/Avro