This document discusses Kafka and the Confluent Schema Registry in Rust. It begins with an introduction to the author and their experience. It then provides short introductions to Apache Kafka as a distributed streaming platform and the Confluent Schema Registry which uses Apache Avro for data serialization and stores schemas in Kafka. The document discusses existing Rust support for Kafka through libraries like rdkafka and challenges in using the Schema Registry in Rust. It outlines the author's process of taking the Schema Registry code and turning it into a Rust crate called schema_registry_converter to improve support for the Schema Registry in Rust projects.

1. www.openweb.nl
Kafka + Schema Registry
in Rust
By Gerard Klijs
Developer at Openweb
gerard@openweb.nl

2. www.openweb.nl
Summary
▰ About me
▰ Short intro to Kafka
▰ Short intro to the Confluent Schema Registry
▰ Rust support for Kafka
▰ Using rdkafka
▰ Turning the Schema Registry code into a Crate
222

3. www.openweb.nl
About me
▰ About 7 years working experience as developer,
mainly java, mainly web.
▰ Some pet projects in Clojure(script)
▰ Became especially interested in rust after
reading Rust for Clojurists
▰ Less than a year since I worked through the
book, no prior C or C++ experience.
33

4. www.openweb.nl
Short introduction to Apache Kafka
▰ A distributed streaming platform.
▰ Developed by LinkedIn, Open sourced in 2011.
▰ Main parts of the platform run on the JVM.
▰ Librdkafka is a C/C++ library implementing both
the producer and the consumer.
▰ Very scalable.
▰ Large ecosystem.
44

5. www.openweb.nl
Short introduction to Schema Registry
▰ Part of the Confluent Platform, from the
company created in 2014 by the creators.
▰ Uses Apache Avro for data serialization.
▰ Schema’s are stored in Kafka, each message
contains a key.
▰ The Schema Registry itself is open source.
▰ Versions the data and at the same time makes
the messages smaller.
55

6. www.openweb.nl
Rust support for Kafka
▰ rdkafka: Rust wrapper for librdkafka.
▰ kafka: Rust client for Apache Kafka.
▰ No support confluent platform, but several
crates working with Apache Avro.
66

7. www.openweb.nl
Using rdkafka
▰ Started with the provided examples from
rdkafka and get it to work.
▰ Tried several avro and serde libraries till I found
a match that worked.
77

8. www.openweb.nl
Turning the Schema Registry code into
a Crate
▰ Frame part which is generic and missing in
current crates.
▰ Create new project and pasted in code.
▰ Add unit tests and documentation.
▰ Some CI, code coverage and tags.
▰ Make it more generic.
▰ Improve error handling.
▰ Use as lib in originating project.
▰ Publish as schema_registry_converter to
crates.io.
88

9. www.openweb.nl
Questions?