Rust with-kafka-07-02-2019

•

0 likes•988 views

Gerard Klijs

Short presentation about using rust with kafka.

Software

www.openweb.nl
Kafka + Schema Registry
in Rust
By Gerard Klijs
Developer at Openweb
gerard@openweb.nl

www.openweb.nl
Summary
▰ About me
▰ Short intro to Kafka
▰ Short intro to the Confluent Schema Registry
▰ Rust support for Kafka
▰ Using rdkafka
▰ Turning the Schema Registry code into a Crate
222

www.openweb.nl
About me
▰ About 7 years working experience as developer,
mainly java, mainly web.
▰ Some pet projects in Clojure(script)
▰ Became especially interested in rust after
reading Rust for Clojurists
▰ Less than a year since I worked through the
book, no prior C or C++ experience.
33

www.openweb.nl
Short introduction to Apache Kafka
▰ A distributed streaming platform.
▰ Developed by LinkedIn, Open sourced in 2011.
▰ Main parts of the platform run on the JVM.
▰ Librdkafka is a C/C++ library implementing both
the producer and the consumer.
▰ Very scalable.
▰ Large ecosystem.
44

www.openweb.nl
Short introduction to Schema Registry
▰ Part of the Confluent Platform, from the
company created in 2014 by the creators.
▰ Uses Apache Avro for data serialization.
▰ Schema’s are stored in Kafka, each message
contains a key.
▰ The Schema Registry itself is open source.
▰ Versions the data and at the same time makes
the messages smaller.
55

www.openweb.nl
Rust support for Kafka
▰ rdkafka: Rust wrapper for librdkafka.
▰ kafka: Rust client for Apache Kafka.
▰ No support confluent platform, but several
crates working with Apache Avro.
66

www.openweb.nl
Using rdkafka
▰ Started with the provided examples from
rdkafka and get it to work.
▰ Tried several avro and serde libraries till I found
a match that worked.
77

www.openweb.nl
Turning the Schema Registry code into
a Crate
▰ Frame part which is generic and missing in
current crates.
▰ Create new project and pasted in code.
▰ Add unit tests and documentation.
▰ Some CI, code coverage and tags.
▰ Make it more generic.
▰ Improve error handling.
▰ Use as lib in originating project.
▰ Publish as schema_registry_converter to
crates.io.
88

What's hot

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...confluent

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia

Developing Secure Scala Applications With Fortify For ScalaLightbend

End to end testing a web application with ClojureGerard Klijs

Event machinealmeidaricardo

RESTEasy Reactive: Why should you care? | DevNation Tech TalkRed Hat Developers

Let the alpakka pull your streamEnno Runne

Integrating microservices with apache camel on kubernetesClaus Ibsen

Ractor's speed is not light-speedSATOSHI TAGOMORI

How to manage large amounts of data with akka streamsIgor Mielientiev

Stateful stream processing with kafka and samzaGeorge Li

Introduction Apache KafkaJoe Stein

Reactive database access with Slick3takezoe

Lessons from managing a Pulsar cluster (Nutanix)StreamNative

Samza at LinkedIn: Taking Stream Processing to the Next LevelMartin Kleppmann

Kafka At Scale in the Cloudconfluent

Apache pulsarKotireddy Sareddy

Apache samzaHumberto Streb

pgWALSyncRumman Iftekhar

Docker and FluentdN Masahiro

What's hot (20)

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...

Developing Secure Scala Applications With Fortify For Scala

End to end testing a web application with Clojure

Event machine

RESTEasy Reactive: Why should you care? | DevNation Tech Talk

Let the alpakka pull your stream

Integrating microservices with apache camel on kubernetes

Ractor's speed is not light-speed

How to manage large amounts of data with akka streams

Stateful stream processing with kafka and samza

Introduction Apache Kafka

Reactive database access with Slick3

Lessons from managing a Pulsar cluster (Nutanix)

Samza at LinkedIn: Taking Stream Processing to the Next Level

Kafka At Scale in the Cloud

Apache pulsar

Apache samza

pgWALSync

Docker and Fluentd

Similar to Rust with-kafka-07-02-2019

Data Engineer’s Lunch #45: Apache LivyAnant Corporation

Kafka ExplainatonNguyenChiHoangMinh

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)My own sweet home!

Introduction to ScalaMohammad Hossein Rimaz

.NET RDF APIsAndrei Iacob

Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent

Apache Kafka 0.8 basic training - VerisignMichael Noll

Polyglot Plugin ProgrammingAtlassian

What's New in AWS Serverless and ContainersAmazon Web Services

Lecture 2_ Intro to laravel.pptxSaziaRahman

7 Apache Process Cloudstack Developer DayKimihiko Kitase

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre ZembStreamNative

Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll

Graph ql subscriptions through the looking glassGerard Klijs

Kafka for DBAsGwen (Chen) Shapira

Apache kafkaKumar Shivam

From java to scala at crowd mixStefano Galarraga

Why scala for data scienceGuglielmo Iozzia

A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende

Similar to Rust with-kafka-07-02-2019 (20)

Data Engineer’s Lunch #45: Apache Livy

Kafka Explainaton

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)

Introduction to Scala

.NET RDF APIs

Kafka Summit SF 2017 - Kafka and the Polyglot Programmer

Apache Kafka 0.8 basic training - Verisign

Polyglot Plugin Programming

What's New in AWS Serverless and Containers

Lecture 2_ Intro to laravel.pptx

7 Apache Process Cloudstack Developer Day

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Graph ql subscriptions through the looking glass

Kafka for DBAs

Apache kafka

From java to scala at crowd mix

Why scala for data science

A Jupyter kernel for Scala and Apache Spark.pdf

Recently uploaded

Unlocking the Future of AI Agents with Large Language Modelsaagamshah0812

Exploring iOS App Development: Simplifying the ProcessEvangelist Apps https://twitter.com/EvangelistSW/

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed DataAlluxio, Inc.

The Ultimate Test Automation Guide_ Best Practices and Tips.pdfkalichargn70th171

5 Signs You Need a Fashion PLM Software.pdfWave PLM

Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH

The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdfkalichargn70th171

Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...stazi3110

DNT_Corporate presentation know about usDynamic Netsoft

BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEOrtus Solutions, Corp

TECUNIQUE: Success Stories: IT Service providermohitmore19

Adobe Marketo Engage Deep Dives: Using Webhooks to Transfer DataBradBedford3

Asset Management Software - InfographicHr365.us smith

Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideChristina Lin

Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsAlberto González Trastoy

(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...gurkirankumar98700

The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...ICS

Optimizing AI for immediate response in Smart CCTVshikhaohhpro

chapter--4-software-project-planning.pptkotipi9215

why an Opensea Clone Script might be your perfect match.pdfjoe51371421

Recently uploaded (20)

Unlocking the Future of AI Agents with Large Language Models

Exploring iOS App Development: Simplifying the Process

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

The Ultimate Test Automation Guide_ Best Practices and Tips.pdf

5 Signs You Need a Fashion PLM Software.pdf

Der Spagat zwischen BIAS und FAIRNESS (2024)

The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf

Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...

DNT_Corporate presentation know about us

BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE

TECUNIQUE: Success Stories: IT Service provider

Adobe Marketo Engage Deep Dives: Using Webhooks to Transfer Data

Asset Management Software - Infographic

Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide

Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...

The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...

Optimizing AI for immediate response in Smart CCTV

chapter--4-software-project-planning.ppt

why an Opensea Clone Script might be your perfect match.pdf

Rust with-kafka-07-02-2019

1. www.openweb.nl Kafka + Schema Registry in Rust By Gerard Klijs Developer at Openweb gerard@openweb.nl

2. www.openweb.nl Summary ▰ About me ▰ Short intro to Kafka ▰ Short intro to the Confluent Schema Registry ▰ Rust support for Kafka ▰ Using rdkafka ▰ Turning the Schema Registry code into a Crate 222

3. www.openweb.nl About me ▰ About 7 years working experience as developer, mainly java, mainly web. ▰ Some pet projects in Clojure(script) ▰ Became especially interested in rust after reading Rust for Clojurists ▰ Less than a year since I worked through the book, no prior C or C++ experience. 33

4. www.openweb.nl Short introduction to Apache Kafka ▰ A distributed streaming platform. ▰ Developed by LinkedIn, Open sourced in 2011. ▰ Main parts of the platform run on the JVM. ▰ Librdkafka is a C/C++ library implementing both the producer and the consumer. ▰ Very scalable. ▰ Large ecosystem. 44

5. www.openweb.nl Short introduction to Schema Registry ▰ Part of the Confluent Platform, from the company created in 2014 by the creators. ▰ Uses Apache Avro for data serialization. ▰ Schema’s are stored in Kafka, each message contains a key. ▰ The Schema Registry itself is open source. ▰ Versions the data and at the same time makes the messages smaller. 55

6. www.openweb.nl Rust support for Kafka ▰ rdkafka: Rust wrapper for librdkafka. ▰ kafka: Rust client for Apache Kafka. ▰ No support confluent platform, but several crates working with Apache Avro. 66

7. www.openweb.nl Using rdkafka ▰ Started with the provided examples from rdkafka and get it to work. ▰ Tried several avro and serde libraries till I found a match that worked. 77

8. www.openweb.nl Turning the Schema Registry code into a Crate ▰ Frame part which is generic and missing in current crates. ▰ Create new project and pasted in code. ▰ Add unit tests and documentation. ▰ Some CI, code coverage and tags. ▰ Make it more generic. ▰ Improve error handling. ▰ Use as lib in originating project. ▰ Publish as schema_registry_converter to crates.io. 88

9. www.openweb.nl Questions?

Rust with-kafka-07-02-2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Rust with-kafka-07-02-2019

Similar to Rust with-kafka-07-02-2019 (20)

Recently uploaded

Recently uploaded (20)

Rust with-kafka-07-02-2019