🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

🎶Bo-Streamian Rhapsody🎶
A musical demonstration of
Kafka Connect and Kafka Streams

Chris Egerton, Aiven

High-level design
Chris Egerton, Aiven
�� (Loop) ��

Plan of attack
1. Play audio out of kafka
2. Record audio into Kafka
3. Looping logic
Chris Egerton, Aiven

● Continuously:
○ Read data from Kafka
○ Convert, process, and/or buffer data
○ Play audio
1. Playing audio from Kafka
Chr...

This sounds like
a job for…
Chris Egerton, Aiven

Kafka Connect
Chris Egerton, Aiven
● Distributed, horizontally-scalable,
fault-tolerant ingest/export tool for Kafka
● Dev...

Developing a Connector
● Two kinds of connector: source and sink
● Kafka Connect gives you a lot for free
○ For sinks, Kaf...

1. Playing audio from Kafka
Chris Egerton, Aiven
● Continuously:
○ Read data from Kafka
○ Convert, process, and/or buffer ...

Java Sound API
Chris Egerton, Aiven
● Kafka Connect is a Java framework
● We can use the javax.sound.sampled
package:
● Au...

Chris Egerton, Aiven
Wait…
…what are samples?

Sound
Chris Egerton, Aiven
👏 👂
Air particles vibrate, vibration spreads out, makes its way to our ears

Audio Sampling
Chris Egerton, Aiven
👏 🎙
● Microphones measure pressure on air particles around them
● Audio sample: measur...

Java Sound API
Chris Egerton, Aiven
● AudioFormat class: defines sample encoding
● SourceDataLine class: play audio
○ void...

Chris Egerton, Aiven
● Continuously:
○ Read data from Kafka
○ Convert, process, and/or buffer data
○ Play audio
✅
✅
1. Pla...

Developing a Connector
Chris Egerton, Aiven
● Have to implement two classes:
○ The Connector class performs config validat...

Connector API
Chris Egerton, Aiven
● ConfigDef config(): how is your connector configured?
● Config validate(Map<String, S...

Speakers sink connector
Chris Egerton, Aiven
public ConfigDef config() {
return new ConfigDef()
.define(
“sample.rate”, //...

Speakers sink connector
Chris Egerton, Aiven
public void start(Map<String, String> props) {
this.props = props;
}
public L...

SinkTask API
Chris Egerton, Aiven
● void start(Map<String, String> props): given connector-generated properties,
prepare t...

Sink Task Lifecycle (Simplified)
Chris Egerton, Aiven
task::start
Connect
runtime
polls Kafka
task::put
task::stop
Task th...

Speakers Task Data Flow (sync)
Chris Egerton, Aiven
task::put
SourceData
Line::
write
Task thread

Chris Egerton, Aiven
✅
● Continuously:
○ Read data from Kafka
○ Convert, process, and/or buffer data
○ Play audio
✅
✅
✅
1....

How about a demo?
Chris Egerton, Aiven
{
"connector.class": "SpeakersSinkConnector",
"name": "speakers-sink",
"tasks.max":...

Chris Egerton, Aiven
● Like playing audio from Kafka, but in reverse
● Continuously:
○ Record audio
○ Convert, process, an...

Chris Egerton, Aiven
This sounds like
another job for…
(Kafka Connect)

Developing a Connector
Chris Egerton, Aiven
● Two kinds of connector: source and sink
● Kafka Connect gives you a lot for ...

2. Recording audio into Kafka
Chris Egerton, Aiven
● Continuously:
○ Record audio
○ Convert, process, and/or buffer data
○...

Java Sound API
Chris Egerton, Aiven
● Uses same AudioFormat class to define sample encoding
● TargetDataLine class: record...

2. Recording audio into Kafka
Chris Egerton, Aiven
● Continuously:
○ Record audio
○ Convert, process, and/or buffer data
○...

Microphone source connector
Chris Egerton, Aiven
public ConfigDef config() {
return new ConfigDef()
.define(
“topic”, // P...

SourceTask API
Chris Egerton, Aiven
● void start(Map<String, String> props): same as SinkTask
● void stop(): same as SinkT...

Source Task Lifecycle (simplified)
Chris Egerton, Aiven
task::start task::poll
Connect
runtime
produces to
Kafka
task::sto...

Microphone Task Data Flow (async)
Chris Egerton, Aiven
TargetData
Line::read
audio read
thread
queue
Task::
poll
Audio rea...

Chris Egerton, Aiven
● Continuously:
○ Record audio
○ Convert, process, and/or buffer data
○ Send data to Kafka
✅
✅
✅
✅
2....

How about a demo?
Chris Egerton, Aiven
{
"connector.class": "MicrophoneSource",
"name": "microphone-source",
"tasks.max": ...

Chris Egerton, Aiven
● Read data from Kafka
● Start off by writing it unaltered to a new topic
● On first loop toggle, rec...

This sounds like
a job for…
Chris Egerton, Aiven

Kafka Streams
Chris Egerton, Aiven
● Client library for building applications and microservices, where the input and outpu...

Our Topology (high-level)
Chris Egerton, Aiven
sound-raw-
input topic
Loop pedal
(Kafka
Streams)
sound-
processed-
output ...

Our Topology (in detail)
Chris Egerton, Aiven
Kafka
topic
Decode
samples
from
bytes
Loop
samples
Encode
samples
as bytes
K...

Our Topology (in code)
Chris Egerton, Aiven
public void loop(StreamsBuilder builder) {
builder.stream(“sound-raw-input”) /...

Hardware
Chris Egerton, Aiven

How about a demo?
Chris Egerton, Aiven

Is that it?
Chris Egerton, Aiven
sound-raw-
input topic
Loop pedal
(Kafka
Streams)
sound-
processed-
output topic

What else can we do?
Chris Egerton, Aiven
● Reverb
○ Record samples
○ After a fixed delay, combine them with real-time sam...

Bo-Streamian Rhapsody
Chris Egerton, Aiven
Input
samples
Distort
(maybe)
Reverb
Output
samples
T
o
g
g
l
e

How about a demo?
Chris Egerton, Aiven
Input
samples
Distort
(maybe)
Reverb
Output
samples
T
o
g
g
l
e

Thank you!
Open Source Program Office @Aiven
chrise@aiven.io
Chris Egerton
/in/chris-egerton-93886148
@C0urante

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1039840 views
Storytelling For The Web: Integrate... by Chiara Aliotta 952098 views
Artificial Intelligence, Data and C... by OECD Directorate ... 876077 views
How to Leverage AI to Boost Employe... by SocialHRCamp 368477 views
2024 State of Marketing Report – by... by Marius Sescu 209319 views
Everything You Need To Know About C... by Expeed Software 229161 views

View on Slideshare

1 of 49 Ad

View on Slideshare

1 of 49 Ad