"Apache Kafka is the core of an amazing ecosystem of tools and frameworks that enable us to get more value from our data. Let's take a tour through this wonderful world and see what we can learn. In this session, we'll have a gentle introduction to Apache Kafka, and then a survey of some of the more popular components in the Kafka ecosystem. We'll look at the Kafka Producer and Consumer libraries, Kafka Connect, Kafka Streams, the Confluent Schema Registry, and more. You will leave loaded with ideas of ways to put Kafka to use in your organization, and a list of resources to help you on your journey."

1. Wonderful World
of
Apache Kafka
@daveklein
dave@tabular.io linkedin.com/in/daveklein19

2. dave@tabular.io
twitter.com/daveklein
linkedin.com/in/daveklein19
data-folks.masto.host/@daveklein
Dave Klein
Senior Developer Advocate at Tabular, an open
table store from the creators of Apache Iceberg
Co-author of Kafka in Action

3. http://tabular.io

4. What is Apache Kafka?

5. Kafka
Connect
Service
Kafka
Connect
Streams App
Service
ksqlDB
MongoDB
Kafka
Connect PostgreSQL
Kafka
Connect Elasticsearch
Event Streaming Platform

6. Noti
fi
cation State
{item:123, price: 29.95,
qty: 2}
{temp:29, unit: F,
time: 1606949836369}
Event
Order Placed
Temperature Read

7. "key": "a1",
"value": {
"eventType": “added-to-cart",
"title": "Kafka Streams in Action”,
"author": "Bill Bejeck",
"price": 44.99
}

8. 1 2 3 4 5 6 7 8 9 10 11
Topic (log)

9. 1 2 3 4 5 6 7 8 9 10 11
Topic Partitions
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
P1
P2
P3

10. Brokers and Replication
Broker 1 Broker 2 Broker 3

11. 1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
{100}
{200}
{300}
Topic
{101}
{103}
{201} {400}
{301} {401}

12. 1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Topic
a1:
{101}
a1:
{103}
b1:
{201}
d1:
{400}
c1:
{301}
d1:
{401}

13. 1 2 3 4 5 6 7 8 9 10 11
Producer

14. 1 2 3 4 5 6 7 8 9 10 11
Producer

15. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer

16. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer

17. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer

18. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
Committed Offset: 3

19. Consumer
1 2 3 4 5 6 7 8 9 10 11
Producer
Committed Offset: 3

20. Consumer
1 2 3 4 5 6 7 8 9 10 11
Producer
Committed Offset: 3

21. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
Committed Offset: 3

22. Producer<String, String> producer = new KafkaProducer<String, String>(props);
String key = "acme";
String value = "some info about acme, probably in JSON”;
ProducerRecord record = new ProducerRecord<String, String>(topic, key, value);
producer.send(record, new Callback(){
@Override
public void onCompletion(RecordMetadata m, Exception e) {
if (e != null) {
e.printStackTrace();
} else {
System.out.printf("Produced record to topic %s partition [%d]",
m.topic(), m.partition());
}
}
});
Sample Kafka Producer

23. Consumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList(topic));
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
String value = record.value();
System.out.printf("Consumed record - key: %s and value: %s", key, value);
}
}
} finally {
consumer.close();
}
Sample Kafka Consumer

24. Consumer Group A
AppInstance01
AppInstance02

25. AppInstance01
AppInstance02
Consumer Group A
AppInstance03
AppInstance04

26. Consumer Group A
AppInstance01
AppInstance02
Consumer Group B
OtherApp01
OthereApp02

27. Kafka Connect
Producer
JDBC

28. Kafka Connect
Consumer
JDBC
con
fl
uent.io/hub

29. Producer
Consumer
stream(input topic)
.
fi
lter()
.map()
.
fl
atMap()
.branch()
.reduce()
.groupByKey()
.peek()
.to(output topic)
Kafka Streams
Input Topic
Output Topic

30. public Topology buildTopology(Properties envProps) {
final StreamsBuilder builder = new StreamsBuilder();
final String movieTopic = envProps.getProperty("movie.topic.name");
final String rekeyedMovieTopic = envProps.getProperty("rekeyed.movie.topic.name");
final String ratingTopic = envProps.getProperty("rating.topic.name");
final String ratedMoviesTopic = envProps.getProperty("rated.movies.topic.name");
final MovieRatingJoiner joiner = new MovieRatingJoiner();
KStream<String, Movie> movieStream = builder.<String, Movie>stream(movieTopic)
.map((key, movie) -> new KeyValue<>(String.valueOf(movie.getId()), movie));
movieStream.to(rekeyedMovieTopic);
KTable<String, Movie> movies = builder.table(rekeyedMovieTopic);
KStream<String, Rating> ratings = builder.<String, Rating>stream(ratingTopic)
.map((key, rating) -> new KeyValue<>(String.valueOf(rating.getId()), rating));
KStream<String, RatedMovie> ratedMovie = ratings.join(movies, joiner);
ratedMovie.to(ratedMoviesTopic, Produced.with(Serdes.String(), ratedMovieAvroSerde(envProps)));
return builder.build();
}

31. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
Stream
Table
a1 {100}

32. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
Stream
Table
a1
b1
{100}
{200}

33. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
Table
a1
b1
c1
{100}
{200}
{300}

34. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
a1:
{101}
Table
a1
b1
c1
{101}
{200}
{300}

35. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
a1:
{101}
b1:
{201}
Table
a1
b1
c1
{101}
{201}
{300}

36. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
a1:
{101}
b1:
{201}
c1:
{301}
Table
a1
b1
c1
{101}
{201}
{301}

37. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
a1:
{101}
a1:
{103}
b1:
{201}
c1:
{301}
Table
a1
b1
c1
{103}
{201}
{301}

38. 1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Stream
a1:
{101}
a1:
{103}
b1:
{201}
d1:
{400}
c1:
{301}
Table
a1
b1
c1
d1
{103}
{201}
{301}
{400}

39. https:/
/www.con
fl
uent.io/blog/kafka-streams-tables-part-1-event-streaming

40. The Broader Kafka Ecosystem

42. Schema Registry
Producer
Consumer
1 2 3 4 5 6 7 8 9 10

43. docs.aiven.io/docs/products/kafka/karapace
docs.aws.amazon.com/glue/latest/dg/schema-registry.html
docs.con
fl
uent.io/platform/current/schema-registry

44. curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json"
-H "Accept: application/vnd.kafka.v2+json"
--data '{"records":[{"value":{"foo":"bar"}}]}'
"http://localhost:8082/topics/jsontest"
Con
fl
uent REST API

45. docs.aiven.io/docs/products/kafka/karapace
docs.conduktor.io/platform/con
fi
guration/api-overview
docs.con
fl
uent.io/platform/current/kafka-rest/api.html

46. Command Line Tools
Kafka Shell Scripts
- Start and stop brokers
- Create / manage topics
- Produce / consume events
Con
fl
uent CLI - https:/
/docs.con
fl
uent.io/con
fl
uent-cli/current/overview.html
- Start / stop services in Con
fl
uent Platform / Cloud
- Manage access control and secrets
kcat (formerly Kafkacat) - https:/
/github.com/edenhill/kcat
- Produce / consume events in style
kcctl (Casey Cuddle) - https:/
/github.com/kcctl/kcctl
- Command-line client for Kafka Connect

47. Learning Resources

48. Kafka
Connect
Service Streams App
Service
ksqlDB
Kafka
Connect MongoDB
Kafka
Connect PostgreSQL
Kafka
Connect Elasticsearch
Event Streaming Platform

50. https:/
/cn
fl
.io/meetup-hub
https:/
/cn
fl
.io/book-bundle
https:/
/cn
fl
.io/community
twitter.com/daveklein
dave@tabular.io
https:/
/aiven.io/developer/kafka
https:/
/www.conduktor.io/kafka
https:/
/developer.con
fl
uent.io

53. data-folks.masto.host/@daveklein
twitter.com/daveklein
dave@tabular.io
linkedin.com/in/daveklein19
Thank you