Apache Flink’s mission is simple: compute over streams of data, this is why combining it with Apache Kafka makes a lot of sense. Kafka is the solid, reliable, scalable backbone and Flink is the engine on top! In 10 minutes you’ll learn all the basics of Flink over Kafka: starting by defining the types of connectors, we’ll explore how to work with various data formats, using pre-defined schemas when appropriate, and storing the pipeline output as standard or compacted topic when needed. Finally we’ll extend the picture, by bringing in additional sources of data, demonstrating how we can join various data sources in streaming mode. If you’re into streaming and want to understand how to define data pipelines using Flink, one of the fastest growing engine in the market, this session is for you.

1. Tips for Apache Flink on Kafka | 2022-04-26
Tips for Apache Flink
on Kafka

2. Flink connectors

3. Kafka Connectors

4. ● Kafka
● Kinesis
● JDBC
● Elasticsearch
● FileSystem (S3, GCS, etc)
● HBase
● Cassandra
● Firehose
● RabbitMQ
● Google Cloud Pub/Sub
● NiFi
● Pulsar
Flink connectors vs Kafka Connectors
● DataGen
● Print
● BlackHole
● Hybrid Source
● Http (unofficial)
● Couchbase
● Official MongoDB preview
● Debezium for MongoDB, MySQL,
PostgreSQL, SQL Server
● Google Cloud Pub/Sub
● JDBC
● Schema Source
● Stream Reactor Cassandra
● Stream Reactor MQTT
● Stream Reactor InfluxDB, Mongo DB,
Redis
● Google BigQuery/ Cloud Storage
● S3
● Elasticsearch
● HTTP preview
● OpenSearch
preview
● Snowflake
preview
● Splunk

5. If only your team uses the Source you can use Flink
If you share data with others, you can use Kafka
Flink + Kafka connectors

6. CSV Useful as a key format
JSON Universal format for various data sources
AVRO Binary compressed data format with embedded schema
Maxwell, OGG,
Canal
Formats over JSON for changelog data capture use cases
AVRO Confluent Support schema registry
AVRO Confluent
Debezium
For data loaded to Kafka using Debezium
Kafka connector for Flink
Supported Formats

7. ● Optimized for compacted topics
● Could improve performance, because transactions are not required
● Flink will guarantee the message ordering on the primary key
Upsert Kafka
Kafka connector for Flink

8. Flink jobs with various Kafka topics

9. Flink jobs with various Kafka topics

10. ● Allocate more resources in advance
● Test your job with a higher parallelism
● Enable checkpoints to preserve a job state.
Flink jobs with various Kafka topics

11. Flink jobs with various Kafka topics

12. Flink jobs with various Kafka topics

13. Flink jobs with various Kafka topics
While you are not watching

14. Flink jobs with various Kafka topics

15. ● If job is slow, Kafka rarely is a bottleneck. But good to monitor in Flink:
currentEmitEventTimeLag, watermarkLag, KafkaSourceReader.currentOffsets , currentSendTime
isBackPressured, currentOutputWatermark, numRecordsIn, numRecordsOut, idleTimeMsPerSecond
● Kafka brokers are rarely the bottleneck… but often skewed data in topics is one
Flink jobs with various Kafka topics

16. Ask me about Flink and Kafka!
Olena Babenko
Senior Software Engineer
olena@aiven.io
kafka-summit-2022