"Kafka Connect and Apache Flink are popular frameworks for data integration and processing. Both frameworks provide for external connectivity through a connector SDK. Kafka connect mainly focuses on moving data in and out of Kafka, whereas connectors in Flink primarily move data from/to external sources /sinks for use in processing directly. While Kafka Connect has a robust and widely covered connector ecosystem and a popular SDK, connector interfaces in Flink are relatively new and still evolving. How different are the two connector ecosystems? Should you write a Kafka Connector or a Flink Connector ? This talk would cover the main connector interfaces in Flink, we will do a high level overview of 2 main FLIPs and briefly touch upon some high level Flink components which enable connectors to work. We will compare and contrast this to connector interfaces in Kafka while touching upon state management, validation models etc, and then briefly dive into some practical use-cases enabled through connectors across the two frameworks. Attendees will learn how to develop a simple source connector in Flink through a sample Datadog connector Existing Kafka Connector developers will be able to appreciate the key differences and features of the Flink connector SDK. This session would also help understand some key considerations involved in writing connectors across these two frameworks along with some best practices."

1. Decoding the data integration matrix
Connectors in Kafka connect and Flink
Sambhav Jain
Software Engineer, Confluent

2. Sink
System
What is a connector framework?
Connector
Framework
Kafka
2
Source
System Kafka

3. Flink connectors
3
Source
System
Sink
System

4. Connector Ecosystem
syslog
SOURCES SINKS
100+ open source connectors ~20 open source connectors
4

5. Source
Systems
Sink
Systems
Worker
Connect Cluster
Connector
Connector Connector
Source
Sink
Source
Operators
Sink
Operators
Task Manager
Job Manager
Source
coordinator
Kafka Connect Flink
Physical Architectures
group by
group by
5

6. Connector Development

7. 7
Source Interfaces in Kafka Connect
Sink
SMT
SMT
Source
Connector
Kafka
Connect
WorkerSourceTask
execute()
SourceTask
poll() SMT/
converters
Offset
store
commitOffsets()
After successful
production
SourceConnector
taskConfigs()
validate()
WorkerConnector
Source system

8. Let’s start with the datadog connector
{
"events": [
{
"alert_type": "info",
"date_happened": "integer",
"device_name": "string",
"host": "string",
"id": "integer",
"id_str": "string",
"payload": "{}",
"priority": "normal",
"source_type_name": "string",
"tags": [
"environment:test"
],
"text": "Oh boy!",
"title": "Did you hear the news today?",
"url": "string"
}
],
"status": "string"
}
API Request API Response
8

9. Source Interfaces in Kafka Connect
9

10. Source Interfaces in Kafka Connect
10

11. Source Interfaces in Flink
Source
Task Manager
Job Manager
handleSplitRequest()
SplitEnumerator
SourceReader
RecordEmitter
Split Fetcher
Records Queue emitRecord()
SplitReader
RecordEmitter
fetch()
11
Source

12. Source Interfaces in Flink
12

13. Source Interfaces in Flink
13

14. Source Flink v/s Source Kafka Connect
fetch()
poll() converter
emitRecord()
Flink
Connect
Record,
Schema
SourceRecord
Schema
Worker 14

15. State Management
"date_happened": "integer" from DD response 15
Source Offsets Split State

16. State Management
State
Store
Offset
topic
Worker
Source offsets
{
task1: {
timestamp: 1708508270
},
task2: {
timestamp: 1708507620
}
}
SourceTask
Kafka Connect Flink
SplitState
snapshotState()
emitRecord()
emitRecord()
RecordEmitter
{
split1: {
timestamp: 1708508270
},
split2: {
timestamp: 1708507620
}
}
16

17. 17
WorkerSinkTask
WorkerConnector
execute()
SinkTask
put()
SMT/
converters
preCommit()
flush()
Consumer
committing
its offsets
Kafka Connect Flink
SinkConnector
taskConfigs()
Sink Interfaces
validate()
flush()
write()
Sink
Periodically called just
before completing
every checkpoint
StatefulSinkWriter

18. Kafka Connect Flink Connector
State
Management
Framework handles source offset
management. Connector can’t
override this behaviour
Handled automatically with default
behaviour but connector can
override this.
Serialization Connectors send data to converters
in workers
Connectors have to serialize and
deserialize
Internal data
Representation
Custom objects: SourceRecord and
SinkRecord
User defined
Delivery
Semantics
(Best possible)
Source: Exactly once
Sink: At Least Once
Source: Exactly once
Sink: Exactly Once
Summary
18

19. TableAPI / SQL in Flink
DynamicTableFactory
identifier()
{ return “kafka”; }
DynamicTableSource
FlinkSource
DynamicTableSink
FlinkSink
19

20. Real time database
Replication
Use cases of Connectors and Kafka
Integrating multiple
systems with Kafka
Capture CDC events
from your source system
Database
Lookups
Real time
Analytics
Enrichment
Joins
Real time
Materialised views
20

21. Q/A

22. References
1. FLIP-27:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
2. FLIP-143:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
3. Example implementation of Datadog Flink Source
https://github.com/sambhav-jain-16/flink/tree/datadog-source--flink-poc/flink-connectors/flink-co
nnector-datadog/src/main/java/org/apache/flink/connector/datadog/source
Special Thanks:
1. Anupam Aggarwal
2. Suchi Amalapurapu
22
LinkedIn: Sambhav Jain