How to Build an Apache Kafka® Connector

@jwfbean | @conﬂuentinc
Building a Kafka Connector
Veriﬁed Integrations Program

Speakers
Jeff Bean
Partner Solution Architect
Lisa Sensmeier
Partner Marketing
Todd McGrath
Partner Solution Architect

Agenda
● Why Kafka Connect
● Kafka Connect Architecture
● Building a Connector
● Common Issues
● Examples
● Veriﬁcation Criteria & Process

5
Apps
Stream
Processing
Search
KV
RDBMS DW
Real-Time
Analytics
Monitoring

6
PRODUCER
CONSUMER
Producer
Application
Consumer
Application

7
PRODUCER
CONSUMER
Sink
ConnectorSMTsSource
Connector ConverterSMTs Converter
KAFKA CONNECT KAFKA CONNECT

8
PRODUCER
CONSUMER
Sink
ConnectorSMTsSource
Connector ConverterSMTs Converter
KAFKA CONNECT KAFKA CONNECT

9
Connectors
● A logical job that copies data in and out of Apache Kafka®
● Maintains tasks
● Provides lifecycle and conﬁguration information
● Ultimately a JAR available to the connect JVM
● Stateless! Use Kafka topics or source / target for state

10RESTAPI
RESTAPI
RESTAPI
C
TTTT
TTTT
Other
Task(s)
Other
Task(s)
Other
Task(s)
start(…)
poll(…) poll(…) poll(…)
poll(…)
Other
Connector(s)
start(…) start(…)
start(…)
TTTT
Running

11
Kafka Connect - Connector and Task Lifecycles
REST
API
REST
API
RESTAPI
KAFKA CONNECT
validate
CC
start(…)
deploy
Starting

12
Kafka Connect - Connector and Task Lifecycles
RESTAPI
RESTAPI
RESTAPI
C
taskConﬁgs(…)
TTTT
Running

13
Tasks
● Lifecycle managed by a connector
● Runs in a worker
● Manages one or more topic partitions
● The assignment is dynamic at runtime
● Does the actual copying and transforming of things

14
Workers
● Actual JVM processes running on a computer of some kind
● Tasks get allocated to workers
● Can run in standalone or distributed mode
○ Standalone is good for dev, one-offs, and conference demos
○ Distributed good for scale and fault tolerance

15RESTAPI
RESTAPI
RESTAPI
C
TTTT
TTTT
Other
Task(s) Other
Task(s)
Other
Task(s)
Other
Connector(s)
Recovery from Failures

16RESTAPI
RESTAPI
C
TTTT
TTTT
Other
Task(s)
Other
Task(s)
Other
Connector(s
)
Recovery from Failures

17RESTAPI
RESTAPI
C
TTTT
TTTT
Stopping
stop() stop() stop()stop()
stop() Other
Task(s)
Other
Connector(s
)
Other
Task(s)

18
Delivery Guarantees
● Framework-managed offsets
● At-least-once default
● Exactly once available with connector support

19
Converters

20
Converters
● Convert input data to bytes for storing in Kafka
● Convert input data from bytes to output to somewhere else
● Sit between the connector and Kafka in either direction

@twitter_handle | #hashtag | @conﬂuentinc
21
Single Message Transform
(SMT)

Single Message
Transforms
● Mask sensitive information
● Add identiﬁers
● Tag events
● Lineage/provenance
● Remove unnecessary columns
● Route high priority events to
faster datastores
● Direct events to different
Elasticsearch indexes
● Cast data types to match
destination
● Remove unnecessary columns
Modify events before storing in Kafka:
Modify events going out of
Kafka:

23
Where SMTs Live

24
Built-in Transformations
● InsertField – Add a field using either static data or record metadata
● ReplaceField – Filter or rename fields
● MaskField – Replace field with valid null value for the type (0, empty string, etc.)
● ValueToKey – Set the key to one of the value’s fields
● HoistField – Wrap the entire event as a single field inside a Struct or a Map
● ExtractField – Extract a specific field from Struct and Map and include only this field in results
● SetSchemaMetadata – Modify the schema name or version
● TimestampRouter – Modify the topic of a record based on original topic and timestamp, which is
useful when using a sink that needs to write to different tables or indexes based on timestamps
● RegexpRouter – Modify the topic of a record based on the original topic, replacement string, and a
regular expression

3
1
SourceConnector and SinkConnector
class Connector {
String version();
ConfigDef config();
Class<? extends Task> taskClass();
void start(Map<String, String> props);
List<Map<String, String>> taskConfigs(int maxTasks);
void stop();
...
}

3
2
SourceConnector and SinkConnector
class SinkConnector extends Connector {}
class SourceConnector extends Connector {}

33
Starting a connector
•The user-specified connector
configuration
•Optionally talk to external system to get
information about tasks
•Determine the task configuration
•Optionally start thread(s) to monitor
external system, and if needed
ask for task reconfiguration

34
Task conﬁgs
List<Map<String, String>> taskConfigs(int maxTasks);
•Tell Connect the conﬁgurations for each
of the tasks

35
Considerations - Testing
•Design for testability
•Unit tests and integration tests in your build
-look for reusable integration tests (PR5516) using
embedded ZK & Kafka
•Continuous integration for regression tests
•System tests use real ZK, Kafka, Connect, and external systems
•Performance and soak tests
•See the Connect Veriﬁcation Guide for more detailed information on testing.

36
SourceConnector and
SinkConnector - Stoppingvoid stop();
•Notiﬁcation that the connector is being
stopped
•Clean up any resources that were
allocated so far

3
7
SourceTask
abstract class SourceTask implements Task {
protected SourceTaskContext context;
List<SourceRecord> poll() throws InterruptedException;
void stop();
...
}

38
SourceTask - Starting
•This task’s conﬁguration that was created
by your connector
•Read previously committed offsets to
know where in the external system to start
reading
•Create any resources it might need
- Connections to external system
- Buffers, queues, threads, etc.

39
SourceTask - getting records
•The topic partitions assigned to this task's
consumer
•Optionally pre-allocate writers and
resources
•Consumer will by default start based on
its own committed offsets
•Or use context.offsets(...) to set the
desired starting point
● Called frequently
● Get the next batch of records
that Connect should write to
Kafka
● block until there are “enough”
records to return
● return null if no records right
now
● For systems that push data
● use separate thread to receive
and process the data and
enqueue
● poll then dequeues and returns
records
List<SourceRecord> poll() throws InterruptedException;

40
SourceRecord
topic : String
partition : Integer
keySchema : Schema
key : Object
valueSchema : Schema
value : Object
timestamp : Long
headers : Headers
sourcePartition : Map
sourceOffset : Map
• Topic where the record is to be written
• Optional partition # for the topic
• Optional key and the schema that describes it
• Optional value and the schema that describes it
• Optional headers
• Optional timestamp
• Source “partition” and “offset"
- Describes where this record originated
- Deﬁned by connector, used only by connector
- Connect captures last partition+offset that it writes, periodically committed to
connect-offsetstopic
- When task starts up, it reads these to know where it should start
- TIP: reuse the same partition Map instance
• Serialized via Converters
SourceRecord

41
Connect Schemas
name : String
version : String
doc : String
type : Type
parameters : Map
fields : List<Field>
name : String
schema : Schema
index : int
Field
Schema
• Name (required), version, and documentation
• Type of schema: primitive, Map, Array, Struct
• Whether the value described by the schema is optional
• Optional metadata parameters
• Information about the structure:
- For Struct schemas, the fields
- For Map schemas, the key schema and value schema
- For Array schemas, the element schema
• Name of field that this schema describes
• Schema for field value
• Index of field within the Struct

42
SourceTask - Stopping
•Clean up resources that were allocated so
farvoid stop();

4
3
SinkTask
abstract class SinkTask implements Task {
protected SingTaskContext context;
void open(Collection<TopicPartition> partitions);
void put(Collection<SinkRecord> records);
Map<TopicPartition, OffsetAndMetadata> preCommit(
Map<TopicPartition, OffsetAndMetadata> currentOffsets
);
void close(Collection<TopicPartition> partitions);
void stop();
...

44
SinkTask - Starting
•This task’s conﬁguration that was created
by your connector
- includes topics and topics.regex property
•Start the task and create any resources it
might need

45
SinkTask - Assigned topic
partitions
void open(Collection<TopicPartition> partitions);
resources

46
SinkTask - Process records
consumer
resources
● Called frequently with next batch of
records returned from the consumer
● Size of batch depends on
● availability of records in our assigned
topic partitions, and
● consumer settings in worker,
including:
● consumer.fetch.min.bytes and
consumer.fetch.max.bytes
● consumer.max.poll.interval.ms and
consumer.max.poll.records
● Either write directly to external
system or
buffer them and write when there are
“enough” to send
void put(Collection<SinkRecord> records);

47
SinkRecord
• Topic, partition, and offset from where the record was
consumed
• Optional key and the schema that describes it
• Optional value and the schema that describes it
• Timestamp and whether its create/append/none
• Headers
- ordered list of name-value pairs
- utility to convert value to desired type, if possible
• Deserialized via Converters
topic : String
partition : Integer
keySchema : Schema
key : Object
valueSchema : Schema
value : Object
timestamp : Long
timestampType : enum
offset : long
headers : Headers
SinkRecord

48
Sink Connectors and Exactly
Once Semantics (EOS)
• Must track the offsets of records that were written to external
system
- typically involves atomically storing offsets in external system
- for convenience can optionally still have consumer commit those offsets
• When a task restarts and is assigned topic partitions, it
- looks in the external system for the next offset for each topic partition
- set in the task context the exact offsets where to start consuming

49
SinkTask - Committing offsets
consumer
resources
● Called periodically based upon
worker’s
offset.ﬂush.interval.ms
● defaults to 60 seconds
● Supplied the current offsets for
consumed records as of the
last call to put(…)
● Return the offsets that the
consumer should actually
commit
● should reﬂect what was
actually written so far to the
external system for EOS
● or return null if consumer
should not commit offsets
Map<TopicPartition, OffsetAndMetadata> preCommit(
Map<TopicPartition, OffsetAndMetadata>
currentOffsets);

50
SinkTask
•Close any writers / resources for these topic
partitions
void close(Collection<TopicPartition> partitions);

5
1
void stop();

53
Considerations - Conﬁguration
•Only way to control a connector
•Make as simple as possible, but no simpler
•Use validators and recommenders in ConﬁgDef
•Strive for backward compatibility

54
Considerations - Ordering
•Each Kafka topic partition is totally ordered
•Design connectors around this notion

55
Considerations - Scaling
•Use tasks to parallelize work
-each task runs in its own thread
-you can start other threads, but always clean them up properly
•Sink connectors - typically use # of tasks speciﬁed by user
•Source connectors - only one task (or several) may make sense

56
Considerations - Version Compatibility
•Connectors are by deﬁnition multi-body problems
-external system
-Connect runtime
-Kafka broker
•Specify what versions can be used together

57
Considerations - Resilience
•What are the failure modes?
•Can you retry (forever with exponential backoff) rather than fail?
-consider RetriableException

58
Considerations - Logging
•Don’t overuse ERROR or WARN
•Use INFO level so users know what’s happening under normal
operation
and when misbehaving
•Use DEBUG so users can diagnose why it’s behaving strangely
•Use TRACE for you

59
Considerations - Security
•Use PASSWORD conﬁguration type in ConﬁgDef
•Communicate securely with external system
•Don’t log records (except maybe at TRACE)

60
Considerations - Naming things
•Don’t use org.apache.kafka.* packages
•These are reserved for use by the Apache Kafka project

61
Considerations - Licensing
•Indicate your license, whether Open Source License or proprietary

Lifecycle Management
● How to make sure systems are ready before sending them
records?
○ SinkTaskContext.pause() / SinkTaskContext.resume() -
when target is not ready
○ SinkTask.open() - called when partition is assigned to a
sink task
○ Connector.start() - Database Creation
○ taskConﬁgs()

Ordering
● Apache Kafka gaurantees ordering but the Connector
Developer has to ensure it as well
● Parallel transactions might complete out of order on the
target system

DLQ
● There’s an open request for API access to the DLQ
● The DLQ is just a Kafka topic, it can be written to by a
producer, but this violates Kafka Connect guidelines
● The DLQ is for records it cannot process, not for errors

Push vs Pull
● Kafka Connect source / sink paradigm “reverses the polarity” of the Kafka
producer / consumer paradigm on purpose.
○ Producer PUSH to Kafka, Source Connector PULLS from sink (Framework
PUSHes to Kafka)
○ Consumer PULL from Kafka, Sink Connector PUSHES to sink (Framework
PULLs from Kafka)
● What if I want a target to PULL via sink connector?
○ The connector PULLs
○ Sink task can call SinkTaskContext.pause, check for liveness on target and
then SinkTaskContext.resume when ready.

68
MongoDB
•Source and Sink connector
•Source: Next release of MongoDB
CDC connector from Debezium
•Sink: Next release of from HPG’s
community connector
•https://www.mongodb.com/blog/post/getting-started-with-the-
mongodb-connector-for-apache-kafka-and-mongodb-atlas
•https://github.com/mongodb/mongo-kafka

69
Snowﬂake
•https://docs.snowflake.net/manuals/user-guide/kafka-connect
or.html
•https://github.com/snowflakedb/snowflake-kafka-connector

70
Neo4J
•https://www.confluent.io/blog/kafka-connect-neo4j-sink-plugin
•https://github.com/neo4j-contrib/neo4j-streams/tree/master/k
afka-connect-neo4j

71
Rockset
•https://rockset.com/blog/kafka-connect-plugin-for-rockset/

72
Conﬂuent Supported
Examples
•S3:
https://github.com/confluentinc/kafka-connect-storage-cloud
•ElasticSearch:
https://github.com/confluentinc/kafka-connect-elasticsearch

Veriﬁcation
Criteria

76
Sign up, ask questions, or start the process
confluent.io/Verified-Integrations-Program/
Email
vip@conﬂuent.io
Kafka Summit San Francisco
kafka-summit.org/
code KS19Online25 for 25% off
Q&A

How to Build an Apache Kafka® Connector

How to Build an Apache Kafka® Connector

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Build an Apache Kafka® Connector

Similar to How to Build an Apache Kafka® Connector (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

How to Build an Apache Kafka® Connector