"Kafka Connect is an ideal tool for building data pipelines. It is both reliable and scalable, with a pluggable interface that lets you flow data between Kafka and any system you need. A Connect pipeline is made up of many different components, and understanding how each of these interact together is essential, even for the simplest setup. In this talk we will introduce the Connect components, from connectors, to transformations to the runtime itself. We will also share some of the new capabilities and best practices that you should be aware of to help you run and manage connectors effectively. Finally we will talk about some different open source projects that have been built on top of Connect that can help you get the most out of the framework."

1. From the basics to the latest features
Getting up to speed with
Kafka Connect
Mickael Maison
Kate Stanley
1

2. 2
Source
System
Sink
System
Source Connector
Sink Connector
Kafka
Connect

3. 3
What is Kafka
Connect?

4. What is Kafka Connect?
4
An API for implementing connectors

5. What is Kafka Connect?
5
A runtime for executing connector processes

6. What is a Connector?
6
Connector

7. What is a Connector?
7
Connector
debezium-connector-mongodb-1.9.0.Final.jar

8. What is a Connector?
8
Connector
Connect
plugin.path
debezium-connector-mongodb-1.9.0.Final.jar
debezium-connector-mongodb-1.9.0.Final.jar

9. What is a Connector?
9
Connector
Connect
plugin.path
Connector
debezium-connector-mongodb-1.9.0.Final.jar
debezium-connector-mongodb-1.9.0.Final.jar

10. What is a Connector?
10
Connector
Connect
Connector
plugin
Connector Connector
debezium-connector-mongodb-1.9.0.Final.jar
plugin.path

11. 11
Why use Kafka
Connect?

12. Why use Kafka Connect?
12

13. Why use Kafka Connect?
13
External
System
External
System

14. Why use Kafka Connect?
14
External
System
External
System

15. Why use Kafka Connect?
15
External
System
External
System

16. Why use Kafka Connect?
16
Kafka
Connect
External
System
External
System
Kafka
Connect

17. Why use Kafka Connect?
17
Kafka
Connect
External
System
External
System
Kafka
Connect

18. Why use Kafka Connect?
18
Source
System
Sink
System
Source Connector
Sink Connector
Kafka
Connect

19. Why use Kafka Connect?
19
Kafka
Connect
External
System
External
System
Kafka
Connect

20. Why use Kafka Connect?
20
Kafka
Connect
External
System
External
System
Kafka
Connect

21. Why use Kafka Connect?
21
Interoperability Extending existing
systems
Data pipelines
Kafka Connect use cases

22. 22
Our top best practice tips

23. Run in Distributed Mode
Optional
section
marker
or
title
23

24. Run in Distributed mode
24
Worker #1 Worker #3
Worker #2
Connect cluster

25. Run in Distributed mode
25
Worker #1 Worker #3
Worker #2
Task
Task
Task
Task
Task
Task
Connect cluster

26. Run in Distributed mode
Worker #1 Worker #3
Worker #2
Task
Task
Task
Task
Task
Worker #4
Task
Task
Connect cluster
Task
26

27. Run in Distributed mode
27
Configuration
Worker
▸ group.id
▸ offset.storage.topic
▸ config.storage.topic
▸ status.storage.topic

28. Use automation for managing
clusters
Optional
section
marker
or
title
28

29. Use automation for managing clusters
29
Responsibilities
▸ Restarting failed connectors and tasks
▸ Scaling up and down
▸ Rolling out upgrades
▸ Adding new connector plugins

30. Use automation for managing clusters
30
Solutions
▸ Operators
▸ Pre-built images
▸ Tools

31. Know the connector plugins
Optional
section
marker
or
title
31

32. 32
Know the connector plugins
Converters Transformations Predicates

33. Know the connector plugins
33
Connector plugins
External
System
Connector
Kafka
Connect

34. Know the connector plugins
34
Converters + Header Converters
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
header.converter=org.apache.kafka.connect.storage.SimpleHeaderConverter
External
System
Connector
Kafka
Connect
Converter

35. Know the connector plugins
35
Transformations
{
"transforms": "filterTombstones",
"transforms.filterTombstones.type": "o.a.k.connect.transforms.Filter",
}
External
System
Connector
Kafka
Connect
Converter
Transforms

36. Know the connector plugins
36
Predicates
{
"transforms": "filterTombstones",
"transforms.filterTombstones.type": "o.a.k.connect.transforms.Filter",
"transforms.filterTombstones.predicate": "isTombstone",
"predicates": "isTombstone",
"predicates.isTombstone.type": "o.a.k...predicates.RecordIsTombstone"
}

37. Know the connector plugins
37
Predicates
{
"transforms": "filterTombstones",
"transforms.filterTombstones.type": "o.a.k.connect.transforms.Filter",
"transforms.filterTombstones.predicate": "isTombstone",
"predicates": "isTombstone",
"predicates.isTombstone.type": "o.a.k...predicates.RecordIsTombstone"
}

38. Know the connector plugins
38
Predicates
{
"transforms": "filterTombstones",
"transforms.filterTombstones.type": "o.a.k.connect.transforms.Filter",
"transforms.filterTombstones.predicate": "isTombstone",
"predicates": "isTombstone",
"predicates.isTombstone.type": "o.a.k...predicates.RecordIsTombstone"
}

39. Consider using a Schema
Registry
Optional
section
marker
or
title
39

40. Consider using a schema registry
40
{
"title": "Getting up to Speed with Kafka Connect"
"speaker": "Mickael and Kate",
"room": "Arora West"
}
My Events

41. Consider using a schema registry
41
{
"type": "object",
"properties": {
"title": {"type": "string"},
"speaker": {"type": "string"},
"room": {
"type": "string",
"enum": ["Arora West", "Arcadia"]
}
},
"required": ["title", "room"]
}
Schema

42. Consider using a schema registry
42
Using Avro Schemas with Apicurio
key.converter=io.apicurio.registry.utils.converter.AvroConverter
key.converter.apicurio.registry.url=http://registry:8080/api
...

43. Know the worker plugins
Optional
section
marker
or
title
43

44. Know the workers plugins
44
Configuration Providers
{
"file": "myfile.txt",
"topic": "source-topic",
"ssl.truststore.password": "secret"
}

45. Know the workers plugins
45
Configuration Providers
{
"file": "myfile.txt",
"topic": "source-topic",
"ssl.truststore.password": "secret"
}

46. Know the workers plugins
46
Configuration Providers
{
"file": "myfile.txt",
"topic": "source-topic",
"truststore.password": "${file:/tmp/ssl.properties:password}"
}
config.providers=file
config.providers.file.class=o.a.k.c.config.provider.FileConfigProvider

47. Know the workers plugins
47
Configuration Providers
{
"file": "myfile.txt",
"topic": "source-topic",
"truststore.password": "secret"
}
{
"file": "myfile.txt",
"topic": "source-topic",
"truststore.password": "${file:/tmp/ssl.properties:password}"
}

48. Know the workers plugins
48
REST Extensions
rest.extension.classes=o.a.k.c...extension.BasicAuthSecurityRestExtension
▸ Authentication
▸ Authorization
▸ Validation
▸ JAX-RS plugins

49. Rely on Connect Metrics
Optional
section
marker
or
title
49

50. Rely on Connect Metrics
50
▸ Tasks/Connectors status
kafka.connect:type=connect-worker-metrics
▸ Connection to Kafka Brokers
kafka.connect:type=connect-node-metrics,client-id=<>,node-id=<>
▸ Connector metrics
Rely on Connect Metrics

51. Master the REST API
Optional
section
marker
or
title
51

52. Master the REST API
52
Manage logger level
GET /admin/loggers
PUT /admin/loggers/org.apache.kafka.connect -d '{"level": "DEBUG"}'

53. Master the REST API
53
Restart connectors and tasks
POST /connectors/file-sink/restart
POST /connectors/file-sink/restart?includeTasks=true&onlyFailed=true
Task
Connector
POST /connectors/file-sink/tasks/0/restart

54. Use Dead Letter Queues
Optional
section
marker
or
title
54

55. Use dead letter queues
55
Sink
Connector
Kafka
Connect
Avro
Converter
input-topic

56. Use dead letter queues
56
Sink
Connector
Kafka
Connect
Avro
Converter
input-topic
my-dlq

57. Use dead letter queues
57
Sink
Connector
Kafka
Connect
Avro
Converter
input-topic
my-dlq
Report
error

58. Try kcctl
Optional
section
marker
or
title
58

59. Try kcctl
59
https://github.com/kcctl/kcctl

60. Don’t forget to Upgrade
Optional
section
marker
or
title
60

61. 61
What coming next in
Connect?

62. CONFIDENTIAL designator
V0000000
62
KIP-769:
APIs to list all connector
plugins and retrieve
their configuration
definitions
Kafka 3.2

63. KIP-769
63
GET /connector-plugins
PUT /connector-plugins/{connector plugin}/config/validate
Existing endpoints

64. KIP-769
64
GET /connector-plugins
PUT /connector-plugins/{connector plugin}/config/validate
GET /connector-plugins/{connector plugin}/config
New endpoint

65. KIP-769
65
$ curl http://localhost:8083/connector-plugins/FileStreamSinkConnector/config
[
{
"name": "file",
"type": "STRING",
"required": false,
"default_value": null,
"importance": "HIGH",
"documentation": "Destination filename. If not specified, the standard output
will be used",
"group": null,
"width": "NONE",
"display_name": "file",
"dependents": [],
"order": -1
}
]

66. CONFIDENTIAL designator
V0000000
66
KIP-618:
Exactly-Once Support
for Source Connectors
Voted

67. KIP-618
67
▸ Source connectors use a transactional producers for
writing records and offsets
▸ Connectors can declare whether they support exactly
once
▸ Workers have exactly once disabled by default

68. CONFIDENTIAL designator
V0000000
68
KIP-802:
Validation Support for
Kafka Connect SMT and
Converter Options
Discussion in
progress

69. KIP-802
69
PUT /connector-plugins/{connector plugin}/config/validate
New capability with existing endpoint
▸ Transformations
▸ Predicates
▸ Converters

70. CONFIDENTIAL designator
V0000000
70
KIP-821:
Connect Transforms
support for nested
structures
Discussion in
progress

71. KIP-821
71
{
"room":"Arcadia",
"speaker": {
"name": {
"speaker-name": "Kate"
}
}
}

72. KIP-821
72
{
"room":"Arcadia",
"speaker": {
"name": {
"speaker-name": "Kate"
}
}
}

73. KIP-821
73
{
"room":"Arcadia",
"speaker": {
"name": {
"speaker-name": "Kate"
}
}
}

74. KIP-821
74
{
"room":"Arcadia",
"speaker-name": "Kate"
}
{
"transforms": "smt1",
"transforms.smt1.type": "org.apache.kafka.connect.transforms.ValueToKey",
"transforms.smt1.field.style": "nested",
"transforms.smt1.fields": "speaker.name"
}

75. 75
https://kafka.apache.org/documentation/#connect
https://www.oreilly.com/library/view/kafka-connect/9781098126520/
https://www.redhat.com/rhdc/managed-files/cl-kafka-connect-analys
t-material-f31379-202203-en.pdf
Thank you
https://twitter.com/katestanley91
https://twitter.com/mickaelmaison