Basic principles of Kafka Connect, recent technology used to ingesta process and data integration.

1. Kafka Connect

2. Contents
Introduction to Kafka1
2
3
Data Ingestion
Kafka Connect

3. 1.
INTRODUCTION
TO KAFKA
Let’s start with the first set of
slides

5. Data Pipeline Problem
Frontend
Server
Metrics
Server
Frontend
Server
Frontend
Server Database
Server
Chat
Server
Metrics
Server
Metrics UI
Inter-process communication channel

6. Data Pipeline Problem
Frontend
Server
Shopping
Cart
Backend
Server
Metrics
Server
Metrics UI Log Search
Database
Server
Frontend
Server
Frontend
Server
Database
Server
Chat
Server
Metrics
Server
Metrics UI
Metrics
Pub/Sub
Metrics
Pub/Sub
Logging
Pub/Sub
A publish/subscribe System Multiple publish/subscribe Systems

7. Kafka Goals
Frontend
Server
Metrics
Server
Metrics UI Log Search
✓ Decouple data pipelines
✓ Provide persistence for
message data to allow
multiple consumers
✓ Optimize for high
throughput of messages
✓ Allow for horizontal scaling
of the system to grow as the
data stream grow
Database
Server Shopping
Cart
Backend
Server

8. Producer
Producer
Producer
Producer
Consumer Consumer Consumer
Broker 1
Topic A
Partition 0
Broker 2
Topic A
Partition 1
Broker 3
Topic B
Z
O
O
K
E
E
P
E
R
Kafka Architecture
Kafka Cluster
Topic C

9. Disk-based retention
Scalable
High throughput

10. USE CASES
Gold
Data
Data
Lake
Data

11. 2.
DATA INGESTION

12. Kafka Ingestion
Kafka
Producer
Kafka
Consumer
Kafka Cluster
Use Case Requirements
DATA LOSS ?
EXCATLY ONCE ?
LATENCY ?
THROUGHPUT ?

13. Producer
Record
Kafka
Broker
Topic
Partition
Key
Value
producer.
send(record).get
Exception/Metadata
Producer
Record
Kafka
Broker
Topic
Partition
Key
Value
producer.
send(record)
Exception/Metadata
Asynchronous sendSynchronous send
Producer

14. Producer
Record
Serializer
Partitioner
Topic A
Partition 0
Batch 0
Batch 1
Topic B
Partition 1
Batch 0
Batch 1
Fail
?
Retry
?
Yes
Yes
MetadataException
Send()Topic
Partition
Key
Value
Topic
Partition
commit
Metadata
Topic
Partition
Offset
Broker
Producer

15. 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12
Consumer
0
Consumer
1
Consumer
2
Consumer
Group
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Partition 0
Partition 1
Partition 2
Partition 3
Consumer 1
Consumer 2
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Consumer 5
Consumer 6
Topic T1 Topic T1 Topic T1Consumer Group 1 Consumer Group 1
Consumer
Group 1
Consumer

16. Channel
Channel
Processor
Interceptor #1
Interceptor #N
SinkSource
Flume Agent
Reliable
Fault Tolerant
Customizable
Manageable
Centralized Sources
Apache Flume

17. Consumer
Kafka as reliable flume channel
Flume + Kafka
Source Sink
Channel
Producer
Flume as kafka producer/consumer

18. 3.
KAFKA CONNECT

19. Data
Source
Schema
Registry
Data
Sink
KafkaSourceConnect
Kafka
Connect
Ingestion integration
Streaming and batch
Scales to the application
Failover control
Accessible connector API
KafkaSinkConnect

20. Worker Mode
Connect
Kafka
Producer
Kafka
Consumer
Worker
Task
Worker
Task
Worker
TaskThread
Standalone Distributed
Connector
Input
N
Input
2
Input
1
Task
Worker

21. Producer
Record
Serializer
Partitioner
Topic A
Partition 0
Batch 0
Batch 1
Topic B
Partition 1
Batch 0
Batch 1
Fail
?
Retry
?
Yes
Yes
MetadataException
Send()Topic
Partition
Key
Value
Topic
Partition
commit
Metadata
Topic
Partition
Offset
Broker
Worker settings to ensure no data loss
request.timeout.ms=MAX_VALUE
retries=MAX_VALUE
max.in.flight.request.per.connection=1
acks=all
max.block.ms=MAX_VALUE

22. Worker 2 Worker 3Worker 1
Worker
Task
Worker
Task
Worker
Task
Worker
config
Source
config
Source
config
Source
config
Worker
config
Worker
Task
Worker
Task
Worker
Task
Conn 1, Task 3
Partitions: 5,6
Conn 2, Task 1
Partitions: 1,2
Conn 2
Conn 1, Task 2
Partitions: 3,4
Conn 1
Conn 1, Task 1
Partitions: 1,2
Conn 2, Task 2
Partitions: 3,4
Standalone
worker
Scalability
Fault tolerance
Share
connectors &
tasks
Distributed Worker
Simple
1 Worker
N conn/tasks

23. Schema
Registry
Consumer
Subject
Topic
Schema
Version
Worker
Task sendRecords()
Source
Record
Source
Record
Source
Record Producer
Record
Producer
Record
Producer
Record
Task
REST API
Serializers
Formaters
Multiple version
of the same
schema
StreamsStreams
Stream
Source
Schema
Id Id
Schema
Serializer
Deserializer

24. THANKS!
Any questions?
@datiobd
rbravo@datiobd.com
datio-big-data