Qlik Replicate can stream data from a source database into Apache Kafka. It supports compression and sends messages in either JSON or Avro format. Kafka topics partition the data across broker nodes. Replicate requires open TCP ports and configured permissions in Kafka. While Kafka supports unlimited throughput, it has limitations like maximum LOB size and lacks some Replicate data apply modes.

1. Full
Load
Source
Database
Transaction
logs
CDC
Qlik Replicate
Data Stream
MSG MSG MSG
n 2 1
Data Stream
MSG MSG MSG
n 2 1
Message
broker
Message
broker
Data Streaming into Kafka
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.

2. • Supports Snappy and Gzip compression
Kafka messages
KEY VALUE
• Two parts to a message
• Optional/custom message keys
Messages sent in two formats:
• JSON
• Avro
Message
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.

3. Example JSON message structure
{
‘task_name”:”task-name”,
“table”: “table-name”,
“schema”: “schema-name”,
“op”: “operation-type”, // F = Full Load, I = Insert, U = Update, D = Delete
“stream_position”: “position”,
“change_seq”: “change-seq”,
“ts”: “change-timestamp”,
“data”: [{“col1”:, “val1”}, {“col2”; “val2”}, …., {“colN”: “valN”}],
“bu_data”: [{col1”: “val1”}, {“col2”: “val2”}, …., {“colN”: “valN”}],
}
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.

4. Kafka topics
Topic 1
Topic 2
Topic 3
Broker 1
Topic 1
Topic 2
Topic 3
Broker 2
Kafka Cluster
Data Stream
KEY VALUE
KEY VALUE
KEY VALUE
Replicate
1 2 3 4 5 6
P1
1 2 3 4
P2
1 2 3 4 5
P3
Topic 1
Consumers Apps
Consumer 1
Consumer 2
Consumer 3
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.

5. Kafka prerequisites
• Open TCP Ports
• Set permissions/ACLs
• Configure:
attrep_apply_exceptions OR auto.create.topics.enable=true
• NOTE: Kafka does not require a client library
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.

6. Limitations of Kafka as a target
• Does not support unlimited LOB size
• Does not support “batch optimized apply” mode
• Does not support “store changes” mode
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are associated.