Kafka High Availability in multi data center setup with floating Ob...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Kafka High Availability in Multi
Data Center Setup with
Floating Observers
Dalibor Blazevic & Phong Pham

Agenda
● What is observer?
● What happen if one data center becomes unavailable?
● How to create Topic that uses Observers...

What is Observer?
In short:
Observer is regular Broker that is NEVER going to be part of “in-sync replica” for
particular ...

Data Center 1 Data Center 2
Broker 1
(leader)
Broker 2
Min In-Sync Replicas: 3
Replication factor: 6
Broker 3
Broker 4
Obs...

Data Center 1 Data Center 2
Broker 1
(leader)
Broker 2
Min In-Sync Replicas: 3
Replication factor: 6
Broker 3 (new leader)...

Data Center 1 Data Center 2
Broker 1
(leader)
Broker 2
Min In-Sync Replicas: 3
Replication factor: 6
Broker 3 (new leader)...

Data Center 1 Data Center 2
Broker 1
(leader)
Broker 2
Min In-Sync Replicas: 3
Replication factor: 6
Broker 3
Broker 4
Obs...

How to create Topic that uses Observers?
kafka-topics --create
--bootstrap-server til0kafka-test-kafka01.skeed.no:9092
-...

What happen with existing topics when Observers are
introduced?
➔ Workload has to be spread evenly (partition shuffling)
➔...

Partition shuffling - before
Broker 1
Topic 1
Partition 1
Partition 2
Partition 3
Partition 4
Topic 2
Partition 1
Partitio...

Partition shuffling - after
Broker 1
Topic 1
Partition 1
Partition 2
Topic 2
Partition 5
Broker 2
Topic 1
Partition 3
Part...

$Auto Data Balancing - name: Reconfigure topic shell: kafka-configs --bootstrap-server {{inventory_hostname}}:{{kafka_port}...$

Self-Balancing (From Confluent Platform ver. 6)
In server.properties
confluent.balancer.enable=true

How we prepared our observers to distribute load evenly and make them floatable?
hosts:
test-kafka01.company.no:
broker_id...

$j2 file that shuffles rack.id per Topic { "version": 2, "replicas": [ {% for replica_id in replica_ids %} { "count": 1, "c...$

Data Center 1 Data Center 2
broker.id=1
rack = 1-dc1
Min In-Sync Replicas: 3
Replication factor: 6
broker.id=2
rack = 2-dc...

Data Center 1 Data Center 2
broker.id=1
rack = 1-dc1
Min In-Sync Replicas: 3
Replication factor: 6
broker.id=2
rack = 2-dc...

Data Center 1 Data Center 2
broker.id=1
rack = 1-dc1
Min In-Sync Replicas: 3
Replication factor: 6
broker.id=2
rack = 2-dc...

Kafka High Availability in multi data center setup with floating Observers with Dalibor Blazevic and Phong Pham | Kafka S...