Data Loss and Duplication in Kafka

© 2015, Conversant, Inc. All rights reserved.
PRESENTED BY
November 10, 2016
Data Loss and Data Duplication
in Kafka
Jayesh Thakrar

© 2015, Conversant, Inc. All rights reserved.2
Kafka is a distributed, partitioned, replicated,
durable commit log service. It provides the
functionality of a messaging system, but with a
unique design.
Exactly once - each message is delivered once
and only once

 Kafka Overview
 Data Loss
 Data Duplication
 Data Loss and Duplicate Prevention
 Monitoring
AGENDA

Kafka Overview

Kafka As A Log Abstraction
Client: Producer
Client: Consumer BClient: Consumer A
Kafka Server = Kafka Broker
Topic: app_events
Source: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Topic Partitioning . . .
Kafka Broker
Client: Producer or Consumer
Source: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Topic: app_events

Topic Partitioning – Scalability
Clients: Producer, Consumer
Leader
Replica
Replica
Leader
Replica
Replica
Leader
Replica
Replica
Kafka Broker 0
Kafka Broker 1
Kafka Broker 2

Topic Partitioning – redundancy
Client: Producer, Consumer
Kafka Broker 2
Leader
Replica
Replica
Leader
Replica
Replica
Leader
Replica
Replica
Kafka Broker 0
Kafka Broker 1

Topic Partitioning – Redundancy/durability
Kafka Broker 2
Leader
Replica
Replica
Leader
Replica
Replica
Leader
Replica
Replica
Kafka Broker 0
Kafka Broker 1
Pull-based inter-broker replication

Topic Partitioning – summary
 Log sharded into partitions
 Messages assigned to partitions by API or custom partitioner
 Partitions assigned to brokers (manual or automatic)
 Partitions replicated (as needed)
 Messages ordered within each partition
 Message offset = absolute position in partition
 Partitions stored on filesystem as
ordered sequence of log segments (files)

Other Key Concepts
 Cluster = collection of brokers
 Broker-id = a unique id (integer) assigned to each broker
 Controller = functionality within each broker responsible for leader
assignment and management, with one being the active controller
 Replica = partition copy, represented (identified) by the broker-id
 Assigned replicas = set of all replicas (broker-ids) for a partition
 ISR = In-Sync Replicas = subset of assigned replicas (brokers) that
are “in-sync/caught-up”* with the leader (ISR always includes the leader)

Data Loss

Data Loss : Inevitable
Upto 0.01% data loss
For 700 billion messages / day,
that's up to 7 million / day

Data loss at the producer
Kafka Producer API
API Call-tree
kafkaProducer.send()
…. accumulator.append() // buffer
…. sender.send() // network I/O
•Messages accumulate in buffer in batches
•Batched by partition, retry at batch level
•Expired batches dropped after retries
•Error count and other metrics via JMX
Data Loss at Producer
•Failure to close / flush producer on
termination
•Dropped batches due to communication
or other errors when acks = 0 or retry
exhaustion
•Data produced faster than delivery,
causing BufferExhaustedException
(deprecated in 0.10+)

dATA LOSS AT The CLUSTER (BY BROKERS)
Was it
a
leader?
Detected by
Controller via
zookeeper
Was it
in ISR?
Other
replicas
in ISR?
Elect another
leader
Allow
unclean
election?
ISR >=
min.insync
.replicas?
Relax, everything will
be fine
Partition
unavailable !!
Other
replicas
available?
Y Y
N
N
Y
Y
Y
Y
N
Broker
Crashes
N
N
N
1
2
4
5 6
3
7

Non-leader broker crash
Was it
a
leader?
Detected by
Controller via
zookeeper
Was it
in ISR?
Other
replicas
in ISR?
Elect another
leader
Allow
unclean
election?
ISR >=
min.insync
.replicas?
be fine
Partition
unavailable !!
Other
replicas
available?
Y Y
N
N
Y
Y
Y
Y
N
Broker
Crashes
N
N
N
1
2
4
5 6
3
7

Leader broker crash: Scenario 1
Was it
a
leader?
Detected by
Controller via
zookeeper
Was it
in ISR?
Other
replicas
in ISR?
Elect another
leader
Allow
unclean
election?
ISR >=
min.insync
.replicas?
be fine
Partition
unavailable !!
Other
replicas
available?
Y Y
N
N
Y
Y
Y
Y
N
Broker
Crashes
N
N
N
1
2
4
5 6
3
7

Leader broker crash: Scenario 2
Was it
a
leader?
Detected by
Controller via
zookeeper
Was it
in ISR?
Other
replicas
in ISR?
Elect another
leader
Allow
unclean
election?
ISR >=
min.insync
.replicas?
be fine
Partition
unavailable !!
Other
replicas
available?
Y Y
N
N
Y
Y
Y
Y
N
Broker
Crashes
N
N
N
1
2
4
5 6
3
7

dATA LOSS AT The CLUSTER (BY BROKERS)
Was it
a
leader?
Detected by
Controller via
zookeeper
Was it
in ISR?
Other
replicas
in ISR?
Elect another
leader
Allow
unclean
election?
ISR >=
min.insync
.replicas?
be fine
Partition
unavailable !!
Other
replicas
available?
Y Y
N
N
Y
Y
Y
Y
N
Potential data-loss
depending upon acks
config at producer. See
KAFKA-3919
KAFKA-4215
Broker
Crashes
N
N
N
1
2
4
5 6
3
7

FROM KAFKA-3919

FROM KAFKA-4215

Config for Data Durability and Consistency
 Producer config
- acks = -1 (or all)
- max.block.ms (blocking on buffer full, default = 60000) and retries
- request.timeout.ms (default = 30000) – it triggers retries
 Topic config
- min.insync.replicas = 2 (or higher)
 Broker config
- unclean.leader.election.enable = false
- timeout.ms (default = 30000) – inter-broker timeout for acks

Config for Availability and Throughput
 Producer config
- acks = 0 (or 1)
- buffer.memory, batch.size, linger.ms (default = 100)
- request.timeout.ms, max.block.ms (default = 60000), retries
- max.in.flight.requests.per.connection
 Topic config
- min.insync.replicas = 1 (default)
 Broker config
- unclean.leader.election.enable = true

Data Duplication

Data Duplication: How it occurs
Client: Producer
Kafka Broker
Topic: app_events
Producer (API)
retries = messages
resent after timeout
when retries > 1
Consumer
consumes messages
more than once after
restart from unclean
shutdown / crash

Data Loss &
Duplication
Detection

How to Detect Data loss & Duplication - 1
Memcache /
HBase /
Cassandra /
Other
Producer Kafka Consumer
Topic, Partition, Offset | Msg Key or
Hash
KEY | VALUE
1) Msg from producer to Kafka
2) Ack from Kafka with details
3) Producer inserts into store
4) Consumer reads msg
5) Consumer validates msg
If exists
not duplicate
consume msg
delete msg
If missing
duplicate msg
Audit: Remaining msgs in store are
"lost" or "unconsumed" msgs
11
22
33
44
55Store

How to Detect Data loss & Duplication - 2
Memcache /
HBase /
Cassandra /
Other
Producer Kafka Consumer
Source, time-window | Msg count or some other checksum (e.g. totals, etc)
KEY | VALUE
1) Msg from producer to Kafka
2) Ack from Kafka with details
3) Producer maintains window stats
4) Consumer reads msg
5) Consumer validates window
stats at end of interval
11
22
33
44
55Store

Data Duplication: How to minimize at consumer
Client: Producer
Kafka Broker
Topic: app_events
If possible,
lookup last
processed offset
in destination at
startup

Monitoring

Monitoring and Operations: JMX Metrics
Producer JMX Consumer JMX

Questions?

Jayesh Thakrar
jthakrar@conversantmedia.com

Data Loss and Duplication in Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Loss and Duplication in Kafka

Similar to Data Loss and Duplication in Kafka (20)

More from Jayesh Thakrar

More from Jayesh Thakrar (7)

Data Loss and Duplication in Kafka