Kafka practical experience

1. Kafka Practical
Experience
RiCo Chen

2. Agenda
◎Kafka Overview
◎AFT Kafka Architecture
.Net Client -Pub/Sub
Key Terms Introduction
Message Delivery
◎Monitor Kafka Architecture
◎Kafka Performance Tuning
Producer
Broker
Consumer
JVM

3. Kafka Overview
• High-performance distributed streaming platform
• Popular project on githup: star=6389 , fork=3909
• Simple installation、Easy scale-out、More resource(Linux)
• High availability、High failover、High reliability、Auto
balance、Message persistence..
• Use Cases:Log Aggregation、Metrics、Event Sourcing、
Messaging…
• Power by: LinkedIn、airbnb、Mozilla、Twitter、LINE、
skyscanner、trivago、Hotel.com、PayPal、Uber、
Yahoo…

4. AFT Kafka Architecture
Kafka cluster(P2P)
broker1 broker2 broker3
zookeeper cluster(Master-Salve)
node1 node2 node3
UGS
API
UGS
WEB
UGS
WinSvr
Producers
Consumers
Serialization
publish message(Batch、
Fire and forget)
Middleware
Consumer
group
Logger
Services
Logger
Services
Logger
Services
deserialization
subscribe message
(message set、
async)
Topic’s configuration、broker status、
Cluster membership、controller process、
Coordinator process
Message(binary) queue、
partition、offset manager、
Leader cache、topic、
ReplicaManager、
GroupCoordinator(rebalance)
Replicat
e
Replicat
e
TCP
TCP
Heartbeat

5. .Net Client -Pub/Sub

6. Key Terms Introduction
• Broker: MQ process(Minimum unit in kafka cluster)
• Topic: Category of message(data is store in)
• Producer: push message to broker(write data)
• Consumer: pull message from broker(read data)
• ConsumerGroup: provide tolerance、 scalability、parallel for
Consumer
• Partition: provide tolerance、 scalability、parallel
• Offset: Message position on each partition

7. Message Delivery
At most once: Messages may be lost but are never redelivered
At least once: Messages are never lost but may be redelivered
Exactly once: Each message is delivered once and only
once(0.11.x)
Messages sent by a producer to a particular topic partition will
be appended in the order they are sent
A consumer instance sees records in the order they are stored in
the log.
Tolerate up to N-1 server failures.(depends replication factors)

8. Monitor Kafka Architecture
Telegraf
http
(every 10 sec)
Influxdb
2.Result
Grafana
JMX
1.QL
http
(every 10 sec)
Kafka-
Manager
Kafka
Eagle
Mysql
2.Result
TCP
1.Collec
t
2.Store
TCP

9. High level architecture blueprint
UGS
Platform
/Producer
Logger
Services
/Consumer
Channel
SQL
Server
Kafka
eagle
/Consumer

10. Kafka Performance Tuning
• Producer
• Broker
• Consumer
• JVM

11. Producer
• Load balancing(sends data directly to the broker that is the leader for the
partition)
• Acks=0 producer no wait any acknowledgment from the broker at all.
Lowest latency at the cost of durability but high data lost.
• Acks=1 producer gets an acknowledgment after the leader wrote the
record to its local log, but will respond without awaiting full
acknowledgement from all followers. Maybe follower will be lost data if
leader commit after.
• Acks=-1 producer gets all acknowledgment after all in-sync replicas has
received the data. Strong guarantee data not be lost.
• batch.size=100 ,net client
• send.buffer.bytes=100*1024
• producer.type=async
• compression.type=none
• max.in.flight.requests.per.connection=3
Note: min.insync.replicas>=2
ACKs Throughput Latency Durability
0 High Low No Gurantee
1 Medium Medium Leader
-1 Low High ISR

12. Broker
• More partition = more concurrent process = more memory =
more io access =increase throughput= increase latency
(brokers have to distribution on each partition) P.S single topic
less than 1024 partitions
• Number of Factors = two brokers at least
num.io.threads=8 num.network.threads=3 background.threads=10
queued.max.requests=500
socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
num.recovery.threads.per.data.dir=2
Log.retention.hours=24 log.flush.interval.messages=10000
log.flush.interval.ms=1000 log.cleanup.policy=delete log.cleaner.enable=true
log.cleaner.threads=1 log.cleaner.backoff.ms=30000
log.segment.bytes=1073741824 replica.fetch.min.bytes=1
replica.high.watermark.checkpoint.interval.ms=5000
replica.fetch.wait.max.ms=500
min.insync.replicas=2

13. Consumer
• Need enough partitions to handle message from producer
• Maximum number of consumer = a multiple of broker(balance
is better)
• max.poll.records=5000
• enable.auto.commit=true
• auto.commit.interval.ms=5000
• fetch.max.wait.ms=500
• fetch.min.bytes=1
• keep small Batch size in our .net client(for real time consumer
data)

14. JVM
• Avoid out of memory
• Avoid high frequency trigger GC
-Xmx8g –Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -
XX:MaxMetaspaceFreeRatio=80 -XX:MinMetaspaceFreeRatio=50 -
XX:G1HeapRegionSize=16M -XX:InitiatingHeapOccupancyPercent=35
-Xms: Set initial Java heap size
-Xmx: Set maximum Java heap size
+UseG1GC: Enable G1 GC
MaxGCPauseMillis: Set maximum pause
MaxMetaspaceFreeRatio: Set maximun metaspace free ratio
MinMetaspaceFreeRatio: Set minimun metaspace free ratio
G1HeapRegionSize: Adjust G1 region on each heap
InitiatingHeapOccupancyPercent: initial Java heap occupancy threshold

15. Q & A

16. Reference
• https://kafka.apache.org/
• https://github.com/apache/kafka
• http://www.oracle.com/technetwork/articles/java/g1gc-
1984535.html
• https://docs.oracle.com/cd/E40972_01/doc.70/e40973/cn
f_jvmgc.htm
• RiCo’s blog