Kafka 為世界級熱門的大數據事件串流處理系統，本次分享將介紹 Kafka 的使用方式、應用案例、以及遇到Kafka疑難雜症時可參考的線上資源。

1. 蔡嘉平
聽說隔壁部⾨開始⽤
KAFKA ???

2. ABOUT 蔡嘉平
www.chia7712.tw
PS5
2
① 技術顧問 @ 原昌、亦思、教育部⼈培
② Committer | PMC @ Kafka、HBase、Yunikorn
③ Member @ Apache
④ PC Member @ Kafka Summit, HBaseConAsia

3. DATA OR
EVENT?
3

4. ACACHE KAFKA
C Clojure C-sharp/.Net Go
Groovy Java Scala Kotlin
Node.js Python Ruby Rust
4
來⾃linkedin
Kafka商業公司
定時舉辦國際
⼤量的貢獻者
Client Library
⾼擴充、⾼吞吐、低延遲

5. PRODUCER AND CONSUMER
Topic
Your Application
Producer
Your Application
Consumer
搜集資料 Push Poll 處理和發佈資料
Your Application
Producer
Your Application
Producer
Your Application
Consumer
Your Application
Consumer
5
A. 設定起始節點
B. 將資料送出
1. 設定起始節點
2. 訂閱topic
3. 開始讀資料

6. 動態擴增訊息吞吐量
Topic A
由1台電腦和1顆硬碟來服務
Topic A
由1台電腦和2顆硬碟來服務
擴充硬碟資源
Topic A
由2台電腦和3顆硬碟來服務
擴充CPU、硬碟和頻寬
6
擴充過程服務都不需要下線 CPU、硬碟、和頻寬皆可⽔平擴充

7. PARTITION REPLICAS
broker_0
Disk A
broker_1 broker_2
Disk B Disk C Disk D Disk E Disk F
Topic A partition 0
Topic B partition 0
Topic C
partition 0
leader
partition 1
partition 1
partition 1
partition 2
partition 2 partition 3
partition 3
partition 4
partition 4
partition 0
follower
7
⼀個Topic
=
多個partitions
副本可確保資料可⽤性
Partition越多越分散

8. 隨時動態調整參數
ssl.keystore.type
ssl.truststore.location
ssl.keystore.certi
fi
cate.chain
ssl.keystore.password
ssl.key.password
ssl.keystore.location
ssl.keystore.key
ssl.truststore.password
ssl.truststore.certi
fi
cates
ssl.truststore.type
log.cleaner.io.buffer.size
log.cleaner.threads
log.cleaner.backoff.ms
log.cleaner.dedupe.buffer.size
log.cleaner.io.max.bytes.per.second
message.max.bytes
log.cleaner.io.buffer.load.factor
log.
fl
ush.interval.ms
log.cleaner.min.compaction.lag.ms
log.retention.ms
log.index.size.max.bytes
log.roll.ms
compression.type
min.insync.replicas
log.preallocate
log.cleaner.min.cleanable.ratio
log.roll.jitter.ms
log.segment.bytes
log.cleaner.max.compaction.lag.ms
log.cleaner.delete.retention.ms
log.
fl
ush.interval.messages
log.cleanup.policy
unclean.leader.election.enable
log.segment.delete.delay.ms
log.index.interval.bytes
log.message.timestamp.type
log.message.timestamp.difference.max.ms
log.retention.bytes
message.max.bytes
log.message.downconversion.enable
num.io.threads
num.replica.fetchers
num.recovery.threads.per.data.dir
background.threads
metric.reporters
sasl.login.refresh.buffer.seconds
max.connections
principal.builder.class
listener.security.protocol.map
listeners
num.network.threads
ssl.truststore.location
ssl.trustmanager.algorithm
ssl.cipher.suites
ssl.key.password
advertised.listeners
ssl.enabled.protocols
sasl.kerberos.service.name
ssl.client.auth
ssl.keymanager.algorithm
sasl.kerberos.min.time.before.relogin
ssl.truststore.type
sasl.kerberos.ticket.renew.window.factor
sasl.kerberos.ticket.renew.jitter
ssl.secure.random.implementation
ssl.engine.factory.class
sasl.enabled.mechanisms
ssl.keystore.type
ssl.protocol
ssl.provider
max.connection.creation.rate
sasl.kerberos.principal.to.local.rules
ssl.truststore.password
sasl.login.refresh.window.jitter
ssl.endpoint.identi
fi
cation.algorithm
sasl.login.refresh.min.period.seconds
ssl.keystore.password
ssl.keystore.location
sasl.jaas.con
fi
g
sasl.kerberos.kinit.cmd
sasl.mechanism.inter.broker.protocol
sasl.login.refresh.window.factor
max.connections.per.ip
max.connections.per.ip.overrides
max.connections
max.connection.creation.rate
Security Log cleaner Log Thread Listener Socket Reporter
8
Compaction Topic
Topic的參數 Partitions/replicas
效能參數
動態調整網路入⼝
監控metrics

9. GROUP_0
CONSUMER AND AVAILABILITY
Topic A
由1台電腦和2顆硬碟來服務
Your Application
Consumer
Your Application
Consumer
Your Application
Consumer
Consumer Partition Assignment
Your Application
Consumer
dead consumer
….
Partition 0 Partition 1 Partition 2
9
⾃動分配給其他活著的consumer

10. THROTTLE
broker_0
Disk A
broker_1
Disk B Disk C
Topic
partition 0
leader
partition 1
leader
partition 0
follower
partition 1
follower
限速5000
leader.replication.throttled.rate=5000 follower.replication.throttled.rate=4000
leader.replication.throttled.replicas=0:0
(partition:broker-id)
Server-Level
此節點的leaders同步副本的吞吐量上限
Server-Level
此節點的followers同步副本的吞吐量上限
Topic-Level
哪些leaders要被限速
限速4000
不限速 不限速
partition 2
future
partition 2
follower.replication.throttled.replicas=0:1
(partition:broker-id)
Topic-Level
哪些followers要被限速
replica.alter.log.dirs.io.max.bytes.per.second=1000
Sever-Level
內部⽬錄移動的速度上限
限速1000
10

11. DATA TO EVENT STREAMING
撈取串流資料
+
定期批次處理
CSV 檔案
CSV檔案轉成
Spark streaming
背景執⾏緒進⾏第⼆次去重
第三次去重
+
MERGE INTO
雲端Spark
第⼀次去重
+
轉成Kafka資料
Data Lake
Kafka streaming
地端Kafka
Kafka streami
推播給下⼀個單位
11

12. 開源專案：Astraea
https://github.com/skiptests/astraea
① Kafka 負載平衡⼯具
② 更現代化的Kafka APIs
③ Web Service over Kafka APIs
④ 效能評測⼯具
⑤ 簡單好⽤的圖形化軟體
12

13. ① 選擇適當的成本
② 給定成本的權重
1. 寫入端的⾃動化負載平衡
① 選擇適當的成本
② 給定成本的權重
2. 伺服器端的⾃動化負載平衡
① 選擇適當的成本
② 給定成本的權重
3. 讀取端的⾃動化負載平衡
https://github.com/skiptests/astraea
吞吐量
延遲
連線數量
leader數量
Partitions數量
資料量
多樣化的成本選擇
⼀致的使⽤⽅式
很⼀致所以說三次!!!
13

14. 萬物皆有數字
1. 給定⼀個叢集描述
broker_0
Disk A
broker_1
Disk B Disk C
Topic A partition 0
Topic B partition 0 partition 1
partition 1
cost function {}
partition 2
2. 輸出叢集各項分數
( 這個叢集本⾝也是⼀個數字 )
broker_0
Disk A
broker_1
Disk B Disk C
Topic A partition 0
Topic B partition 0 partition 1
partition 1
partition 2
50 70
20
20
40
50 60
30 30 20
叢集資訊
伺服器效能指標
寫入端效能指標
讀取端效能指標
14

15. 平均吞吐提升：約 1.98 倍
15

16. 感謝聆聽、歡迎拷問