This document discusses BIGO's evolution from using open-source Kafka to Apache Pulsar for its real-time messaging system. It describes the challenges BIGO faced with Kafka as data scales rapidly grew, including poor scalability and degraded I/O performance. BIGO chose Pulsar for its lightweight horizontal scalability, excellent read-write isolation, and ability to support over a million topics. Typical application scenarios discussed include high throughput event tracking, lightweight traffic balancing, and high performance catch-up reads for machine learning tasks. Future work may involve optimizations for different read/write models and combining SSD and HDD storage.

1. The Evolution of Trillion-level Real-
time Messaging System in BIGO
Zhanpeng Wu, Staff engineer at BIGO

2. Agenda
• BIGO Messaging System Introduction
• Challenges, Opportunities, and Choices
• Typical Application Scenarios
• High Throughput Event Tracking System
• Lightweight Traffic Balancing in DevOps
• High Performance Catch-up Read in Machine Learning
Tasks
• Conclusions
• Future Work

3. BIGO Messaging System Introduction

4. Development Timeline
2019
Pure Open-source
Kafka Solution
BIGO Messaging System Introduction
2021
5+ Pulsar Production Clusters
2022
Switch to Pulsar on
A Large Scale
2020
Introduced Pulsar
for the first time

5. Messaging System
Platform
BIGO Messaging System Introduction
APPs
Event Data Realtime Service Data
RecSys Biz Ad Monitoring
DB Bin-log Data
SQL KV
Others
Streaming System
Distributed Storage Computation Data Warehouse Others Applications

6. Challenges, Opportunities, and Choices

7. Challenges in Open-source Kafka Framework
Rapidly growing data scale
Challenges, Opportunities, and Choices
1
2
4
5
0
1
3
4
5
6
2019 2020 2021 2022
BIGO Messaging System Data Scale per Day

8. Challenges in Open-source Kafka Framework
• Poor scalability when encountering a sudden increase in traffic
• The I/O performance is being deteriorate as the #partitions increases
• Mixed reading and writing during catch-up read leading performance
degradation
• …
Challenges, Opportunities, and Choices

9. Why We Decided to Choose Apache Pulsar
• Lightweight and seamless horizontal scalability
Challenges, Opportunities, and Choices

10. Why We Decided to Choose Apache Pulsar
• Excellent read-write isolation effect
provided by Apache BookKeeper
• Ability to support over a million topics
in production env
Challenges, Opportunities, and Choices
BookKeeper I/O Isolation

11. Why We Decided to Choose Apache Pulsar
• Multiple subscription
modes for topics
Challenges, Opportunities, and Choices

12. Typical Application Scenarios

13. High Throughput Event Tracking System
Serious Difficulties:
• High throughput
• Huge amount of producers
Typical Application Scenarios
EU
AS
NA
BE-1
BE-2
Topic-A-partition-0
Topic-A-partition-1
Topic-B-partition-0
… …
…
100TB+ / 50Bil+ Msg-In per Day for one App
300K+ producers among topics for one App

14. High Throughput Event Tracking System
Typical Application Scenarios
Pulsar ETL System
Topic-A-partition-0
Topic-A-partition-1
Topic-B-partition-0
…
Flink job with Pulsar Flink Connector
Event-A
Event-B
…
Data Warehouse
ETL task
ETL task
…

15. High Throughput Event Tracking System
Entries from different ledgers are aggregated and written
sequentially, while their offsets are kept as pointers in a
ledger cache for fast lookup.
Typical Application Scenarios

16. High Throughput Event Tracking System
Kafka V.S. Pulsar: Catch-up read under high throughput env
Typical Application Scenarios
Pulsar has a more efficient catch-up capability while maintaining the stability of the
cluster.

17. High Throughput Event Tracking System
Optimization of adding a large number of producers in a short time
Typical Application Scenarios
https://github.com/apache/pulsar/pull/12
202
No exceptions after lock optimization even frequently reloading bundles
No response to R/W due to deadlock

18. Lightweight Traffic Balancing in DevOps
Cluster Scaling-up in Kafka
Typical Application Scenarios
Data is replicated across the disks and networks

19. Lightweight Traffic Balancing in DevOps
Pulsar-side Balancing
Typical Application Scenarios
A common shedding strategy is to determine whether it needs to be balanced by
comprehensively calculating multiple indicators to obtain a score

20. Lightweight Traffic Balancing in DevOps
Pulsar-side Balancing – Throughput-aware Shedding Optimization
Typical Application Scenarios
Focus on the traffic of the cluster to avoid redundant reload operations.
Be aware that even if it is lightweight, frequent reload will hurt performance

21. Lightweight Traffic Balancing in DevOps
Typical Application Scenarios
BookKeeper-side Balancing
Bookies X and Y are instantly
discovered by broker 2.
The traffic instantly ramps up on the
newly added bookies, without
recopying any data.

22. High Performance Catch-up Read in Machine Learning Tasks
The relationship between the #partitions and the degree of parallelism (Kafka)
Typical Application Scenarios
Topic-A
Topic-A-partition-0
Volume: ■■□
Topic-A-partition-1
Volume: ■■■
Topic-A-partition-2
Volume: ■□□
Group-1
Consumer-0
Consumer-1
Consumer-2
Consumer-3
ML Task
Worker-0
Worker-1
Worker-2
Worker-3
IDLE
Data
Skew

23. High Performance Catch-up Read in Machine Learning Tasks
The relationship between the #partitions and the degree of parallelism (Pulsar)
Typical Application Scenarios
Topic-A
Topic-A-partition-0
Volume: ■■□
Topic-A-partition-1
Volume: ■■■
Topic-A-partition-2
Volume: ■□□
Subscription-2
Consumer-0
Consumer-1
Consumer-2
Consumer-3
ML Task
Worker-0
Worker-1
Worker-2
Worker-3
Dispatcher
Shared Mode

24. High Performance Catch-up Read in Machine Learning Tasks
Ability to support huge amount of consumers (~200K+)
Typical Application Scenarios
With the growth of business, #consumers connected to the cluster has
increased to 200K+, and consumption performance has not significantly
affected.

25. >>>> Message Publish Timeline >>>>
High Performance Catch-up Read in Machine Learning Tasks
Efficient data replay with concurrent multitasking
Typical Application Scenarios
Data replay for a
specified time
period

26. Conclusions

27. • Valuable horizontal scalability in the production environment.
• Thanks to the I/O isolation, message writing and catch-up reading both
show the characteristics of low latency and high throughput.
• Producers, consumers, and topics all have excellent capacity expansion
capabilities
Conclusions

28. Future Work

29. • Combining SSD and HDD to ensure read and write performance while
reasonably controlling hardware costs.
• Latency optimization for different R/W models (read amplification, full
sequential reading, full catch-up reading, etc.).
Future Work

30. • Apache Pulsar Website: https://pulsar.apache.org/
• Apache Kafka Website: https://kafka.apache.org/
• https://jack-vanlightly.com/sketches/2018/10/2/kafka-vs-pulsar-rebalancing-sketch
• https://bookkeeper.apache.org/distributedlog/technical-review/2016/09/19/kafka-vs-
distributedlog.html
• https://www.splunk.com/en_us/blog/it/comparing-pulsar-and-kafka-how-a-segment-based-
architecture-delivers-better-performance-scalability-and-resilience.html
References

31. THANK YOU FOR
WATCHING
Zhanpeng Wu, Staff engineer at BIGO
The Evolution of Trillion-level Real-time Messaging System in BIGO