The document discusses challenges and solutions related to big data streams and provides an overview of Kafka, highlighting its integration with various big data technologies. It covers aspects such as batch processing, compression techniques, and structural optimization to improve data handling efficiency. The author emphasizes the significance of high throughput and storage cost in big data ecosystems.

1. Performance and
BigData dongeforever@apache.org

2. dongeforever
Apache RocketMQ PMC/Committer.
Interested in algorithms and
performance.
Now spending more time in open
source, including MQ and BigData

3. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

4. Challenge of Big Data Stream

5. Challenge of Big Data Stream
❖ High Throughput —— Million TPS
❖ IO BandWidth —— Network & Disk
❖ Storage Cost

6. Kafka Overview
❖ Open sourced in early 2011, graduate from the Apache
Incubator on 23 October 2012
❖ Log Aggregation, Messaging, Stream Processing, Event
Sourcing
❖ Widely used in BigData processing, integrate with
Storm, Spark, Flink, Samza, Hadoop, Flume, etc.

7. Kafka Domain Model

8. Kafka Store Structure

9. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

10. Batch Through

11. Kafka Producer TPS vs Msg Size

12. Proxy Producer TPS vs Msg Size

13. RocketMQ Batch
❖ send(Collection<Message> msgs)
❖ Atomic
❖ Get 150w TPS
❖ https://rocketmq.apache.org/docs/batch-example/

14. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

15. Compression Through
• Clients handle the Compress/DeCompress
• Only Need One Operation in Broker

16. Compression Through
• For log collection, about 5~10 compression ratio

17. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

18. Structural Compression

19. Structural Compression

20. Structural Compression
❖ Assume each msg has N bytes, each batch has B msgs
❖ Size of 0.10: (34 + N) * B
❖ Size of 0.11: 61 + (7 + N) * B
❖ For N <= 100, save storage upper to 20%~50%

21. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

22. External ecosystem

23. External ecosystem
https://github.com/apache/rocketmq-externals