Jamie Grier presents findings on extending the Yahoo! streaming benchmark for Apache Flink, showcasing its capabilities compared to Storm and Spark. His benchmarks demonstrate that Flink achieved 15 million messages per second with stronger processing guarantees, significantly outperforming previous results. The presentation also highlights Flink's compatibility with existing Storm applications, allowing for easy migration.

1. Extending the Yahoo!
Streaming Benchmark
Jamie Grier
@jamiegrier
jamie@data-artisans.com

2. Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

3. Overview
• Yahoo! performed a benchmark comparing
Apache Flink, Storm and Spark
• The benchmark never actually pushed Flink to it’s
throughput limits but stopped at Storms limits
• I knew Flink was capable of much more so I
repeated the benchmarks myself
• I did a follow up blog post explaining my findings
and will summarize them here

4. Yahoo! Benchmark
• Count ad impressions grouped by campaign
• Compute aggregates over a 10 second window
• Emit current value of window aggregates to
Redis every second for query
• Map ads to campaigns using Redis as well

5. Any questions so far?

6. Storm Code

7. Flink Code

8. Hardware Specs
• 10 Kafka brokers with 2 partitions each
• 10 compute nodes (Flink / Storm)
• Each machine has 1 Xeon E3-1230-V2@3.30GHz CPU
• 4 cores w/ hyperthreading
• 32 GB RAM (only 8GB allocated to JVMs)
• 10 GigE Ethernet between compute nodes
• 1 GigE Ethernet between Kafka cluster and compute nodes

9. Logical Deployment
Data
Generat
or
Kafka Source Filter Project Join
Redis
Windo
w
Sink Redis
Stream Processor

10. Redis
Apache Storm
Deployment
Kafka
Kafka
Kafka
Source Filter Project Join Window Sink
Flink
Data Generator
Redis
Shuffle
Apache Storm
10 Gige Link
1 Gige Link

11. Redis
Kafka
Kafka
Kafka
Source Filter Project Join Window Sink
Flink
Data Generator
Redis
Shuffle
10 Gige Link
1 Gige Link

12. Redis
Kafka
Kafka
Kafka
Source / Filter Project Join Window Sink
Flink
Data Generator
Redis
Shuffle
10 Gige Link
1 Gige Link

13. Redis
Kafka
Kafka
Kafka
Source / Filter / Project Join Window Sink
Flink
Data Generator
Redis
Shuffle
10 Gige Link
1 Gige Link

14. Redis
Kafka
Kafka
Kafka
Source / Filter / Project / Join Window Sink
Flink
Data Generator
Redis
Shuffle
10 Gige Link
1 Gige Link

15. Redis
Kafka
Kafka
Kafka
Window / Sink
Flink
Data Generator
Redis
Shuffle
Source / Filter / Project / Join
10 Gige Link
1 Gige Link

16. Redis
Kafka
Kafka
Kafka
Flink
Data Generator
Redis
Shuffle
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link

17. Redis
Kafka
Kafka
Kafka
Flink
Data Generator
Redis
Shuffle
Apache Flink
Deployment
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link

18. Processing Guarantees
Apples and Oranges
Apache Storm Apache Flink
At least once
semantics
Exactly once
semantics
Double counting after
failures
No double counting
Lost state after
failures
No state loss

19. Benchmark
0 750,000 1,500,000 2,250,000 3,000,000 3,750,000
Storm
Flink
Throughput: msgs/sec
Baseline

20. Bottleneck Analysis
Apache Storm
Kafka
Kafka
Kafka
Source Filter Project Join Window Sink
Flink
Data Generator
Shuffle
Apache Storm
10 Gige Link
1 Gige Link
Redis
Redis

21. Bottleneck Analysis
Apache Storm
Kafka
Kafka
Kafka
Source Filter Project Join Window Sink
Flink
Data Generator
Shuffle
Apache Storm
10 Gige Link
1 Gige Link
Redis
Redis
CPU

22. Redis
Kafka
Kafka
Kafka
Flink
Data Generator
Redis
Shuffle
Bottleneck Analysis
Apache Flink
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link

23. Redis
Kafka
Kafka
Kafka
Flink
Data Generator
Redis
Shuffle
Bottleneck Analysis
Apache Flink
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link
Network

24. Redis
Kafka
Kafka
Kafka
Flink
Data Generator
Redis
Shuffle
Eliminate the
Bottleneck
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link

25. Redis
Flink
Data Generator
Redis
Shuffle
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link
Eliminate the
Bottleneck

26. Redis
Redis
Shuffle
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link
Data
Generator
Eliminate the
Bottleneck

27. Redis
Redis
Shuffle
Apache Flink
Window / SinkSource / Filter / Project / Join
10 Gige Link
1 Gige Link
Data
Generator
Apache Flink
Deployment
Round 2

28. Benchmark
0 750,000 1,500,000 2,250,000 3,000,000 3,750,000
Storm
Flink
Throughput: msgs/sec
Baseline

29. Benchmark
Round 2
0 4,000,000 8,000,000 12,000,000 16,000,000
Storm
Flink
Flink (10 GigE)
Throughput: msgs/sec
10 GigE end-to-end

30. Results
• Apache Flink achieved 15 million messages / sec
on Yahoo! benchmark
• Much stronger processing guarantees: Exactly
once
• 80x higher than what was reported in the original
Yahoo! benchmark on similar hardware

31. Questions?

32. Storm Compatibility
• Lot’s of companies already have applications written
using the Storm API
• Flink provides a Storm compatibility layer
• Run your Storm jobs on Flink with a one line code
change
• Flink also allows you to reuse your existing Storm
spout and bolt code from a Flink job
• Give it a try!

33. Thanks!