2nd version

1. Streaming Benchmark
Vinaya M S
Insight Data Science

2. m4.large: 3
r3.large: 4
r3.large: 4
r3.large: 1

3. Program
Read
tweets
Parse
tweets
Count
characters
Store
(r_ts, w_ts)

4. How many tweets are processed/second?
Insert read_ts Insert write_ts
100
tweets/second
Subset of 100
tweets/second

5. Latency:
∑ (𝑤𝑟𝑖𝑡𝑒() − 𝑟𝑒𝑎𝑑())./0102
34./056786 / total_tweets_processed
winStart: window start time; winEnd: window End time

6. RESULTS

9. What about storm?

12. • Total number of tweets processed in storm
are more.
•~41000 in 160sec (Flink: ~18500 in 160sec)

13. Storm:
Maximise per window based throughput/latency??
•Needs performance tunings.

14. Few tunings I considered
§Number of compute bolts
§Consumer spouts.
§Java heap size. Plays an important role.
§Tried 2 of the groupings.

15. Challenges:
•Visualizing the storm data distribution.
•NTP server sync.
•SQL queries.

16. About Me
3 years experience.
MS- Computer Science
Enjoy:
Table Tennis
Cooking

17. Thank you 😊

18. Recommendation
Storm
• Low level apis:
• builder.setBolt("parse-bolt", new
ParseTweetBolt(), parallel*3)
.shuffleGrouping("tweet-spout");
• Better understanding of architecture
is required.
• Configuration tunings
Flink
• Hides details:
• flatMap(new ParseTweetBolt())

19. • Storm: Round robin scheduling.
• Flink: Each task slot in task manager can run one pipeline of parallel
task.

20. Storm: