Comparison of Processing Frameworks. Project done in three weeks for Data Insight

1. Testing Processing Frameworks
Streaming
and
Gabriela Choy

2. Spark / Storm
Spark Storm
Implemented in Scala Clojure, Java
Delivery Semantics
Exactly once At least once. Exactly
once with Trident
API
Python, Java, Scala Java, Scala, Clojure,
Python, etc. Trident:
Java, Scala, Clojure.
Processing Model
Batch. Micro-batches
with Spark Streaming.
~ 500ms
Record at a time/
Trident allows for
micro-batches.
Latency 1 - 2 seconds sub-seconds

3. Pipeline
Streaming
MySQL
MySQL
Producer/s
Producer/s
Each pipeline is run
independently

4. Node Specifications
Spark Streaming Storm
4 AWS nodes m3.
medium
Zookeeper 3.4.6
Kafka 0.8.2.1
Spark (streaming) 1.3
4 AWS nodes m3.
medium
Zookeeper 3.4.6
Kafka 0.8.2.1
Storm 0.9.5

5. Spark Streaming: 1 master node, 3 workers
Cluster Configuration
Master node
Worker 1
Worker 2
Worker 3

6. Storm : 1 nimbus, 3 Supervisors
Cluster Configuration
Nimbus
Supervisor 1
Supervisor 2
Supervisor 3

7. Metric
Throughput: amount of data that is being
processed.
● By changing batch size
● By changing load (i.e. Scaling up)
● Programs used for benchmarking will be wordcount.

8. # Producers Batch Interval
1 1s, 2s, 3s, 4s, 6s
4
1s, 2s, 3s, 4s, 6s
8
1s, 2s, 3s, 4s, 6s
Tests for Spark Streaming

9. Throughput for 1 producer with 95% CI

10. Throughput for 1 producer

11. Throughput for 4 producers with 95% CI

12. Throughput for 4 producers

13. Throughput for 8 producers

14. Throughput for 8 producers

15. # Producers Tuples Emitted-Acked
1 10 min
4 10 min
8 10 min
Tests for Storm
Preliminary results for storm

16. Tuples Emitted per Second

17. Tuples Acked per Second

18. Spout Latency

19. Takeaways
● Setting the batch interval in spark streaming
should be done by monitoring processing times
and load size
● For Storm as numbers of producers increase so
does throughput and spout latency.

20. Would like to add:
● Increase number of producers. Use real data.
● Add a graph as a second use case.
● Dashboard to monitor live streaming.

21. Gabriela Choy
Bsc. in Chem. Engineering. ULA, Vnzla.
Msc in Statistics. UT Dallas
Previously: Worked in Device Reliability Engineering at View, Inc.
About Me