Download as PDF, PPTX
![stream
extract
hashtags
Constant time and space
approximate frequent
itemset
countmin
sketch
[Cormode and Muthukrishnan, 2005]
File Sink](https://image.slidesharecdn.com/slides-131212113520-phpapp02/75/Development-of-a-Distributed-Stream-Processing-System-39-2048.jpg)
Uploaded byMaycon Viana Bordin
Development of a Distributed Stream Processing System
The document discusses the development of a distributed stream processing system, detailing the architecture, operator classification, and the implementation framework for a trending topics application. It highlights key concepts such as stream sources, operators, data windowing, and the pub/sub communication pattern between operators. Performance metrics are analyzed through experiments comparing runtime, latency, and throughput under varying conditions, emphasizing the trade-offs between data processing efficiency and latency when scaling nodes.
More Related Content
Similar to Development of a Distributed Stream Processing System
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Development of a Distributed Stream Processing System
- 1. Development of aDistributed Stream Processing System Maycon Viana Bordin Final Assignment Instituto de Informática Universidade Federal do Rio Grande do Sul CMP157 – PDP 2013/2, Claudio Geyer
- 2.
- 4. Stream Source: emits datacontinuously and sequentially
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. B Tuples are orderedby a timestamp or other attribute 7 6 5 4 3 2 1
- 11. Data from thestream source may or may not be structured
- 12. The amount ofdata is usually unbounded in size
- 13. The input rateis variable and typically unpredictable
- 14.
- 15.
- 16. Receives one or moredata streams OP Sends one or more data streams
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. OPERATORS Stateless Stateful (map, filter) Blocking Non-Blocking (join, freq.itemset) (count, sum)
- 23. Blocking operators needall input in order to generate a result
- 24. but that’s notpossible since data streams are unbounded
- 25. To solve thisissue, tuples are grouped in windows
- 26. Range in timeunits or number of tuples window start (ws) window end (we)
- 27. old ws old we advance newws new we
- 28.
- 29.
- 30. The heartbeat carriesthe status of each worker in the slave
- 31. Tuples processed Throughput Latency The heartbeatcarries the status of each worker in the slave
- 32.
- 33. Applications are composedas a DAG (Directed Acyclic Graph)
- 34. To illustrate, let’slook at the graph of a Trending Topics application
- 35.
- 36. stream Stream source emits tweetsin JSON format extract hashtags countmin sketch File Sink
- 37. Extract the textfrom the tweet and add a timestamp to each tuple stream extract hashtags countmin sketch File Sink
- 38. stream extract hashtags Extract and emit each#hashtag in the tweet countmin sketch File Sink
- 39. stream extract hashtags Constant time andspace approximate frequent itemset countmin sketch [Cormode and Muthukrishnan, 2005] File Sink
- 40. stream extract hashtags Without a window,it will emit all top-k items each time a hashtag is received countmin sketch File Sink
- 41. stream extract hashtags With a windowthe number of tuples emitted is reduced, but the latency is increasead countmin sketch File Sink
- 42. The second stepin building an application is to set the number of instances of each operator:
- 43.
- 44. But the userhas to choose the way tuples are going to be partitioned among the operators
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60. The communication between operatorsis done with the pub/sub pattern
- 61.
- 62. stream extract extract extract extract extract U countmin countmin countmin countmin countmin The operator subscribes toall upstream operators, with his ID as a filter File Sink
- 63. stream extract extract extract extract extract U countmin countmin countmin countmin countmin The operator willonly receive tuples with his ID as prefix File Sink
- 64. The last stepis to get each operator instance from the graph and assign it to a node
- 65.
- 66. Currently the scheduleris static and only balances the number of operators per node
- 67.
- 68.
- 69.
- 70. Application Trending Topics Dataset of40GB from Twitter
- 71. Test Environment GridRS -PUCRS 3 nodes 4 x 3.52 GHz (Intel Xeon) 2 GB RAM Linux 2.6.32-5-amd64 Gigabit Ethernet
- 72. Metrics Runtime Latency: time toa tuple traverse the graph Throughput: no. of tuples processed per sec. Loss of Tuples Methodology 5 runs per test. Every 3s each operator sends its status with no. of tuples processed. The PerfMon sink collects a tuple every 100ms, and sends the average latency every 3s (and cleans up the collected tuples). Variables Number of nodes Number of operator instances Window size
- 73.
- 74. Runtime vs Latency 90.00 2000.00 80.00 1800.00 1600.00 70.00 1400.00 1200.00 50.00 1000.00 40.00 800.00 30.00 600.00 20.00 400.00 10.00 runtime(min) 200.00 latency (ms) 0.00 0.00 1 2 No. of nodes 3 Latency (ms) Runtime (minutes) 60.00
- 75. Runtime vs StreamRate 90.00 25.00 80.00 20.00 70.00 15.00 50.00 40.00 10.00 30.00 20.00 5.00 10.00 runtime (min) stream rate (MB/s) 0.00 0.00 1 2 No. of nodes 3 Stream rate (MB/s) Runtime (minutes) 60.00
- 76. Throughput 7000.00 stream extractor 6000.00 countmin filesink perfmon Tuples per second(tps) 5000.00 4000.00 3000.00 2000.00 1000.00 0.00 1 2 No. of nodes 3
- 77. Loss of Tuples 10000.00 stream extractor countmin 8000.00 filesink perfmon Losttuples 6000.00 4000.00 2000.00 0.00 1 -2000.00 2 No. of nodes 3
- 78. Throughput and LatencyOver Time (nodes=3, instances=5, window=20) 14000 100000 stream 90000 extractor countmin 80000 10000 latency 70000 60000 8000 50000 6000 40000 30000 4000 20000 2000 10000 0 0 3 76 117 158 207 274 315 356 417 466 507 548 609 657 698 738 811 852 893 936 1008 1048 1089 1156 1197 1238 1308 1348 1389 1446 1501 1542 1583 1617 1657 1698 1728 1759 1800 1838 1865 1899 1939 1980 2011 2038 2079 Throughput (tuples/ps) filesink Time (seconds) Latency (ms) 12000
- 79.
- 80. Runtime vs Latency 27.20 700.00 27.00 600.00 26.80 Runtime(minutes) 26.40 400.00 26.20 300.00 26.00 25.80 200.00 25.60 100.00 runtime (min) 25.40 latency (ms) 25.20 0.00 20 80 120 Window Size 200 Latency (ms) 500.00 26.60
- 81.
- 82. Runtime vs StreamRate 40 35 35 30 30 25 20 20 15 15 10 10 5 5 runtime (min) stream rate (MB/s) 0 0 1 5 No. of Instances Stream rate (MB/s) Runtime (minutes) 25
- 83.
- 84. The system wasable to process more data with the inclusion of more nodes
- 85. On the otherhand, distributing the load increased the latency
- 86. The scheduler hasto reduce the network communication
- 87. The communication betweenworkers in the same node has to happen through main memory
- 88. References Chakravarthy, Sharma. Streamdata processing: a quality of service perspective: modeling, scheduling, load shedding, and complex event processing. Vol. 36. Springer, 2009. Cormode, Graham, and S. Muthukrishnan. "An improved data stream summary: the count-min sketch and its applications." Journal of Algorithms 55.1 (2005): 58-75. Gulisano, Vincenzo Massimiliano, Ricardo Jiménez Peris, and Patrick Valduriez. StreamCloud: An Elastic Parallel-Distributed Stream Processing Engine. Diss. Informatica, 2012. Source code @ github.com/mayconbordin/tempest