The document discusses stream processing in cloud computing, specifically using the Storm framework, which is a fast, scalable, and fault-tolerant distributed system for real-time computation. It explains key concepts like streams, spouts, bolts, and topologies, outlining how to set up and operate a Storm cluster. Additionally, it covers advanced topics such as distributed RPC and transactional topologies, and showcases real-time analytics applications like Twitter sentiment analysis.

1. STORM ANATOMY
Cloud Computing Course
Prof Hanku Lee
Social Media Cloud Computing lab
MSAkhmedov Khumoyun

2. What is Stream processing
 Stream processing is a technical paradigm to process big volume of
unbound sequence of tuples in realtime
= stream
Source Stream Processor
• Continuous analytics
• Online machine learning
• Sensor data monitoring
• Financial trading …

3. Storm at Twitter
Twitter Web Analytics

4. What is Storm?
Storm is
• Fast & scalable
• Fault-tolerant
• Guarantees messages will be processed
• Easy to setup & operate
• Free & open source
distributed realtime computation system
- Originally developed by Nathan Marz at
BackType (acquired by Twitter)
- Written in Java and Clojure

5. Conceptual View

6. Physical View

7. Concepts
 Streams
 Spouts
 Bolts
 Topologies

8. Streams
Unbounded sequence of tuples

9. Spouts
Source of streams
• Read from Kafka queue
• Read from Twitter Streaming API

10. Bolts
Processes input streams and produces new streams

11. Bolts
• Functions
• Filters
• Aggregation
• Joins
• Talk to databases

12. Topology
Network of spouts and bolts

13. Tasks
Spouts and bolts execute as
many tasks across the cluster

14. Stream grouping
When a tuple is emitted, which task does it go to?

15. Stream grouping
• Shuffle grouping: pick a random task
• Fields grouping: consistent hashing on a
subset of tuple fields
• All grouping: send to all tasks
• Global grouping: pick task with lowest id

16. Starting topology

17. Starting topology

18. Storm : Fault-tolerance

19. Storm : Fault-tolerance

20. Storm : Fault-tolerance

21. Storm : Fault-tolerance

22. Storm : Fault-tolerance

23. Guarantees messages
will be processed

24. Message Passing (ZeroMQ)

25. Easy to setup & operate
• Setup ZooKeeper cluster
• Install dependencies on Nimbus and worker
machines
- ZeroMQ 2.1.7 and JZMQ
- Java 6 and Python 2.6.6
- unzip
• Download and extract a Storm release to Nimbus
and worker machines
• Fill in mandatory configuration into storm.yaml
• Launch daemons under supervision using “storm”
script

26. Cluster Summary

27. Topology Summary

28. Component Summary

29. Advanced Topics
• Distributed RPC
• Transactional topologies
• Trident
• Using non-JVM languages with Storm
• Unit testing
• Patterns

30. Real-time TwitterAnalytics
Trending Topics and SentimentAnalysis
Twitter
MySQL
Kafka
Storm Cluster
Hadoop (HDFS and HBase )
Twitter Crawler

32. THANK YOU FOR ATTENTION
Any Questions AreWelcome…