![15
Tuples
[“Colonoscopy”, 14106]
• Storm Data structure
• List of elements](https://image.slidesharecdn.com/storm-introv8-130716154811-phpapp02/75/Introduction-to-Storm-15-2048.jpg)
![16
Stream
Unbounded sequence of tuples
[“Colonoscopy”, 14091][“Cancer”,42651]
[“Oncology”, 14417]](https://image.slidesharecdn.com/storm-introv8-130716154811-phpapp02/75/Introduction-to-Storm-16-2048.jpg)
![63
Example
[Physicians, 79]
[Oncology:78]
[Cancer:237]
…….](https://image.slidesharecdn.com/storm-introv8-130716154811-phpapp02/75/Introduction-to-Storm-63-2048.jpg)
![72
Example
[Physicians, 79]
[Oncology:78]
[Cancer:237]
…….](https://image.slidesharecdn.com/storm-introv8-130716154811-phpapp02/75/Introduction-to-Storm-72-2048.jpg)
Uploaded byEugene Dvorkin
Introduction to Storm
Storm is an open-source distributed real-time computation system. It provides a framework for processing unbounded streams of data reliably and fault-tolerantly. Storm allows data to be analyzed in real-time using spouts, bolts, and topologies. It is scalable, fault-tolerant, guarantees processing, and is easy to code. Storm powers many real-time systems at Twitter and is useful for applications like analytics, personalization, and ETL.
More Related Content
Similar to Introduction to Storm
![[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels](https://cdn.slidesharecdn.com/ss_thumbnails/md-exploringappleson-devicefoundationmodels-251124030840-d690542c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...](https://cdn.slidesharecdn.com/ss_thumbnails/md-mobileengineerandsoftwareengineerarewestillrelevantsidiqpermana-251127010650-55224ef1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DevFest Strasbourg 2025] - NodeJs Can do that !!](https://cdn.slidesharecdn.com/ss_thumbnails/devfeststrasbourg2025-nodejscandothat-251127142731-da65b6fd-thumbnail.jpg?width=640&height=640&fit=bounds)
Introduction to Storm
- 1. 1 Distributed, real-time, fault-tolerantframework Introduction to Storm Eugene Dvorkin Coding Architect, WebMD edvorkin@gmail.com #edvorkin eugenedvorkin.com
- 2. 2 Big Data “Big Datais the capability to manage a huge volume of disparate data, at the right speed, and within the right time frame to allow real-time analysis and reaction”
- 3.
- 4. 4 Enablers of BigData Map/Reduce frameworks – Hadoop Scalable storage – HDFS, NoSQL databases Cheap computing power – Cloud computing
- 5. 5 Why Real Time? Betterend-user experience - Ex: View an ad, see the counter move. Operational intelligence - Low latency analysis - Real time Dashboards ŸEvent response - Rule Engine, Personalization, Predictions - Scalable analysis Example: Trend analysis to recommend „hot‟ articles.
- 6. 6 Requirements Fast Scalable by processparallelization and distribution Fault-tolerant Guarantees data processing Easy to learn, code and operate Robust Doing scalable real time processing require framework that:
- 7. 7 Storm • Storm –open source distributed Real- time computation system. • Developed by Nathan Marz – acquired by Twitter
- 8. 8 Storm Fast Scalable by processparallelization and distribution Fault-tolerant Guarantees data processing Runs on JVM Easy to learn, code and operate Supports development in multiple languages
- 9. 9 Hadoop Storm Storm forReal-Time processing Storm is to real-time computation what Hadoop is to batch computation.
- 10.
- 11. 11 Storm Use Cases “Stormpowers a wide variety of Twitter systems, ranging in applications from discovery, real-time analytics, personalization, search, revenue optimization, and many more.” “Storm empowers stream/micro-batch processing of user events, content feeds, and application logs” - Yahoo “ETL – move data from MongoDB to BI”
- 12.
- 13.
- 14. 14 Storm Abstractions Tuples, Streams,Spouts, Bolts and Topologies
- 15. 15 Tuples [“Colonoscopy”, 14106] • StormData structure • List of elements
- 16. 16 Stream Unbounded sequence oftuples [“Colonoscopy”, 14091][“Cancer”,42651] [“Oncology”, 14417]
- 17. 17 Spout Read from streamof data – queues, web logs, API calls, databases Emit streams of tuples
- 18. 18 Bolts Process tuples andcreate new streams
- 19. 19 Bolts Apply functions /transforms Calculateand aggregate data (word count!) Access DB, API , etc. Filter data Process tuples and create new streams
- 20.
- 21. 21 Storm is Easyto Code How to write storm components? Storm is easy to use
- 22.
- 23. 23 How to createa spout
- 24. 24 How to createa spout
- 25. 25 Spouts Available onGitHub Integration with Redis, Kafka, MongoDB, Amazon SQS, JMS and some others are readily available
- 26. 26 How to Createa Bolt
- 27.
- 28.
- 29.
- 30. 30 Problem What about parallelprocessing?
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37. 37 Storm rebalance > stormrebalance demo -n 3 -e myspout=5 -e mybolt=1
- 38. 38 Creating Cluster Topology >stormjar HashTagTopology.jar org.javameetup.topology. HashTagCountTopology
- 39. 39 Stream groupings Shuffle grouping:Tuples are randomly distributed across the bolt's tasks Fields grouping: The stream is partitioned by the fields specified in the grouping Custom grouping
- 40.
- 41.
- 42.
- 43.
- 44. 44 Storm deployment Out ofbox configuration are suitable for production One-click deploy with storm-deploy project to EC2 Once deployed, easy to operate – designed to be robust Storm daemons, Nimbus and Supervisors are stateless and fail-fast Useful UI
- 45.
- 46.
- 47. 47 Storm is Fault- Tolerant
- 48.
- 49. 49 Nimbus down • Processingwill continue. But topology lifecycle operations and reassignment facility are lost. • Run under system supervision.
- 50. 50 Worker node down •Nimbus will reassign tasks to other machines
- 51. 51 Supervisor goes down Processingwill still continue. But assignment is never synchronized
- 52. 52 Worker process down •Supervisor will restart the worker process and the processing will continue
- 53.
- 54.
- 55. 55 Reliability API When emittinga tuple, the Spout provides a "message id" that will be used to identify the tuple later.
- 56.
- 57. 57 Reliability API –finishing processing
- 58.
- 59.
- 60.
- 61. 61 Advanced Topics -Trident Trident is a high-level abstraction for doing real-time computing on top of Storm.
- 62. 62 Trident- Higher levelconstructs Joins Aggregations Grouping Functions Filters Consistent, exactly one semantics
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75. 75 DRPC Server We wantto know the aggregate count of tweets with hashtags #cancer and #Physician at this moment
- 76.
- 77.
- 78.
- 79. 79 Conclusion Storm allows usto solve a wide range of business problems in real time Thriving open-source community
- 80. 80 Resources Storm Project wiki Stormstarter project Storm contributions project Running a Multi-Node Storm cluster tutorial Implementing real-time trending topic A Hadoop Alternative: Building a real-time data pipeline with Storm Storm Use cases
- 81. 81 Resources (cont’d) Understanding theParallelism of a Storm Topology Trident – high level Storm abstraction A practical Storm‟s Trident API Storm online forum Project source code New York City Storm Meetup Image credits: US NASA
- 82. 82 Questions Eugene Dvorkin, Architect WebMD edvorkin@gmail.com Twitter:#edvorkin Introduction to Storm
Editor's Notes
- #3 Average enterprises now can process and make sense of big data
- #4 Variety – the various types of dataVelocity – how fast this data is processedVolume – how much data
- #7 Running if component die and self healing
- #9 Running if component die and self healing
- #11 Stream – read tuples, do some processing and update database and drop tuples. Move data from operational db into BI or process log file, ETL processingYou ask storm for really expensive computation query online – for example, how many events I got since last week.Trending topics or most popular articles
- #21 Graph of spouts and bolts with streams connection
- #37 Number of worker processes per clusterFinally, you can change the number of workers and/or number of executors for components using the "storm rebalance" command. The following command changes the number of workers for the "demo" topology to 3, the number of executors for the "myspout" component to 5, and the number of executors for the "mybolt" component to 1: storm rebalance demo -n 3 -e myspout=5 -e mybolt=1 The number of executor threads can be changed after the topology has been started (see storm rebalance command).The number of tasks of a topology is static.So one reason for having 2+ tasks per executor thread is to give you the flexibility to expand/scale up the topology through the storm rebalance command in the future without taking the topology offline. For instance, imagine you start out with a Storm cluster of 15 machines but already know that next week another 10 boxes will be added. Here you could opt for running the topology at the anticipated parallelism level of 25 machines already on the 15 initial boxes (which is of course slower than 25 boxes). Once the additional 10 boxes are integrated you can then storm rebalance the topology to make full use of all 25 boxes without any downtime.Another reason to run 2+ tasks per executor is for (primarily functional) testing. For instance, if your dev machine or CI server is only powerful enough to run, say, 2 executors alongside all the other stuff running on the machine, you can still run 30 tasks (here: 15 per executor) to see whether code such as your custom Storm grouping is working as expected.
- #40 Question
- #44 Submitter - Uploads topology JAR to Nimbus inbox with dependencies Nimbus - Makes assignment, Starts topology
- #55 Storm considers a tuple coming of a spout fully processed when every message in the tree has been processed. A tuple is considered failed when its tree of messages fails to be fully processed within a configurable timeout. The default is 30 seconds.
- #56 For example, mongoDB _id
- #57 There's two things you have to do as a user to benefit from Storm's reliability capabilities. First, you need to tell Storm whenever you're creating a new link in the tree of tuples. Second, you need to tell Storm when you have finished processing an individual tuple. By doing both these things, Storm can detect when the tree of tuples is fully processed and can ack or fail the spout tuple appropriately. Storm's API provides a concise way of doing both of these tasks.Specifying a link in the tuple tree is called anchoring.
- #58 Second, you need to tell Storm when you have finished processing an individual tuple.