Your SlideShare is downloading. ×
Apache Storm
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Apache Storm


Published on

Published in: Education, Technology, Business

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Module-1 Introduction to Big Data and STORM
  • 2. LIVE On-line Class Class Recording in LMS 24/7 Post Class Support Module Wise Quiz and Assignment Project Work on Large Data Set Verifiable Certificate How it Works? Slide 2
  • 3. Course Topics Slide 3  Module 1 » Introduction to Big Data and Storm  Module 2 » Storm Technology Stack and Groupings  Module 3 » Spouts and Bolts  Module 4 » Trident Topologies  Module 5 » Real Life Storm Project -1  Module 6 » Real Life Storm Project -2
  • 4. Objectives Slide 4 At the end of this module, you will be able to: Recall Big Data and Hadoop Understand Batch and Real-time Analytics of Big Data Investigate Shortcoming of Hadoop Understand Lambda Architecture Develop a basic knowledge of Apache Storm and its components Explain the Use Cases and Key Differentiators of Storm
  • 5. Big Data Slide 5 Storm is a open source computing system used for processing Real-time Big Data Analytics. Lets understand Big Data first to learn STORM.
  • 6.  Lots of Data - Terabytes or Petabytes  Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.  The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. What is Big Data? Slide 6
  • 7.  Systems / Enterprises generate huge amount of data from Terabytes and even Petabytes of information. Stock market generates about one terabyte of new trade data per day to perform stock trading analytics to determine trends for optimal trades. What is Big Data? Slide 7
  • 8.  2,500 exabytes of new information in 2012 with Internet as primary driver.  Digital universe grew by 62% last year to 800K petabytes and will grow to 1.2 “zettabytes” this year. Slide 8 Un-structured Data is Exploding
  • 9. IBM’s Definition – Big Data Characteristics IBM’s Definition Web logs Images Videos Sensor Data Audios VOLUME VELOCITY VARIETY Slide 9
  • 10. Annie’s Introduction Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Slide 10
  • 11. Annie’s Question Map the following to correspolnodinTghdeatraet!y!pe: Slide 11 My name is Annie. I lo quizzes and Data from EpnutezrpzrliseessyastnemdsI(EaRmP, CRhMereetc.)to make you guys think and answer my questions. - XML files - Word docs, PDF files, Text files - - E-Mail body
  • 12. Annie’s Answer XML files -> Semi-structureldodTathaere!! Slide 12 Word docs, PDF filesM, Tyextnfailems -e> UisnsAtrnunctiuer.ed DataE-Mail body -> Unstructured Data Data from EnterpriseIsylostems q(EuRiPz, zCReMs eatcn.)d-> Structured Data puzzles and I am here to make you guys think and answer my questions.
  • 13. Hadoop and its primary programming model, Map-Reduce, are great for batch-oriented processing of huge amount of data. With growing data, Hadoop enables you to horizontally scale your cluster by adding commodity nodes and thus keep up with query workloads. is primary programming model great for batch-oriented processing of huge amount of data Big Data Batch Analytics Slide 13
  • 14. What is Hadoop? Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model. It is an Open-source Data Management with scale-out storage and distributed processing. Slide 14
  • 15. Hadoop Eco-System Apache Oozie (Workflow) HDFS (Hadoop Distributed File System) HIVE DW System Pig Latin Data Analysis Other YARN Frameworks (MPI,GIRAPH)MapReduce Framework HBase YARN Cluster Resource Management Slide 15
  • 16. This evolution has forced the addition of support for Higher Level Languages (Pig & Hive) New Real-time Storage Engines (HBase) Big Data Batch Analytics Extensions for Streaming Data (Hadoop Streaming) Slide 16
  • 17. Due to batch processing, Hadoop should be deployed in situations such as Index Building Pattern Recognitions Creating Recommendation Engine Sentiment Analysis Situations generate huge amount of data stored queried Hadoop for Batch Analytics Slide 17
  • 18. Real-time Big Data Analytics Social Networking: » Pick your own Big Data database (RDBMS or NoSQL) » Measure the immediate impact to your site traffic from social media, whether a new blog post, a tweet, a “Like”, or even a comment. » Knowing this information translates to better conversion and more effective online campaigns. Slide 18
  • 19. Real-time Big Data Analytics SaaS: » Measuring user behaviour and acting upon it is crucial for improving customer satisfaction and conversion rates – which represent immediate increases in revenue. Slide 19
  • 20. Real-time Big Data Analytics Financial Services: » Determining in real time whether your portfolio is losing money, or if there is fraud in your system means that you can prevent disasters as they occur, not after the damage is done. » Correlating multiple sources from the market in real-time results in a more accurate view of the market and enables more accurate actions to maximize your profit. Slide 20
  • 21. Real Time Big Data Analytics - Options Apache StormAmazon Kinesis Slide 21
  • 22. Problem Statement: To find the total number of page views of Edureka’s blog over a range of time. Google Analytics can provide you this information. Example: For a particular day, the data can be: Need for Real-time Analytics Slide 22
  • 23. petabyte – scale All Data Slide 23 Need for Real-time Analytics Challenge: Querying huge amount of Historical Data is slow
  • 24. Precomputed View All Data Query Slide 24 Need for Real-time Analytics Solution: Precompute historical data
  • 25. Need for Real-time Analytics Google Analytics might have to keep the historical data for each hour as precompiled view Page view Page view Page view Page view Page view All Data Query Slide 25 URL Hr of the day No. of pageviews 1 250 2 300 3 455 4 460 5 320 6 111 7 129 Precomputed View
  • 26. Need for Real-time Analytics Precomputed View All Data Query Slide 26 using Hadoop
  • 27. But, what about the data generated after last precompiled view? Slide 27 Need for Real-time Analytics
  • 28. Compensating for last few hours of data Need for Real-time Analytics spout bolt bolt bolt Real-time View Storm Real-time Data Stored Or Slide 28 Or
  • 29. All Data Precomputed Batch View Precomputed Real-time View Query New Data Stream Storm Slide 29 Hadoop Need for Real-time Analytics
  • 30. Lambda Architecture All data entering the system is dispatched to both the batch layer and the speed layer for processing. New Data Speed Layer Slide 30 Batch Layer 1 Serving Layer
  • 31. Lambda Architecture Batch View Batch View Master Dataset The batch layer has two functions: » managing the master dataset (an immutable, append-only set of raw data), and » to pre-compute the batch views. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way. Batch Layer Serving Layer New Data Speed Layer 1 2 3 Slide 31
  • 32. Lambda Architecture The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only. Batch View Batch View Real-time View Master Dataset New Data Speed Layer Real-time View Batch Layer Serving Layer 1 2 3 Slide 32 4
  • 33. Lambda Architecture Any incoming query can be answered by merging results from batch views and real-time views. Batch View Batch View Real-time View Master Dataset New Data Query Speed Layer Query Real-time View Batch Layer Serving Layer 1 2 3 Slide 33 4 5
  • 34. Storm is a distributed, reliable, fault-tolerant system for processing streams of data. Fault-tolerant STORM processing Streams of Data What is Storm? Slide 34
  • 35. The work is delegated to different types of components that are each responsible for a simple specific processing task. The input stream of a Storm cluster is handled by a component called a spout. The spout passes the data to a component called a bolt, which transforms it in some way. A bolt either persists the data in some sort of storage, or passes it to some other bolt. transforms data bolt bolt spout spout bolt bolt passes data passes data data storage Input Data Source What is Storm? Slide 35
  • 36. Annie’s Question Storm can be used in: - Real-time Processing - Batch Processing - Both Slide 36
  • 37. Annie’s Answer Real-time Processing Slide 37
  • 38. Annie’s Question Which of them can be a source of Stream? - Spout - Bolt - Both Slide 38
  • 39. Annie’s Answer Both Slide 39
  • 40. Annie’s Question It is not possible to run Storm process along with MapReduce jobs inside a Hadoop Cluster. - True - False Slide 40
  • 41. Annie’s Answer False. With Hadoop 2.0, it is possible. Slide 41
  • 42. ZooKeeper Nimbus ZooKeeper ZooKeeper Supervisor Supervisor Supervisor Supervisor Supervisor Nimbus node (master node, similar to the Hadoop JobTracker): » Uploads computations for execution » Distributes code across the cluster » Launches workers across the cluster » Monitors computation and reallocates workers as needed ZooKeeper nodes: » Coordinates the Storm cluster Supervisor nodes : » Communicates with Nimbus through Zookeeper, starts and stops workers according to signals from Nimbus Storm Components A Storm cluster has 3 sets of nodes 1. Nimbus node 2. Zookeeper nodes 3. Supervisor nodes Slide 42
  • 43. Annie’s Question A Nimbus Node is similar to TaskTracker Node in Hadoop Cluster. - True - False Slide 43
  • 44. Annie’s Answer No. A Nimbus Node is more like a JobTracker Node in Hadoop Slide 44
  • 45. Five key abstractions help to understand how Storm processes data: Tuples – an ordered list of elements. For example, a “4-tuple” might be (7, 1, 3, 7) Streams – an unbounded sequence of tuples Spouts – sources of streams in a computation (e.g. a Twitter API) Bolts – process input streams and produce output streams. They can: run functions; filter, aggregate, or join data; or talk to databases Topologies – the overall calculation, represented visually as a network of spouts and bolts spout spout bolt bolt bolt bolt Storm users define topologies for how to process the data when it comes streaming in from the spout. Slide 45 Storm Components
  • 46. Annie’s Question A Storm topology is defined in terms of - Nimbus, Zookeeper, Supervisor nodes - Spout, Bolt - Spout, Bolt, Nimbus, Zookeeper, Supervisor nodes - Spout, Bolt, Zookeeper node Slide 46
  • 47. Annie’s Answer Spout and Bolt Slide 47
  • 48. Use Cases of Storm Processing Streams Distributed Remote Procedure Call Unlike other stream processing systems, with Storm there’s no need for intermediate queues. Send data to clients continuously so they can update and show results in real time, such as site metrics. Easily parallelize CPU- intensive operations. Continuous Computation Use Cases of Storm Slide 48
  • 49. Use Cases of Storm Slide 49 Financial Services » Securities Fraud » Compliance Violations » Order Routing » Pricing  Telecom » Security Breaches » Network Outages » Bandwidth Allocation » Customer Service  Retail » Shrinkage » Stock outs » Offers » Pricing Web » Application Failure » Operational Issues » Personalized Content Use Storm to prevent certain outcomes or to optimize their objectives.
  • 50. Key Differentiators Simple to Program Fault-tolerant It’s painful to do real- time processing from scratch. With storm, complexity is reduced drastically. It’s easier to develop in a JVM-based language, but Storm supports any language as long as you use or implement a small intermediary library. The Storm cluster takes care of workers going down, reassigning tasks when necessary. Support for Multiple Programming Languages Key Differentiators Slide 50
  • 51. Assignment Slide 51 Try setting up single-node Storm cluster on your system as shown in LMS Apache Storm single-node cluster installation document.
  • 52. Pre-work Slide 52 Install Ubuntu Vmware Player on your System. Install single-node Storm cluster on your system.
  • 53. What’s within the LMS This section will give you an insight of Apache Storm course Slide 53
  • 54. What’s within the LMS Click here to expand and view all the elements of this Module Slide 54
  • 55. What’s within the LMS Assignment Pre-work Slide 55 Quiz
  • 56. edureka ! /•