Storm @ Fifth Elephant 2013
•
7 likes•5,159 views
Workshop on "Big Data, Real-time Processing and Storm" at The Fifth Elephant, 2013 Bangalore, India on 11th July, 2013. http://fifthelephant.in/2013/workshops Session proposal: https://funnel.hasgeek.com/fifthel2013/652-big-data-real-time-processing-and-storm Code for the workshop can be found at: https://github.com/P7h
Recommended
More Related Content
What's hot
Similar to Storm @ Fifth Elephant 2013
Similar to Storm @ Fifth Elephant 2013 (20)
Recently uploaded
Recently uploaded (20)
Storm @ Fifth Elephant 2013
- 4. Laptop with anyOS JDK v7.x installed Mavenv3.0.5+ installed IDE[either Eclipse with m2eclipse plugin or IntelliJ IDEA] CreatedTwitter app for retrieving tweets Cloned ordownloaded Storm Projectsfrom my GitHub Account: https://github.com/P7h/StormWordCount https://github.com/P7h/StormTweetsWordCount Prerequisites for Workshop
- 5. Big Data Batchvs. Real-time processing Intro to Storm CompaniesusingStorm Storm Dependencies Storm Concepts Anatomyof Storm Cluster Live codinga use caseusingStormTopology Storm vs. Hadoop Ag e n d a
- 10. Batch vs. Real-time Processing Batch processing Gatheringof dataand processingas a group at one time. Real-time processing Processingof datathat takes place as theinformation is being entered.
- 11. Event Processing Simple Event Processing Actingon a single event,filter in theESP Event Stream Processing Looking across multiple events Complex Event Processing Looking across multiple eventsfrom multipleevent streams
- 13. Storm Createdby Nathan Marz @BackType Analyzetweets, links,users on Twitter Open sourced on 19th September, 2011 EclipsePublic License 1.0 Stormv0.5.2 16k Java and 7k Clojure LOC Latest Updates Currentstable release v0.8.2released on 11th January,2013 Major core improvementsplanned for v0.9.0 Stormwill be anApacheProject [soon..]
- 14. Storm Open source distributedreal-time computation system Hadoop of real-time Fast Scalable Fault-tolerant Guarantees datawill be processed Programminglanguage agnostic Easyto set up and operate Excellent documentation
- 16. Polyglotism(language agnostic) – Clojure,Java, Python, Ruby, PHP, Perl,… and yes, evenJavaScript https://github.com/nathanmarz/storm-starter/blob/master/multilang/resources/splitsentence.py https://github.com/nathanmarz/storm-starter/blob/master/multilang/resources/splitsentence.rb
- 17. Real-time analytics Stream processing Online machine learning Continuous computation Distributed RPC Extract,Transform and Load (ETL) Use cases
- 23. enables the convergenceof Big Data and low-latency processing. Empowers stream / micro-batch processing of user events, content feeds and application logs.
- 27. Clojure a dialect of the Lisp programminglanguagerunson the JVM, CLR, andJavaScript engines ApacheThrift Cross languagebridge,RPC; Frameworktobuild services ØMQ Asynchronous messagetransportlayer Jetty Embeddedweb server Stormunderthe hood
- 28. Stormunderthe hood ApacheZooKeeper Distributed system, used to store metadata LMAX Disruptor High performancequeuesharedbythreads Kryo Serializationframework Misc. SLF4J,Python, Java 5+,JZMQ, JODA,Guava
- 30. Tuples Main data structure in Storm. An ordered list of objects. (“user”, “Prashanth”, “Babu”, “Engineer”, “Bangalore“) Key-valuepairs – keys are strings, values canbe of anytype. Tuple
- 31. Streams Unbounded sequence of tuples. Edges in the topology. Defined with a schema. Tuple Tuple Tuple Tuple Tuple
- 32. Spouts Source of streams. Spouts are like sourcesin a graph. Examples are API Calls, log files, event data,queues, Kestrel,AMQP, JMS, Kafka, etc.
- 33. BaseRichSpout
- 34. Bolts Processinput streams and [might] produce new streams. Can do anything i.e. filtering, streaming joins, aggregations,readfrom / write to databases, APIs, run arbitraryfunctions, etc. All sinks in the topology are bolts but not all bolts are sinks. Tuple Tuple Tuple
- 35. Bolts
- 36. Topology Networkof spouts and bolts. Can be visualized like a graph. Container for application logic. Analogous to a MapReduce job. But runs forever.
- 37. Sample Topology [Sentence] [Word, Count] [Sentence] RandomSentenceSpout SplitSentenceBolt SplitSentenceBolt WordCountBolt DBBolt / JMSBolt ………….. More such bolts RandomSentenceSpout https://github.com/P7h/StormWordCount
- 38. Stream Groupings Each Spout or Bolt might be running n instances in parallel[tasks]. Groupings are used to decide which task in the subscribing bolt, the tuple is sent to. Grouping Feature Shuffle Randomgrouping Fields Groupedby valuesuch that equal valueresults in sametask All Replicates to all tasks Global Makesall tuples goto one task None MakesBolt run in the samethread as the Bolt / Spout it subscribesto Direct Producer(task that emits) controls which Consumerwill receive Local or Shuffle If the targetbolt has one or moretasks in the sameworkerprocess, tuples will be shuffled to just those in-process tasks
- 41. Storm Cluster Nimbus daemon is the masterof this cluster. Managestopologies. Comparable to HadoopJobTracker. Supervisor daemon spawns workers. Comparable to HadoopTaskTracker. Workersare spawned bysupervisors. One per port defined in storm.yaml configuration.
- 42. Storm Cluster [contd..] Task is run as a thread in workers. Zookeeperis a distributed system, used to store metadata. UI is a webapp which gives diagnostics on the cluster and topologies. Nimbus and Supervisor daemons are fail-fast and stateless. State is storedin Zookeeper.
- 43. Storm – Modes of operation Localmode Develop,test and debug topologies onyour localmachine. Mavenis used to include Storm asa dev dependency for the project. mvn clean compile package && java -jar target/storm-wordcount-1.0-SNAPSHOT-jar- with-dependencies.jar
- 44. Remote [or Production]mode Topologies are submitted for executionon a clusterof machines. Cluster information is added in storm.yaml file. More details on storm.yaml file canbe found here: https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster#fill-in-mandatory-configurations- into-stormyaml storm jar target/storm-wordcount-1.0-SNAPSHOT.jar org.p7h.storm.offline.wordcount.topology.WordCountTopology WordCount Storm – Modes of operation [contd..]
- 49. Code Sample – Topology
- 54. Problem#1 – WordCount [if there are internetissues] Create a Spout which feeds randomsentences [you can define your own set of randomsentences]. Create a Bolt which receivessentencesfrom the Spout and then splits them into words and forwardsthem to next bolt. Create another Bolt to count the words. https://github.com/P7h/StormWordCount
- 55. Create a Spout which gets data from Twitter [please use Twitter4Jand OAUTH Credentials to get tweets using Streaming API]. Forsimplicity consideronly tweets which arein English. Emit only the stuff which we areinterested,i.e.A tweet’s getRetweetedStatus(). Create another Bolt to count the count the retweetsof a particular tweet. Makean in-memoryMapwith retweetscreen name andthe counterof the retweet asthe value. Logthe counterevery few seconds / minutes [should be configurable]. https://github.com/P7h/StormTopRetweets Problem#2 – Top5 retweetedtweets[if internetworks fine]
- 57. Storm vs. Hadoop Batch processing Jobs runto completion [Pre-YARN]NameNode is SPOF Statefulnodes Scalable Guarantees nodata loss Open source Real-time processing Topologies run forever No SPOF Stateless nodes Scalable Gurantees nodataloss Open source
- 58. t now Hadoop works great back here Stormworks here Hadoop AND Storm Blended view Blended view Blended View
- 59. Convergenceofbatchand low-latencyprocessing PersonalizationbasedonUserInterests HadoopAND Stormat Yahoo
- 60. Advanced Topics [not covered in this session] Distributed RPC Transactional topologies Trident Unit testing Patterns
- 61. References This Slide deck[on slideshare] – http://j.mp/5thEleStorm_SS This Slide deck[on speakerdeck] – http://j.mp/5thEleStorm_SD MyGitHub Account for code repos – https://github.com/P7h Bit.ly Bundle for Storm curatedby me – http://j.mp/YrDgcs
- 63. PrashanthBabuV V Follow me on Twitter: @P7h
Editor's Notes
- https://fifthelephant.in/2013/workshops
- https://twitter.com/jeremyckahn/status/322390046491688960
- http://visual.ly/what-big-data
- https://twitter.com/Bill_Gross/statuses/329069954911580160
- Advantages of batch processing:>> Once the data process is started, the computer can be left running without supervision.>> Large amount of transactions can be combined into a batch rather than processing them each individually.>> Is cost efficient for an organization. Batch processing is only carried out when needed and uses less computer processing time.Disadvantages of batch processing:>> With batch processing there is a time delay before the work is processed and returned.>> Jobs must be run through to completion, and any changes in the data require reprocessing of the batch job itself.>> In batch processing, the master file is not always kept up to date.>> Batch processing can be slow.>> Complete or no insight into the outcomeAdvantages of real-time processing:>> Any errors in the processing of data are caught at the time the data is entered and can be fixed when real-time processing is used.>> Real-time processing provides better control over inventory and inventory turnover.>> Real-time processing reduces the delay in analyzing.Disadvantages of real-time processing:>> Real-time processing is more complex and expensive than batch systems. >> A real-time processing system is harder to audit.>> The backup of real time processing is necessary to retain the data.
- Open sourced on 19th September, 2011 [post Twitter acquisition of BackType]Current stable release [as of 11th July, 2013] v0.8.2 released on 11th January, 2013
- https://github.com/nathanmarz/storm-starter/tree/master/multilang/resourceshttp://storm-project.net/about/multi-language.html
- Real-time analytics – trending topics in Twitter
- http://tweitgeist.colinsurprenant.com/
- http://storm-project.net/https://github.com/nathanmarz/storm/wiki/Powered-By
- https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
- https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
- https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
- http://developer.yahoo.com/blogs/ydn/storm-yarn-released-open-source-143745133.htmlhttps://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
- https://github.com/nathanmarz/storm/wiki/Concepts
- https://github.com/P7h/StormTopRetweets
- http://developer.yahoo.com/blogs/ydn/storm-hadoop-convergence-big-data-low-latency-processing-54503.html
- https://github.com/nathanmarz/storm/wiki/Tutorial
- https://github.com/nathanmarz/storm/wiki/Tutorial