Storm @ Fifth Elephant 2013
Upcoming SlideShare
Loading in...5
×
 

Storm @ Fifth Elephant 2013

on

  • 3,796 views

Workshop on "Big Data, Real-time Processing and Storm" at The Fifth Elephant, 2013 Bangalore, India on 11th July, 2013. ...

Workshop on "Big Data, Real-time Processing and Storm" at The Fifth Elephant, 2013 Bangalore, India on 11th July, 2013.
http://fifthelephant.in/2013/workshops

Session proposal: https://funnel.hasgeek.com/fifthel2013/652-big-data-real-time-processing-and-storm

Code for the workshop can be found at: https://github.com/P7h

Statistics

Views

Total Views
3,796
Views on SlideShare
3,467
Embed Views
329

Actions

Likes
6
Downloads
74
Comments
0

30 Embeds 329

http://p7h.blogspot.in 173
http://p7h.blogspot.com 62
http://p7h.blogspot.com.br 13
http://p7h.blogspot.co.uk 12
http://p7h.blogspot.ca 9
http://p7h.blogspot.de 7
http://p7h.blogspot.sg 5
http://p7h.blogspot.com.au 5
https://twitter.com 4
http://p7h.blogspot.jp 4
http://p7h.blogspot.ie 3
http://p7h.blogspot.fr 3
https://www.linkedin.com 3
http://p7h.blogspot.ch 3
http://www.linkedin.com 3
http://p7h.blogspot.ru 2
http://p7h.blogspot.co.il 2
http://p7h.blogspot.tw 2
http://p7h.blogspot.gr 2
http://p7h.blogspot.com.es 2
http://p7h.blogspot.it 1
http://p7h.blogspot.hk 1
http://p7h.blogspot.com.tr 1
http://p7h.blogspot.se 1
http://p7h.blogspot.kr 1
http://p7h.blogspot.hu 1
http://p7h.blogspot.com.ar 1
http://p7h.blogspot.fi 1
http://p7h.blogspot.mx 1
http://p7h.blogspot.dk 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • https://fifthelephant.in/2013/workshops
  • https://twitter.com/jeremyckahn/status/322390046491688960
  • http://visual.ly/what-big-data
  • https://twitter.com/Bill_Gross/statuses/329069954911580160
  • Advantages of batch processing:>> Once the data process is started, the computer can be left running without supervision.>> Large amount of transactions can be combined into a batch rather than processing them each individually.>> Is cost efficient for an organization. Batch processing is only carried out when needed and uses less computer processing time.Disadvantages of batch processing:>> With batch processing there is a time delay before the work is processed and returned.>> Jobs must be run through to completion, and any changes in the data require reprocessing of the batch job itself.>> In batch processing, the master file is not always kept up to date.>> Batch processing can be slow.>> Complete or no insight into the outcomeAdvantages of real-time processing:>> Any errors in the processing of data are caught at the time the data is entered and can be fixed when real-time processing is used.>> Real-time processing provides better control over inventory and inventory turnover.>> Real-time processing reduces the delay in analyzing.Disadvantages of real-time processing:>> Real-time processing is more complex and expensive than batch systems. >> A real-time processing system is harder to audit.>> The backup of real time processing is necessary to retain the data.
  • Open sourced on 19th September, 2011 [post Twitter acquisition of BackType]Current stable release [as of 11th July, 2013] v0.8.2 released on 11th January, 2013
  • https://github.com/nathanmarz/storm-starter/tree/master/multilang/resourceshttp://storm-project.net/about/multi-language.html
  • Real-time analytics – trending topics in Twitter
  • http://tweitgeist.colinsurprenant.com/
  • http://storm-project.net/https://github.com/nathanmarz/storm/wiki/Powered-By
  • https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
  • https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
  • https://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
  • http://developer.yahoo.com/blogs/ydn/storm-yarn-released-open-source-143745133.htmlhttps://github.com/nathanmarz/storm/wiki/Powered-Byhttps://speakerdeck.com/mariuszgil/streams-processing-with-strom
  • https://github.com/nathanmarz/storm/wiki/Concepts
  • https://github.com/P7h/StormTopRetweets
  • http://developer.yahoo.com/blogs/ydn/storm-hadoop-convergence-big-data-low-latency-processing-54503.html
  • https://github.com/nathanmarz/storm/wiki/Tutorial
  • https://github.com/nathanmarz/storm/wiki/Tutorial

Storm @ Fifth Elephant 2013 Storm @ Fifth Elephant 2013 Presentation Transcript

  • PrashanthBabuV V @P7h
  • https://twitter.com/P7h https://github.com/P7h http://P7h.org http://about.me/Prashanth
  • Laptop with anyOS JDK v7.x installed Mavenv3.0.5+ installed IDE[either Eclipse with m2eclipse plugin or IntelliJ IDEA] CreatedTwitter app for retrieving tweets Cloned ordownloaded Storm Projectsfrom my GitHub Account:  https://github.com/P7h/StormWordCount  https://github.com/P7h/StormTweetsWordCount Prerequisites for Workshop
  • Big Data Batchvs. Real-time processing Intro to Storm CompaniesusingStorm Storm Dependencies Storm Concepts Anatomyof Storm Cluster Live codinga use caseusingStormTopology Storm vs. Hadoop Ag e n d a
  • Dataoverload everysecond
  • Batch vs. Real-time Processing Batch processing  Gatheringof dataand processingas a group at one time. Real-time processing  Processingof datathat takes place as theinformation is being entered.
  • Event Processing Simple Event Processing  Actingon a single event,filter in theESP Event Stream Processing  Looking across multiple events Complex Event Processing  Looking across multiple eventsfrom multipleevent streams
  • Storm Createdby Nathan Marz @BackType  Analyzetweets, links,users on Twitter Open sourced on 19th September, 2011  EclipsePublic License 1.0  Stormv0.5.2  16k Java and 7k Clojure LOC Latest Updates  Currentstable release v0.8.2released on 11th January,2013  Major core improvementsplanned for v0.9.0  Stormwill be anApacheProject [soon..]
  • Storm Open source distributedreal-time computation system Hadoop of real-time Fast Scalable Fault-tolerant Guarantees datawill be processed Programminglanguage agnostic Easyto set up and operate Excellent documentation
  • Polyglotism(language agnostic) – Clojure,Java, Python, Ruby, PHP, Perl,… and yes, evenJavaScript https://github.com/nathanmarz/storm-starter/blob/master/multilang/resources/splitsentence.py https://github.com/nathanmarz/storm-starter/blob/master/multilang/resources/splitsentence.rb
  • Real-time analytics Stream processing Online machine learning Continuous computation Distributed RPC Extract,Transform and Load (ETL) Use cases
  • http://tweitgeist.colinsurprenant.com/
  • https://github.com/nathanmarz/storm/wiki/Powered-By CompaniesusingStorm
  • enables the convergenceof Big Data and low-latency processing. Empowers stream / micro-batch processing of user events, content feeds and application logs.
  • https://github.com/P7h/storm-camel-example
  • Clojure  a dialect of the Lisp programminglanguagerunson the JVM, CLR, andJavaScript engines ApacheThrift  Cross languagebridge,RPC; Frameworktobuild services ØMQ  Asynchronous messagetransportlayer Jetty  Embeddedweb server Stormunderthe hood
  • Stormunderthe hood ApacheZooKeeper  Distributed system, used to store metadata LMAX Disruptor  High performancequeuesharedbythreads Kryo  Serializationframework Misc.  SLF4J,Python, Java 5+,JZMQ, JODA,Guava
  • Tuples Main data structure in Storm. An ordered list of objects.  (“user”, “Prashanth”, “Babu”, “Engineer”, “Bangalore“) Key-valuepairs – keys are strings, values canbe of anytype. Tuple
  • Streams Unbounded sequence of tuples. Edges in the topology. Defined with a schema. Tuple Tuple Tuple Tuple Tuple
  • Spouts Source of streams. Spouts are like sourcesin a graph. Examples are API Calls, log files, event data,queues, Kestrel,AMQP, JMS, Kafka, etc.
  • BaseRichSpout
  • Bolts Processinput streams and [might] produce new streams. Can do anything i.e. filtering, streaming joins, aggregations,readfrom / write to databases, APIs, run arbitraryfunctions, etc. All sinks in the topology are bolts but not all bolts are sinks. Tuple Tuple Tuple
  • Bolts
  • Topology Networkof spouts and bolts. Can be visualized like a graph. Container for application logic. Analogous to a MapReduce job. But runs forever.
  • Sample Topology [Sentence] [Word, Count] [Sentence] RandomSentenceSpout SplitSentenceBolt SplitSentenceBolt WordCountBolt DBBolt / JMSBolt ………….. More such bolts RandomSentenceSpout https://github.com/P7h/StormWordCount
  • Stream Groupings Each Spout or Bolt might be running n instances in parallel[tasks]. Groupings are used to decide which task in the subscribing bolt, the tuple is sent to. Grouping Feature Shuffle Randomgrouping Fields Groupedby valuesuch that equal valueresults in sametask All Replicates to all tasks Global Makesall tuples goto one task None MakesBolt run in the samethread as the Bolt / Spout it subscribesto Direct Producer(task that emits) controls which Consumerwill receive Local or Shuffle If the targetbolt has one or moretasks in the sameworkerprocess, tuples will be shuffled to just those in-process tasks
  • Storm Cluster NIMBUS ZooKeeper#1 ZooKeeper#2 ZooKeeper#n Supervisor#1 Supervisor#2 Supervisor#3 Supervisor#n UI Workers Workers Workers Workers
  • Storm Cluster Nimbus daemon is the masterof this cluster.  Managestopologies.  Comparable to HadoopJobTracker. Supervisor daemon spawns workers.  Comparable to HadoopTaskTracker. Workersare spawned bysupervisors.  One per port defined in storm.yaml configuration.
  • Storm Cluster [contd..] Task is run as a thread in workers. Zookeeperis a distributed system, used to store metadata. UI is a webapp which gives diagnostics on the cluster and topologies. Nimbus and Supervisor daemons are fail-fast and stateless.  State is storedin Zookeeper.
  • Storm – Modes of operation Localmode  Develop,test and debug topologies onyour localmachine.  Mavenis used to include Storm asa dev dependency for the project. mvn clean compile package && java -jar target/storm-wordcount-1.0-SNAPSHOT-jar- with-dependencies.jar
  • Remote [or Production]mode  Topologies are submitted for executionon a clusterof machines.  Cluster information is added in storm.yaml file.  More details on storm.yaml file canbe found here: https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster#fill-in-mandatory-configurations- into-stormyaml storm jar target/storm-wordcount-1.0-SNAPSHOT.jar org.p7h.storm.offline.wordcount.topology.WordCountTopology WordCount Storm – Modes of operation [contd..]
  • StormUI – ClusterSummary
  • StormUI – TopologySummary
  • StormUI – ComponentSummary
  • Code Sample – Topology
  • Code Sample– Spout
  • Code Sample– Bolt#1
  • Code Sample– Bolt#2
  • Problem#1 – WordCount [if there are internetissues] Create a Spout which feeds randomsentences [you can define your own set of randomsentences]. Create a Bolt which receivessentencesfrom the Spout and then splits them into words and forwardsthem to next bolt. Create another Bolt to count the words. https://github.com/P7h/StormWordCount
  • Create a Spout which gets data from Twitter [please use Twitter4Jand OAUTH Credentials to get tweets using Streaming API].  Forsimplicity consideronly tweets which arein English.  Emit only the stuff which we areinterested,i.e.A tweet’s getRetweetedStatus(). Create another Bolt to count the count the retweetsof a particular tweet.  Makean in-memoryMapwith retweetscreen name andthe counterof the retweet asthe value.  Logthe counterevery few seconds / minutes [should be configurable]. https://github.com/P7h/StormTopRetweets Problem#2 – Top5 retweetedtweets[if internetworks fine]
  • Storm vs. Hadoop Batch processing Jobs runto completion [Pre-YARN]NameNode is SPOF Statefulnodes Scalable Guarantees nodata loss Open source Real-time processing Topologies run forever No SPOF Stateless nodes Scalable Gurantees nodataloss Open source
  • t now Hadoop works great back here Stormworks here Hadoop AND Storm Blended view Blended view Blended View
  • Convergenceofbatchand low-latencyprocessing PersonalizationbasedonUserInterests HadoopAND Stormat Yahoo
  • Advanced Topics [not covered in this session] Distributed RPC Transactional topologies Trident Unit testing Patterns
  • References This Slide deck[on slideshare] – http://j.mp/5thEleStorm_SS This Slide deck[on speakerdeck] – http://j.mp/5thEleStorm_SD MyGitHub Account for code repos – https://github.com/P7h Bit.ly Bundle for Storm curatedby me – http://j.mp/YrDgcs
  • PrashanthBabuV V Follow me on Twitter: @P7h