Workshop on "Big Data, Real-time Processing and Storm" at The Fifth Elephant, 2013 Bangalore, India on 11th July, 2013.
http://fifthelephant.in/2013/workshops
Session proposal: https://funnel.hasgeek.com/fifthel2013/652-big-data-real-time-processing-and-storm
Code for the workshop can be found at: https://github.com/P7h
4. Laptop with anyOS
JDK v7.x installed
Mavenv3.0.5+ installed
IDE[either Eclipse with m2eclipse plugin or IntelliJ IDEA]
CreatedTwitter app for retrieving tweets
Cloned ordownloaded Storm Projectsfrom my GitHub Account:
https://github.com/P7h/StormWordCount
https://github.com/P7h/StormTweetsWordCount
Prerequisites for Workshop
5. Big Data
Batchvs. Real-time processing
Intro to Storm
CompaniesusingStorm
Storm Dependencies
Storm Concepts
Anatomyof Storm Cluster
Live codinga use caseusingStormTopology
Storm vs. Hadoop
Ag e n d a
10. Batch vs. Real-time Processing
Batch processing
Gatheringof dataand processingas a group at one time.
Real-time processing
Processingof datathat takes place as theinformation is being entered.
11. Event Processing
Simple Event Processing
Actingon a single event,filter in theESP
Event Stream Processing
Looking across multiple events
Complex Event Processing
Looking across multiple eventsfrom multipleevent streams
12.
13. Storm
Createdby Nathan Marz @BackType
Analyzetweets, links,users on Twitter
Open sourced on 19th September, 2011
EclipsePublic License 1.0
Stormv0.5.2
16k Java and 7k Clojure LOC
Latest Updates
Currentstable release v0.8.2released on 11th January,2013
Major core improvementsplanned for v0.9.0
Stormwill be anApacheProject [soon..]
14. Storm
Open source distributedreal-time computation system
Hadoop of real-time
Fast
Scalable
Fault-tolerant
Guarantees datawill be processed
Programminglanguage agnostic
Easyto set up and operate
Excellent documentation
23. enables the
convergenceof Big Data and
low-latency processing.
Empowers stream / micro-batch
processing of user events,
content feeds and application
logs.
27. Clojure
a dialect of the Lisp programminglanguagerunson the JVM, CLR, andJavaScript engines
ApacheThrift
Cross languagebridge,RPC; Frameworktobuild services
ØMQ
Asynchronous messagetransportlayer
Jetty
Embeddedweb server
Stormunderthe hood
28. Stormunderthe hood
ApacheZooKeeper
Distributed system, used to store metadata
LMAX Disruptor
High performancequeuesharedbythreads
Kryo
Serializationframework
Misc.
SLF4J,Python, Java 5+,JZMQ, JODA,Guava
29.
30. Tuples
Main data structure in Storm.
An ordered list of objects.
(“user”, “Prashanth”, “Babu”, “Engineer”, “Bangalore“)
Key-valuepairs – keys are strings, values canbe of anytype.
Tuple
34. Bolts
Processinput streams and [might] produce new streams.
Can do anything i.e. filtering, streaming joins, aggregations,readfrom
/ write to databases, APIs, run arbitraryfunctions, etc.
All sinks in the topology are bolts but not all bolts are sinks.
Tuple Tuple Tuple
36. Topology
Networkof spouts and bolts.
Can be visualized like a graph.
Container for application logic.
Analogous to a MapReduce job. But runs forever.
38. Stream Groupings
Each Spout or Bolt might be running n instances in parallel[tasks].
Groupings are used to decide which task in the subscribing bolt, the
tuple is sent to.
Grouping Feature
Shuffle Randomgrouping
Fields Groupedby valuesuch that equal valueresults in sametask
All Replicates to all tasks
Global Makesall tuples goto one task
None MakesBolt run in the samethread as the Bolt / Spout it subscribesto
Direct Producer(task that emits) controls which Consumerwill receive
Local or Shuffle If the targetbolt has one or moretasks in the sameworkerprocess,
tuples will be shuffled to just those in-process tasks
41. Storm Cluster
Nimbus daemon is the masterof this cluster.
Managestopologies.
Comparable to HadoopJobTracker.
Supervisor daemon spawns workers.
Comparable to HadoopTaskTracker.
Workersare spawned bysupervisors.
One per port defined in storm.yaml configuration.
42. Storm Cluster [contd..]
Task is run as a thread in workers.
Zookeeperis a distributed system, used to store metadata.
UI is a webapp which gives diagnostics on the cluster and topologies.
Nimbus and Supervisor daemons are fail-fast and stateless.
State is storedin Zookeeper.
43. Storm – Modes of operation
Localmode
Develop,test and debug topologies onyour localmachine.
Mavenis used to include Storm asa dev dependency for the project.
mvn clean compile package && java -jar target/storm-wordcount-1.0-SNAPSHOT-jar-
with-dependencies.jar
44. Remote [or Production]mode
Topologies are submitted for executionon a clusterof machines.
Cluster information is added in storm.yaml file.
More details on storm.yaml file canbe found here:
https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster#fill-in-mandatory-configurations-
into-stormyaml
storm jar target/storm-wordcount-1.0-SNAPSHOT.jar
org.p7h.storm.offline.wordcount.topology.WordCountTopology WordCount
Storm – Modes of operation [contd..]
54. Problem#1 – WordCount [if there are internetissues]
Create a Spout which feeds randomsentences [you can define your own
set of randomsentences].
Create a Bolt which receivessentencesfrom the Spout and then splits
them into words and forwardsthem to next bolt.
Create another Bolt to count the words.
https://github.com/P7h/StormWordCount
55. Create a Spout which gets data from Twitter [please use Twitter4Jand
OAUTH Credentials to get tweets using Streaming API].
Forsimplicity consideronly tweets which arein English.
Emit only the stuff which we areinterested,i.e.A tweet’s getRetweetedStatus().
Create another Bolt to count the count the retweetsof a particular
tweet.
Makean in-memoryMapwith retweetscreen name andthe counterof the retweet asthe
value.
Logthe counterevery few seconds / minutes [should be configurable].
https://github.com/P7h/StormTopRetweets
Problem#2 – Top5 retweetedtweets[if internetworks fine]
56.
57. Storm vs. Hadoop
Batch processing
Jobs runto completion
[Pre-YARN]NameNode is SPOF
Statefulnodes
Scalable
Guarantees nodata loss
Open source
Real-time processing
Topologies run forever
No SPOF
Stateless nodes
Scalable
Gurantees nodataloss
Open source
58. t
now
Hadoop works great back here Stormworks here
Hadoop AND Storm
Blended
view
Blended
view
Blended View
Advantages of batch processing:>> Once the data process is started, the computer can be left running without supervision.>> Large amount of transactions can be combined into a batch rather than processing them each individually.>> Is cost efficient for an organization. Batch processing is only carried out when needed and uses less computer processing time.Disadvantages of batch processing:>> With batch processing there is a time delay before the work is processed and returned.>> Jobs must be run through to completion, and any changes in the data require reprocessing of the batch job itself.>> In batch processing, the master file is not always kept up to date.>> Batch processing can be slow.>> Complete or no insight into the outcomeAdvantages of real-time processing:>> Any errors in the processing of data are caught at the time the data is entered and can be fixed when real-time processing is used.>> Real-time processing provides better control over inventory and inventory turnover.>> Real-time processing reduces the delay in analyzing.Disadvantages of real-time processing:>> Real-time processing is more complex and expensive than batch systems. >> A real-time processing system is harder to audit.>> The backup of real time processing is necessary to retain the data.
Open sourced on 19th September, 2011 [post Twitter acquisition of BackType]Current stable release [as of 11th July, 2013] v0.8.2 released on 11th January, 2013