Storm: distributed and fault-tolerant realtime computation

StormDistributed and fault-tolerant realtime computation Nathan Marz Twitter

Storm at Twitter Twitter Web Analytics

Before StormQueues Workers

Example (simpliﬁed)

ExampleWorkers schemify tweets and append to Hadoop

ExampleWorkers update statistics on URLs byincrementing counters in Cassandra

ExampleDistribute tweets randomly on multiple queues

ExampleWorkers share the load of schemifying tweets

ExampleDesire all updates for same URL go to same worker

Message locality• Because: • No transactions in Cassandra (and no atomic increments at the time) • More effective batching of updates

Implementing message locality• Have a queue for each consuming worker• Choose queue for a URL using consistent hashing

ExampleWorkers choose queue to enqueue to using hash/mod of URL

Example All updates for same URLguaranteed to go to same worker

Adding a worker

Adding a worker DeployReconﬁgure/redeploy

Problems• Scaling is painful• Poor fault-tolerance• Coding is tedious

What we want• Guaranteed data processing• Horizontal scalability• Fault-tolerance• No intermediate message brokers!• Higher level abstraction than message passing• “Just works”

StormGuaranteed data processingHorizontal scalabilityFault-toleranceNo intermediate message brokers!Higher level abstraction than message passing“Just works”

Use cases Stream Distributed Continuousprocessing RPC computation

Storm Cluster

Storm ClusterMaster node (similar to Hadoop JobTracker)

Storm ClusterUsed for cluster coordination

Storm Cluster Run worker processes

Starting a topology

Killing a topology

Concepts• Streams• Spouts• Bolts• Topologies

StreamsTuple Tuple Tuple Tuple Tuple Tuple Tuple Unbounded sequence of tuples

SpoutsSource of streams

Spout examples• Read from Kestrel queue• Read from Twitter streaming API

BoltsProcesses input streams and produces new streams

Bolts• Functions• Filters• Aggregation• Joins• Talk to databases

TopologyNetwork of spouts and bolts

TasksSpouts and bolts execute asmany tasks across the cluster

Stream groupingWhen a tuple is emitted, which task does it go to?

Stream grouping• Shufﬂe grouping: pick a random task• Fields grouping: consistent hashing on a subset of tuple ﬁelds• All grouping: send to all tasks• Global grouping: pick task with lowest id

Topologyshuffle [“id1”, “id2”] shuffle[“url”] shuffle all

Streaming word countTopologyBuilder is used to construct topologies in Java

Streaming word countDeﬁne a spout in the topology with parallelism of 5 tasks

Streaming word countSplit sentences into words with parallelism of 8 tasks

Streaming word countConsumer decides what data it receives and how it gets groupedSplit sentences into words with parallelism of 8 tasks

Streaming word count Create a word count stream

Streaming word count splitsentence.py

Streaming word count

Streaming word count Submitting topology to a cluster

Streaming word count Running topology in local mode

Traditional data processing

Traditional data processing Intense processing (Hadoop, databases, etc.)

Traditional data processingLight processing on a single machine to resolve queries

Distributed RPCDistributed RPC lets you do intense processing at query-time

Game changer

Distributed RPCData ﬂow for Distributed RPC

DRPC ExampleComputing “reach” of a URL on the ﬂy

ReachReach is the number of unique people exposed to a URL on Twitter

Computing reach Follower Distinct Tweeter Follower follower Follower DistinctURL Tweeter follower Count Reach Follower Follower Distinct Tweeter follower Follower

Reach topology

Guaranteeing message processing “Tuple tree”

Guaranteeing message processing• A spout tuple is not fully processed until all tuples in the tree have been completed

Guaranteeing message processing• If the tuple tree is not completed within a speciﬁed timeout, the spout tuple is replayed

Guaranteeing message processing Reliability API

Guaranteeing message processing“Anchoring” creates a new edge in the tuple tree

Guaranteeing message processing Marks a single node in the tree as complete

Guaranteeing message processing• Storm tracks tuple trees for you in an extremely efﬁcient way

Storm UI

Storm on EC2https://github.com/nathanmarz/storm-deploy One-click deploy tool

Documentation

State spout (almost done) Synchronize a large amount of frequently changing state into a topology

State spout (almost done)Optimizing reach topology by eliminating the database calls

State spout (almost done) Each GetFollowers task keeps a synchronous cache of a subset of the social graph

State spout (almost done)This works because GetFollowers repartitions the social graph the same way it partitions GetTweeter’s stream

Future work• Storm on Mesos• “Swapping”• Auto-scaling• Higher level abstractions

Questions?http://github.com/nathanmarz/storm

What Storm does• Distributes code and conﬁgurations• Robust process management• Monitors topologies and reassigns failed tasks• Provides reliability by tracking tuple trees• Routing and partitioning of streams• Serialization• Fine-grained performance stats of topologies

http://www.readwriteweb.com	4646
http://readwrite.com	671
http://1.bigdata.sinaapp.com	580
http://architecturalatrocities.com	497
http://paper.li	151
http://irr.posterous.com	124
http://lanyrd.com	115
http://feeds.feedburner.com	107
http://gaurko-haurrak.blogspot.com	75
http://m.readwriteweb.com	68
http://feedproxy.google.com	56
http://allenjoe.net	54
http://willolbrys.com	50
http://a0.twimg.com	35
http://us-w1.rockmelt.com	28
http://bigdata.sinaapp.com	28
http://www.linkedin.com	23
http://webcache.googleusercontent.com	23
http://www.cnblogs.com	22
http://snstouch.com	20
http://twitter.com	17
http://www.hanrss.com	14
http://mrmcfarlen.blogspot.com	13
http://chrisnews-unknown.blogspot.com	12
http://www.twylah.com	11
http://wiki.kthcorp.com	10
https://twitter.com	10
http://nuevospowerpoints.blogspot.com	9
http://potap-m.com	9
http://semanticweb.collected.info	8
http://oschuba.cedar.buffalo.edu	8
http://www.newsisfree.com	6
http://blog.fasoulas.com	6
http://technohidelic.posterous.com	5
http://readwriteweb.com	5
http://presentworth.blogspot.com	5
http://digitalapprentices.collected.info	4
http://spacelounge.collected.info	4
http://zizindrin.com	4
http://socialmediasuperstars.collected.info	4
http://www.blogger.com	4
http://www.techgig.com	4
http://mobilemarketing.collected.info	4
http://translate.googleusercontent.com	4
http://edu.ashdod.muni.il	4
http://aitao.collected.info	3
http://www.seofacts.biz	3
http://ideafarming.collected.info	3
http://elfarodemedianoche.blogspot.com	3
http://futureweb.collected.info	3
http://www.onlydoo.com	3

Storm: distributed and fault-tolerant realtime computation

by nathanmarz on Sep 20, 2011

Accessibility

Categories

Upload Details

Usage Rights

85 Embeds 7,623

Statistics

Storm: distributed and fault-tolerant realtime computation Presentation Transcript