Storm: distributed and fault-tolerant realtime computation
Upcoming SlideShare
Loading in...5
×
 

Storm: distributed and fault-tolerant realtime computation

on

  • 90,404 views

 

Statistics

Views

Total Views
90,404
Views on SlideShare
81,646
Embed Views
8,758

Actions

Likes
167
Downloads
2,198
Comments
4

92 Embeds 8,758

http://www.readwriteweb.com 4646
http://readwrite.com 1694
http://1.bigdata.sinaapp.com 580
http://architecturalatrocities.com 568
http://paper.li 151
http://lanyrd.com 129
http://irr.posterous.com 124
http://feeds.feedburner.com 107
http://gaurko-haurrak.blogspot.com 75
http://m.readwriteweb.com 68
http://willolbrys.com 57
http://feedproxy.google.com 56
http://allenjoe.net 54
http://a0.twimg.com 35
http://bigdata.sinaapp.com 28
http://us-w1.rockmelt.com 28
http://www.linkedin.com 23
http://www.cnblogs.com 23
http://webcache.googleusercontent.com 23
http://snstouch.com 20
http://twitter.com 17
https://twitter.com 16
http://www.hanrss.com 14
http://mrmcfarlen.blogspot.com 13
http://chrisnews-unknown.blogspot.com 12
http://www.twylah.com 11
http://wiki.kthcorp.com 10
http://potap-m.com 9
http://nuevospowerpoints.blogspot.com 9
http://oschuba.cedar.buffalo.edu 8
http://semanticweb.collected.info 8
http://translate.googleusercontent.com 6
http://blog.fasoulas.com 6
http://www.newsisfree.com 6
http://presentworth.blogspot.com 5
http://readwriteweb.com 5
http://www.techgig.com 5
http://technohidelic.posterous.com 5
http://mobilemarketing.collected.info 4
http://socialmediasuperstars.collected.info 4
http://zizindrin.com 4
http://spacelounge.collected.info 4
http://digitalapprentices.collected.info 4
http://www.blogger.com 4
http://edu.ashdod.muni.il 4
http://elfarodemedianoche.blogspot.com 4
http://futureweb.collected.info 3
http://aitao.collected.info 3
http://local.einstein.golfe 3
http://ideafarming.collected.info 3
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Storm: distributed and fault-tolerant realtime computation Storm: distributed and fault-tolerant realtime computation Presentation Transcript

  • StormDistributed and fault-tolerant realtime computation Nathan Marz Twitter
  • Storm at Twitter Twitter Web Analytics
  • Before StormQueues Workers
  • Example (simplified)
  • ExampleWorkers schemify tweets and append to Hadoop
  • ExampleWorkers update statistics on URLs byincrementing counters in Cassandra
  • ExampleDistribute tweets randomly on multiple queues
  • ExampleWorkers share the load of schemifying tweets
  • ExampleDesire all updates for same URL go to same worker
  • Message locality• Because: • No transactions in Cassandra (and no atomic increments at the time) • More effective batching of updates
  • Implementing message locality• Have a queue for each consuming worker• Choose queue for a URL using consistent hashing
  • ExampleWorkers choose queue to enqueue to using hash/mod of URL
  • Example All updates for same URLguaranteed to go to same worker
  • Adding a worker
  • Adding a worker DeployReconfigure/redeploy
  • Problems• Scaling is painful• Poor fault-tolerance• Coding is tedious
  • What we want• Guaranteed data processing• Horizontal scalability• Fault-tolerance• No intermediate message brokers!• Higher level abstraction than message passing• “Just works”
  • StormGuaranteed data processingHorizontal scalabilityFault-toleranceNo intermediate message brokers!Higher level abstraction than message passing“Just works”
  • Use cases Stream Distributed Continuousprocessing RPC computation
  • Storm Cluster
  • Storm ClusterMaster node (similar to Hadoop JobTracker)
  • Storm ClusterUsed for cluster coordination
  • Storm Cluster Run worker processes
  • Starting a topology
  • Killing a topology
  • Concepts• Streams• Spouts• Bolts• Topologies
  • StreamsTuple Tuple Tuple Tuple Tuple Tuple Tuple Unbounded sequence of tuples
  • SpoutsSource of streams
  • Spout examples• Read from Kestrel queue• Read from Twitter streaming API
  • BoltsProcesses input streams and produces new streams
  • Bolts• Functions• Filters• Aggregation• Joins• Talk to databases
  • TopologyNetwork of spouts and bolts
  • TasksSpouts and bolts execute asmany tasks across the cluster
  • Stream groupingWhen a tuple is emitted, which task does it go to?
  • Stream grouping• Shuffle grouping: pick a random task• Fields grouping: consistent hashing on a subset of tuple fields• All grouping: send to all tasks• Global grouping: pick task with lowest id
  • Topologyshuffle [“id1”, “id2”] shuffle[“url”] shuffle all
  • Streaming word countTopologyBuilder is used to construct topologies in Java
  • Streaming word countDefine a spout in the topology with parallelism of 5 tasks
  • Streaming word countSplit sentences into words with parallelism of 8 tasks
  • Streaming word countConsumer decides what data it receives and how it gets groupedSplit sentences into words with parallelism of 8 tasks
  • Streaming word count Create a word count stream
  • Streaming word count splitsentence.py
  • Streaming word count
  • Streaming word count Submitting topology to a cluster
  • Streaming word count Running topology in local mode
  • Demo
  • Traditional data processing
  • Traditional data processing Intense processing (Hadoop, databases, etc.)
  • Traditional data processingLight processing on a single machine to resolve queries
  • Distributed RPCDistributed RPC lets you do intense processing at query-time
  • Game changer
  • Distributed RPCData flow for Distributed RPC
  • DRPC ExampleComputing “reach” of a URL on the fly
  • ReachReach is the number of unique people exposed to a URL on Twitter
  • Computing reach Follower Distinct Tweeter Follower follower Follower DistinctURL Tweeter follower Count Reach Follower Follower Distinct Tweeter follower Follower
  • Reach topology
  • Guaranteeing message processing “Tuple tree”
  • Guaranteeing message processing• A spout tuple is not fully processed until all tuples in the tree have been completed
  • Guaranteeing message processing• If the tuple tree is not completed within a specified timeout, the spout tuple is replayed
  • Guaranteeing message processing Reliability API
  • Guaranteeing message processing“Anchoring” creates a new edge in the tuple tree
  • Guaranteeing message processing Marks a single node in the tree as complete
  • Guaranteeing message processing• Storm tracks tuple trees for you in an extremely efficient way
  • Storm UI
  • Storm UI
  • Storm UI
  • Storm on EC2https://github.com/nathanmarz/storm-deploy One-click deploy tool
  • Documentation
  • State spout (almost done) Synchronize a large amount of frequently changing state into a topology
  • State spout (almost done)Optimizing reach topology by eliminating the database calls
  • State spout (almost done) Each GetFollowers task keeps a synchronous cache of a subset of the social graph
  • State spout (almost done)This works because GetFollowers repartitions the social graph the same way it partitions GetTweeter’s stream
  • Future work• Storm on Mesos• “Swapping”• Auto-scaling• Higher level abstractions
  • Questions?http://github.com/nathanmarz/storm
  • What Storm does• Distributes code and configurations• Robust process management• Monitors topologies and reassigns failed tasks• Provides reliability by tracking tuple trees• Routing and partitioning of streams• Serialization• Fine-grained performance stats of topologies