This document provides an overview of Storm, an open source distributed real-time computation system. It describes key Storm concepts like tuples, streams, spouts, bolts and topologies. It provides an example of a streaming word count topology and discusses tips for using Storm like using log aggregation, rolling deployments, tuning parallelism and anchoring tuples. It also briefly introduces the Trident abstraction for Storm.

1. Storm
Dan Lynn
dan@fullcontact.com
@danklynn
As deep into real-time data processing as you can get*
*in 30 minutes.

2. Keeps Contact Information Current and Complete
Based in Denver, Colorado
CTO & Co-Founder
dan@fullcontact.com
@danklynn

3. Turn Partial Contacts
Into Full Contacts

4. Storm

5. StormDistributed
and
fault-­‐tolerant
real-­‐3me
computa3on

6. StormDistributed
and
fault-­‐tolerant
real-­‐3me
computa3on

7. StormDistributed
and
fault-­‐tolerant
real-­‐3me
computa3on

8. StormDistributed
and
fault-­‐tolerant
real-­‐3me
computa3on

9. THE HARD WAY
Queues
Workers

10. THE HARD WAY

11. Key Concepts

12. Tuples
Ordered list of elements

13. Tuples
Ordered list of elements
("search-01384", "e:dan@fullcontact.com")

14. StreamsUnbounded sequence of tuples

15. StreamsUnbounded sequence of tuples
Tuple Tuple Tuple Tuple Tuple Tuple

16. SpoutsSource of streams

17. SpoutsSource of streams

18. SpoutsSource of streams
Tuple Tuple Tuple Tuple Tuple Tuple

19. Spouts can talk with
some
images
from
h,p://commons.wikimedia.org
•Queues
•Web
logs
•API
calls
•Event
data

20. BoltsProcess tuples and create new streams

21. Bolts
some
images
from
h,p://commons.wikimedia.org
•Apply
funcBons
/
transforms
•Filter
•AggregaBon
•Streaming
joins
•Access
DBs,
APIs,
etc...

22. Bolts
Tuple Tuple Tuple Tuple Tuple Tuple
some
images
from
h,p://commons.wikimedia.org
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple

23. TopologiesA directed graph of Spouts and Bolts

24. This is a Topology
some
images
from
h,p://commons.wikimedia.org

25. This is also a topology
some
images
from
h,p://commons.wikimedia.org

26. TasksExecute Streams or Bolts

27. Running a Topology
$ storm jar my-code.jar com.example.MyTopology arg1 arg2

28. Storm Cluster
Nathan
Marz

29. Storm Cluster
Nathan
Marz
If this were
Hadoop...

30. Storm Cluster
Nathan
Marz
Job Tracker
If this were
Hadoop...

31. Storm Cluster
Nathan
MarzTask Trackers
If this were
Hadoop...

32. Storm Cluster
Nathan
Marz
Coordinates everything
But it’s not Hadoop

33. Example:
Streaming Word Count

34. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));

35. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));

36. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java

37. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java
splitsentence.py

38. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java

39. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
java

40. Streaming Word Count
public static class WordCount extends BaseBasicBolt {
Map<String, Integer> counts = new HashMap<String, Integer>();
@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.get(word);
if(count==null) count = 0;
count++;
counts.put(word, count);
collector.emit(new Values(word, count));
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}
WordCount.java

41. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
java
Groupings control how tuples are routed

42. Shuffle grouping
Tuples are randomly distributed across all of the
tasks running the bolt

43. Fields grouping
Groups tuples by specific named fields and routes
them to the same task

44. Fields grouping
Groups tuples by specific named fields and routes
them to the same task
Analogous to Hadoop’s
partitioning behavior

45. Trending Topics

46. Twitter Trending Topics
TwitterStreamingTopicSpout
parallelism = 1 (unless you use GNip)
(word)
RollingCountsBolt
parallelism = n
(word, count)
IntermediateRankingsBolt
parallelism = n
(rankings)
(tweets)
(JSON rankings)
RankingsReportBolt
parallelism = 1
TotalRankingsBolt
parallelism = 1
(rankings)

47. Live Coding!

48. Twitter Trending Topics
TwitterStreamingTopicSpout
parallelism = 1 (unless you use GNip)
(word)
RollingCountsBolt
parallelism = n
(word, count)
IntermediateRankingsBolt
parallelism = n
(rankings)
(tweets)
(JSON rankings)
RankingsReportBolt
parallelism = 1
TotalRankingsBolt
parallelism = 1
(rankings)

49. Tips

50. loggly.com
Graylog2
logstash
Use a log aggregator

51. "$topologyName-$buildNumber"
Rolling Deploys

52. 1.
Launch
new
topology
2.
Wait
for
it
to
be
healthy
3.
Kill
the
old
one
Rolling Deploys

53. These are under active
development
Rolling Deploys

54. TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
java
see:
https://github.com/nathanmarz/storm/wiki/Understanding-the-parallelism-of-a-Storm-topology
Tune your parallelism

55. Tune your parallelism
Supervisor
Worker
Process
(JVM)
Executor
(thread)
Task
Task
Executor
(thread)
Task
Task
Worker
Process
(JVM)
Executor
(thread)
Task
Task
Executor
(thread)
Task
Task
Parallelism hints control
the number of Executors

56. collector.emit(new Values(word, count));
see:
https://github.com/nathanmarz/storm/wiki/Understanding-the-parallelism-of-a-Storm-topology
Anchor your tuples (or not)
collector.emit(tuple, new Values(word, count));

57. But Dan, you
left out
Trident!

58. if (storm == hadoop) {
trident = pig / cascading
}

59. A little taste of Trident
TridentState
urlToTweeters
=
topology.newStaticState(getUrlToTweetersState());
TridentState
tweetersToFollowers
=
topology.newStaticState(getTweeterToFollowersState());
topology.newDRPCStream("reach")
.stateQuery(urlToTweeters,
new
Fields("args"),
new
MapGet(),
new
Fields("tweeters"))
.each(new
Fields("tweeters"),
new
ExpandList(),
new
Fields("tweeter"))
.shuffle()
.stateQuery(tweetersToFollowers,
new
Fields("tweeter"),
new
MapGet(),
new
Fields("followers"))
.parallelismHint(200)
.each(new
Fields("followers"),
new
ExpandList(),
new
Fields("follower"))
.groupBy(new
Fields("follower"))
.aggregate(new
One(),
new
Fields("one"))
.parallelismHint(20)
.aggregate(new
Count(),
new
Fields("reach"));
h,ps://github.com/nathanmarz/storm/wiki/Trident-­‐tutorial

60. Thanks:
@stormprocessor
http://github.com/nathanmarz/storm
Nathan Marz - @nathanmarz
http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-
trending-topics-in-storm/
Michael Knoll - @miguno
Michael Rose - @xorlev
http://github.com/xorlev

61. Questions?
dan@fullcontact.com
https://github.com/danklynn/storm-starter/tree/gluecon2013