1. Storm
The Real-Time Layer Your Big
Data’s Been Missing
Dan Lynn
dan@fullcontact.com
@danklynn

2. Keeps Contact Information Current and Complete
Based in Denver, Colorado
CTO & Co-Founder
dan@fullcontact.com
@danklynn

3. Turn Partial Contacts
Into Full Contacts

4. Your data is old

5. Old data is confusing

6. First, there was the spreadsheet

7. Next, there was SQL

8. Then, there wasn’t SQL

9. Batch Processing
=
Stale Data

10. Streaming
Computation

11. Queues / Workers

12. Queues / Workers
Messages
Messages
Messages
Messages
Messages
Messages Queue
Messages
Workers

13. Queues / Workers
Me ssage rou ting
can be com ple x
Messages
Messages
Messages
Messages
Messages
Messages Queue
Messages
Workers

14. Queues / Workers

15. Brittle

16. Hard to scale

17. Queues / Workers

18. Queues / Workers

19. Queues / Workers

20. Queues / Workers
Rou ting mus t be
reco nfig ured whe n
sca ling out

21. Storm

22. Storm
Distributed and fault-tolerant real-time computation

23. Storm
Distributed and fault-tolerant real-time computation

24. Storm
Distributed and fault-tolerant real-time computation

25. Storm
Distributed and fault-tolerant real-time computation

26. Key Concepts

27. Tuples
Ordered list of elements

28. Tuples
Ordered list of elements
("search-01384", "e:dan@fullcontact.com")

29. Streams
Unbounded sequence of tuples

30. Streams
Unbounded sequence of tuples
Tuple Tuple Tuple Tuple Tuple Tuple

31. Spouts
Source of streams

32. Spouts
Source of streams

33. Spouts Source of streams
Tuple Tuple Tuple Tuple Tuple Tuple

34. Spouts can talk with
some images from http://commons.wikimedia.org

35. Spouts can talk with
•Queues
some images from http://commons.wikimedia.org

36. Spouts can talk with
•Queues
•Web logs
some images from http://commons.wikimedia.org

37. Spouts can talk with
•Queues
•Web logs
•API calls
some images from http://commons.wikimedia.org

38. Spouts can talk with
•Queues
•Web logs
•API calls
•Event data
some images from http://commons.wikimedia.org

39. Bolts
Process tuples and create new streams

40. Bolts
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple Tuple Tuple Tuple Tuple Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
some images from http://commons.wikimedia.org

41. Bolts
some images from http://commons.wikimedia.org

42. Bolts
•Apply functions / transforms
some images from http://commons.wikimedia.org

43. Bolts
•Apply functions / transforms
•Filter
some images from http://commons.wikimedia.org

44. Bolts
•Apply functions / transforms
•Filter
•Aggregation
some images from http://commons.wikimedia.org

45. Bolts
•Apply functions / transforms
•Filter
•Aggregation
•Streaming joins
some images from http://commons.wikimedia.org

46. Bolts
•Apply functions / transforms
•Filter
•Aggregation
•Streaming joins
•Access DBs, APIs, etc...
some images from http://commons.wikimedia.org

47. Topologies
A directed graph of Spouts and Bolts

48. This is a Topology
some images from http://commons.wikimedia.org

49. This is also a topology
some images from http://commons.wikimedia.org

50. Tasks
Processes which execute Streams or Bolts

51. Running a Topology
$ storm jar my-code.jar com.example.MyTopology arg1 arg2

52. Storm Cluster
Nathan Marz

53. Storm Cluster
If thi s we re
Hadoo p...
Nathan Marz

54. Storm Cluster
If thi s we re
Hadoo p...
Job Tracke r
Nathan Marz

55. Storm Cluster
If thi s we re
Hadoo p...
Tas k Tracke rs Nathan Marz

56. Storm Cluster
But it’s not Hado op
Coo rdi nates eve ry thi ng
Nathan Marz

57. Example:
Streaming Word Count

58. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));

59. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));

60. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java

61. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java
splitsentence.py

62. Streaming Word Count
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
SplitSentence.java

63. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
java

64. Streaming Word Count
public static class WordCount extends BaseBasicBolt {
Map<String, Integer> counts = new HashMap<String, Integer>();
@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.get(word);
if(count==null) count = 0;
count++;
counts.put(word, count);
collector.emit(new Values(word, count));
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}
WordCount.java

65. Streaming Word Count
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("sentences");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
java
Gro upings con tro l how tup les are rou ted

66. Shuffle grouping
Tuples are randomly distributed across all of the
tasks running the bolt

67. Fields grouping
Groups tuples by specific named fields and routes
them to the same task

68. o p’s
ado r
t o H av i o
o us b e h
An a lo g i ng
rt i t io n
pa
Fields grouping
Groups tuples by specific named fields and routes
them to the same task

69. Distributed RPC

70. Before Distributed RPC,
time-sensitive queries relied on a
pre-computed index

71. What if you didn’t need an index?

72. Distributed RPC

73. Try it out!
Huge thanks to Nathan Marz - @nathanmarz
http://github.com/nathanmarz/storm
https://github.com/nathanmarz/storm-starter
@stormprocessor

74. Questions?
dan@fullcontact.com