The slides from my session on Apache Storm architecture at Hadoop Summit Europe 2014.

1. © Hortonworks Inc. 2011
P. Taylor Goetz
Apache Storm Committer
tgoetz@hortonworks.com
@ptgoetz
Apache Storm Architecture and Integration
Real-Time Big Data

2. Shedding Light on Data

3. Shedding Light on Big Data

4. Shedding Light on Big Data
In Real Time

5. What is Storm?

6. Storm is Streaming

7. Storm is Streaming
Key enabler of the Lamda Architecture

8. Storm is Fast

9. Storm is Fast
Clocked at 1M+ messages per second per node

10. Storm is Scalable

11. Storm is Scalable
Thousands of workers per cluster

12. Storm is Fault Tolerant

13. Storm is Fault Tolerant
Failure is expected, and embraced

14. Storm is Reliable

15. Storm is Reliable
Guaranteed message delivery

16. Storm is Reliable
Exactly-once semantics

17. Conceptual Model

18. Tuple
{…}

19. Tuple
{…} • Core Unit of Data
• Immutable Set of Key/Value
Pairs

20. Streams
{…} {…} {…} {…} {…} {…} {…}
Unbounded Sequence of Tuples

21. Spouts

22. Spouts
• Source of Streams
• Wraps a streaming data source
and emits Tuples
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}
{…}

23. Spout API
public interface ISpout extends Serializable {!
!
void open(Map conf, !
! TopologyContext context, !
! ! ! SpoutOutputCollector collector);!
!
void close();!
!
void activate();!
!
void deactivate();!
!
void nextTuple();!
!
void ack(Object msgId);!
!
void fail(Object msgId);!
}
Lifecycle API

24. Spout API
public interface ISpout extends Serializable {!
!
void open(Map conf, !
! TopologyContext context, !
! ! ! SpoutOutputCollector collector);!
!
void close();!
!
void activate();!
!
void deactivate();!
!
void nextTuple();!
!
void ack(Object msgId);!
!
void fail(Object msgId);!
}
Core API

25. Spout API
public interface ISpout extends Serializable {!
!
void open(Map conf, !
! TopologyContext context, !
! ! ! SpoutOutputCollector collector);!
!
void close();!
!
void activate();!
!
void deactivate();!
!
void nextTuple();!
!
void ack(Object msgId);!
!
void fail(Object msgId);!
}
Reliability API

26. Bolts

27. Bolts
• Core functions of a
streaming computation
• Receive tuples and do stuff
• Optionally emit additional
tuples

28. Bolts
• Write to a data store

29. Bolts
• Read from a data store

30. Bolts
• Perform arbitrary
computation
Compute

31. {…}
{…}
{…}
{…}
{…}
{…}
{…}
Bolts
• (Optionally) Emit additional
streams
{…}
{…}
{…}
{…}
{…}
{…}
{…}

32. Bolt API
public interface IBolt extends Serializable {!
!
void prepare(Map stormConf, !
TopologyContext context, !
OutputCollector collector);!
!
void cleanup();!
! !
void execute(Tuple input);!
! !
}
Lifecycle API

33. Bolt API
public interface IBolt extends Serializable {!
!
void prepare(Map stormConf, !
TopologyContext context, !
OutputCollector collector);!
!
void cleanup();!
! !
void execute(Tuple input);!
! !
}
Core API

34. Bolt Output API
public interface IOutputCollector extends IErrorReporter {!
!
List<Integer> emit(String streamId, !
Collection<Tuple> anchors, !
List<Object> tuple);!
! !
void emitDirect(int taskId, !
String streamId, !
Collection<Tuple> anchors, !
List<Object> tuple);!
! !
void ack(Tuple input);!
! !
void fail(Tuple input);!
}
Core API

35. Bolt Output API
public interface IOutputCollector extends IErrorReporter {!
!
List<Integer> emit(String streamId, !
Collection<Tuple> anchors, !
List<Object> tuple);!
! !
void emitDirect(int taskId, !
String streamId, !
Collection<Tuple> anchors, !
List<Object> tuple);!
! !
void ack(Tuple input);!
! !
void fail(Tuple input);!
}
Reliability API

36. Topologies

37. Topologies

38. Topologies
• DAG of Spouts and Bolts
• Data Flow Representation
• Streaming Computation

39. Topologies
• Storm executes spouts
and bolts as individual
Tasks that run in parallel
on multiple machines.

40. Stream Groupings

41. Stream Groupings
Stream Groupings determine how Storm routes
Tuples between tasks in a topology

42. Stream Groupings
Shuffle!
!
Randomized round-robin.

43. Stream Groupings
LocalOrShuffle!
!
Randomized round-robin.
(With a preference for intra-worker Tasks)

44. Stream Groupings
Fields Grouping!
!
Ensures all Tuples with with the same field value(s)
are always routed to the same task.

45. Stream Groupings
Fields Grouping!
!
Ensures all Tuples with with the same field value(s)
are always routed to the same task.
!
(this is a simple hash of the field values,
modulo the number of tasks)

46. Physical View

47. Physical View
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Worker* Worker* Worker* Worker*

48. Topology Deployment
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Topology Submitter uploads topology:!
• topology.jar!
• topology.ser!
• conf.ser
$ bin/storm jar

49. Topology Deployment
Nimbus calculates assignments and sends to Zookeeper
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter

50. Topology Deployment
Supervisor nodes receive assignment information !
via Zookeeper watches.
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter

51. Topology Deployment
Supervisor nodes download topology from Nimbus:!
• topology.jar!
• topology.ser!
• conf.ser
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter

52. Topology Deployment
Supervisors spawn workers (JVM processes) to start the topology
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker

53. Fault Tolerance

54. Fault Tolerance
Workers heartbeat back to Supervisors and Nimbus via ZooKeeper, !
as well as locally.
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker

55. Fault Tolerance
If a worker dies (fails to heartbeat), the Supervisor will restart it
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker
X

56. Fault Tolerance
If a worker dies repeatedly, Nimbus will reassign the work to other!
nodes in the cluster.
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker
X

57. Fault Tolerance
If a supervisor node dies, Nimbus will reassign the work to other nodes.
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker
X
X

58. Fault Tolerance
If Nimbus dies, topologies will continue to function normally,!
but won’t be able to perform reassignments.
ZooKeeperNimbus
Supervisor Supervisor Supervisor Supervisor
Topology
Submitter
Worker Worker Worker Worker
X

59. Parallelism
Scaling a Distributed Computation

60. Parallelism
Worker (JVM)
Executor (Thread) Executor (Thread) Executor (Thread)
Task Task Task
1 Worker,
Parallelism = 1

61. Parallelism
Worker (JVM)
Executor (Thread) Executor (Thread) Executor (Thread)
Task Task Task
Executor (Thread)
Task
1 Worker,
Parallelism = 2

62. Parallelism
Worker (JVM)
Executor (Thread) Executor (Thread)
Task Task
Executor (Thread)
Task
Task
1 Worker,
Parallelism = 2, NumTasks = 2

63. Parallelism
3 Workers,
Parallelism = 1, NumTasks = 1
Worker (JVM)Worker (JVM)Worker (JVM)
Executor (Thread) Executor (Thread) Executor (Thread)
Task Task Task

64. Internal Messaging

65. Internal Messaging
Worker Mechanics

66. Worker Internal Messaging
Worker Receive Thread
Worker Port
List<List<Tuple>>
Receive Buffer
Executor Thread *
Inbound Queue Outbound Queue
Router Send
Thread
Worker Transfer Thread
List<List<Tuple>>
Transfer Buffer
To Other Workers
Task
(Spout/Bolt)
Task
(Spout/Bolt)
Task(s)
(Spout/Bolt)

67. Reliable Processing
At Least Once

68. Reliable Processing
Bolts may emit Tuples Anchored to one received.
Tuple “B” is a descendant of Tuple “A”
{A} {B}

69. Reliable Processing
Multiple Anchorings form a Tuple tree
(bolts not shown)
{A} {B}
{C}
{D}
{E}
{F}
{G}
{H}

70. Reliable Processing
Bolts can Acknowledge that a tuple
has been processed successfully.
{A} {B}
ACK

71. Reliable Processing
Acks are delivered via a system-level bolt
ACK
{A} {B}
Acker Bolt
ackack

72. Reliable Processing
Bolts can also Fail a tuple to trigger a spout to
replay the original.
FAIL
{A} {B}
Acker Bolt
failfail

73. Reliable Processing
Any failure in the Tuple tree will trigger a
replay of the original tuple
{A} {B}
{C}
{D}
{E}
{F}
{G}
{H}
X
X

74. Reliable Processing
How to track a large-scale tuple tree efficiently?

75. Reliable Processing
A single 64-bit integer.

76. XOR Magic
Long a, b, c = Random.nextLong();

77. XOR Magic
Long a, b, c = Random.nextLong();!
!
a ^ a == 0

78. XOR Magic
Long a, b, c = Random.nextLong();!
!
a ^ a == 0!
!
a ^ a ^ b != 0

79. XOR Magic
Long a, b, c = Random.nextLong();!
!
a ^ a == 0!
!
a ^ a ^ b != 0!
!
a ^ a ^ b ^ b == 0

80. XOR Magic
Long a, b, c = Random.nextLong();!
!
a ^ (a ^ b) ^ c ^ (b ^ c) == 0

81. XOR Magic
Long a, b, c = Random.nextLong();!
!
a ^ (a ^ b) ^ c ^ (b ^ c) == 0
Acks can arrive asynchronously, in any order

82. Trident

83. Trident
High-level abstraction built on Storm’s core primitives.

84. Trident
Built-in support for:
• Merges and Joins
• Aggregations
• Groupings
• Functions
• Filters

85. Trident
Stateful, incremental processing on top
of any persistence store.

86. Trident
Trident is Storm

87. Trident
Fluent, Stream-oriented API

88. Trident
Fluent, Stream-Oriented API
TridentTopology topology = new TridentTopology();!
FixedBatchSpout spout = new FixedBatchSpout(…);!
Stream stream = topology.newStream("words", spout);!
!
stream.each(…, new MyFunction())!
.groupBy()!
.each(…, new MyFilter())!
.persistentAggregate(…);!
User-defined functions

89. Trident
Micro-Batch Oriented
Tuple Micro-Batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}

90. Trident
Trident Batches are Ordered
Tuple Micro-Batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
Tuple Micro-Batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
Batch #1 Batch #2

91. Trident
Trident Batches can be Partitioned
Tuple Micro-Batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}

92. Trident
Trident Batches can be Partitioned
Tuple Micro-Batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
Partition Operation
Partition A
{…} {…}
{…}{…}
Partition B
{…} {…}
{…}{…}
Partition C
{…} {…}
{…}{…}
Partition D
{…} {…}
{…}{…}

93. Trident Operation Types
1. Local Operations (Functions/Filters)
2. Repartitioning Operations (Stream Groupings,
etc.)
3. Aggregations
4. Merges/Joins

94. Trident Topologies
each
each
shuffle
Function
Filter
partition
persist

95. Trident Toplogies
Partitioning operations define the boundaries
between bolts, and thus network transfer
and parallelism

96. Trident Topologies
each
each
shuffle
Function
Filter
partition
persist
Bolt 1
Bolt 2
shuffleGrouping()
Partitioning!
Operation

97. Trident Batch
Coordination

98. Trident Batch Coordination
Trident SpoutMaster Batch Coordinator User Logic
next
batch
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
{…} {…} {…} {…}
commit

99. Controlling
Deployment

100. Controlling Deployment
How do you control where spouts
and bolts get deployed in a cluster?

101. Controlling Deployment
How do you control where spouts
and bolts get deployed in a cluster?
Plug-able Schedulers

102. Controlling Deployment
How do you control where spouts
and bolts get deployed in a cluster?
Isolation Scheduler

103. Wait… Nimbus, Supervisor, Schedulers…
!
Doesn’t that sound kind of like
resource negotiation?

104. Storm on YARN
HDFS2
(redundant,
reliable
storage)
YARN
(cluster
resource
management)
MapReduce
(batch)
Apache
STORM
(streaming)
HADOOP 2.0
Tez
(interactive)
Multi Use Data Platform
Batch, Interactive, Online, Streaming, …

105. Storm on YARN
HDFS2
(redundant,
reliable
storage)
YARN
(cluster
resource
management)
MapReduce
(batch)
Apache
STORM
(streaming)
HADOOP 2.0
Tez
(interactive)
Multi Use Data Platform
Batch, Interactive, Online, Streaming, …
Batch and real-time on the same cluster

106. Storm on YARN
HDFS2
(redundant,
reliable
storage)
YARN
(cluster
resource
management)
MapReduce
(batch)
Apache
STORM
(streaming)
HADOOP 2.0
Tez
(interactive)
Multi Use Data Platform
Batch, Interactive, Online, Streaming, …
Security and Multi-tenancy

107. Storm on YARN
HDFS2
(redundant,
reliable
storage)
YARN
(cluster
resource
management)
MapReduce
(batch)
Apache
STORM
(streaming)
HADOOP 2.0
Tez
(interactive)
Multi Use Data Platform
Batch, Interactive, Online, Streaming, …
Elasticity

108. Storm on YARN
Nimbus
Resource Management, Scheduling
Supervisor
Node and Process management
Workers
Runs topology tasks
YARN RM
Resource Management
Storm AM
Manage Topology
Containers
Runs topology tasks
YARN NM
Process Management
Storm’s resource management system
maps very naturally to the YARN model.

109. Storm on YARN
Nimbus
Resource Management, Scheduling
Supervisor
Node and Process management
Workers
Runs topology tasks
YARN RM
Resource Management
Storm AM
Manage Topology
Containers
Runs topology tasks
YARN NM
Process Management
High Availability

110. Storm on YARN
Nimbus
Resource Management, Scheduling
Supervisor
Node and Process management
Workers
Runs topology tasks
YARN RM
Resource Management
Storm AM
Manage Topology
Containers
Runs topology tasks
YARN NM
Process Management
Detect and scale around bottlenecks

111. Storm on YARN
Nimbus
Resource Management, Scheduling
Supervisor
Node and Process management
Workers
Runs topology tasks
YARN RM
Resource Management
Storm AM
Manage Topology
Containers
Runs topology tasks
YARN NM
Process Management
Optimize for available resources

112. Shameless
Plug
https://www.packtpub.com/
storm-distributed-real-time-
computation-blueprints/book

113. Thank You!
Contributions welcome.
Join the storm community at:
http://storm.incubator.apache.org
P. Taylor Goetz
tgoetz@hortonworks.com
@ptgoetz