Nimbus similar to JobTracker
Distributes code around cluster, assign tasks, handle failures
Supervisor similar to TaskTracker
ZooKeeper for Nimbus to Supervisor co-ordination…. The nimbus and supervisors find each other through zookeper. So they use zookeper to find each other.. A hugely common use case for zookeeper
Communication between workers using ZeroMQ
The number of tasks for a spout/bolt is always the same throughout the lifetime of a topology, but the number of executors (threads) for a spout/bolt can change over time. This allows a topology to scale to more or less resources without redeploying the topology or violating the constraints of Storm (such as a fields grouping guaranteeing that the same value goes to the same task).
Supervisor has one or more worker processes. Each worker process has one or more executor threads and each executor runs one or more tasks of the same component ( spout or bolt)
NOTE: Running multiple tasks per executor thread does not give any performance benefit at all but it was built more so that you can expand the cluster – processing capacity wise
NOTE: Having less number of tasks than executors does not make sense. Typically #tasks = #executors
Source: http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
Executors
Supervisors
Tasks
Workers
#threads
Slots?
Nimbus