Storm is an open source distributed real-time computation system. It provides fault tolerance and allows processing of large datasets in parallel. A Storm topology defines the flow of data between spouts that read data streams and bolts that process the data. Storm can process data from multiple data sources and output results to various data stores. Trident is a high-level abstraction that provides functions like aggregation on top of Storm. The demo showed writing random sentences to Elasticsearch using a Storm topology.

1. APACHE STORM
Presenter: Kyle Lin
Date: Dec, 2014 1

2. Agenda
2
 Introduction
 Architecture
 Topology
 Related Packages
 Read from/Write to multiple data store
 Demo

3. Profile
3
 History
 Initial release in 2011
 Open sourced after being acquired by Twitter.
 An Apache Incubator project since September
2013
 Became an Apache Top-Level Project in
September 2014
 Development
 Backtype, Twitter
 On 5 July 2011 BackType was acquired by Twitter.
Ref: http://en.wikipedia.org/wiki/Storm_(event_processor)
http://en.wikipedia.org/wiki/BackType

4. Architecture
4
Ref: http://doc.mapr.com/display/MapR/Storm

5. Architecture
5
Hadoop Storm
JobTracker Nimbus
TaskTracker Supervisor
Child Worker
Job Topology
Mapper/Reducer Spout/Bolt
Ref: http://www.biaodianfu.com/storm.html

6. Architecture
6
 Topology
 Like MR job
 MapReduce job eventually finishes, whereas a
topology processes messages forever (or until
you kill it).
 A topology is a graph of stream transformations
where each node is a spout or bolt.
Ref: http://storm.apache.org/documentation/Tutorial.html

7. Architecture - Topology
7
Ref: https://blog.safaribooksonline.com/2013/06/11/your-guide-to-storm/

8. Features
8
 Scalability
 Fault tolerant
 When workers die, Storm will automatically restart
them. If a node dies, the worker will be restarted
on another node.
 Nimbus and the Supervisors, are designed to be
stateless and fail-fast.
Ref: https://storm.apache.org/about/fault-tolerant.html
https://storm.apache.org/documentation/Fault-tolerance.html

9. Features
9
 Real-time processing
 “Storm is more for real-time computation (e.g.
streaming analytics) where you analyze data in
flight and don't necessarily land it anywhere.”
 HDP included
 Delivered from Hortonworks HDP 2.1
Ref: https://groups.google.com/forum/#!topic/storm-user/sjGoDf2FMCs

10. Topology
10
 Topology
 DAG of spouts and bolts
 Spout
 Source of streams
 “Typically a spout reads from a queuing broker such as
Kestrel, RabbitMQ, or Kafka, but a spout can also generate
its own stream or read from somewhere like the Twitter
streaming API.”
 Bolt
 Function
 Core functions of a streaming computation
 Metrics
 Report summary statistics across the full topology.
Ref: https://storm.apache.org/about/simple-api.html

11. Topology
11
Ref: http://jansipke.nl/storm-in-pictures/

12. Topology
12
Ref: https://blog.safaribooksonline.com/2013/06/11/your-guide-to-storm/

13. Stream grouping
13
 Telling Storm how to send tuples between sets
of tasks.
Ref: https://storm.apache.org/documentation/Tutorial.html

14. Stream grouping
14
 Shuffle grouping
 Stream tuples are randomly distributed such that each
bolt is guaranteed to get an equal number of tuples.
 Fields grouping
 The tuples are partitioned by the fields specified in the
grouping.
 All grouping
 The stream tuples are replicated across all the bolts.
 Global grouping
 The entire stream goes to a single bolt.
 Direct Grouping
 The source decides which component will receive the
tuple
Ref: https://blog.safaribooksonline.com/2013/06/11/your-guide-to-storm/

15. Stream grouping
15
Ref: https://blog.safaribooksonline.com/2013/06/11/your-guide-to-storm/

16. Topology
16
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new SentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split”, new Fields("wo
builder.setBolt("es-bolt", new EsBolt("storm/docs"), 5).shuffleGrouping("count");
Config conf = new Config();
conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);
conf.setNumWorkers(3);
StormSubmitter.submitTopology(args[0], conf, builder.createTopology());

17. Trident on Storm
17
 Trident
 “A high-level abstraction for doing realtime
computing on top of Storm.”
 “An abstraction on storm, just like pig over
hadoop, which provides us with various useful
functions like aggregation, filter etc.”
Ref:
https://storm.apache.org/documentation/Trident-tutorial.html
http://realtime-cachedmind.tumblr.com/post/89974796387/real-time-processing-storm-tride

18. Trident
18
TridentTopology topology = new TridentTopology();
TridentState wordCounts =
topology.newStream("spout1", spout)
.each(new Fields("sentence"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("c
.parallelismHint(6);
Ref: https://storm.apache.org/documentation/Trident-tutorial.html

19. Packages
19
 $STORM_HOME/contrib
 storm-cassandra, storm-hbase, storm-hdfs,
storm-hive, storm-jms, storm-jmxetric, storm-
kafka, storm-kafka-example, storm-starter
 Access Elasticsearch
 Elasticsearch-hadoop
 Access HDFS
 Hadoop-hdfs
 Storm-hdfs

20. Demo
20
 Start a topology
 Write random sentence into ES
 See the status of topology
 From web UI

21. Steps
21
 Done
 Test(Arrange maven dependencies, Usability of
packages)
 Write/Read ES
 Write/Read HDFS
 Next
 Integrate Kafka and Storm
 Kafka
 Distributed messaging system

22. 22
Q and A

23. Reference
23
 Storm concepts
 https://storm.apache.org/documentation/Concepts
.html