The document provides an overview of real-time computation using Apache Storm, highlighting its features such as guaranteed data processing, horizontal scalability, and fault-tolerance. It explains key components like spouts and bolts for processing streams and offers examples of interoperability with various systems. The document also discusses use cases and provides code examples illustrating how to implement functionalities within Storm.

1. Realtime Computation
with Storm
Brad Anderson
banderson@maprtech.com
@boorad

3. Definition & Overview
Interoperability
Use Cases

4. Stream Processing
CEP
Distributed RPC

5. Before Storm
Queues Workers

6. Example
(simplified)

7. Storm
Guaranteed data processing
Horizontal scalability
Fault-tolerance
No intermediate message brokers!
Higher level abstraction than message passing

8. Concepts

9. streams
Tuple Tuple Tuple Tuple Tuple Tuple Tuple
Unbounded sequence of tuples

10. spouts
Source of streams

11. spouts
public interface ISpout extends Serializable {
void open(Map conf,
TopologyContext context,
SpoutOutputCollector collector);
void close();
void nextTuple();
void ack(Object msgId);
void fail(Object msgId);
}

12. bolts
Processes input streams and produces new streams

13. bolts
public class DoubleAndTripleBolt extends BaseRichBolt {
private OutputCollectorBase _collector;
public void prepare(Map conf,
TopologyContext context,
OutputCollectorBase collector) {
_collector = collector;
}
public void execute(Tuple input) {
int val = input.getInteger(0);
_collector.emit(input, new Values(val*2, val*3));
_collector.ack(input);
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("double", "triple"));
}
}

14. topologies
Network of spouts and bolts

15. Trident
Cascading for Storm

16. TridentTopology topology = new TridentTopology();
TridentState wordCounts =
topology.newStream("spout1", spout)
.each(new Fields("sentence"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(new MemoryMapState.Factory(),
new Count(),
new Fields("count"))
.parallelismHint(6);

17. Interoperability

18. spouts
•Kafka (with transactions)
• Kestrel
• JMS
• AMQP
• Beanstalkd

19. bolts
• Functions
• Filters
• Aggregation
• Joins
• Talk to databases, Hadoop write-behind

20. Storm
realtime
processes
Queue Apps
Raw
Data Business
Value
Hadoop
batch
processes

21. Storm
realtime
processes
Queue Apps
Raw
Data Business
Value
Hadoop
Parallel Cluster Ingest
batch
processes

22. Storm
realtime
processes
Queue Apps
Raw
Data Business
Value
Hadoop
batch
processes

23. Storm
realtime
processes
Apps
Raw
Data Business
Value
Hadoop
batch
processes

24. Use Cases

25. Twitter
Follower
Distinct
Tweeter Follower follower
Follower
Distinct
URL Tweeter follower Reach
Follower
Follower
Distinct
Tweeter follower
Follower

26. Heartbyte

27. Fleet Logistics

28. http://github.com/{tdunning | boorad}/mapr-spout
Brad Anderson
banderson@maprtech.com
@boorad

29. Thank you.
http://github.com/{tdunning | boorad}/mapr-spout
Brad Anderson
banderson@maprtech.com
@boorad