5. What is stream processing?
“Streaming processing” is the ideal platform to process data
streams or sensor data (usually a high ratio of event
throughput versus numbers of queries), whereas “complex
event processing” (CEP) utilizes event-by-event processing
and aggregation
http://www.infoq.com/articles/stream-processing-hadoop
14. Message Delivery Semantics
1. At most once: messages may be lost but never redelivered
2. At least once: messages will never be lost but may be
redelivered
3. Exactly once: messages are never lost and never redelivered
15. Complex solution needed?
1. scaling
2. message delivery
3. message grouping
4. message aggregation
5. cost of development and maintenance
16.
17.
18. ➔ distributed real-time processing system
➔ scalable
➔ fault-tolerant
➔ simplify working with queues & workers
➔ Written in Clojure
21. Nimbus and Supervisor daemons must be run under supervision
using a tool like daemontools or monit.
No worker processes are affected by the death of Nimbus or the
Supervisors.
So the answer is that Nimbus is "sort of" a SPOF
31. Apache Storm vs. Spark Streaming – Two Stream Processing Platforms compared. Guido Schmutz.
How to scale?
q2
ProcessingB
ProcessingB
Collecting
Collecting
qn
q1 ProcessingA
ProcessingA
ProcessingA
q1
q2
32. TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("ftp-spout", new FTPSpout(config), 1);
builder.setSpout("kafka-spout", new KafkaSpout(config), 4);
builder.setBolt("processingA-bolt", new ProcessingABolt())
.shuffleGrouping("ftp-spout");
builder.setBolt("processingB-bolt", new ProcessingBBolt())
.shuffleGrouping("kafka-spout");
builder.setBolt("processingC-bolt", new ProcessingCBolt())
.shuffleGrouping("kafka-spout");
builder.setBolt("merge-bolt", new MergeBolt()).shuffleGrouping("processingA-bolt");
builder.setBolt("merge-bolt", new MergeBolt()).shuffleGrouping("processingB-bolt");
builder.setBolt("collecting-bolt", new CollectingBolt())
.shuffleGrouping("processingC-bolt");
39. Message delivery/reliability
Storm guarantees that every spout tuple will be fully processed
by the topology. It does this by tracking the tree of tuples
triggered by every spout tuple and determining when that tree
of tuples has been successfully completed. Every topology has
a "message timeout" associated with it. If Storm fails to detect
that a spout tuple has been completed within that timeout,
then it fails the tuple and replays it later.
40. public interface IOutputCollector extends IErrorReporter {
/**
* Returns the task ids that received the tuples.
*/
List<Integer> emit(String streamId, Collection<Tuple> anchors,
List<Object> tuple);
void emitDirect(int taskId, String streamId, Collection<Tuple>
anchors,
List<Object> tuple);
void ack(Tuple input);
void fail(Tuple input);
}
41. public class JUGBolt implements IRichBolt {
OutputCollector collector;
@Override
public void execute(Tuple input) {
try {
process(input);
collector.ack(input);
} catch (Exception e) {
collector.fail(input);
}
}
}
46. Config config = new Config();
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("APR-30-JUGTopology", config,
new JUGTopology().build());
47. Scaling
Storm's usage of Zookeeper for cluster coordination
add machines
increase the parallelism
49. Config conf = new Config();
conf.setNumWorkers(2); // use two worker processes
topologyBuilder.setSpout("blue-spout", new BlueSpout(), 2);
// set parallelism hint to 2
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
.setNumTasks(4)
.shuffleGrouping("blue-
spout");
topologyBuilder.setBolt("yellow-bolt", new YellowBolt(), 6)
.shuffleGrouping("green-
bolt");
StormSubmitter.submitTopology(
"mytopology",
conf,
topologyBuilder.createTopology()
);
http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
52. Trident
Trident is a high-level abstraction for doing realtime
computing on top of Storm. It allows you to seamlessly
intermix high throughput (millions of messages per second),
stateful stream processing with low latency distributed
querying
https://storm.apache.org/documentation/Trident-tutorial.html
53. topology.newStream("spout1", spout)
.each(new Fields("sentence"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count"))
.parallelismHint(6);
https://storm.apache.org/documentation/Trident-tutorial.html
59. Latency
• Is performance of streaming application paramount
Development Cost
• Is it desired to have similar code bases for batch and stream processing =>
lambda architecture
Message Delivery Guarantees
• Is there high importance on processing every single record, or is some normal
amount of data loss acceptable
Process Fault Tolerance
• Is high-availability of primary concern
Choice?
1.Scala group presentation
2.Marrog presentation - zeromq/protocol buffeers
3.jacek laskowski
4. adam kawa - spotify
5.186k$
6.storm is difficult
Dlaczego chce powiedziec o stormie jezeli na reszcie tych rzeczy spedzam 90% czasu. Nikt nie zakwestionuje hadoopa, nawet goscuiu ktory nie wie dokladnie co to jest ale kiedys sciskal dlon ceo cloudery, jak uslyczy ze to sie ‘nie nadaje’, wybuchnie smiechem.
jak ja slysze takie bzdury ze spark jest lepszy bo mozna go na yarn-ie uruchomic… albo dlatego ze mozna w scali pisac
jak slysze ze development w stormi jest trudny, i moga to robic tylko wybrancy, bo 99% devow to glaby i nie dadza rade. Jest dokladnie odwrotnie i to zamierzam pokazac.
slyszalem nawet ze developerzy znajacy storm-a zarabiaja srednio 186k $ rocznie, to ja powiedzialem mojemu managerowi “to juz wiesz dlaczego w tym stormie chce robic”
bacouse of latency, batch cannot provide real-time results.
source -> processors -> queues-> result ->>>
more processors
more queues
hard to maintain:
apachestormvsapachespark-v1-141203182123-conversion-gate02.pdf
-fail-over
-must recover fast
-top-layer project - discussion on scaling on jboss, hazelcast or sth else
-persistent queues - state to recover after fail
Batch Processing
• Familiar concept of processing data en masse
• Generally incurs a high-latency
(Event-) Stream Processing
• A one-at-a-time processing model
• A datum is processed as it arrives
• Sub-second latency
• Difficult to process state data efficiently
Micro-Batching
• A special case of batch processing with very small batch sizes (tiny)
• A nice mix between batching and streaming
• At cost of latency
• Gives stateful computation, making windowing an easy task
Batch Processing
• Familiar concept of processing data en masse
• Generally incurs a high-latency
(Event-) Stream Processing
• A one-at-a-time processing model
• A datum is processed as it arrives
• Sub-second latency
• Difficult to process state data efficiently
Micro-Batching
• A special case of batch processing with very small batch sizes (tiny)
• A nice mix between batching and streaming
• At cost of latency
• Gives stateful computation, making windowing an easy task
Batch Processing
• Familiar concept of processing data en masse
• Generally incurs a high-latency
(Event-) Stream Processing
• A one-at-a-time processing model
• A datum is processed as it arrives
• Sub-second latency
• Difficult to process state data efficiently
Micro-Batching
• A special case of batch processing with very small batch sizes (tiny)
• A nice mix between batching and streaming
• At cost of latency
• Gives stateful computation, making windowing an easy task
Batch Processing
• Familiar concept of processing data en masse
• Generally incurs a high-latency
(Event-) Stream Processing
• A one-at-a-time processing model
• A datum is processed as it arrives
• Sub-second latency
• Difficult to process state data efficiently
Micro-Batching
• A special case of batch processing with very small batch sizes (tiny)
• A nice mix between batching and streaming
• At cost of latency
• Gives stateful computation, making windowing an easy task
dobra, wiemy co to jest ten stream processing,
wiemy jak mozemy przetwarzac te dane (filozoficznie)
wiemy ze wiadomosci moga ginac, i to moze byc dopuszczalne lub nie.
czy potrzebne jest kompleksowe rozwiazanie?
w aktualnym stanie rzeczy sa liczne problemy….
BLOG Nathana - wpis o histori storma
-baktype acquired by twitter
-storm opensourced
-nathan marz start-up
-taylor goetz
no tak ale to mialo byc foult tolerant, a jest jeden nimbus tylko….
=============
nimbus, zookeeper, supervisor, (worker, executors, tasks)
Storm cluster:
S1: hadoop job tracker structue - question, do you know what is this?
S2: storm nimbus + supervisor - similar? what is wrong reqarding to fault tolerant?
S3: is nimbus a SPOF?
S3: for cluster... yes
S4: for running topology... no
S4:but... it fail fast
S5: is supervisor a SPOF?
S5: what happens when Nimbus or supervisor fails?
What happens when Nimbus or Supervisor daemons die?
The Nimbus and Supervisor daemons are designed to be fail-fast (process self-destructs whenever any unexpected situation is encountered) and stateless (all state is kept in Zookeeper or on disk). As described in Setting up a Storm cluster, the Nimbus and Supervisor daemons must be run under supervision using a tool like daemontools or monit. So if the Nimbus or Supervisor daemons die, they restart like nothing happened.
Most notably, no worker processes are affected by the death of Nimbus or the Supervisors. This is in contrast to Hadoop, where if the JobTracker dies, all the running jobs are lost.
If you lose the Nimbus node, the workers will still continue to function. Additionally, supervisors will continue to restart workers if they die. However, without Nimbus, workers won't be reassigned to other machines when necessary (like if you lose a worker machine).
So the answer is that Nimbus is "sort of" a SPOF. In practice, it's not a big deal since nothing catastrophic happens when the Nimbus daemon dies. There are plans to make Nimbus highly available in the future.
stormhadoopsummit2014-140407145559-phpapp01.pdf
about 20 slides
core data unit (single queue message)
topology
topology
topology
topology
topology
topology
FTP
Kafka
percolate
alertin
-fail-over
-must recover fast
-top-layer project - discussion on scaling on jboss, hazelcast or sth else
-persistent queues - state to recover after fail
Stream groupings
Part of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks.
There are seven built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the CustomStreamGrouping interface:
Shuffle grouping: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
Fields grouping: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks.
Partial Key grouping: The stream is partitioned by the fields specified in the grouping, like the Fields grouping, but are load balanced between two downstream bolts, which provides better utilization of resources when the incoming data is skewed. This paper provides a good explanation of how it works and the advantages it provides.
All grouping: The stream is replicated across all the bolt's tasks. Use this grouping with care.
Global grouping: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id.
None grouping: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible).
Direct grouping: This is a special kind of grouping. A stream grouped this way means that the producer of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the [emitDirect](/javadoc/apidocs/backtype/storm/task/OutputCollector.html#emitDirect(int, int, java.util.List) methods. A bolt can get the task ids of its consumers by either using the provided TopologyContext or by keeping track of the output of the emit method in OutputCollector (which returns the task ids that the tuple was sent to).
Local or shuffle grouping: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.
==================
Shuffle grouping is random grouping
Fields grouping is grouped by value, such that equal value results in equal task
All grouping replicates to all tasks
Global grouping makes all tuples go to one task
None grouping makes bolt run in the same thread as bolt/spout it subscribes to
Direct grouping producer (task that emits) controls which consumer will receive
Local or Shuffle
grouping
similar to the shuffle grouping but will shuffle tuples among bolt tasks
running in the same worker process, if any. Falls back to shuffle
grouping behavior.
fail / ack
timeout
understanding of storm parallelism - more deeply
rebalance
obrazek - worker, executors, tasks
request-reply w asseco , no tutaj duzo trzeba bylo duzo pic, zeby sie nie odwodnic.
Latency
• Is performance of streaming application paramount
Development Cost
• Is it desired to have similar code bases for batch and stream processing =>
lambda architecture
Message Delivery Guarantees
• Is there high importance on processing every single record, or is some normal
amount of data loss acceptable
Process Fault Tolerance
• Is high-availability of primary concern
