Large-Scale Stream Processing in the Hadoop Ecosystem

Large-Scale Stream Processing
in the Hadoop Ecosystem
Gyula Fóra
gyfora@apache.org
Márton Balassi
mbalassi@apache.org

This talk
§ Stream processing by example
§ Open source stream processors
§ Runtime architecture and programming
model
§ Counting words…
§ Fault tolerance and stateful processing
§ Closing
2Apache: Big Data Europe2015-‐‑09-‐‑28

Stream processing
by example
2015-‐‑09-‐‑28 Apache: Big Data Europe 3

Streaming applications
ETL style operations
• Filter incoming data,
Log analysis
• High throughput, connectors,
at-least-once processing
Window aggregations
• Trending tweets,
User sessions, Stream joins
• Window abstractions
Inpu
t
Inpu
t
Inpu
tInput
Process/Enrich

Streaming applications
Machine learning
• Fitting trends to the evolving
stream, Stream clustering
• Model state, cyclic flows
Pattern recognition
• Fraud detection, Triggering
signals based on activity
• Exactly-once processing

Open source stream
processors

Apache Streaming landscape
72015-‐‑09-‐‑28 Apache: Big Data Europe

Apache Storm
§ Started in 2010, development driven by
BackType, then Twitter
§ Pioneer in large scale stream processing
§ Distributed dataflow abstraction (spouts &
bolts)

Apache Flink
§ Started in 2008 as a research project
(Stratosphere) at European universities
§ Unique combination of low latency streaming
and high throughput batch analysis
§ Flexible operator states and windowing
9
Batch data
Kafka,
RabbitMQ,

...
HDFS,
JDBC,

...
Stream
Data

Apache Spark
§ Started in 2009 at UC Berkley, Apache since 2013
§ Very strong community, wide adoption
§ Unified batch and stream processing over a
batch runtime
§ Good integration with batch programs

Apache Samza
§ Developed at LinkedIn, open sourced in 2013
§ Builds heavily on Kafka’s log based philosophy
§ Pluggable messaging system and execution
backend

System comparison
12
Streaming
model
Native Micro-batching Native Native
API Compositional Declarative Compositional Declarative
Fault tolerance Record ACKs RDD-based Log-based Checkpoints
Guarantee At-least-once Exactly-once At-least-once Exactly-once
State Only in Trident
State as
DStream
Stateful
operators
Stateful
operators
Windowing Not built-in Time based Not built-in Policy based
Latency Very-Low Medium Low Low
Throughput Medium High High High

Runtime and
programming model

Native Streaming

Distributed dataflow runtime
§ Storm, Samza and Flink
§ General properties
• Long standing operators
• Pipelined execution
• Usually possible to create
cyclic flows
Pros
• Full expressivity
• Low-latency execution
• Stateful operators
Cons
• Fault-tolerance is hard
• Throughput may suffer
• Load balancing is an
issue

Distributed dataflow runtime
§ Storm
• Dynamic typing + Kryo
• Dynamic topology rebalancing
§ Samza
• Almost every component pluggable
• Full task isolation, no backpressure (buffering
handled by the messaging layer)
§ Flink
• Strongly typed streams + custom serializers
• Flow control mechanism
• Memory management

Micro-batching

Micro-batch runtime
§ Implemented by Apache Spark
§ General properties
• Computation broken down
to time intervals
• Load aware scheduling
• Easy interaction with batch
Pros
• Easy to reason about
• High-throughput
• FT comes for “free”
• Dynamic load balancing
Cons
• Latency depends on
batch size
• Limited expressivity
• Stateless by nature

Programming model
Declarative
§ Expose a high-level API
§ Operators are higher order
functions on abstract data
stream types
§ Advanced behavior such as
windowing is supported
§ Query optimization
Compositional
§ Offer basic building blocks
for composing custom
operators and topologies
§ Advanced behavior such as
windowing is often missing
§ Topology needs to be hand-
optimized

Programming model
DStream, DataStream
§ Transformations abstract
operator details
§ Suitable for engineers and data
analysts
Spout, Consumer,
Bolt, Task, Topology
§ Direct access to the execution
graph / topology
• Suitable for engineers

Counting words…

WordCount
storm budapest flink
apache storm spark
streaming samza storm
flink apache flink
bigdata storm
flink streaming
(storm, 4)
(budapest, 1)
(flink, 4)
(apache, 2)
(spark, 1)
(streaming, 2)
(samza, 1)
(bigdata, 1)

Storm
Assembling the topology
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new SentenceSpout(), 5);
builder.setBolt("split", new Splitter(), 8).shuffleGrouping("spout");
builder.setBolt("count", new Counter(), 12)
.fieldsGrouping("split", new Fields("word"));
public class Counter extends BaseBasicBolt {
Map<String, Integer> counts = new HashMap<String, Integer>();
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.containsKey(word) ? counts.get(word) + 1 : 1;
counts.put(word, count);
collector.emit(new Values(word, count));
}
}
Rolling word count bolt

Samza
public class WordCountTask implements StreamTask {
private KeyValueStore<String, Integer> store;
public void process( IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator) {
String word = envelope.getMessage();
Integer count = store.get(word);
if(count == null){count = 0;}
store.put(word, count + 1);
collector.send(new OutgoingMessageEnvelope(new
SystemStream("kafka", ”wc"), Tuple2.of(word, count)));
}
}
Rolling word count task

Flink
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.groupBy("word").sum("frequency")
.print()
case class Word (word: String, frequency: Int)
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ")
.map(word => Word(word,1))}
.window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS))
.groupBy("word").sum("frequency")
.print()
Rolling word count
Window word count

Spark
Window word count
Rolling word count (kind of)

Fault tolerance and
stateful processing

Fault tolerance intro
§ Fault-tolerance in streaming systems is
inherently harder than in batch
• Can’t just restart computation
• State is a problem
• Fast recovery is crucial
• Streaming topologies run 24/7 for a long period
§ Fault-tolerance is a complex issue
• No single point of failure is allowed
• Guaranteeing input processing
• Consistent operator state
• Fast recovery
• At-least-once vs Exactly-once semantics

Storm record acknowledgements
§ Track the lineage of tuples as they are
processed (anchors and acks)
§ Special “acker” bolts track each lineage
DAG (efficient xor based algorithm)
§ Replay the root of failed (or timed out)
tuples

Samza offset tracking
§ Exploits the properties of a durable, offset
based messaging layer
§ Each task maintains its current offset, which
moves forward as it processes elements
§ The offset is checkpointed and restored on
failure (some messages might be repeated)

Flink checkpointing
§ Based on consistent global snapshots
§ Algorithm designed for stateful dataflows
(minimal runtime overhead)
§ Exactly-once semantics

Spark RDD recomputation
§ Immutable data model with
repeatable computation
§ Failed RDDs are recomputed
using their lineage
§ Checkpoint RDDs to reduce
lineage length
§ Parallel recovery of failed
RDDs

State in streaming programs
§ Almost all non-trivial streaming programs are
stateful
§ Stateful operators (in essence):
𝒇:
𝒊𝒏, 𝒔𝒕𝒂𝒕𝒆 ⟶ 𝒐𝒖𝒕, 𝒔𝒕𝒂𝒕𝒆.
§ State hangs around and can be read and
modified as the stream evolves
§ Goal: Get as close as possible while
maintaining scalability and fault-tolerance

§ States available only in Trident API
§ Dedicated operators for state updates and
queries
§ State access methods
• stateQuery(…)
• partitionPersist(…)
• persistentAggregate(…)
§ It’s very difficult to
implement transactional
states
Exactly-‐‑once guarantee

§ Stateless runtime by design
• No continuous operators
• UDFs are assumed to be stateless
§ State can be generated as a separate
stream of RDDs: updateStateByKey(…)
𝒇:
𝑺𝒆𝒒[𝒊𝒏 𝒌], 𝒔𝒕𝒂𝒕𝒆 𝒌 ⟶ 𝒔𝒕𝒂𝒕𝒆.
𝒌
§ 𝒇 is scoped to a specific key

§ Stateful dataflow operators
(Any task can hold state)
§ State changes are stored
as a log by Kafka
§ Custom storage engines can
be plugged in to the log
§ 𝒇 is scoped to a specific task
§ At-least-once processing
semantics

§ Stateful dataflow operators (conceptually
similar to Samza)
§ Two state access patterns
• Local (Task) state
• Partitioned (Key) state
§ Proper API integration
• Java: OperatorState interface
• Scala: mapWithState, flatMapWithState…
§ Exactly-once semantics by checkpointing

Performance
§ Throughput/Latency
• A cost of a network hop is 25+ msecs
• 1 million records/sec/core is nice
§ Size of Network Buffers/Batching
§ Buffer Timeout
§ Cost of Fault Tolerance
§ Operator chaining/Stages
§ Serialization/Types

Closing

Comparison revisited
40
Streaming
model
Native Micro-batching Native Native
API Compositional Declarative Compositional Declarative
Fault tolerance Record ACKs RDD-based Log-based Checkpoints
Guarantee At-least-once Exactly-once At-least-once Exactly-once
State Only in Trident
State as
DStream
Stateful
operators
Stateful
operators
Windowing Not built-in Time based Not built-in Policy based
Latency Very-Low Medium Low Low
Throughput Medium High High High

Summary
§ Streaming applications and stream
processors are very diverse
§ 2 main runtime designs
• Dataflow based (Storm, Samza, Flink)
• Micro-batch based (Spark)
§ The best framework varies based on
application specific needs
§ But high-level APIs are nice J

List of Figures (in order of usage)
§ https://upload.wikimedia.org/wikipedia/commons/thumb/2/2a/CPT-FSM-
abcd.svg/326px-CPT-FSM-abcd.svg.png
§ https://storm.apache.org/images/topology.png
§ https://databricks.com/wp-content/uploads/2015/07/image11-1024x655.png
§ https://people.csail.mit.edu/matei/papers/2012/hotcloud_spark_streaming.pdf,
page 2.
§ http://www.slideshare.net/ptgoetz/storm-hadoop-summit2014, page 69-71.
§ http://samza.apache.org/img/0.9/learn/documentation/container/checkpointi
ng.svg
§ https://storm.apache.org/documentation/images/spout-vs-state.png
§ http://samza.apache.org/img/0.9/learn/documentation/container/stateful_job.
png

Large-Scale Stream Processing in the Hadoop Ecosystem

More Related Content

What's hot

Viewers also liked

Similar to Large-Scale Stream Processing in the Hadoop Ecosystem

Recently uploaded

Large-Scale Stream Processing in the Hadoop Ecosystem