SlideShare a Scribd company logo
Kostas Tzoumas
Flink committer
co-founder, data Artisans
kostas@data-artisans.com
@kostas_tzoumas
Apache
Flink
Apache Flink
2
flink.apache.org
github.com/apache/flink
user@flink.apache.org
Cool squirrel logo (also open source)
The Flink Community
 Over 70 contributors from academia &
industry; growing fast
 Apache Top-Level Project since December
3
Using Flink
4
DataSet and transformations
Input First SecondX Y
Operator X Operator Y
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> input = env.readTextFile(input);
DataSet<String> first = input
.filter (str -> str.contains(“Apache Flink“));
DataSet<String> second = first
.filter (str -> str.length() > 40);
second.print();
env.execute();
5
DataSet
 Central notion of the programming API
 Files and other data sources are read into
DataSets
• DataSet<String> text = env.readTextFile(…)
 Transformations on DataSets produce DataSets
• DataSet<String> first = text.map(…)
 DataSets are printed to files or on stdout
• first.writeAsCsv(…)
 Execution is triggered with env.execute()
6
WordCount
7
text
.flatMap((line,out) -> {
String[] tokens = value.toLowerCase().split(“ “);
for (String token : tokens) {
if (token.length() > 0) {
out.collect(new Tuple2<String, Integer>(token, 1));
}
}
})
.groupBy(0)
.sum(1);
Transitive Closure
8
IterativeDataSet<Tuple2<Long,Long>> paths = edges.iterate (10);
DataSet<Tuple2<Long,Long>> nextPaths = paths
.join(edges).where(1).equalTo(0)
.with((left, right) -> return new Tuple2<Long, Long>(left.f0, right.f1);)
.union(paths)
.distinct();
DataSet<Tuple2<Long, Long>> tc = paths.closeWith(nextPaths);
Scala API
9
text
.flatMap {line => line.split(" ").map(word => Word(word,1)}
.groupBy("word").sum("frequency")
case class Path (from: Long, to: Long)
val tc = edges.iterate(10) { paths: DataSet[Path] =>
val next = paths
.join(edges).where("to").equalTo("from") {
(path, edge) => Path(path.from, edge.to)
}
.union(paths).distinct()
next
}
WordCount
Transitive Closure
Data sources
Batch API
 Files
• HDFS, Local file system,
MapR file system
• Text, Csv, Avro, Hadoop
input formats
 JDBC
 HBase
 Collections
Stream API
 Files
 Socket streams
 Kafka
 RabbitMQ
 Flume
 Collections
 Implement your own
• SourceFunction.collect
10
Available transformations
 map
 flatMap
 filter
 reduce
 reduceGroup
 join
 coGroup
 aggregate
 cross
 project
 distinct
 union
 iterate
 iterateDelta
 repartition
 …
11
Data types and grouping
 Bean-style Java classes & field names
 Tuples and position addressing
 Any data type with key selector function
public static class Access {
public int userId;
public String url;
...
}
public static class User {
public int userId;
public int region;
public Date customerSince;
...
}
DataSet<Tuple2<Access,User>> campaign = access.join(users)
.where(“userId“).equalTo(“userId“)
DataSet<Tuple3<Integer,String,String> someLog;
someLog.groupBy(0,1).reduceGroup(...);
12
Real-time stream processing
13
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> tweets = env.socketTextStream(host,port);
DataStream<Tuple2<String,Integer>> filteredTweets = tweets
.flatMap(new SelectLanguageAndTokenize())
.partition(0)
.map(s -> new Tuple2<String,Integer>(s, 1))
.groupBy(0).sum(1)
.flatMap(new SelectMaxOccurence());
tweets.print();
env.execute();
DataStream instead of DataSet
StreamExecutionEnvironment instead of ExecutionEnvironment
Streaming operators
 Most DataSet
operators can be
used
• map, filter, flatMap,
reduce, reduceGroup,
join, cross, coGroup,
iterate, project,
grouping, partitioning,
aggregations, union
(merge), …
 DataStream-specific
operators (snip)
• CoMap, CoReduce,
etc: share state
between streams
• Temporal binary ops:
join, cross, …
• Windows: policy-
based flexible
windowing
• Time, Count, Delta
14
Windowing
 Trigger policy
• When to trigger the computation on current window
 Eviction policy
• When data points should leave the window
• Defines window width/size
 E.g., count-based policy
• evict when #elements > n
• start a new window every n-th element
 Built-in: Count, Time, Delta policies
15
Windowing example
//Build new model every minute on the last 5 minutes
//worth of data
val model = trainingData
.window(Time.of(5,TimeUnit.MINUTES))
.every(Time.of(1,TimeUnit.MINUTES))
.reduceGroup(buildModel)
//Predict new data using the most up-to-date model
val prediction = newData
.connect(model)
.map(predict); M
P
Training Data
New Data Prediction
12/01/15 16
Window Join example
case class Name(id: Long, name: String)
case class Age(id: Long, age: Int)
case class Person(name: String, age: Int)
val names = ...
val ages = ...
names.join(ages)
.onWindow(5, TimeUnit.SECONDS)
.where("id")
.equalTo("id") {(n, a) => Person(n.name, a.age)}
12/01/15 17
Iterative processing example
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.generateSequence(1, 10).iterate(incrementToTen, 1000)
.print
env.execute("Iterative example")
def incrementToTen(input: DataStream[Long]) = {
val incremented = input.map {_ + 1}
val split = incremented.split
{x => if (x >= 10) "out" else "feedback"}
(split.select("feedback"), split.select("out"))
}
12/01/15 18
Number
stream
Map Reduce
Output
stream
“out”
“feedback”
Spargel: Flink’s Graph API
19
DataSet<Tuple2<Long, Long>> result = vertices
.runOperation(VertexCentricIteration.withPlainEdges(
edges, new CCUpdater(), new CCMessager(), 100));
class CCUpdater extends VertexUpdateFunction …
class CCMessenger extends MessagingFunction …
Spargel is a very thin layer (~500 LOC) on top of
Flink’s delta iterations
Other API elements & tools
 Accumulators and counters
• Int, Long, Double counters
• Histogram accumulator
• Define your own
 Broadcast variables
 Plan visualization
 Local debugging/testing mode
20
Hadoop Compatibility
 Flink runs on YARN, can read from HDFS, HBase
 Can use all Hadoop input and output formats
 Can use unmodified Hadoop Mappers and Reducers
and mix-and-match them with Flink operators
 Coming up: can run unmodified Hadoop jobs (faster)
21
Visualization tools
22
Visualization tools
23
Visualization tools
24
Flink runtime
features
25
Task
Manager
Job
Manager
Task
Manager
Flink Client &
Optimizer
DataSet<String> text = env.readTextFile(input);
DataSet<Tuple2<String, Integer>> result = text
.flatMap((str, out) -> {
for (String token : value.split("W")) {
out.collect(new Tuple2<>(token, 1));
})
.groupBy(0)
.aggregate(SUM, 1);
O Romeo,
Romeo,
wherefore
art thou
Romeo?
O, 1
Romeo, 3
wherefore, 1
art, 1
thou, 1
Apache Flink
26
Nor arm,
nor face,
nor any
other part
nor, 3
arm, 1
face, 1,
any, 1,
other, 1
part, 1
If you need to know one
thing about Flink is that
you don’t need to know
the internals of Flink.
27
Philosophy
 Flink “hides” its internal workings from the
user
 This is good
• User does not worry about how jobs are executed
• Internals can be changed without breaking
changes
 … and bad
• Execution model more complicated to explain
compared to MapReduce or Spark RDD
28
Recap: DataSet
Input First SecondX Y
Operator X Operator Y
29
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> input = env.readTextFile(input);
DataSet<String> first = input
.filter (str -> str.contains(“Apache Flink“));
DataSet<String> second = first
.filter (str -> str.length() > 40);
second.print()
env.execute();
Common misconception
 Programs are not executed eagerly
 Instead, system compiles program to an
execution plan and executes that plan
Input First SecondX Y
30
DataSet<String>
 Think of it as a PCollection<String>, or a
Spark RDD[String]
 With a major difference: it can be
produced/recovered in several ways
• … like a Java collection
• … like an RDD
• … perhaps it is never fully materialized (because
the program does not need it to)
• … implicitly updated in an iteration
 And this is transparent to the user
31
Example: grep
Romeo,
Romeo,
where art
thou Romeo?
Load Log
Search
for str1
Search
for str2
Search
for str3
Grep 1
Grep 2
Grep 3
32
Staged (batch) execution
Romeo,
Romeo,
where art
thou Romeo?
Load Log
Search
for str1
Search
for str2
Search
for str3
Grep 1
Grep 2
Grep 3
Stage 1:
Create/cache Log
Subseqent stages:
Grep log for matches
Caching in-memory
and disk if needed
33
Pipelined execution
Romeo,
Romeo,
where art
thou Romeo?
Load Log
Search
for str1
Search
for str2
Search
for str3
Grep 1
Grep 2
Grep 3
001100110011001100110011
Stage 1:
Deploy and start operators
Data transfer in-
memory and disk if
needed 34
Note: Log
DataSet is
never
“created”!
Benefits of pipelining
 25 node cluster
 Grep log for 3
terms
 Scale data size
from 100GB to
1TB
0
250
500
750
1000
1250
1500
1750
2000
2250
2500
0 100 200 300 400 500 600 700 800 900 1000
Timetocompletegrep(sec)
Data size (GB)Cluster memory
exceeded 35
36
Drawbacks of pipelining
 Long pipelines may be active at the same time leading to
memory fragmentation
• FLINK-1101: Changes memory allocation from static to adaptive
 Fault-tolerance harder to get right
• FLINK-986: Adds intermediate data sets (similar to RDDS) as
first-class citizen to Flink Runtime. Will lead to fine-grained fault-
tolerance among other features.
37
Example: Iterative processing
DataSet<Page> pages = ...
DataSet<Neighborhood> edges = ...
DataSet<Page> oldRanks = pages; DataSet<Page> newRanks;
for (i = 0; i < maxIterations; i++) {
newRanks = update(oldRanks, edges)
oldRanks = newRanks
}
DataSet<Page> result = newRanks;
DataSet<Page> update (DataSet<Page> ranks, DataSet<Neighborhood> adjacency) {
return oldRanks
.join(adjacency)
.where(“id“).equalTo(“id“)
.with ( (page, adj, out) -> {
for (long n : adj.neighbors)
out.collect(new Page(n, df * page.rank / adj.neighbors.length))
})
.groupBy(“id“)
.reduce ( (a, b) -> new Page(a.id, a.rank + b.rank) );
38
Iterate by unrolling
 for/while loop in client submits one job per iteration
step
 Data reuse by caching in memory and/or disk
Step Step Step Step Step
Client
39
Iterate natively
DataSet<Page> pages = ...
DataSet<Neighborhood> edges = ...
IterativeDataSet<Page> pagesIter = pages.iterate(maxIterations);
DataSet<Page> newRanks = update (pagesIter, edges);
DataSet<Page> result = pagesIter.closeWith(newRanks)
40
partial
solution
partial
solutionX
other
datasets
Y
initial
solution
iteration
result
Replace
Step function
Iterate natively with deltas
DeltaIteration<...> pagesIter = pages.iterateDelta(initialDeltas, maxIterations, 0);
DataSet<...> newRanks = update (pagesIter, edges);
DataSet<...> newRanks = ...
DataSet<...> result = pagesIter.closeWith(newRanks, deltas)
See http://data-artisans.com/data-analysis-with-flink.html 41
partial
solution
delta
setX
other
datasets
Y
initial
solution
iteration
result
workset A B workset
Merge deltas
Replace
initial
workset
Native, unrolling, and delta
42
Optimization of iterative algorithms
43
Caching Loop-invariant Data
Pushing work
„out of the loop“
Maintain state as index
Dissecting
Flink
44
45
The growing Flink stack
46
Flink Optimizer Flink Stream Builder
Common API
Scala API Java API
Python API
(upcoming)
Graph API
Apache
MRQL
Flink Local Runtime
Embedded
environment
(Java collections)
Local
Environment
(for debugging)
Remote environment
(Regular cluster execution)
Apache Tez
Data
storage
HDFSFiles S3 JDBC Redis
Rabbit
MQ
Kafka
Azure
tables
…
Single node execution Standalone or YARN cluster
Program lifecycle
47
val source1 = …
val source2 = …
val maxed = source1
.map(v => (v._1,v._2,
math.max(v._1,v._2))
val filtered = source2
.filter(v => (v._1 > 4))
val result = maxed
.join(filtered).where(0).equalTo(0)
.filter(_1 > 3)
.groupBy(0)
.reduceGroup {……}
1
3
4
5
2
 The optimizer is the
component that selects
an execution plan for a
Common API program
 Think of an AI system
manipulating your
program for you 
 But don’t be scared – it
works
• Relational databases have
been doing this for
decades – Flink ports the
technology to API-based
systems
Flink Optimizer
48
A simple program
49
DataSet<Tuple5<Integer, String, String, String, Integer>> orders = …
DataSet<Tuple2<Integer, Double>> lineitems = …
DataSet<Tuple2<Integer, Integer>> filteredOrders = orders
.filter(. . .)
.project(0,4).types(Integer.class, Integer.class);
DataSet<Tuple3<Integer, Integer, Double>> lineitemsOfOrders = filteredOrders
.join(lineitems)
.where(0).equalTo(0)
.projectFirst(0,1).projectSecond(1)
.types(Integer.class, Integer.class, Double.class);
DataSet<Tuple3<Integer, Integer, Double>> priceSums = lineitemsOfOrders
.groupBy(0,1).aggregate(Aggregations.SUM, 2);
priceSums.writeAsCsv(outputPath);
Two execution plans
50
DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
broadcast forward
Combine
GroupRed
sort
DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
hash-part [0] hash-part [0]
hash-part [0,1]
GroupRed
sort
forwardBest plan
depends on
relative sizes
of input files
Flink Local Runtime
51
 Local runtime, not
the distributed
execution engine
 Aka: what happens
inside every parallel
task
Flink runtime operators
 Sorting and hashing data
• Necessary for grouping, aggregation, reduce,
join, cogroup, delta iterations
 Flink contains tailored implementations of
hybrid hashing and external sorting in
Java
• Scale well with both abundant and restricted
memory sizes
52
Internal data representation
53
JVM Heap
map
JVM Heap
reduce
O Romeo,
Romeo,
wherefore
art thou
Romeo?
00110011
00110011
00010111
01110001
01111010
00010111
art, 1
O, 1
Romeo, 1
Romeo, 1
00110011
Network transfer
Local sort
How is intermediate data internally represented?
Internal data representation
 Two options: Java objects or raw bytes
 Java objects
• Easier to program
• Can suffer from GC overhead
• Hard to de-stage data to disk, may suffer from “out of
memory exceptions”
 Raw bytes
• Harder to program (customer serialization stack, more
involved runtime operators)
• Solves most of memory and GC problems
• Overhead from object (de)serialization
 Flink follows the raw byte approach
54
Memory in Flink
public class WC {
public String word;
public int count;
}
empty
page
Pool of Memory Pages
JVM Heap
Sorting,
hashing,
caching
Shuffling,
broadcasts
User code
objects
Network
buffers
Managed
heap
Unmanaged
heap
55
Memory in Flink (2)
 Internal memory management
• Flink initially allocates 70% of the free heap as byte[]
segments
• Internal operators allocate() and release() these
segments
 Flink has its own serialization stack
• All accepted data types serialized to data segments
 Easy to reason about memory, (almost) no
OutOfMemory errors, reduces the pressure to the
GC (smooth performance)
56
Operating on serialized data
Microbenchmark
 Sorting 1GB worth of (long, double) tuples
 67,108,864 elements
 Simple quicksort
57
Flink distributed execution
58
 Pipelined
• Same engine for Flink
and Flink streaming
 Pluggable
• Local runtime can be
executed on other
engines
• E.g., Java collections
and Apache Tez
Coordination built on Akka library
Streaming overview
 Streaming and batch use same code
paths in runtime
 Differences
• Streaming does not use Flink’s memory
management right now
• Streaming uses its own compiler/optimizer
59
Roadmap
60
Flink Roadmap
 Currently being discussed by the Flink
community
 Flink has a major release every 3 months,
and one or more bug-fixing releases between
major releases
 Caveat: rough roadmap, depends on
volunteer work, outcome of community
discussion, and Apache open source
processes
61
Roadmap for 2015 (highlights)
Q1 Q2 Q3
APIs Logical Query
integration
Additional
operators
Interactive
programs
Interactive
Scala shell
SQL-on-Flink
Optimizer Semantic
annotations
HCatalog
integration
Optimizer
hints
Runtime Dual engine
(blocking &
pipelining)
Fine-grained
fault
tolerance
Dynamic
memory
allocation
Streaming Better
memory
management
More
operators in
API
At-least-once
processing
guarantees
Unify batch
and
streaming
Exactly-once
processing
guarantees
ML library First version Additional
algorithms
Mahout
integration
Graph
library
First version
Integration Tez, Samoa Mahout
62
Dual streaming and batch engine
 Runtime will support
• Current pipelined
execution mode
• Blocking (RDD-like)
execution
 Batch programs use a
combination
 Stream programs use
pipelining
 Interactive programs
need blocking
 Expected Q1 2015
Blocking
runtime
execution
Pipelined
runtime
execution
Batch API
(DataSet) ✔ ✔
Stream API
(DataStream) ✔
63
Fault tolerance improvements
 Fine-grained fault tolerance for batch
programs
• Currently lineage goes back to sources unless
user explicitly persists intermediate data set
• In the future replay will restart from
automatically checkpointed intermediate data
set
 Expected Q1 2015
64
Streaming fault tolerance
 At-least-once guarantees
• Reset of operator state
• Source in-memory replication
• Expected Q1 2015
 Exactly-once guarantees
• Checkpoint operator state
• Rest similar to batch fine-grained recovery
• Expected Q2 2015
65
Interactive programs & shells
Support interactive ad-hoc data analysis
 Programs that are executed partially in the cluster
and partially in the client
• Needs same runtime code as fine-grained batch fault
tolerance
• Expected Q1 2015
 Interactive Scala shell
• Needs to transfer class files between shell and
master
• Expected Q2 2015
• Want to contribute?
66
Machine Learning
Machine learning capabilities on Flink will stand
on two legs
 Dedicated Machine Learning library
• Optimized algorithms
• First version (with infrastructure code, ALS, k-
means, logistic regression) open for contributions
in Q1 2015
 Mahout linear algebra DSL integration
• Expected Q2 2015
67
Graph Library
 Codename “Gelly”
 Library with graph operations
• Common graph stats, PageRank, SSSP,
Connected Components, label propagation
• Vertex-centric API
• Gather-apply-scatter API
 Expected Q1 2015
68
Logical Query Integration
 Define queries in SQL style inside
Scala/Java similar to Microsoft LINQ
 Expected Q1 2015
69
DataSet<Row> dates = env.readCsv(...).as("order_id", "date");
dates.filter("date < 2010-7-14")
.join(other).where("order_id").equalto("id");
SQL on Flink
 Allow HiveQL queries to run on Flink directly,
both standalone and embedded in batch
programs
 Approach: translate SQL to Logical Query
Interface
• Hey, Flink already has an optimizer 
 Currently in design phase
 Expected Q3/Q4 2015
70
Integration with other projects
 Apache Tez
• Expected Q1 2015
 Apache Mahout
• Expected Q2 2015
 Tachyon
• Basic integration
available, waiting for
Tachyon API
 Apache Zepelin (inc.)
• Interest from both
communities
 Apache Samoa (inc.)
• Expected Q1 2015
• Interest from both
communities
 H2O
• In consideration
 Apache Hive
• Expected Q3/Q4 2015
71
And many more…
 Runtime: even better performance and
robustness
• Using off-heap memory, dynamic memory
allocation
 Improvements to the Flink optimizer
• Integration with HCatalog, better statistics
• Runtime optimization
 Streaming graph and ML pipeline libraries
72
Closing
73
Stay informed
 flink. apache.org
• Subscribe to the mailing lists!
• http://flink.apache.org/community.html#mailing-lists
 Blogs
• flink.apache.org/blog
• data-artisans.com/blog
 Twitter
• follow @ApacheFlink
74
flink.apache.org

More Related Content

What's hot

Apache airflow
Apache airflowApache airflow
Apache airflow
Purna Chander
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
Kafka streams windowing behind the curtain
Kafka streams windowing behind the curtain Kafka streams windowing behind the curtain
Kafka streams windowing behind the curtain
confluent
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
Michael Noll
 
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan EwenAdvanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
confluent
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
Ververica
 
Advanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applicationsAdvanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applications
Aljoscha Krettek
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
DataWorks Summit
 
Extending Flink SQL for stream processing use cases
Extending Flink SQL for stream processing use casesExtending Flink SQL for stream processing use cases
Extending Flink SQL for stream processing use cases
Flink Forward
 
Designing Structured Streaming Pipelines—How to Architect Things Right
Designing Structured Streaming Pipelines—How to Architect Things RightDesigning Structured Streaming Pipelines—How to Architect Things Right
Designing Structured Streaming Pipelines—How to Architect Things Right
Databricks
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
Aljoscha Krettek
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
Araf Karsh Hamid
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
Jonas Bonér
 
Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.
Dan Harvey
 
Airflow Clustering and High Availability
Airflow Clustering and High AvailabilityAirflow Clustering and High Availability
Airflow Clustering and High Availability
Robert Sanders
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
DataWorks Summit/Hadoop Summit
 
Introducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using itIntroducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using it
Bruno Faria
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
HostedbyConfluent
 

What's hot (20)

Apache airflow
Apache airflowApache airflow
Apache airflow
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
 
Kafka streams windowing behind the curtain
Kafka streams windowing behind the curtain Kafka streams windowing behind the curtain
Kafka streams windowing behind the curtain
 
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan EwenAdvanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
 
Advanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applicationsAdvanced Flink Training - Design patterns for streaming applications
Advanced Flink Training - Design patterns for streaming applications
 
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data ProcessingApache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
 
Extending Flink SQL for stream processing use cases
Extending Flink SQL for stream processing use casesExtending Flink SQL for stream processing use cases
Extending Flink SQL for stream processing use cases
 
Designing Structured Streaming Pipelines—How to Architect Things Right
Designing Structured Streaming Pipelines—How to Architect Things RightDesigning Structured Streaming Pipelines—How to Architect Things Right
Designing Structured Streaming Pipelines—How to Architect Things Right
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
 
Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.
 
Airflow Clustering and High Availability
Airflow Clustering and High AvailabilityAirflow Clustering and High Availability
Airflow Clustering and High Availability
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
 
Introducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using itIntroducing Apache Airflow and how we are using it
Introducing Apache Airflow and how we are using it
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
 
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...
 

Viewers also liked

Alexander Kolb – Flink. Yet another Streaming Framework?
Alexander Kolb – Flink. Yet another Streaming Framework?Alexander Kolb – Flink. Yet another Streaming Framework?
Alexander Kolb – Flink. Yet another Streaming Framework?
Flink Forward
 
Christian Kreuzfeld – Static vs Dynamic Stream Processing
Christian Kreuzfeld – Static vs Dynamic Stream ProcessingChristian Kreuzfeld – Static vs Dynamic Stream Processing
Christian Kreuzfeld – Static vs Dynamic Stream Processing
Flink Forward
 
Aljoscha Krettek – Notions of Time
Aljoscha Krettek – Notions of TimeAljoscha Krettek – Notions of Time
Aljoscha Krettek – Notions of Time
Flink Forward
 
Apache Flink Training: System Overview
Apache Flink Training: System OverviewApache Flink Training: System Overview
Apache Flink Training: System Overview
Flink Forward
 
Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School
Flink Forward
 
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache FlinkMartin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
Flink Forward
 
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache FlinkAlbert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
Flink Forward
 
Matthias J. Sax – A Tale of Squirrels and Storms
Matthias J. Sax – A Tale of Squirrels and StormsMatthias J. Sax – A Tale of Squirrels and Storms
Matthias J. Sax – A Tale of Squirrels and Storms
Flink Forward
 
Flink 0.10 @ Bay Area Meetup (October 2015)
Flink 0.10 @ Bay Area Meetup (October 2015)Flink 0.10 @ Bay Area Meetup (October 2015)
Flink 0.10 @ Bay Area Meetup (October 2015)
Stephan Ewen
 
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Flink Forward
 
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
S. Bartoli & F. Pompermaier – A Semantic Big Data CompanionS. Bartoli & F. Pompermaier – A Semantic Big Data Companion
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
Flink Forward
 
Fabian Hueske – Cascading on Flink
Fabian Hueske – Cascading on FlinkFabian Hueske – Cascading on Flink
Fabian Hueske – Cascading on Flink
Flink Forward
 
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
Anwar Rizal – Streaming & Parallel Decision Tree in FlinkAnwar Rizal – Streaming & Parallel Decision Tree in Flink
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
Flink Forward
 
Apache Flink Training: DataStream API Part 2 Advanced
Apache Flink Training: DataStream API Part 2 Advanced Apache Flink Training: DataStream API Part 2 Advanced
Apache Flink Training: DataStream API Part 2 Advanced
Flink Forward
 
Kamal Hakimzadeh – Reproducible Distributed Experiments
Kamal Hakimzadeh – Reproducible Distributed ExperimentsKamal Hakimzadeh – Reproducible Distributed Experiments
Kamal Hakimzadeh – Reproducible Distributed Experiments
Flink Forward
 
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSLSebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Flink Forward
 
Assaf Araki – Real Time Analytics at Scale
Assaf Araki – Real Time Analytics at ScaleAssaf Araki – Real Time Analytics at Scale
Assaf Araki – Real Time Analytics at Scale
Flink Forward
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward KeynoteK. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward Keynote
Flink Forward
 
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Till Rohrmann – Fault Tolerance and Job Recovery in Apache FlinkTill Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Flink Forward
 
Fabian Hueske – Juggling with Bits and Bytes
Fabian Hueske – Juggling with Bits and BytesFabian Hueske – Juggling with Bits and Bytes
Fabian Hueske – Juggling with Bits and Bytes
Flink Forward
 

Viewers also liked (20)

Alexander Kolb – Flink. Yet another Streaming Framework?
Alexander Kolb – Flink. Yet another Streaming Framework?Alexander Kolb – Flink. Yet another Streaming Framework?
Alexander Kolb – Flink. Yet another Streaming Framework?
 
Christian Kreuzfeld – Static vs Dynamic Stream Processing
Christian Kreuzfeld – Static vs Dynamic Stream ProcessingChristian Kreuzfeld – Static vs Dynamic Stream Processing
Christian Kreuzfeld – Static vs Dynamic Stream Processing
 
Aljoscha Krettek – Notions of Time
Aljoscha Krettek – Notions of TimeAljoscha Krettek – Notions of Time
Aljoscha Krettek – Notions of Time
 
Apache Flink Training: System Overview
Apache Flink Training: System OverviewApache Flink Training: System Overview
Apache Flink Training: System Overview
 
Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School Vasia Kalavri – Training: Gelly School
Vasia Kalavri – Training: Gelly School
 
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache FlinkMartin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink
 
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache FlinkAlbert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
 
Matthias J. Sax – A Tale of Squirrels and Storms
Matthias J. Sax – A Tale of Squirrels and StormsMatthias J. Sax – A Tale of Squirrels and Storms
Matthias J. Sax – A Tale of Squirrels and Storms
 
Flink 0.10 @ Bay Area Meetup (October 2015)
Flink 0.10 @ Bay Area Meetup (October 2015)Flink 0.10 @ Bay Area Meetup (October 2015)
Flink 0.10 @ Bay Area Meetup (October 2015)
 
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
 
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
S. Bartoli & F. Pompermaier – A Semantic Big Data CompanionS. Bartoli & F. Pompermaier – A Semantic Big Data Companion
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
 
Fabian Hueske – Cascading on Flink
Fabian Hueske – Cascading on FlinkFabian Hueske – Cascading on Flink
Fabian Hueske – Cascading on Flink
 
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
Anwar Rizal – Streaming & Parallel Decision Tree in FlinkAnwar Rizal – Streaming & Parallel Decision Tree in Flink
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
 
Apache Flink Training: DataStream API Part 2 Advanced
Apache Flink Training: DataStream API Part 2 Advanced Apache Flink Training: DataStream API Part 2 Advanced
Apache Flink Training: DataStream API Part 2 Advanced
 
Kamal Hakimzadeh – Reproducible Distributed Experiments
Kamal Hakimzadeh – Reproducible Distributed ExperimentsKamal Hakimzadeh – Reproducible Distributed Experiments
Kamal Hakimzadeh – Reproducible Distributed Experiments
 
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSLSebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
 
Assaf Araki – Real Time Analytics at Scale
Assaf Araki – Real Time Analytics at ScaleAssaf Araki – Real Time Analytics at Scale
Assaf Araki – Real Time Analytics at Scale
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward KeynoteK. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward Keynote
 
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Till Rohrmann – Fault Tolerance and Job Recovery in Apache FlinkTill Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
 
Fabian Hueske – Juggling with Bits and Bytes
Fabian Hueske – Juggling with Bits and BytesFabian Hueske – Juggling with Bits and Bytes
Fabian Hueske – Juggling with Bits and Bytes
 

Similar to Apache Flink: API, runtime, and project roadmap

Flink internals web
Flink internals web Flink internals web
Flink internals web
Kostas Tzoumas
 
Apache Flink Deep Dive
Apache Flink Deep DiveApache Flink Deep Dive
Apache Flink Deep Dive
Vasia Kalavri
 
Introduction to Apache Flink
Introduction to Apache FlinkIntroduction to Apache Flink
Introduction to Apache Flink
mxmxm
 
Apache Flink Stream Processing
Apache Flink Stream ProcessingApache Flink Stream Processing
Apache Flink Stream Processing
Suneel Marthi
 
Hadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log projectHadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log project
Mao Geng
 
So you think you can stream.pptx
So you think you can stream.pptxSo you think you can stream.pptx
So you think you can stream.pptx
Prakash Chockalingam
 
Spark what's new what's coming
Spark what's new what's comingSpark what's new what's coming
Spark what's new what's coming
Databricks
 
Hack Like It's 2013 (The Workshop)
Hack Like It's 2013 (The Workshop)Hack Like It's 2013 (The Workshop)
Hack Like It's 2013 (The Workshop)
Itzik Kotler
 
Data race
Data raceData race
Data race
James Wong
 
Apache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathurApache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathur
Siddharth Mathur
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
spark-project
 
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Strata Singapore: GearpumpReal time DAG-Processing with Akka at ScaleStrata Singapore: GearpumpReal time DAG-Processing with Akka at Scale
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Sean Zhong
 
AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)
Paul Chao
 
Hadoop-Introduction
Hadoop-IntroductionHadoop-Introduction
Hadoop-Introduction
Sandeep Deshmukh
 
Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoop
Apache Apex
 
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbsSystem Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
ashukiller7
 
Hierarchical free monads and software design in fp
Hierarchical free monads and software design in fpHierarchical free monads and software design in fp
Hierarchical free monads and software design in fp
Alexander Granin
 
Apache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathurApache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathur
Siddharth Mathur
 
Influx data basic
Influx data basicInflux data basic
Influx data basic
Сергій Саварин
 
Airflow tutorials hands_on
Airflow tutorials hands_onAirflow tutorials hands_on
Airflow tutorials hands_on
pko89403
 

Similar to Apache Flink: API, runtime, and project roadmap (20)

Flink internals web
Flink internals web Flink internals web
Flink internals web
 
Apache Flink Deep Dive
Apache Flink Deep DiveApache Flink Deep Dive
Apache Flink Deep Dive
 
Introduction to Apache Flink
Introduction to Apache FlinkIntroduction to Apache Flink
Introduction to Apache Flink
 
Apache Flink Stream Processing
Apache Flink Stream ProcessingApache Flink Stream Processing
Apache Flink Stream Processing
 
Hadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log projectHadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log project
 
So you think you can stream.pptx
So you think you can stream.pptxSo you think you can stream.pptx
So you think you can stream.pptx
 
Spark what's new what's coming
Spark what's new what's comingSpark what's new what's coming
Spark what's new what's coming
 
Hack Like It's 2013 (The Workshop)
Hack Like It's 2013 (The Workshop)Hack Like It's 2013 (The Workshop)
Hack Like It's 2013 (The Workshop)
 
Data race
Data raceData race
Data race
 
Apache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathurApache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathur
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
 
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Strata Singapore: GearpumpReal time DAG-Processing with Akka at ScaleStrata Singapore: GearpumpReal time DAG-Processing with Akka at Scale
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
 
AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)
 
Hadoop-Introduction
Hadoop-IntroductionHadoop-Introduction
Hadoop-Introduction
 
Introduction to Hadoop
Introduction to HadoopIntroduction to Hadoop
Introduction to Hadoop
 
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbsSystem Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
System Calls.pptxnsjsnssbhsbbebdbdbshshsbshsbbs
 
Hierarchical free monads and software design in fp
Hierarchical free monads and software design in fpHierarchical free monads and software design in fp
Hierarchical free monads and software design in fp
 
Apache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathurApache pig presentation_siddharth_mathur
Apache pig presentation_siddharth_mathur
 
Influx data basic
Influx data basicInflux data basic
Influx data basic
 
Airflow tutorials hands_on
Airflow tutorials hands_onAirflow tutorials hands_on
Airflow tutorials hands_on
 

More from Kostas Tzoumas

Debunking Common Myths in Stream Processing
Debunking Common Myths in Stream ProcessingDebunking Common Myths in Stream Processing
Debunking Common Myths in Stream Processing
Kostas Tzoumas
 
Debunking Six Common Myths in Stream Processing
Debunking Six Common Myths in Stream ProcessingDebunking Six Common Myths in Stream Processing
Debunking Six Common Myths in Stream Processing
Kostas Tzoumas
 
Streaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkStreaming in the Wild with Apache Flink
Streaming in the Wild with Apache Flink
Kostas Tzoumas
 
Apache Flink at Strata San Jose 2016
Apache Flink at Strata San Jose 2016Apache Flink at Strata San Jose 2016
Apache Flink at Strata San Jose 2016
Kostas Tzoumas
 
First Flink Bay Area meetup
First Flink Bay Area meetupFirst Flink Bay Area meetup
First Flink Bay Area meetup
Kostas Tzoumas
 
Flink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseFlink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San Jose
Kostas Tzoumas
 

More from Kostas Tzoumas (6)

Debunking Common Myths in Stream Processing
Debunking Common Myths in Stream ProcessingDebunking Common Myths in Stream Processing
Debunking Common Myths in Stream Processing
 
Debunking Six Common Myths in Stream Processing
Debunking Six Common Myths in Stream ProcessingDebunking Six Common Myths in Stream Processing
Debunking Six Common Myths in Stream Processing
 
Streaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkStreaming in the Wild with Apache Flink
Streaming in the Wild with Apache Flink
 
Apache Flink at Strata San Jose 2016
Apache Flink at Strata San Jose 2016Apache Flink at Strata San Jose 2016
Apache Flink at Strata San Jose 2016
 
First Flink Bay Area meetup
First Flink Bay Area meetupFirst Flink Bay Area meetup
First Flink Bay Area meetup
 
Flink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseFlink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San Jose
 

Recently uploaded

NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
Alison B. Lowndes
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
Michael Price
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Intel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdfIntel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdf
Tech Guru
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
Zilliz
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
Priyanka Aash
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
FIDO Alliance
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3
DianaGray10
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Zilliz
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
Priyanka Aash
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1
DianaGray10
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 

Recently uploaded (20)

NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Intel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdfIntel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdf
 
The History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal EmbeddingsThe History of Embeddings & Multimodal Embeddings
The History of Embeddings & Multimodal Embeddings
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1Discovery Series - Zero to Hero - Task Mining Session 1
Discovery Series - Zero to Hero - Task Mining Session 1
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 

Apache Flink: API, runtime, and project roadmap

  • 1. Kostas Tzoumas Flink committer co-founder, data Artisans kostas@data-artisans.com @kostas_tzoumas Apache Flink
  • 3. The Flink Community  Over 70 contributors from academia & industry; growing fast  Apache Top-Level Project since December 3
  • 5. DataSet and transformations Input First SecondX Y Operator X Operator Y ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<String> input = env.readTextFile(input); DataSet<String> first = input .filter (str -> str.contains(“Apache Flink“)); DataSet<String> second = first .filter (str -> str.length() > 40); second.print(); env.execute(); 5
  • 6. DataSet  Central notion of the programming API  Files and other data sources are read into DataSets • DataSet<String> text = env.readTextFile(…)  Transformations on DataSets produce DataSets • DataSet<String> first = text.map(…)  DataSets are printed to files or on stdout • first.writeAsCsv(…)  Execution is triggered with env.execute() 6
  • 7. WordCount 7 text .flatMap((line,out) -> { String[] tokens = value.toLowerCase().split(“ “); for (String token : tokens) { if (token.length() > 0) { out.collect(new Tuple2<String, Integer>(token, 1)); } } }) .groupBy(0) .sum(1);
  • 8. Transitive Closure 8 IterativeDataSet<Tuple2<Long,Long>> paths = edges.iterate (10); DataSet<Tuple2<Long,Long>> nextPaths = paths .join(edges).where(1).equalTo(0) .with((left, right) -> return new Tuple2<Long, Long>(left.f0, right.f1);) .union(paths) .distinct(); DataSet<Tuple2<Long, Long>> tc = paths.closeWith(nextPaths);
  • 9. Scala API 9 text .flatMap {line => line.split(" ").map(word => Word(word,1)} .groupBy("word").sum("frequency") case class Path (from: Long, to: Long) val tc = edges.iterate(10) { paths: DataSet[Path] => val next = paths .join(edges).where("to").equalTo("from") { (path, edge) => Path(path.from, edge.to) } .union(paths).distinct() next } WordCount Transitive Closure
  • 10. Data sources Batch API  Files • HDFS, Local file system, MapR file system • Text, Csv, Avro, Hadoop input formats  JDBC  HBase  Collections Stream API  Files  Socket streams  Kafka  RabbitMQ  Flume  Collections  Implement your own • SourceFunction.collect 10
  • 11. Available transformations  map  flatMap  filter  reduce  reduceGroup  join  coGroup  aggregate  cross  project  distinct  union  iterate  iterateDelta  repartition  … 11
  • 12. Data types and grouping  Bean-style Java classes & field names  Tuples and position addressing  Any data type with key selector function public static class Access { public int userId; public String url; ... } public static class User { public int userId; public int region; public Date customerSince; ... } DataSet<Tuple2<Access,User>> campaign = access.join(users) .where(“userId“).equalTo(“userId“) DataSet<Tuple3<Integer,String,String> someLog; someLog.groupBy(0,1).reduceGroup(...); 12
  • 13. Real-time stream processing 13 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<String> tweets = env.socketTextStream(host,port); DataStream<Tuple2<String,Integer>> filteredTweets = tweets .flatMap(new SelectLanguageAndTokenize()) .partition(0) .map(s -> new Tuple2<String,Integer>(s, 1)) .groupBy(0).sum(1) .flatMap(new SelectMaxOccurence()); tweets.print(); env.execute(); DataStream instead of DataSet StreamExecutionEnvironment instead of ExecutionEnvironment
  • 14. Streaming operators  Most DataSet operators can be used • map, filter, flatMap, reduce, reduceGroup, join, cross, coGroup, iterate, project, grouping, partitioning, aggregations, union (merge), …  DataStream-specific operators (snip) • CoMap, CoReduce, etc: share state between streams • Temporal binary ops: join, cross, … • Windows: policy- based flexible windowing • Time, Count, Delta 14
  • 15. Windowing  Trigger policy • When to trigger the computation on current window  Eviction policy • When data points should leave the window • Defines window width/size  E.g., count-based policy • evict when #elements > n • start a new window every n-th element  Built-in: Count, Time, Delta policies 15
  • 16. Windowing example //Build new model every minute on the last 5 minutes //worth of data val model = trainingData .window(Time.of(5,TimeUnit.MINUTES)) .every(Time.of(1,TimeUnit.MINUTES)) .reduceGroup(buildModel) //Predict new data using the most up-to-date model val prediction = newData .connect(model) .map(predict); M P Training Data New Data Prediction 12/01/15 16
  • 17. Window Join example case class Name(id: Long, name: String) case class Age(id: Long, age: Int) case class Person(name: String, age: Int) val names = ... val ages = ... names.join(ages) .onWindow(5, TimeUnit.SECONDS) .where("id") .equalTo("id") {(n, a) => Person(n.name, a.age)} 12/01/15 17
  • 18. Iterative processing example val env = StreamExecutionEnvironment.getExecutionEnvironment env.generateSequence(1, 10).iterate(incrementToTen, 1000) .print env.execute("Iterative example") def incrementToTen(input: DataStream[Long]) = { val incremented = input.map {_ + 1} val split = incremented.split {x => if (x >= 10) "out" else "feedback"} (split.select("feedback"), split.select("out")) } 12/01/15 18 Number stream Map Reduce Output stream “out” “feedback”
  • 19. Spargel: Flink’s Graph API 19 DataSet<Tuple2<Long, Long>> result = vertices .runOperation(VertexCentricIteration.withPlainEdges( edges, new CCUpdater(), new CCMessager(), 100)); class CCUpdater extends VertexUpdateFunction … class CCMessenger extends MessagingFunction … Spargel is a very thin layer (~500 LOC) on top of Flink’s delta iterations
  • 20. Other API elements & tools  Accumulators and counters • Int, Long, Double counters • Histogram accumulator • Define your own  Broadcast variables  Plan visualization  Local debugging/testing mode 20
  • 21. Hadoop Compatibility  Flink runs on YARN, can read from HDFS, HBase  Can use all Hadoop input and output formats  Can use unmodified Hadoop Mappers and Reducers and mix-and-match them with Flink operators  Coming up: can run unmodified Hadoop jobs (faster) 21
  • 26. Task Manager Job Manager Task Manager Flink Client & Optimizer DataSet<String> text = env.readTextFile(input); DataSet<Tuple2<String, Integer>> result = text .flatMap((str, out) -> { for (String token : value.split("W")) { out.collect(new Tuple2<>(token, 1)); }) .groupBy(0) .aggregate(SUM, 1); O Romeo, Romeo, wherefore art thou Romeo? O, 1 Romeo, 3 wherefore, 1 art, 1 thou, 1 Apache Flink 26 Nor arm, nor face, nor any other part nor, 3 arm, 1 face, 1, any, 1, other, 1 part, 1
  • 27. If you need to know one thing about Flink is that you don’t need to know the internals of Flink. 27
  • 28. Philosophy  Flink “hides” its internal workings from the user  This is good • User does not worry about how jobs are executed • Internals can be changed without breaking changes  … and bad • Execution model more complicated to explain compared to MapReduce or Spark RDD 28
  • 29. Recap: DataSet Input First SecondX Y Operator X Operator Y 29 ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<String> input = env.readTextFile(input); DataSet<String> first = input .filter (str -> str.contains(“Apache Flink“)); DataSet<String> second = first .filter (str -> str.length() > 40); second.print() env.execute();
  • 30. Common misconception  Programs are not executed eagerly  Instead, system compiles program to an execution plan and executes that plan Input First SecondX Y 30
  • 31. DataSet<String>  Think of it as a PCollection<String>, or a Spark RDD[String]  With a major difference: it can be produced/recovered in several ways • … like a Java collection • … like an RDD • … perhaps it is never fully materialized (because the program does not need it to) • … implicitly updated in an iteration  And this is transparent to the user 31
  • 32. Example: grep Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 32
  • 33. Staged (batch) execution Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 Stage 1: Create/cache Log Subseqent stages: Grep log for matches Caching in-memory and disk if needed 33
  • 34. Pipelined execution Romeo, Romeo, where art thou Romeo? Load Log Search for str1 Search for str2 Search for str3 Grep 1 Grep 2 Grep 3 001100110011001100110011 Stage 1: Deploy and start operators Data transfer in- memory and disk if needed 34 Note: Log DataSet is never “created”!
  • 35. Benefits of pipelining  25 node cluster  Grep log for 3 terms  Scale data size from 100GB to 1TB 0 250 500 750 1000 1250 1500 1750 2000 2250 2500 0 100 200 300 400 500 600 700 800 900 1000 Timetocompletegrep(sec) Data size (GB)Cluster memory exceeded 35
  • 36. 36
  • 37. Drawbacks of pipelining  Long pipelines may be active at the same time leading to memory fragmentation • FLINK-1101: Changes memory allocation from static to adaptive  Fault-tolerance harder to get right • FLINK-986: Adds intermediate data sets (similar to RDDS) as first-class citizen to Flink Runtime. Will lead to fine-grained fault- tolerance among other features. 37
  • 38. Example: Iterative processing DataSet<Page> pages = ... DataSet<Neighborhood> edges = ... DataSet<Page> oldRanks = pages; DataSet<Page> newRanks; for (i = 0; i < maxIterations; i++) { newRanks = update(oldRanks, edges) oldRanks = newRanks } DataSet<Page> result = newRanks; DataSet<Page> update (DataSet<Page> ranks, DataSet<Neighborhood> adjacency) { return oldRanks .join(adjacency) .where(“id“).equalTo(“id“) .with ( (page, adj, out) -> { for (long n : adj.neighbors) out.collect(new Page(n, df * page.rank / adj.neighbors.length)) }) .groupBy(“id“) .reduce ( (a, b) -> new Page(a.id, a.rank + b.rank) ); 38
  • 39. Iterate by unrolling  for/while loop in client submits one job per iteration step  Data reuse by caching in memory and/or disk Step Step Step Step Step Client 39
  • 40. Iterate natively DataSet<Page> pages = ... DataSet<Neighborhood> edges = ... IterativeDataSet<Page> pagesIter = pages.iterate(maxIterations); DataSet<Page> newRanks = update (pagesIter, edges); DataSet<Page> result = pagesIter.closeWith(newRanks) 40 partial solution partial solutionX other datasets Y initial solution iteration result Replace Step function
  • 41. Iterate natively with deltas DeltaIteration<...> pagesIter = pages.iterateDelta(initialDeltas, maxIterations, 0); DataSet<...> newRanks = update (pagesIter, edges); DataSet<...> newRanks = ... DataSet<...> result = pagesIter.closeWith(newRanks, deltas) See http://data-artisans.com/data-analysis-with-flink.html 41 partial solution delta setX other datasets Y initial solution iteration result workset A B workset Merge deltas Replace initial workset
  • 43. Optimization of iterative algorithms 43 Caching Loop-invariant Data Pushing work „out of the loop“ Maintain state as index
  • 45. 45
  • 46. The growing Flink stack 46 Flink Optimizer Flink Stream Builder Common API Scala API Java API Python API (upcoming) Graph API Apache MRQL Flink Local Runtime Embedded environment (Java collections) Local Environment (for debugging) Remote environment (Regular cluster execution) Apache Tez Data storage HDFSFiles S3 JDBC Redis Rabbit MQ Kafka Azure tables … Single node execution Standalone or YARN cluster
  • 47. Program lifecycle 47 val source1 = … val source2 = … val maxed = source1 .map(v => (v._1,v._2, math.max(v._1,v._2)) val filtered = source2 .filter(v => (v._1 > 4)) val result = maxed .join(filtered).where(0).equalTo(0) .filter(_1 > 3) .groupBy(0) .reduceGroup {……} 1 3 4 5 2
  • 48.  The optimizer is the component that selects an execution plan for a Common API program  Think of an AI system manipulating your program for you   But don’t be scared – it works • Relational databases have been doing this for decades – Flink ports the technology to API-based systems Flink Optimizer 48
  • 49. A simple program 49 DataSet<Tuple5<Integer, String, String, String, Integer>> orders = … DataSet<Tuple2<Integer, Double>> lineitems = … DataSet<Tuple2<Integer, Integer>> filteredOrders = orders .filter(. . .) .project(0,4).types(Integer.class, Integer.class); DataSet<Tuple3<Integer, Integer, Double>> lineitemsOfOrders = filteredOrders .join(lineitems) .where(0).equalTo(0) .projectFirst(0,1).projectSecond(1) .types(Integer.class, Integer.class, Double.class); DataSet<Tuple3<Integer, Integer, Double>> priceSums = lineitemsOfOrders .groupBy(0,1).aggregate(Aggregations.SUM, 2); priceSums.writeAsCsv(outputPath);
  • 50. Two execution plans 50 DataSource orders.tbl Filter Map DataSource lineitem.tbl Join Hybrid Hash buildHT probe broadcast forward Combine GroupRed sort DataSource orders.tbl Filter Map DataSource lineitem.tbl Join Hybrid Hash buildHT probe hash-part [0] hash-part [0] hash-part [0,1] GroupRed sort forwardBest plan depends on relative sizes of input files
  • 51. Flink Local Runtime 51  Local runtime, not the distributed execution engine  Aka: what happens inside every parallel task
  • 52. Flink runtime operators  Sorting and hashing data • Necessary for grouping, aggregation, reduce, join, cogroup, delta iterations  Flink contains tailored implementations of hybrid hashing and external sorting in Java • Scale well with both abundant and restricted memory sizes 52
  • 53. Internal data representation 53 JVM Heap map JVM Heap reduce O Romeo, Romeo, wherefore art thou Romeo? 00110011 00110011 00010111 01110001 01111010 00010111 art, 1 O, 1 Romeo, 1 Romeo, 1 00110011 Network transfer Local sort How is intermediate data internally represented?
  • 54. Internal data representation  Two options: Java objects or raw bytes  Java objects • Easier to program • Can suffer from GC overhead • Hard to de-stage data to disk, may suffer from “out of memory exceptions”  Raw bytes • Harder to program (customer serialization stack, more involved runtime operators) • Solves most of memory and GC problems • Overhead from object (de)serialization  Flink follows the raw byte approach 54
  • 55. Memory in Flink public class WC { public String word; public int count; } empty page Pool of Memory Pages JVM Heap Sorting, hashing, caching Shuffling, broadcasts User code objects Network buffers Managed heap Unmanaged heap 55
  • 56. Memory in Flink (2)  Internal memory management • Flink initially allocates 70% of the free heap as byte[] segments • Internal operators allocate() and release() these segments  Flink has its own serialization stack • All accepted data types serialized to data segments  Easy to reason about memory, (almost) no OutOfMemory errors, reduces the pressure to the GC (smooth performance) 56
  • 57. Operating on serialized data Microbenchmark  Sorting 1GB worth of (long, double) tuples  67,108,864 elements  Simple quicksort 57
  • 58. Flink distributed execution 58  Pipelined • Same engine for Flink and Flink streaming  Pluggable • Local runtime can be executed on other engines • E.g., Java collections and Apache Tez Coordination built on Akka library
  • 59. Streaming overview  Streaming and batch use same code paths in runtime  Differences • Streaming does not use Flink’s memory management right now • Streaming uses its own compiler/optimizer 59
  • 61. Flink Roadmap  Currently being discussed by the Flink community  Flink has a major release every 3 months, and one or more bug-fixing releases between major releases  Caveat: rough roadmap, depends on volunteer work, outcome of community discussion, and Apache open source processes 61
  • 62. Roadmap for 2015 (highlights) Q1 Q2 Q3 APIs Logical Query integration Additional operators Interactive programs Interactive Scala shell SQL-on-Flink Optimizer Semantic annotations HCatalog integration Optimizer hints Runtime Dual engine (blocking & pipelining) Fine-grained fault tolerance Dynamic memory allocation Streaming Better memory management More operators in API At-least-once processing guarantees Unify batch and streaming Exactly-once processing guarantees ML library First version Additional algorithms Mahout integration Graph library First version Integration Tez, Samoa Mahout 62
  • 63. Dual streaming and batch engine  Runtime will support • Current pipelined execution mode • Blocking (RDD-like) execution  Batch programs use a combination  Stream programs use pipelining  Interactive programs need blocking  Expected Q1 2015 Blocking runtime execution Pipelined runtime execution Batch API (DataSet) ✔ ✔ Stream API (DataStream) ✔ 63
  • 64. Fault tolerance improvements  Fine-grained fault tolerance for batch programs • Currently lineage goes back to sources unless user explicitly persists intermediate data set • In the future replay will restart from automatically checkpointed intermediate data set  Expected Q1 2015 64
  • 65. Streaming fault tolerance  At-least-once guarantees • Reset of operator state • Source in-memory replication • Expected Q1 2015  Exactly-once guarantees • Checkpoint operator state • Rest similar to batch fine-grained recovery • Expected Q2 2015 65
  • 66. Interactive programs & shells Support interactive ad-hoc data analysis  Programs that are executed partially in the cluster and partially in the client • Needs same runtime code as fine-grained batch fault tolerance • Expected Q1 2015  Interactive Scala shell • Needs to transfer class files between shell and master • Expected Q2 2015 • Want to contribute? 66
  • 67. Machine Learning Machine learning capabilities on Flink will stand on two legs  Dedicated Machine Learning library • Optimized algorithms • First version (with infrastructure code, ALS, k- means, logistic regression) open for contributions in Q1 2015  Mahout linear algebra DSL integration • Expected Q2 2015 67
  • 68. Graph Library  Codename “Gelly”  Library with graph operations • Common graph stats, PageRank, SSSP, Connected Components, label propagation • Vertex-centric API • Gather-apply-scatter API  Expected Q1 2015 68
  • 69. Logical Query Integration  Define queries in SQL style inside Scala/Java similar to Microsoft LINQ  Expected Q1 2015 69 DataSet<Row> dates = env.readCsv(...).as("order_id", "date"); dates.filter("date < 2010-7-14") .join(other).where("order_id").equalto("id");
  • 70. SQL on Flink  Allow HiveQL queries to run on Flink directly, both standalone and embedded in batch programs  Approach: translate SQL to Logical Query Interface • Hey, Flink already has an optimizer   Currently in design phase  Expected Q3/Q4 2015 70
  • 71. Integration with other projects  Apache Tez • Expected Q1 2015  Apache Mahout • Expected Q2 2015  Tachyon • Basic integration available, waiting for Tachyon API  Apache Zepelin (inc.) • Interest from both communities  Apache Samoa (inc.) • Expected Q1 2015 • Interest from both communities  H2O • In consideration  Apache Hive • Expected Q3/Q4 2015 71
  • 72. And many more…  Runtime: even better performance and robustness • Using off-heap memory, dynamic memory allocation  Improvements to the Flink optimizer • Integration with HCatalog, better statistics • Runtime optimization  Streaming graph and ML pipeline libraries 72
  • 74. Stay informed  flink. apache.org • Subscribe to the mailing lists! • http://flink.apache.org/community.html#mailing-lists  Blogs • flink.apache.org/blog • data-artisans.com/blog  Twitter • follow @ApacheFlink 74