Apache Flink is an open source stream processing framework that originated from a research project at TU Berlin in 2009. It is undergoing incubation at the Apache Software Foundation and allows for fast and reliable processing of big data. Flink provides high-level APIs for the Java and Scala programming languages to define and execute dataflow programs on top of distributed datasets.

1. Apache Flink
Fast and reliable big data processing
Aljoscha Krettek
aljoscha@apache.org

2. What is Apache Flink?
• Project undergoing incubation in the Apache Software
Foundation
• Originating from the Stratosphere research project
started at TU Berlin in 2009
• http://flink.incubator.apache.org
• 59 contributors (doubled in ~4 months)
• Has awesome squirrel logo

3. What is Apache Flink?
Flink Client

4. Apache Flink
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
flink.incubator.apache.org
…
env.execute();

5. Apache Flink
DataSet<String> input = env.readTextFile(“/hello/there”);
flink.incubator.apache.org

6. Apache Flink
DataSet<String> input = env.readTextFile(“hdfs:///hello/there”);
flink.incubator.apache.org

7. Apache Flink
DataSet<Tuple2<String,Integer> input = env.readCsvFile(“/hello/there”)
flink.incubator.apache.org
.fieldDelimiter(‘|’)
.lineDelimiter(“n")
.ignoreFirstLine()
.types(String.class, Integer.class);

8. Apache Flink
DataSet<Beer> beers = input.map( new MapFunction<?, Beer>() {
flink.incubator.apache.org
public Beer map(? in) {
return new Beer(in.f0, in.f1);
}
});

9. Apache Flink
DataSet<Beer> beers = input.map( in -> new Beer(in.f0, in.f1) );
flink.incubator.apache.org

10. Apache Flink
val beers = input.map( in => new Beer(in._1, in._2) )
flink.incubator.apache.org

11. Apache Flink
DataSet<Beer> filtered = beers.filter( new FilterFunction<Beer>() {
flink.incubator.apache.org
public boolean filter(Beer in) {
return beer.name.contains(“augustiner”);
}
});

12. Apache Flink
flink.incubator.apache.org
DataSet<Beer> grouped = beers
.groupBy(“name”)
.sortGroup(“rating”, Order.DESCENDING)
.reduceGroup( new GroupReduceFunction<Beer, Beer>() {
public Beer reduceGroup(Iterable<Beer> in, Collector<Beer> out) {
out.collect(in.iterator().next());
}
});

13. Apache Flink
DataSet<Tuple2<String, Integer>> aggregated = input.groupBy(0).sum(1);
flink.incubator.apache.org

14. Apache Flink
flink.incubator.apache.org
result.print();

15. Apache Flink
result.writeAsText(“/ciao/for/now”);
flink.incubator.apache.org

16. github.com/aljoscha/beer-analysis
www.filedropper.com/beerdatacsv
www.filedropper.com/beerdata (large)
flink.incubator.apache.org
github.com/apache/incubator-flink
meetup.com/Apache-Flink-Meetup