Beginning with MapReduce and its first popular open-source implementation in Apache Hadoop the data processing landscape has evolved quite a bit. Since then we have seen several paradigm shifts and open-source systems evolved to support new types of applications and to attract new audiences. We will follow developments using the example of the open-source stream processing system Apache Flink and in the end we will see how expressive APIs, support for event-driven applications, Flink SQL for seamless batch and stream processing, and a powerful runtime enable a wide range of applications.

1. Aljoscha Krettek - Co-Founder & Software Engineer at data Artisans
THE EVOLUTION OF
(OPEN SOURCE) DATA PROCESSING

2. © 2018 data Artisans2
ABOUT DATA ARTISANS
Original Creators of
Apache Flink®
RealTime Stream Processing Enterprise
Ready

3. © 2018 data Artisans3
POWERED BY APACHE FLINK

4. © 2018 data Artisans4
Disclaimer:
I might forget systems or misrepresent their use or
when they were created.This is not intentional. Please
come discuss with me afterwards!

5. © 2018 data Artisans5
How do we process data and what are the systems
available to us?

6. © 2018 data Artisans6
PRE-HISTORIC

7. © 2018 data Artisans7
Purpose-built
programs
Since the beginning of computers.

8. © 2018 data Artisans8
Programming is kinda hard.
Data analysis is only available to a
small circle of
programmers/engineers.

9. © 2018 data Artisans9
(Big) Data Bases
Since the 1970s

10. © 2018 data Artisans10
SQL is approachable to a wider
range of people.
Data analysis is no longer
restricted to “programmers”.
There are even tools that create
SQL: BI tools and whatnot.

11. © 2018 data Artisans11
Application Services
talking to data
bases, event-driven
applications
Since quite a while… 😉

12. © 2018 data Artisans12
THE ADVENT OF BIG DATA

13. © 2018 data Artisans13
MapReduce
2004

14. © 2018 data Artisans14
Apache Hadoop®
2006

15. © 2018 data Artisans15
Store first, ask
questions later*
* we’ll get back to this later

16. © 2018 data Artisans16
Programming is kinda hard.
Data analysis is only available to a
small circle of
programmers/engineers.

17. © 2018 data Artisans17
Apache Hive™ 2009
Apache Pig™
2008
*it’s tricky with release dates and when they incubated and whatnot

18. © 2018 data Artisans18
SQL is approachable to a wider
range of people.
Data analysis is no longer
restricted to “programmers”.
There are even tools that create
SQL: BI tools and whatnot.

19. © 2018 data Artisans19
Apache Spark™
2012? – non-apache release
2014 – first apache release

20. © 2018 data Artisans20
THE RISE OF STREAMING

21. © 2018 data Artisans21
Apache Storm™
2011 – first non-apache release
2014 – Storm 0.9.1, first Apache release

22. © 2018 data Artisans22
Apache Kafka®
2011 – non-apache release
2013 – first apache release

23. © 2018 data Artisans23
Lambda Architecture
At some point in between.
Was a bit of a dead end.

24. © 2018 data Artisans24
Apache Flink®
2010 - under the name Stratosphere
2014 - Flink 0.6, first Apache release
2015 – Flink 0.9, first release with exactly-once stream processing

25. © 2018 data Artisans25
Reliable Stream Processing
No more need for the lambda architecture.

26. © 2018 data Artisans26
Ask questions
first, then wait for
things to happen*
* i.e., we put in place a program, and get real-
time updates when things happen

27. © 2018 data Artisans27
And of course…
Programming this was hard.
Then we had “SQL” on streams.

28. © 2018 data Artisans28
APACHE FLINK

29. © 2018 data Artisans29
batch
streaming analytics &
continuous processing
event-driven applications
offline real-time
The processing landscape

30. © 2018 data Artisans30
What’s in a processing system/framework?
1.
Engine
2.
APIs
3.
Connectors

31. © 2018 data Artisans31
1. Flink Engine
Deployment
• YARN
• Mesos
• Kubernetes
• Resource elasticity
Stateful stream processing
• Network shuffle
• State & timers
• Fault tolerance
• Exactly once
• Savepoints

32. © 2018 data Artisans32
2. Flink APIs
DataSet API
DataStream API
Table API/SQL
and more …

33. © 2018 data Artisans33
2. Flink APIs – DataStream API
• Stateful stream processing
• Windowing
• State & timers
• Complete control over what is
going on

34. © 2018 data Artisans34
2. Flink APIs –Table API/SQL
• Declarative/relational API
• “No programming required” SQL (ANSI SQL)
• Same SQL for batch and streaming
• Pluggable connectors / data formats

35. © 2018 data Artisans35 https://data-artisans.com/blog/flink-sql-powerful-querying-of-data-streams

36. © 2018 data Artisans36
3. Flink Connectors
The usual suspects: Kafka, Kinesis, HDFS/S3,
Elasticsearch, Cassandra, …
Table API / SQL has a modular library of connectors &
formats that can be extended by users.

37. © 2018 data Artisans37
SQL connector definition
- name:TaxiRides
type: source
update-mode: append
schema:
- name: rideId
type: LONG
- name: rowTime
type:TIMESTAMP
rowtime:
timestamps:
type: "from-field"
from: "rideTime"
watermarks:
type: "periodic-bounded"
delay: "60000"
- name: isStart
type: BOOLEAN
- name: lon
type: FLOAT
- name: lat
type: FLOAT
- name: taxiId
type: LONG
- name: driverId
type: LONG
- name: psgCnt
type: INT
connector:
property-version: 1
type: kafka
version: 0.11
topic:TaxiRides
startup-mode: earliest-offset
properties:
- key: zookeeper.connect
value: zookeeper:2181
- key: bootstrap.servers
value: kafka:9092
- key: group.id
value: testGroup
format:
property-version: 1
type: json
schema: "ROW(rideId LONG, isStart,
BOOLEAN, rideTimeTIMESTAMP,
lon FLOAT, lat FLOAT, psgCnt INT,
taxiId LONG, driverId LONG)"

38. © 2018 data Artisans38
DataSetAPI DataStreamAPI
TableAPI / SQL
batch
streaming analytics &
continuous processing
event-driven applications
offline real-time

39. © 2018 data Artisans39
What’s the next
evolution?

40. © 2018 data Artisans40
DataSetAPI DataStreamAPI
TableAPI / SQL
batch
streaming analytics &
continuous processing
event-driven applications
offline real-time
* this is where we are now
Different algorithms/data
structures optimized for the
use case.

41. © 2018 data Artisans41
Grand Unification
Truly unified runtime that adapts to the workload.
Seamless integration of batch and streaming data
sources.

42. © 2018 data Artisans42
DataStreamAPITableAPI / SQL
batch
streaming analytics &
continuous processing
event-driven applications
offline real-time
* possible future evolution

43. © 2018 data Artisans43
http://flink.apache.org

44. THANK YOU!
aljoscha@apache.org
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers

45. © 2018 data Artisans45
FREE TRIAL DOWNLOAD
data-artisans.com/download

46. © 2018 data Artisans46
DOWNLOAD REPORT
data-artisans.com/download-report-
stream-processing-da-platform-apache-flink
Stream processing for real-time businesses
powered by Apache FlinkⓇ

47. © 2018 data Artisans47
BACKUP

48. © 2018 data Artisans48
Akka
2010 – 0.5, first public release