Introduction to Apache Flink talk for meetup: https://www.meetup.com/futureofdata-vienna/events/254138753

1. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
An Introduction to
Apache Flink®
Andrey Zagrebin, Software Engineer at Data Artisans

2. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Agenda
● Overview
● Use Cases & Users
● Stream Processing
● Fault Tolerant Stateful Processing
● Working With Event-Time
● APIs & Deployment
● Community
● Demo

3. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Stateful Computations over Data Streams

4. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Use Case & Users

5. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Use Case: Batch & Stream Analytics/Pipeline

6. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Use Case: Event-Driven Applications

7. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Powered By
Details about their use cases and more users are listed on Flink’s website at https://flink.apache.org/poweredby.html

8. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Features

9. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Example of Stream Processing Application
Inject Extract Group by Increment
click UserId UserId counter

10. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Anatomy of Stream Processing Application
Inject Extract Group by Increment
click UserId UserId counter

11. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
What is State in a Streaming Application?
● Functions need to remember records or temporary results
● State is either per operator instance or per record key
● State backends: JVM heap or RocksDB on local disk
● Local == fast access
User A is
an active user
User A: (3 + 1) clicks
User A clicks

12. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Maintaining and Checkpointing State
● State is periodically checkpointed to durable storage
○ A checkpoint is a consistent snapshot of the state of all operators

13. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Processing-Time
21 3 4 5 7
Processing
time
Active user
Next time
discount
buy
6

14. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Event-Time vs Processing-Time
Buy
late
21 3 4 5 7
Processing
time
21 4 5 3
Event
time
Not active
anymore!
No discount!
Timemismatch!!!
6

15. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Watermark
Buy
late
21 3 4 5 7
Processing
time
21 4 5 3
Event
time
Active
again!
Discount!
-0-1 2 3 3 5
Watermark
-2 delay
7
8

16. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
APIs and Deployment

17. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
APIs
Connectors

18. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Deployments

19. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Community
● Apache open source project
● Time-based releases every 3 months
● user mailing list in TOP-5 most active of ASF
● Join the community and contribute!
● https://flink.apache.org

20. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Demo!
http://training.data-artisans.com/exercises/fault-recovery.html

21. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Demo Task
CSV compressed input:
id, type start timestamp end timestamp longitude latitude
6, START, 2013-01-01 04:17:00, 1970-01-01 00:00:00, -73.979645,40.784046,
-73.973434,40.79026,6
6, END, 2013-01-01 04:20:00, 2013-01-01 04:17:00, -73.979645,40.784046, -73.973434,40.79026,6
Normal ride
starts
Normal ride
ends
2pm1pm 3pm 4pm 5pm
time
Long ride
starts
6pm
Long ride
ends
More than 2 hours1 hour is ok

22. Apache Flink, Flink and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Source code

25. THANK YOU! Q&A
andrey@data-artisans.com
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers