Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
1. Apache Flink in
the Cloud-Native
Era
David Moravek
-
@davidmoravek
-
Flink Forward
22
2. ● Co-Founder and Cloud Engineer @ immerok
● Apache Flink & Apache Beam Committer
● I worked with Big Data before it was cool
● I <3 Running
About me
2
Ex…
8. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers
10. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers
12. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers
14. Delay by in-flight data
Unaligned checkpoints
“not blocking on in-flight data”
Buffer debloating
“having less in-flight data”
Delay by state snapshot
Incremental checkpoints
“doing smaller snapshots”
Checkpoint while firing timers
Yet to be solved.
Stay tuned!