1.
Piotr Nowojski
piotr@data-artisans.com
@PiotrNowojski
“Hit me, baby, just one
time”
Building End-to-End Exactly-Once
Applications with Flink

2.
Overview of delivering
guarantees
2

3.
No guarantees and failures
3

4.
At-Least-Once and failures
4

5.
Exactly-Once and failures
5

6.
How to achieve Exactly-Once
6

7.
Exactly-Once on a single node
▪ Simple transactions
▪ Write data in transaction
▪ Include read offsets in transaction
▪ On success - commit transaction
▪ On failure - abort transaction and restart from
last committed read offset
7

8.
Exactly-Once on a single node
▪ What about application state?
▪ For example running average
▪ Include in the transaction, by adding as a
separate file/table just before commit
8

9.
Exactly-Once in a distributed system
▪ Persistent communication channels
9

10.
Exactly-Once in a distributed system
▪ Persistent communication channels
▪ Allow to re-process messages from last
committed read offsets
▪ High costs of communication
▪ Processes operate independent of each
other
10

11.
Exactly-Once in Flink
1
1

12.
Exactly-Once two phase commit
▪ Can we do better? Yes! Two phase commit
protocol
12
Input data Process 1 Process 2 Output data

13.
Exactly-Once two phase commit
13
Input data Process 1 Process 2 Output data
State
backend
Job
manager

14.
Exactly-Once two phase commit
14
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Start external
transaction

15.
Exactly-Once two phase commit
15
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Inject checkpoint
barrier (1)
Pre-commit
(checkpoint starts)

16.
Exactly-Once two phase commit
16
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Inject checkpoint
barrier (1)
Snapshot offsets &
state (2)
Pass checkpoint
barrier (2)
Pre-commit

17.
Exactly-Once two phase commit
17
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Inject checkpoint
barrier (1)
Snapshot offsets &
state (2)
Pass checkpoint
barrier (2)
Snapshot state (3)
Pre-commit external
transaction (3)
Pre-commit

18.
Exactly-Once two phase commit
▪ Without side effects (only internal state)
•Example: calculating running average
•State managed by Flink
▪ With side effects
•Operator must manage external state on it’s own
18

19.
Exactly-Once two phase commit
19
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Inject checkpoint
barrier (1)
Snapshot offsets &
state (2)
Pass checkpoint
barrier (2)
Snapshot state (3)
Pre-commit external
transaction (3)
Pre-commit

20.
Exactly-Once two phase commit
20
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Notify checkpoint
completed (1)
Commit

21.
Exactly-Once two phase commit
21
Input data Process 1 Process 2 Output data
State
backend
Job
manager
Notify checkpoint
completed (1) Commit external
transaction (2)
Commit

22.
Exactly-Once two phase commit
▪ Once all processes successfully complete pre-
commit, commit is issued
▪ If at least one pre-commit fails others are aborted
▪ After a successful pre-commit, commit MUST be
guaranteed to eventually succeed
▪ This guarantees that all processes agree on the final
outcome
22

23.
Two phase commit example
implementation
▪ Begin transaction - create a temporary file
▪ Write element - write data to that file
▪ Pre-commit - flush the file
▪ And begin new transaction for subsequent
writes
▪ Commit - move the file to a target directory
▪ Can increase latency
23

24.
Exactly-Once two phase commit
▪ In case of system crash, we can restore
state of the application to the latest
snapshot
▪ We could/should abort previous “pending”
transactions
•Delete temporary files
24

25.
TwoPhaseCommitSinkFunction
▪ Flink’s snapshots/checkpoints are very
similar to two phase commit
▪ There is a TwoPhaseCommitSinkFunction
which maps checkpoint calls to
beginTransaction, preCommit, commit and
abort calls
25

26.
Exactly-Once producers in Flink
▪ BucketingSink
▪ Pravega connector
▪ Kafka 0.11 connector
▪ Kafka 0.11 introduced transactions
▪ Implemented on top of the
TwoPhaseCommitSinkFunction
▪ Low overhead
26

27.
Summarizon
▪ Many ways to achieve Exactly-Once
▪ Flink checkpointing is similar to the two
phase commit protocol
▪ No materialization of the data in transit
▪ Lower latency
▪ Higher throughput
27

28.
Questions?
2
8

29.
29
Thank you!
@PiotrNowojski
@ApacheFlink
@dataArtisans

30.
We are hiring!
data-artisans.com/careers