An instant world requires instant decisions at scale. This includes the ability to digest and react to changes in real-time. Thus, event logs such as Apache Kafka can be found in almost every architecture, while databases and similar systems still provide the foundation. Change Data Capture (CDC) has become popular for propagating changes. Nevertheless, integrating all these systems, which often have slightly different semantics, can be a challenge. In this talk, we highlight what it means for Apache Flink to be a general data processor that acts as a data integration hub. Looking under the hood, we demonstrate Flink's SQL engine as a changelog processor that ships with an ecosystem tailored to processing CDC data and maintaining materialized views. We will discuss the semantics of different data sources and how to perform joins or stream enrichment between them. This talk illustrates how Flink can be used with systems such as Kafka (for upsert logging), Debezium, JDBC, and others.

1. CDC Stream
Processing with
Apache Flink
Timo Walther
@twalthr
–
Current 2022
2022-10-05

2. About me
Open source
● Long-term committer since 2014 (before ASF)
● Member of the project management committee (PMC)
● Top 5 contributor (commits), top 1 contributor (additions)
● Among core architects of Flink SQL
Career
● Early Software Engineer @ DataArtisans
● SDK Team @ DataArtisans/Ververica (acquisition by Alibaba)
● SQL Team Lead @ Ververica
● Co-Founder @ Immerok
2
Visit us at
booth S14!

3. What is Apache Flink?
3

4. Building Blocks for Stream Processing
4
Time
● Synchronize
● Progress
● Wait
● Timeout
● Fast-forward
● Replay
State
● Store
● Buffer
● Cache
● Model
● Grow
● Expire
Streams
● Pipeline
● Distribute
● Join
● Enrich
● Control
● Replay
Snapshots
● Backup
● Version
● Fork
● A/B test
● Time-travel
● Restore

5. What makes Apache Flink unique?
5
© 2022
Source 1 Normalize
Join Sink
Source 2 Filter
Shard 1
Shard 2
Subtask 1
Subtask 2
Partition 1 Subtask 1
Subtask 1
Subtask 2
Partition 1
Partition 2
fast local state that scales with the
operator
long-term durable storage

6. What is Apache Flink used for?
6
Transactions
Logs
IoT
Interactions
Events
…
Analytics
Event-driven
Applications
Data
Integration
ETL
Messaging
Systems
Files
Databases
Key/Value Stores
Applications
Messaging
Systems
Files
Databases
Key/Value Stores

7. Apache Flink’s APIs
7

8. API Stack
8
Dataflow Runtime
Low-Level Stream Operator API
Optimizer / Planner
Table / SQL API
DataStream API Stateful Functions

9. DataStream API
9
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(STREAMING);
DataStream<Integer> stream = env.fromElements(1, 2, 3);
stream.executeAndCollect().forEachRemaining(System.out::println);
Properties
● Exposes the building blocks for stream processing
● Arbitrary operator topologies using map(), process(), connect(), ...
● Business logic is written in user-defined functions
● Arbitrary user-defined record types flow in-between
● Conceptually always an append-only / insert-only log!
1
2
3
Output

10. Table / SQL API
10
TableEnvironment env =
TableEnvironment.create(EnvironmentSettings.inStreamingMode());
// Programmatic
Table table = env.fromValues(row(1), row(2), row(3));
// SQL
Table table = env.sqlQuery("SELECT * FROM (VALUES (1), (2), (3))");
table.execute().print();
Properties
● Abstracts the building blocks for stream processing
● Operator topology is determined by planner
● Business logic is declared in SQL and/or Table API
● Internal record types flow, Flink’s Row type is exposed in Table API
● Conceptually a table, but a changelog under the hood!
+----+-------------+
| op | f0 |
+----+-------------+
| +I | 1 |
| +I | 2 |
| +I | 3 |
Output

11. DataStream API ↔Table / SQL API
11
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
// Stream -> Table
DataStream<?> inStream1 = ...
Table appendOnlyTable = tableEnv.fromDataStream(inStream1)
DataStream<Row> inStream2 = ...
Table anyTable = tableEnv.fromChangelogStream(inStream2)
// Table -> Stream
DataStream<T> appendOnlyStream = tableEnv.toDataStream(insertOnlyTable, T.class)
DataStream<Row> changelogStream = tableEnv.toChangelogStream(anyTable)
Mix and match APIs!

12. Changelog Stream
Processing
12

13. Data Processing is a Stream of Changes
13
● Business data is always a stream: bounded or unbounded
● Every record is a changelog entry: insertion as the default
● Batch processing is just a special case in the runtime
now
past future
start end of stream
bounded stream unbounded stream
unbounded stream

14. How do I Work with Streams in Flink SQL?
14
● You don’t. You work with dynamic tables!
● A concept similar to materialized views
CREATE TABLE Revenue
(name STRING, total INT)
WITH (…)
INSERT INTO Revenue
SELECT name, SUM(amount)
FROM Transactions
GROUP BY name
CREATE TABLE Transactions
(name STRING, amount INT)
WITH (…)
name amount
Alice 56
Bob 10
Alice 89
name total
Alice 145
Bob 10
So, is Flink SQL a database? No, bring your own data and systems!

15. Stream-Table Duality - Basics
15
● A stream is the changelog of a dynamic table
● Sources, operators, and sinks work on changelogs under the hood
● Each component declares the kind of changes it consumes/produces
only +I Appending/Insert-only
contains -… Updating
contains -U Retracting
never –U but +U Upserting
Short name Long name
+I Insertion Default for scans + output of bounded results.
-U Update Before Retracts a previously emitted result.
+U Update After Updates a previously emitted result.
Requires a primary key if -U is omitted for idempotent updates.
-D Delete Removes the last result.

16. Stream-Table Duality - Example
16
An applied changelog becomes a real (materialized) table.
name amount
Alice 56
Bob 10
Alice 89
name total
Alice 56
Bob 10
changelog
+I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56]
145
materialization
CREATE TABLE Revenue
(name STRING, total INT)
WITH (…)
INSERT INTO Revenue
SELECT name, SUM(amount)
FROM Transactions
GROUP BY name
CREATE TABLE Transactions
(name STRING, amount INT)
WITH (…)

17. Stream-Table Duality - Example
17
An applied changelog becomes a real (materialized) table.
name amount
Alice 56
Bob 10
Alice 89
name total
Alice 56
Bob 10
+I[Alice, 89] +I[Bob, 10] +I[Alice, 56] +U[Alice, 145] -U[Alice, 56] +I[Bob, 10] +I[Alice, 56]
145
materialization
CREATE TABLE Revenue
(PRIMARY KEY(name) …)
WITH (…)
INSERT INTO Revenue
SELECT name, SUM(amount)
FROM Transactions
GROUP BY name
CREATE TABLE Transactions
(name STRING, amount INT)
WITH (…)
Save ~50% of traffic if downstream system supports upserting!

18. Stream-Table Duality - Propagation
18
● Source declares set of emitted changes i.e. changelog mode
● Optimizer tracks changelog mode and primary key through pipeline
● Sink declares changes it can digest
CREATE TABLE …
… WITH ('connector'='filesystem')
… WITH ('connector'='kafka')
… WITH ('connector'='kafka-upsert')
… WITH ('connector'='jdbc')
… WITH ('connector'='kafka', 'format' = 'debezium-json')
+I
+I
+I -D
+I -U +U -D
+I
(for sources)

19. Retract vs. Upsert
19
Retract
● No primary key requirements
● Works for almost every external system
● Supports duplicate rows
● In distributed system often unavoidable
à most flexible changelog mode
à default mode
Upsert
● Traffic + computation optimization
● In-place updates (idempotency)
SELECT c, COUNT(*) FROM (
SELECT COUNT(*) AS c
FROM T
GROUP BY user
)
GROUP BY c
Count 1
Subtask 1
Count 2
Subtask 1
Subtask 2
+U[1]
+U[2]
+I[…]
1=>1
2=>1
Subtask 2
+I[…]

20. Changelog Insights – Append-only
20
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …); // accepts all changes
INSERT INTO Result SELECT * FROM Transactions T JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])

21. Changelog Insights – Updating
21
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])

22. Changelog Insights – Updating with PK
22
CREATE TABLE Transaction (tid BIGINT, amount INT);
CREATE TABLE Payment (tid BIGINT, method STRING);
CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE], upsertMaterialize=[true])
+- Join(leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey], changelogMode=[I,UB,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])

23. Changelog Insights – Updating with PK
23
CREATE TABLE Transaction (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
CREATE TABLE Payment (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
CREATE TABLE Result (tid BIGINT, …, PRIMARY KEY(tid) NOT ENFORCED);
INSERT INTO Result SELECT * FROM Transactions T LEFT JOIN Payments P ON T.tid = P.tid;
Sink(table=[Result], changelogMode=[NONE])
+- Join(leftInputSpec=[UniqueKey], rightInputSpec=[UniqueKey], changelogMode=[I,UA,D])
:- Exchange(changelogMode=[I])
: +- TableSourceScan(table=[[Transaction]], changelogMode=[I])
+- Exchange(changelogMode=[I])
+- TableSourceScan(table=[[Payment]], changelogMode=[I])

24. Mode Transitions
24
Append-only
Retracting
Updating
through operation
if operator/sink requires it
ChangelogNormalize
if sink requires it
UpsertMaterialize

25. Mode Transitions – Characteristics
25
Append-only
● Event-time column backed
by watermarks
● Highly state efficient due to
notion of completeness
● Usually no event-time
column
● State usage needs to
be kept in mind
● Pure materialized view
maintenance
Retracting
Updating
aka "TABLE"
aka "STREAM"
aka ?

26. Mode Transitions – Joins
26
Append-only Append-only
regular join
Append-only Updating
Append-only
Updating
Append-only Append-only
regular
outer join
Updating
regular join
Append-only Updating
temporal
join
Append-only

27. Mode Transitions – Temporal Join
27
SELECT
order_id,
price,
currency,
conversion_rate,
order_time
FROM Orders
LEFT JOIN CurrencyRates FOR SYSTEM_TIME AS OF Orders.order_time
ON Orders.currency = CurrencyRates.currency;
CREATE TABLE CurrencyRates (
WATERMARK FOR update_time AS …, PRIMARY KEY(currency) NOT ENFORCED,…);

28. Mode Transitions – Explicit Transition without PK
28
Append-only Updating
op update_time currency rate
== =========== ======== ====
+I 09:00:00 Yen 102
+I 09:00:00 Euro 114
+I 09:00:00 USD 1
+I 11:15:00 Euro 119
+I 11:49:00 Pounds 108
op update_time currency rate
== =========== ======== ====
+I 09:00:00 Yen 102
+I 09:00:00 Euro 114
+I 09:00:00 USD 1
+U 11:15:00 Euro 119
+I 11:49:00 Pounds 108

29. Mode Transitions – Explicit Transition without PK
29
Append-only Updating
CREATE VIEW versioned_rates AS
SELECT currency, rate, update_time
FROM (
SELECT
*,
ROW_NUMBER() OVER (PARTITION BY currency ORDER BY update_time DESC) AS rownum
FROM currency_rates
)
WHERE rownum = 1;

30. Demo
https://github.com/twalthr/flink-api-examples
30

31. Summary
TLDR
● Flink's SQL engine is a powerful changelog processor
● Flexible tool for integrating systems with different semantics
There is more…
● CDC connector ecosystem
à 2.6k Github stars
https://flink-packages.org/packages/cdc-connectors
● Table Store
à unified storage engine for dynamic tables
https://flink.apache.org/news/2022/05/11/release-table-store-0.1.0.html
● SQL Gateway
https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway
31

32. Thanks
Timo Walther
@twalthr
mrsql@immerok.io