"In the stream processing context, event-time processing means the events are processed based on when the events occurred, rather than when the events are observed (processing-time) in the system. Apache Flink has a powerful framework for event-time processing, which plays a pivotal role in ensuring temporal order and result accuracy. In this talk, we will introduce Flink event-time semantics and demonstrate how watermarks as a means of handling late-arriving events are generated, propagated, and triggered using Flink SQL. We will explore operators such as window and join that are often used with event time processing, and how different configurations can impact the processing speed, cost and correctness. Join us for this exploration where event-time theory meets practical SQL implementation, providing you with the tools to make informed decisions for making optimal trade-offs."

1. Timing is Everything
Understanding Event-Time Processing in Flink SQL
Sharon Xie，Flink Babysitter
Founding Engineer @ Decodable

2. Agenda
- Flink’s event time processing model
- Use cases in Flink SQL

3. What is Apache Flink
Stateful Computations over Data Streams
● Highly Scalable
● Exactly-once processing semantics
● Event time semantics and watermarks
● Layered APIs: Streaming SQL (easy to use) ↔ DataStream (expressive)

4. Context
● Flink streaming mode
● Flink SQL
● Current Flink version - 1.18 1.19

5. When is event time used?
● Monitoring and alerting
● Time-based compute or analytics

6. Event
Immutable record containing
the detail of something that
happened at some point in
time.

7. Time in Flink
Event Time
● The time at which the event happened
Processing Time
● The time at which the event is observed by
Flink

8. Event time vs Processing Time
● Event time is <
processing time
● The lag is arbitrary
● Events can be
out-of-order

9. Challenges
How do you know
when all of the events
are received for a
particular window?

10. Watermark
● Measures the progress of event time
● Tracks the maximum event time seen
● Indicates the completeness of the event time

12. Create table sensors (
id bigint,
`value` integer
_time timestamp(3),
watermark for _time as _time - interval '3' minutes
) WITH (
'scan.watermark.emit.strategy'='on-event',
...
);
Define Watermark

13. Watermark Generation (on-event)

14. There is a
window that
ends at 1:05.
When can the
window close?
Quiz

15. There is a
window that
ends at 1:05.
When can the
window close?
Quiz - Answer

16. Multiple sources/partitions

17. Idle source/partition
● If a partition is idle (no events), the watermark
will not advance
● No result will be produced
● Solutions
○ Configure source idle timeout
■ set table.exec.source.idle-timeout = 1m
○ Balance the partitions

18. Implications
● Tradeoff between Correctness and Latency
● Latency
○ Results of a window is only seen after the window
closes
● Correctness
○ Late arriving events are discarded after the window
is closed

19. Correctness VS Latency
In general:
Alerting and monitoring: latency
Timely analytics: correctness

20. But…can I have both?
● Yes! Flink can process & emit
“updates” (changelog)
● No watermark is needed
● Downstream system must support
“updates”
● It’s costly - need to store global state

21. Trade-offs

22. Quick Summary
● Timely response & analytics are based on event time
● Flink uses watermark to account for out-of-order
events
● Watermark allows trade-off between accuracy and
latency

23. Flink SQL Event-time Usage
● Windowed Aggregations
● Window join
● Temporal join

24. Windowing
Put unbounded events into finite-sized temporal
buckets, over which computation is applied.

25. Window Types
● Tumble / Fixed
● Hop / Sliding
● Cumulative
● Session

26. Window Types - Tumble/Fixed
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/
● Fixed window
size
● No overlapping
● Each event
belongs to
exactly 1 window

27. Flink SQL (Window TVF)
● TVF - Table-Valued Function
● Returns a new relation with all columns of original stream and
additional 3 columns:
○ window_start, window_end, window_time

28. Example - Tumble Window

29. Window Types - Hop/Slide
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/
● Fixed
window size
● Overlaps
when slide <
window size

30. Window Types - Cumulative
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/
● Similar to tumble
window, but with
early firing at the
defined interval
● Defined by max
window size and
window step

31. Window Types - Session
😃 Supported in Flink
1.19
● A new window is
created when the
consecutive event
time > session gap
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#session

32. Window Join
● A window join adds the dimension of time into
the join criteria themselves.
● Use case: compute click-through events

33. Example - tumble window join

34. Example - hop window join

35. Temporal Join
● Enrich a stream with the value of the joined
record at the event time.
● Example: Continuously computing the price for
each order based on the exchange rate
happened when the order is placed

36. Example - temporal join
Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins/#temporal-joins

38. Summary
● Event time is essential for timely response and
analytics
● Watermark and windowing are the key concepts
● Flink SQL simplifies event time processing

41. Q&A
@sharon_rxie