The document discusses stateful stream processing using Apache Flink, including its definition, capabilities, and types of state management (operator, keyed, broadcast, and queryable state). It details how Flink handles data streams both in real-time and from bounded sources, emphasizing its scalability, fault tolerance, and support for large states. Additionally, the document outlines different state backends (memory, file system, and rocksdb) and the checkpointing process, which allows for replaying data from specific points in the stream.

1. Presented By:
Kundan Kumar
Software Consultant
Stateful Stream
Processing with
Apache Flink

2. Lack of etiquette and manners is a huge turn off.
KnolX Etiquettes
Punctuality
Respect Knolx session timings, you
are requested not to join sessions
after a 5 minutes threshold post
the session start time.
Feedback
Make sure to submit a constructive
feedback for all sessions as it is
very helpful for the presenter.
Mute
Be on mute until you have
questions or concerns.

3. Agenda
01 What is stateful stream processing
02 Flink takes on stateful stream processing
Demo
03

4. What is Stateful Stream Processing?
Streaming and Stream Processing:
Stream processing is the processing of data in motion, or in other words, computing on data directly
as it is produced or received.
The systems that receive and send the data streams and execute the application or analytics logic are
called stream processors.

5. Stateful Stream Processing:
Stateful stream processing is a subset of stream processing in which the computation maintains
contextual state. This state is used to store information derived from the previously-seen events.
Stateful stream processing means a “State” is shared between events(stream entities). And therefore
past events can influence the way the current events are processed.

6. Flink takes on stateful stream processing
Flink in nutshell-
● Apache Flink is a Big Data framework and distributed processing engine for stateful
computations over unbounded and bounded data streams.

7. ➢ A Flink application may consume real-time data from streaming sources such as
message queues or distributed logs, like Apache Kafka or Kinesis.
➢ Flink can also consume bounded, historic data from a variety of data sources.
➢ The streams of results being produced by a Flink application can be sent to a wide
variety of systems that can be connected as sinks.
➢ Fast, In memory, scalable, large state, fault tolerant, event time, exactly once.

8. Source
Transformations
Sink

9. ➢ Programs in Flink are inherently parallel and distributed.
➢ During execution, a stream has one or more stream partitions, and each
operator has one or more operator subtasks.

10. ➢ Flink facilitate stateful operations.
➢ Current handling event can depend on the accumulated effect of all the events that
came before it.
➢ The set of parallel instances of a stateful operator is effectively a sharded key-value
store. Each parallel instance is responsible for handling events for a specific group of
keys, and the state for those keys is kept locally.

12. States in Flink
➢ Operator State: State is maintained on per operator basis on stream. Special type of
state used in source and sink implementations.
➢ Keyed State: Maintaining state on per key basis on stream. Stores state associated
with the same key. Embedded key value store.
➢ Broadcast State: Special type of operator state used where records of one stream will
be broadcast to all downstream task which needs access to those records.
➢ Queryable State: Feature that allow client API’s to query Jobstate from outside Flink.

13. Stateful streaming application in Flink

15. State Backends
1. Memory state backend:
➢ This is the default backend used by Flink in case nothing is configured.
➢ Persists the data in the memory of each task manager’s Heap.
➢ This state should never be used in production jobs.
➢ The state creates a backup of the data (also known as checkpointing) in the job
manager memory which puts unnecessary pressure on the job manager's operational
stability.

16. 2. File System Backend
➢ This backend is similar to Memory state backend except, it stores the backup on the
filesystem rather than job manager’s memory.
➢ The filesystem can be task manager's local filesystem or a durable store such as
HDFS/S3.
3. RocksDB backend
➢ This backend uses RocksDB by Facebook to store the data
➢ RocksDB maintains an in-memory table (also known as mem-table) along with bloom
filters, reading recent data also is extremely fast.
➢ Each task manager maintains its own Rocks DB file and the backup of this state is
checkpointed to a durable store such as HDFS/S3.
➢ This is the only backend which offers support for incremental checkpointing i.e. taking a
backup of only modified data rather than complete data.

17. Checkpointing
Checkpoint: Specific marked point in each input stream from which stream can
replayed. Flink implements it by persisting state of all stateful operator. Periodically
save state to reliable storage system.
Stream Barriers: Lightweight stream marker with unique ID’s. Injected by Flink into
input stream and flow with stream in line.

18. Checkpointing mechanism
Aligned Checkpointing-

19. Unaligned Checkpointing-

20. Demo

21. Q/A

22. References
1. https://flink.apache.org
2. https://ci.apache.org/projects/flink/flink-docs-release-1.12/con
cepts/stateful-stream-processing.html#unaligned-checkpointin
g
3. Book: Learning Apache Flink By Tanmay Deshpande

23. Thank You !