"Stream processing technologies have been around for more than a decade. Nowadays, Apache Flink has become the de facto standard for stream processing and is being used in many enterprises. In this talk, we summarize our five-year experience in supporting enterprises to implement, deploy, and operate their stream processing solutions based on Apache Flink. We will cover topics such as Flink application lifecycle management, Flink SQL development, multi-tenancy, security, cost optimization, business continuity, customer support, deployment options, and how they are supported in our product Ververica Platform. Finally, we will conclude by discussing the outlook for the future."

1. Stream Processing Solutions for the Enterprise
Jun Qin,
2023-09-26

2. Agenda
Introduction
1
Solutions for the Enterprise
2
Summary
4
● Simplicity
● Eiciency
● Flexibility
● Security
Ververica Platform
3

3. Ⓒ 2023 Ververica
Ververica - the Original Creators of Apache FlinkⓇ
Donate Flink to Apache
Incubating 1st
COTS Product
2.11
Apache Flink
1.0
2014
2016
2017
2019
2023
(Today)
1.4
1st
Flink Forward
2015
3
Seale 2023
November 6-8

4. Ⓒ 2023 Ververica
Simplicity
● Application, instead of Jobs
● Advanced SQLEditor
● Kubernetes Operator
● Smart Advisor
4

5. Ⓒ 2023 Ververica
Applications, instead of Jobs
5
Load Increase Poison Pill
spec:
… parallelism: 3
spec:
… parallelism: 4
spec:
… jarUri:
myapp-1.0.1.jar
time
create update update
deploy stateful upgrade stateful upgrade

6. Ⓒ 2023 Ververica
6
A deployment links all historical jobs
A deployment keeps the history of all state snapshots
15
apps
300
apps
before using
our solution
when using
our solution
One team
can manage

7. Ⓒ 2023 Ververica
7
Advanced SQLEditor
● Web UI
○ Auto completion
○ Table DDLdeduction
● Validation
○ Syntax, Existence
● Preview on a session cluster
● Connectors & formats & UDFs mgmt
● Persistent catalogs
○ built-in /JDBC/Hive/CDC catalog
● Versioning

8. Ⓒ 2023 Ververica
8
Kubernetes Operator
● Kubernetes Operator
○ Kubernetes native
○ allows deploying Flink jobs in the
gitops style
● Manage
○ streaming & batch jobs
○ Jar/SQL/PyFlink deployments
● Integrate with Web UI

9. Ⓒ 2023 Ververica
● What
○ A knowledge base
○ A process to update KB by log ETL,
aggregation, annotation
○ A process to collect diagnosis feedbacks
○ UI Components
● Support
○ Diagnose errors & exceptions
■ development, operation
○ Analyze potential risks of running jobs
○ Apply the proposed solution with one click
9
Smart Advisor

10. Ⓒ 2023 Ververica
10
cause
solution
feedback

11. Ⓒ 2023 Ververica
Eiciency
● Faster and Stronger Flink
● Autopilot
11

12. Ⓒ 2023 Ververica
12
Faster and Stronger Flink
● State backend plays an important factor
● Gemini: tailored for stream processing
○ Hybrid state store -> No need to flush to disk
upon checkpointing
○ Tiered storage with Hash/sorted index ->
■ Fast access for frequent used state
■ Optimized point & range lookups
■ No strong depedency on local disk size
○ KV separation -> Fast stream join
○ Lazy restore -> much shorter time to restore
large state

13. Ⓒ 2023 Ververica
13
Autopilot
● Works with Flink 1.11 or later
● Monitor operator metrics
continuously
● Use sophisticated algorithms to
calculate the target parallelism
● Monitoring only & Activated
mode
● Cooldown time
● Limit job parallelism when
scaling up

14. Ⓒ 2023 Ververica
Flexibility
● Java/Scala
● Python
● SQL
14
Language Support
Deployment Options
● Fully-managed/public cloud
● Self-managed/private cloud
● Semi-manged/hybrid cloud
SQL

15. Ⓒ 2023 Ververica
15
Security
● Job Level
○ One-click SSLSetup
● Platform Level
○ Authentication
○ Authorization
○ Multi-tenancy
○ Fix security vulnerabilities regularly
○ Audit Logs

16. Ⓒ 2023 Ververica
Ververica Platform
16
We Run
Where You Run

17. Ⓒ 2023 Ververica
Ververica Platform Main Features
17
Poison Pill

18. Ⓒ 2023 Ververica
➢ One-click SSL
➢ Authentication
➢ Authorization
➢ Multi-tenancy
➢ Regular Vulnerability Fixes
➢ Audit logs
Summary
18
S
i
m
p
l
i
c
i
t
y
Eiciency
F
l
e
x
i
b
l
i
l
i
t
y
S
e
c
u
r
i
t
y
➢ Application, Instead of Jobs
➢ Advanced SQLEditor
➢ Kubernetes Operator
➢ Smart Advisor
➢ Java/Scala/Python/SQL
➢ Public/Private/Hybrid Cloud
➢ Faster and Stronger Flink
➢ Autopilot
Since 2017

19. Thank You
Question?
❏ Stop by Booth 202
Esp. Tue 4-6pm Ververica Oktoberfest
❏ Join us at
in Seale on November 6-8, 2023
❏ Visit www.ververica.com