The document discusses the Flink operator for Kubernetes, which allows Flink jobs to be deployed and managed on Kubernetes clusters. The Flink operator uses custom resources and control loops to create Kubernetes objects like deployments and services that run the Flink job manager and task managers. It provides features like job recovery after failures, updating jobs, and auto-scaling of resources. An example use case described is running real-time Flink jobs on Kafka streams deployed on Kubernetes and managed by the Flink operator.

1. Flink on
Kubernetes operator
SK telecom
Eui Heo

2. Outline
• Background
• Flink operator
• Use case
• Future works

3. Kubernetes
“Kubernetes is a portable, extensible, open-source platform for
managing containerized workloads and services, that facilitates
both declarative configuration and automation.”
https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
• Declarative configurations
• Automation of operation
• Extensible platform

4. Kubernetes
control loop
Desired state or actual state
Is updated?
Diff the two
Act if the two is not equal
Observed actual state Act to change the state
Control loop
(declarative configuration)
Desired state

5. Kubernetes architecture
K8s
Nodes

6. K8s Master
Kubernetes architecture
apiVersion: core/v1
kind: Deployment
metadata:
name: flink-jm
spec:
replicas: 1
template:
…
image: flink:1.9.3
API server
K8s
Nodes
Persist
Cache
Submit spec
Deployment
Deployment
Deployment
Deployment
etcd

7. K8s Master
Kubernetes architecture
apiVersion: core/v1
kind: Deployment
metadata:
name: flink-jm
spec:
replicas: 1
template:
…
image: flink:1.9.3
API server
K8s
Nodes
Controller manager
(control loops for each objects)
ReplicaSet
Persist
Cache
Submit spec
Deployment
Deployment
Deployment
Deployment
etcd
Pod
Create pods

8. K8s Master
Kubernetes architecture
apiVersion: core/v1
kind: Deployment
metadata:
name: flink-jm
spec:
replicas: 1
template:
…
image: flink:1.9.3
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
ReplicaSet
Persist
Cache
Submit spec
Scheduler
Kubelet Kubelet
Deployment
Deployment
Deployment
Deployment
etcd
Pod
Create pods

9. K8s Master
Kubernetes architecture
apiVersion: core/v1
kind: Deployment
metadata:
name: flink-jm
spec:
replicas: 1
template:
…
image: flink:1.9.3
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
ReplicaSet
Persist
Cache
Submit spec
Scheduler
Kubelet Kubelet
Deployment
Deployment
Deployment
Deployment
etcd
Pod
<job manager>
Pod
Update deploy status

10. Outline
• Background
• Flink operator
• Use case
• Future works

11. K8s Master
Flink deploy with manifests
Job manager
Deployment
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
submit
manifests
Scheduler
Kubelet Kubelet
etcd
Task manager
Deployment
ConfigMap
Job manager
Service
kubectl apply –f manifest.yaml

12. K8s Master
Flink native Kubernetes support
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
Scheduler
Kubelet Kubelet
etcd

13. K8s Master
Flink native Kubernetes support
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
Scheduler
Kubelet Kubelet
etcd
Job manager
Deployment
Flink CLI
submits
manifests
Task manager
Deployment
ConfigMap
Job manager
Service
X
<JobManager>
Deployment
ConfigMap Service
$ ./bin/kubernetes-session.sh …

14. K8s Master
Flink native Kubernetes support
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
Scheduler
Kubelet Kubelet
etcd
<job manager>
Pod
Job manager
Deployment
Flink CLI
submits
manifests
Task manager
Deployment
ConfigMap
Job manager
Service
X
<JobManager>
Deployment
ConfigMap Service
Service
$ ./bin/kubernetes-session.sh …

15. K8s Master
Flink native Kubernetes support
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
Scheduler
Kubelet Kubelet
etcd
<job manager>
Pod
Job manager
Deployment
Flink CLI
submits
manifests
Task manager
Deployment
ConfigMap
Job manager
Service
X
Flink CLI
submits
job
Create
taskmanager pod
on demand
<JobManager>
Deployment
ConfigMap Service
Pod
<task
manager>
Pod
Service
$ ./bin/kubernetes-session.sh …
$ ./bin/flink run -d …

16. K8s Master
Flink native Kubernetes support
API server
Kubelet
K8s
Nodes
Controller manager
(control loops for each objects)
Scheduler
Kubelet Kubelet
etcd
<job manager>
Pod
Job manager
Deployment
Flink CLI
submits
manifests
Task manager
Deployment
ConfigMap
Job manager
Service
X
Flink CLI
submits
job
Create
taskmanager pod
on demand
<task manager>
Pod
<task manager>
Pod
<JobManager>
Deployment
ConfigMap Service
Pod
<task
manager>
Pod
Service
$ ./bin/kubernetes-session.sh …
$ ./bin/flink run -d …

17. Kubernetes operator
• Operator pattern
• Operators are software extensions to Kubernetes
• Make use of custom resources to manage applications
• Operators follow Kubernetes principles, notably the control loop.
• Motivation
• Capture the key aim of a human operator
• Automation to take care of repeatable tasks
Note: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

18. K8s Master
Flink operator (GoogleCloudPlatform/flink-on-k8s-operator)
API server
K8s
Nodes
Controller manager
(control loops for each objects)
etcd
FlinkCluster
controller
Scheduler
Kubelet Kubelet Kubelet
FlinkCluster
FlinkCluster CRD
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
taskManager:
replicas: 2

19. K8s Master
Flink operator (GoogleCloudPlatform/flink-on-k8s-operator)
API server
K8s
Nodes
Controller manager
(control loops for each objects)
etcd
FlinkCluster
controller
FlinkCluster
Scheduler
Kubelet Kubelet Kubelet
FlinkCluster
FlinkCluster CRD
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
taskManager:
replicas: 2

20. K8s Master
Flink operator (GoogleCloudPlatform/flink-on-k8s-operator)
API server
K8s
Nodes
Controller manager
(control loops for each objects)
etcd
FlinkCluster
controller
FlinkCluster
Scheduler
Kubelet Kubelet Kubelet
Deployment Deployment ConfigMap Service Job
FlinkCluster
FlinkCluster CRD
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
taskManager:
replicas: 2

21. K8s Master
Flink operator (GoogleCloudPlatform/flink-on-k8s-operator)
API server
K8s
Nodes
Controller manager
(control loops for each objects)
etcd
FlinkCluster
controller
FlinkCluster
Scheduler
Kubelet Kubelet Kubelet
Deployment Deployment ConfigMap Service Job
FlinkCluster
<job manager>
Pod
<task manager>
Pod
<task manager>
Pod
FlinkCluster CRD
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
taskManager:
replicas: 2

22. K8s Master
Flink operator (GoogleCloudPlatform/flink-on-k8s-operator)
API server
K8s
Nodes
Controller manager
(control loops for each objects)
etcd
FlinkCluster
controller
FlinkCluster
Scheduler
Kubelet Kubelet Kubelet
Deployment Deployment ConfigMap Service Job
FlinkCluster
user
program
Java
Python
Javascript
Go
… <job manager>
Pod
<task manager>
Pod
<task manager>
Pod
FlinkCluster CRD
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
taskManager:
replicas: 2

23. Flink operator features
• Replay job from savepoint
• Take savepoint periodically
• Update Flink job program
• Update Flink cluster (e.g, Flink version)
• Recover job from Fink cluster failure
• Manage FlinkCluster revision history
• Re-scaling
• Rollback FlinkCluster to specific revision
• Auto-scaling
• Support Apache Beam on Flink per-job cluster

24. FlinkCluster custom resource
apiVersion: flinkoperator.k8s.io/v1beta1
kind: FlinkCluster
metadata:
name: wordcount
spec:
image:
name: flink:1.9.2
jobManager:
resources:
limits:
memory: "1024Mi"
cpu: "200m"
taskManager:
replicas: 2
job:
jarFile: ./examples/streaming/WordCount.jar
className: org.apache.flink.streaming.examples.wordcount.WordCount
args: ["--input", "./README.txt"]
parallelism: 2
fromSavepoint: /savepoints/savepoint-d299f1-9bd71f023b4a
autoSavepointSeconds: 30
flinkProperties:
taskmanager.numberOfTaskSlots: "1"

25. Outline
• Background
• Flink operator
• Use case
• Future works

26. Outline
• Background
• Flink operator
• Use case
• Future works
Use case
Kafka
topic
Stream manager
Flink real tiem job
Real-time
data stream
Kafka
topic
Druid
Kafka management
Druid management
Flink job management
Kubernetes cluster
Flink operator
FlinkCluster
specs
Deep
storage
Metatron Grandview
Request Flink jobs for new assets

27. Outline
• Background
• Flink operator
• Use case
• Future works

28. • Integrate Flink native Kubernetes support
• Auto scaling
• Apache beam support