Operating Flink on Kubernetes can be challenging. Which products are available? Do we need to build our own tool? Which tool do we need? How do we build it? This presentation provides some ideas how to build a Flink Operator and it contains a link to a proof of concept available on GitHub

1. How to build a tool for
operating Flink on Kubernetes
Andrea Medeghini
Software Engineer / Contractor

2. Which products are available?
Ververica dA Platform:
● Automated deployments
● Easy management of jobs
● Monitoring
● Logging (ELK)
● Trial version available

3. Are there free alternatives?
There are few projects on GitHub, however…
● They mainly focus on deployment
● They do not provide a complete solution
● They might not work for all use cases

4. Can we use Helm?
It’s good, but it doesn’t help with…
● Jobs and Savepoints
● Monitoring / Alerting
● Automatic Scaling
# helm install --name my-flink-cluster charts/flink

5. Wait for next Flink release?
Better integration with Kubernetes it’s coming:
● Reactive container mode
https://issues.apache.org/jira/browse/FLINK-10407
● Active Kubernetes integration
https://issues.apache.org/jira/browse/FLINK-9953

6. Shall we build our own tool?
It’s going to be challenging! Because…
● Flink is a distributed engine
● Flink is a stateful engine
● Jobs need to be packaged and uploaded
● Jobs need to be monitored to detect failures
● Resources need to be adjusted according to workload

7. Do we need an open source tool?
Everybody likes open source tools…
… how do we build one?

8. Overview of a Flink Cluster
● One or more JobManagers
(typically one)
● One or more TaskManagers
(typically many)
● One or more jobs packaged as
JAR files
● Storage for savepoints

9. Exploiting Kubernetes API
It’s all REST!
There are clients libraries…
… for many languages not only Go!
See Kubernetes Documentation:
https://kubernetes.io/docs/reference/

10. Control Kubernetes Programmatically
val jobmanagerStatefulSet = V1StatefulSet()
.metadata(jobmanagerMetadata)
.spec(
V1StatefulSetSpec()
.replicas(1)
.template(
V1PodTemplateSpec().spec(jobmanagerPodSpec).metadata(jobmanagerMetadata)
)
.updateStrategy(updateStrategy)
.serviceName("jobmanager")
.selector(jobmanagerSelector)
.addVolumeClaimTemplatesItem(persistentVolumeClaim)
)
api.createNamespacedStatefulSet(namespace, jobmanagerStatefulSet, null, null, null)

11. What resources do we need?
● StatefulSet for JobManager (1 replica)
● StatefulSet for TaskManager (N replicas)
● Services for JobManager (headless, NodePort, ...)
● PersistentVolumeClaims
● …

12. What configuration do we need?
● Set JOB_MANAGER_RPC_ADDRESS to JobManager service
● Set TASK_MANAGER_NUMBER_OF_TASK_SLOTS to 1
● Set memory limits of container higher than max heap
● Set CPU limits to sensible value
● Configure pod affinity to spread workload
● Expose relevant ports (usually only internally)
● Add sensible labels to identify resources

13. Run exec against the Job Manager
How does it work?
● Kubernetes Client for
managing clusters
● Exec for executing
commands in the
containers

14. Easy to implement but...
● It depends on commands installed in the container
● It seems too consuming in terms of resources (we need to
run a process inside the container for each operation)
● It doesn’t enforce any protocol (stdin/stdout)

15. Flink Monitoring API to the rescue!
Flink has a pretty useful REST API:
● Endpoints for managing jobs
● Endpoints for managing savepoints
● Endpoints for monitoring the cluster
Is there a client library? I am afraid not…

16. Create client using OpenAPI
I manually crafted an
OpenAPI specification file…
… It’s tedious but the
generated client works fine!
See Swagger Documentation:
https://swagger.io/docs/specification/about/

17. Swagger Editor and Code generator
/v1/jobs:
get:
operationId: getJobs
summary: Returns an overview over all jobs and their current state
responses:
'200':
description: |-
200 response
content:
application/json:
schema:
$ref: '#/components/schemas/JobIdsWithStatusOverview'
...
See full specification on GitHub:
https://github.com/nextbreakpoint/flink-client/blob/master/flink-openapi.yaml

18. Combine all in one application
We can combine the APIs:
● Kubernetes Client for
managing clusters
● Flink Client for
managing jobs

19. What are the limitations?
● Where does the client live?
● Still no monitoring or automatic scaling
● NodePort or Port Forward required for each
JobManager (for each Flink Cluster)
● Port Forward doesn’t work well with file
upload (there is a problem with timeout in
the Kubernetes Client for Java)

20. Run controller inside Kubernetes
What are the benefits?
● It can easily access
internal resources
● It runs with its own service
account
● It can monitor the clusters
● It can rescale the clusters

21. Better than before but...
● One port forward is still required
● Authorization is required for API
● It doesn’t follow best practises!

22. We need a Kubernetes Operator!
Everybody think we need Go, but…
… an Operator is like a pattern…
… and we can use any programming language!
Operator SDK for Go:
https://github.com/operator-framework/operator-sdk
Operator Pattern:
https://coreos.com/blog/introducing-operators.html

23. Custom Resource Definition
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: flinkclusters.beta.nextbreakpoint.com
spec:
group: beta.nextbreakpoint.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: flinkclusters
singular: flinkcluster
kind: FlinkCluster
shortNames:
- fc
/api/beta.nextbreakpoint.com/v1/namespaces/*/flinkclusters
# kubectl create -f flink-crd.yaml
# kubectl get crd
NAME AGE
flinkclusters.beta.nextbreakpoint.com 1d
…

24. Custom Objects
apiVersion: "beta.nextbreakpoint.com/v1"
kind: FlinkCluster
metadata:
name: test
spec:
clusterName: test
environment: test
pullSecrets: regcred
pullPolicy: Always
flinkImage: nextbreakpoint/flink:1.7.2-1
sidecarImage: flink-workshop-jobs:2
sidecarServiceAccount: flink-operator
sidecarClassName: com.nextbreakpoint.flink.jobs.TestJob
sidecarJarPath: /com.nextbreakpoint.flinkworkshop-1.0.0.jar
sidecarParallelism: 1
sidecarArguments:
- --BUCKET_BASE_PATH
- file:///var/tmp
# kubectl create -f cluster.yaml
# kubectl get flinkclusters
NAME AGE
test 4s

25. The Operator Loop
1. Receive updates of Custom Objects
2. Receive updates of StatefulSets,
Services, PVCs, …
3. Compare desired state to actual
state
4. Adjust current state to match
desired state
5. Repeat from 1

26. Run a Flink Operator
What are the benefits?
● It follows Kubernetes
best practises
● It runs with its own
service account
● We only need to create
cluster objects

27. Operator meets Controller
They can operate together:
● Use operator with CD pipeline
● Use controller for manual ops
● Use controller for monitoring
● Use controller for alerting
● Use controller for scaling

28. Time for a demo !
A preview of Flink K8S Toolbox:
● Easy installation
● Easy deployments
● Jobs management
● Cluster metrics
● Cluster scaling

29. Monitoring and Scaling
We can use Flink API for:
● Watching jobs status and
alerting when something is
broken
● Observing cluster metrics
and scaling cluster when
required

30. Checkpoints/Savepoints
We can use Flink API for:
● Monitoring checkpoints
● Managing savepoints
● Retrieving last savepoint

31. Continuous Delivery
We can use tools like Flux:
● Push changes into Git repo
● Changes are automatically
applied to resources
Flux (I haven’t actually tried it)
https://github.com/weaveworks/flux

32. Nice features to have...
● Pluggable alerting strategy
● Pluggable scaling strategy
● Web console
● Secure access
● Support for HA mode
● …

33. It’s all free!
Flink Kubernetes Toolbox:
https://github.com/nextbreakpoint/flink-k8s-toolbox
Related projects:
https://github.com/nextbreakpoint/flink-client
https://github.com/nextbreakpoint/flink-workshop
https://github.com/nextbreakpoint/kubernetes-playground

34. Fine.
Where to follow:
@AndreaMedeghini
nextbreakpoint.com