An Operator is an application that encodes the domain knowledge of the application and extends the Kubernetes API through custom resources. They enable users to create, configure, and manage their applications. Operators have been around for a while now, and that has allowed for patterns and best practices to be developed. In this talk, Lili will explain what operators are in the context of Kubernetes and present the different tools out there to create and maintain operators over time. She will end by demoing the building of an operator from scratch, and also using the helper tools available out there.

1. An intro to Kubernetes operators
Lili Cosic
Software Engineer at Red Hat
Twitter: @LiliCosic
Github: lilic

2. An Intro to Kubernetes Operators - Lili Cosic2
Intro to Kubernetes

3. An Intro to Kubernetes Operators - Lili Cosic3
Kubernetes
What is Kubernetes?
● Open source platform for managing containerized
workloads and services
● Containers, containers, containers
● Name originates from Greek - helmsman or pilot
● Google open-sourced in 2014
● Based on Borg - Google’ internal project
● K_ _ _ _ _ _ _ _S -> k8s

4. An Intro to Kubernetes Operators - Lili Cosic4
Kubernetes
What is great about Kubernetes?
● Scalability of workloads
● Separation of workloads
● Native stable resources (Deployments, Pods)
● API / Custom Resources

5. An Intro to Kubernetes Operators - Lili Cosic5
Kubernetes
What is Kubernetes not?
● Not a PaaS
● Not limited to the types of apps supported
● Not opinionated on:
○ Deploying
○ Run CI/CD
○ Logging, monitoring or alerting

6. An Intro to Kubernetes Operators - Lili Cosic6
Kubernetes controllers

7. An Intro to Kubernetes Operators - Lili Cosic7
What is a controller?
● Reconciles given state
● Controller pattern
○ Non terminating loop that regulates the state of the system
○ A control loop that watches the shared state of the cluster via the API server and
makes changes to move from current to desired state

8. An Intro to Kubernetes Operators - Lili Cosic8
ReplicaSet controller
Example controller
● ReplicaSet
○ Created by a Deployments
○ Creates Pods
● Reconciles to the correct (specified) number of pods running the cluster

9. An Intro to Kubernetes Operators - Lili Cosic9
Controller
Credit: github.com/kubernetes/sample-controller

10. An Intro to Kubernetes Operators - Lili Cosic10
Informer pattern
More important API functions
● ListWatcher
○ Helps you filter resources you want to watch
● ResourceEventHandler
○ Add, Update and Delete event trigger functions

11. An Intro to Kubernetes Operators - Lili Cosic11
Workqueue?
● Resource Event Handler puts an item to the workqueue
● Workqueue consists of keys:
○ <resource_namespace>/<resource_name>
● Only one worker works on one item at a time
● First In First Out

12. An Intro to Kubernetes Operators - Lili Cosic12
CRD - Custom Resource Definition

13. An Intro to Kubernetes Operators - Lili Cosic13
CRD - custom resource definition
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# list of versions supported by this CustomResourceDefinition
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct

14. An Intro to Kubernetes Operators - Lili Cosic14
Operators, operators, operators

15. An Intro to Kubernetes Operators - Lili Cosic15

16. An Intro to Kubernetes Operators - Lili Cosic16
What is an “operator”?
Operator is a K8s controller - specific to operating an application

17. An Intro to Kubernetes Operators - Lili Cosic17
What is an “operator”?
● Non core Kubernetes controller
● Makes use of CRDs - custom resource definition
● Holds the knowledge of how an application needs to be deployed, managed and
packaged
● Reconcile loop
● Extends the Kubernetes API to make the application a custom resource - part of the
Kubernetes ecosystem
● Concept introduced by CoreOS

18. An Intro to Kubernetes Operators - Lili Cosic18
Good examples of operators
Awesome operators!
● github.com/coreos/prometheus-operator
● github.com/zalando-incubator/postgres-operator
● github.com/coreos/etcd-operator

19. An Intro to Kubernetes Operators - Lili Cosic19
When to choose creating an operator?
● Application uses declarative API
● Resources are scoped to a namespace or a cluster
● Encapsulate business logic
● Build automation that watches for updates of Kubernetes objects
● Create or update resources via the Kubernetes native API
● Top level support from kubectl

20. An Intro to Kubernetes Operators - Lili Cosic20
When to just use a ConfigMap or Secret?
Sometimes creating an operator is not needed
● Existing well known config file format (e.g. mysql.cnf )
● Config file is used to just configure an application running in a Pod
● No need to reconcile to the state

21. An Intro to Kubernetes Operators - Lili Cosic21
101 ways to build an operator
● go language:
○ client-go & co.
○ operator-sdk
○ kubebuilder
● Other languages:
○ Kubernetes python and java clients and others
● Other tools:
○ helm
○ ansible

22. An Intro to Kubernetes Operators - Lili Cosic22
Building operators using Kubernetes native
clients

23. An Intro to Kubernetes Operators - Lili Cosic23
Kubernetes native clients
● k8s.io/client-go
○ rest
○ discovery
○ dynamic
● k8s.io/api
● ks8.io/apimachinery

24. An Intro to Kubernetes Operators - Lili Cosic24
Kubernetes native clients
Pros
● Same as upstream controllers use
● Stability of K8s code
● Versioning based on Kubernetes releases
● Ability to finetune
Cons
● Large ecosystem
● No abstractions/helpers
● A lot of inside knowledge to optimize correctly
● New major version on every Kubernetes minor
version release

25. An Intro to Kubernetes Operators - Lili Cosic25
Using Kubernetes native go clients
Example of an “operator”
github.com/kubernetes/sample-controller

26. An Intro to Kubernetes Operators - Lili Cosic26
kubebuilder

27. An Intro to Kubernetes Operators - Lili Cosic27
kubebuilder
● github.com/kubernetes-sigs/kubebuilder
● SDK for building Kubernetes APIs using CRDs
● Part of kubernetes-sigs repo
● Uses controller-runtime under the hood
● Doesn’t strictly advertise itself as operator builder tool

28. An Intro to Kubernetes Operators - Lili Cosic28
Using kubebuilder
Example of a go operator
$ mkdir kubebuilder-operator && cd kubebuilder-operator
$ kubebuilder init --domain k8s.io --license apache2 --owner "The JOnTheBeach
Audience"
$ kubebuilder create api --group ships --version v1beta1 --kind Sloop
$ # Edit the logic code
$ pkg/controller/sloop/sloop_controller.go

29. An Intro to Kubernetes Operators - Lili Cosic29
operator-sdk

30. An Intro to Kubernetes Operators - Lili Cosic30
operator-sdk
● github.com/operator-framework/operator-sdk
● Aimed at creating operators
● Part of operator-framework
● Operator types you can create:
○ go
○ helm
○ ansible
● Testing framework

31. An Intro to Kubernetes Operators - Lili Cosic31
Using operator-sdk
Example of a go operator
$ operator-sdk new app-operator
$ cd app-operator
$ # Add a new API for the custom resource AppService
$ operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService
$ # Add a new controller that watches for AppService
$ operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService
$ # Build operator
$ operator-sdk build quay.io/example/app-operator

32. An Intro to Kubernetes Operators - Lili Cosic32
operator-sdk - helm

33. An Intro to Kubernetes Operators - Lili Cosic33
Using operator-sdk
Example of a helm operator
$ # Create a new helm operator project
$ operator-sdk new nginx-operator --api-version=example.com/v1alpha1 --kind=Nginx --type=helm
$ # Edit watches.yaml file to customize the operator logic

34. An Intro to Kubernetes Operators - Lili Cosic34
operator-sdk - ansible

35. An Intro to Kubernetes Operators - Lili Cosic35
Using operator-sdk
Example of an ansible operator
$ # Create a new ansible based operator project:
$ operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1
--kind=Memcached --type=ansible
$ # Edit watches file to customize the logic

36. An Intro to Kubernetes Operators - Lili Cosic36
Deploying an operator

37. An Intro to Kubernetes Operators - Lili Cosic37
Deploying (operator) on Kubernetes
● YAML
● Kustomize
● Jsonnet
● OLM
● Helm

38. An Intro to Kubernetes Operators - Lili Cosic38
YAML
$ # Register the CRD
$ kubectl apply -f crd.yaml
$ # Create any needed Role Based Access Control
$ kubectl apply -f rbac.yaml
$ # Deploy the operator
$ kubectl apply -f deploy.yaml
$ # Create an instance
$ kubectl apply -f cr.yaml

39. An Intro to Kubernetes Operators - Lili Cosic39
YAML - CRD manifest file
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: memcacheds.cache.example.com
spec:
group: cache.example.com
names:
kind: Memcached
listKind: MemcachedList
plural: memcacheds
singular: memcached
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
properties:
size:
format: int32
type: integer
required:
- size
type: object
status:
properties:
nodes:
items:
type: string
type: array
required:
- nodes
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true

40. An Intro to Kubernetes Operators - Lili Cosic40
YAML - deployment manifest file
apiVersion: apps/v1
kind: Deployment
metadata:
name: memcached-operator
spec:
replicas: 1
selector:
matchLabels:
name: memcached-operator
template:
metadata:
labels:
name: memcached-operator
spec:
serviceAccountName: memcached-operator
containers:
- name: memcached-operator
image: memached:123
command:
- memcached-operator
imagePullPolicy: Always
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "memcached-operator"

41. An Intro to Kubernetes Operators - Lili Cosic41
YAML - CR instance manifest file
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
name: example-memcached
spec:
size: 3

42. An Intro to Kubernetes Operators - Lili Cosic42
OLM
● Upgrades
● Descriptions and metadata
● Dependency resolution
● Multiple versions of operator to install
Key features

43. An Intro to Kubernetes Operators - Lili Cosic43
Debugging an operator

44. An Intro to Kubernetes Operators - Lili Cosic44
Debugging an operator
● `operator-sdk up local` - run
● Log all the things!
● Get logs of an operator
○ `kubectl logs <pod-name>`
● Use CR name to label any pods the operator creates
● `kubectl events -n NAMESPACE`
● `Spec.Paused`

45. An Intro to Kubernetes Operators - Lili Cosic45
Kubernetes cluster stack for big data

46. An Intro to Kubernetes Operators - Lili Cosic46
More information
● Kubernetes Special Interest Group Big Data
● User group meeting: Wednesdays at 18:00 UTC (biweekly)
● Slack channel - #ug-big-data

47. An Intro to Kubernetes Operators - Lili Cosic47
Spark operator
Spark on Kubernetes
github.com/GoogleCloudPlatform/spark-on-k8s-operator

48. An Intro to Kubernetes Operators - Lili Cosic48
Future
Whats next for operators?
● OLM
● cluster-addons
● CRDs
○ GA in 1.16

49. QUESTIONS
Lili Cosic
Twitter: @LiliCosic
Github: lilic