This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.

1. Building within Kubernetes
CRDs and Operators
9 / 3 / 2018 - Incontro Devops Italia

2. whoami
Giacomo Tirabassi
Devops Engineer @ Sighup Srl
twitter.com/gitirabassi
github.com/gitirabassi
I like to automate, cook and git together

3. Sighup Srl
All things Kubernetes
sighup.io
twitter.com/sighup_

4. Why are we here?
- Operations are boring
- Repeating work over and over
- Distributed Systems are hard to manage by humans
- Application Lifecycle Management as Code

6. Agenda
OPERATOR
Pattern
Kubernetes
Resources
Kubernetes
Controllers
Kubernetes
Pluggability

7. OPERATOR
Pattern
Kubernetes
Resources
Kubernetes
Controllers
Kubernetes
Pluggability

8. K8S Extensibility -- Infrastructure
● Container Runtime (CRI)
● Network Plugin (CNI)
● Volumes (CSI) → alpha in 1.9
● Cloud Controller Manager (Aws, GCP, Azure, Openstack, oVirt) → alpha in 1.10
● Device Plugins (GPU, FPGAs, QRNGs) → alpha in 1.8

9. K8S Extensibility -- Infrastructure
● Secrets Backend (Vault, Azure AWS GCP KMS ) → alpha in 1.9
● Replace Scheduler
● Multiple Scheduler
● Scheduler Extender
● Custom Controller

10. K8S Extensibility -- API
● API(-server) Aggregation
● Custom Resource Definition

11. OPERATOR
Pattern
Kubernetes
Resources
Kubernetes
Controllers
Kubernetes
Pluggability

14. K8s Resource Types
● API-Server is a REST Application with CRUD interface
● API-Server doesn’t know anything about infrastructure
● API-Server manages Pods/Deployments/Services like apples and pears (no special meaning)
● Object Properties:
○ Apiversion
○ Kind
○ Metadata
○ Specs

15. K8s Resource Types
https://MASTER_NODE_URL/apis/extensions/v1beta1/namespaces/default/deployments
https://MASTER_NODE_URL/api/ v1 /nodes
https://MASTER_NODE_URL/api/ v1 /namespaces/default/services
Master url GROUP VERSION NAMESPACE KIND

16. Custom Resource Definition
● Previously known as TPR ( 1.2 -- 1.7 )
● Stable from kubernetes 1.8
● C-R-U-D Object
● RBAC
● State definition API: declarative
● Versioning
● Automatic `kubectl` compatibility
demo

17. K8s Resources
https://MASTER_NODE_URL/apis/sighup.io/v1alpha1/namespaces/default/apples
Master url GROUP VERSION NAMESPACE KIND

18. OPERATOR
Pattern
Kubernetes
Resources
Kubernetes
Controllers
Kubernetes
Pluggability

21. Controllers
- Reconciliation loops: Observe + Analyze + Act
- Converge desidered state with real state
- Attached to infrastructure events (getters & listers & informers)
- Simple Interface to implement: ADD, UPDATE, DELETE
- Cluster aware
- Actions can be internal to the cluster or external (beginning of service catalog)

22. Controllers Properties
- Fully typed
- Cache
- Workqueues
- Chaos
- Leader election
- Code generation
- Clientsets

23. Custom controllers examples
- Automatic notification on resources allocation, role bindings, errors
- Automatic service creation for every new deployment
- Automatic Secret distribution
- Automatic new namespace configurations (conservatives defaults)
- Automatic Custom horizontal pod scaler
- Automatic ingress controller configuration
demo

24. OPERATOR
Pattern
Kubernetes
Resources
Kubernetes
Controllers
Kubernetes
Pluggability

25. Patterns
- CRD + Volume Plugin + Controller ⇒ Rook
- CRD + Network Plugin ⇒ Calico Canal
- CRD + Controller ⇒ Operator
- CRD + Controller ⇒ core features prototyping

26. Operators
- Not a kubernetes Object: just a pattern
- Introduced by COREOS → 3/11/2016
- Go program running in the cluster as deployment (other language supported)

27. Operators available
- Etcd / kafka / kubedb / prometheus / influx / elasticsearch / mongodb / memcached / postgres
- Vitess / redis
- Rook
- Cert-manager
- Api gateway: kanali / kong
- All operations infrastructure facing: backup, dr, snapshots, vpn

28. Why Operators?
- Custom Bootstrap, Backup, Restore, DR, Monitoring, Telemetry
- Manual recurrent cleanups
- Ad-hoc commands for scaling/join/leave
- Clustering
- Stateful (writeOnceReadMany, writeManyReadMany)
- Custom Management (dedicated hardware or people)

29. Why Operators?
- How to automate application lifecycle without operators?
- Many Wheels:
- Accessibility
- Availability (HA)
- Authorization
- Events
- Reconciliation Loop ( real state → desired state )

30. Operator’s alternatives?
- YAML manifest → not templatable, very verbose
- Helm Charts → only deploy, no cluster aware
- Third party automation → reinvent wheels and polling on api-server for changes
- Operator → complex business logic to deal with, simple for multiple instances of object
demo

31. Problems
● Bring together people who knows about go, kubernetes and the specific application
● Convince people to trust automation
● Like kernel modules → very powerful, but it can crash the whole system
● Kubernetes Go SDK is a mess

32. Recap
- Manual operations are expensive, boring and error prone
- Abstracting apps livecycle management from underlying infrastructure (k8s feature)
- Non-cloud application can become simple objects
- Stateful is hard but not blocking

33. Tools
● github.com/golang/dep
● github.com/kubernetes/code-generator
● github.com/giantswarm/operatorkit
● github.com/rool/operator-kit
● github.com/spotahome/kooper

34. the end