Extending Kubernetes

Kubernetes Meetup Frankfurt
@ Meshcloud, Mar 25th 2019

why, what and how
Extending Kubernetes

@meshstack
Kubernetes orchestrates Infrastructure & Containers...
… meshcloud orchestrates Cloud Tenants
...Cloud Tenants?
Motivation

@meshstack
81%
We live in a Multi-Cloud world
@meshstack

Orchestrating Clouds & Tenants
Integrierte Prozesse mit
meshcloud
Heterogene
Prozesse!

- OpenShift PaaS (OKD = “OpenShift Kubernetes Distribution”)
- https://www.okd.io/
- Fission “Kubernetes-native Serverless Framework”
- https://fission.io/
- Kubernetes Operators
- ArangoDB
- Couchbase
- MySQL
- … many more
Extending Kubernetes - Examples

@meshstack
- Control Plane
- kube-apisever: provides the API
- backed by etcd (& only component talking to etcd)
- scales horizontally
- think of it as a “dumb” object server (authn/authz, validation, defaulting etc.)
- kube-controller-manager
- runs the main control-loops
- HA, but only a single active leader at a time
- kube-scheduler
- distributes pods on nodes: think “pod controller”
- Workload Plane
- Kubelet & kube-proxy: execute containers, report status etc.
Kubernetes - Architecture recap

@meshstack
1. kubectl plugins
2. API Extensions
3. Custom Resources
4. Scheduler Extensions
5. Controllers (& Operators)
6. Network Plugins
7. Storage Plugins
Goal: Provide an Overview and Examples
Extending Kubernetes - Overview
https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/

@meshstack
API Groups are a way to … group APIs for
- Versioning e.g. /api/v1 or
/apis/extensions/v1beta1
- Extending Kubernetes with vendor or
distribution-specific APIs
API Extensions - Basics

@meshstack
- Runs in-process with the kube-apiserver
- APIService object “claims” URL path
- Enables auto-discovery of APIs by clients
- Checkout Kubernetes sample-apiserver
- Example: OpenShift with a “dumb” kubectl client
- kubectl api-resources
- kubectl get apiservices
- kubectl get apiservices v1.build.openshift.io -o=yaml
- kubectl --v=8 get build
API Extensions - Aggregation Layer

@meshstack
API Extensions - when to use?

@meshstack
- Intercept Kubernetes API requests in order to
- validate: e.g. rate limit api access EventRateLimit
- mutate: e.g. AlwaysPullImages
- Kubernetes ships with default Admission Controllers that
implement core functionality
- e.g. NamespaceExists
Admission Controllers

@meshstack
- Define name and Schema for new types of K8s objects
- Simpler than writing your own API for some cases
- e.g. built-in validation using JSON Schema (K8s > v1.8)
- Typically goes hand in hand with a Controller
- kubectl get customresourcedefinitions
Learn more from the documentation
Custom Resource Definitions

@meshstack
- Pretty much everything is an Object in Kubernetes. Objects follow
conventions
- .apiVersion version of the API that produced this object
- .kind what kind of Object it is, e.g. pod
- .metadata common metadata like .name, .annotations etc.
- .spec desired state of the Object
- .status last observed state of the object
- kubectl get pods -o=yaml --namespace=openshift-web-console
Learn more from the documentation
Custom Resource Definitions - Objects

@meshstack
- Example: Fission (sorry, no kubectl demo)
Custom Resource Definitions

@meshstack
- Controllers make the “magic” happen
- Think CRDs as a way to teach kube-apiserver to store new objects
- Controllers process these objects as clients of the api-server
- Control Loop
- watch relevant objects via kube-apiserver
- compare .status and .spec
- take action to converge actual to desired state
- update managed object’s .status
Controllers

@meshstack
- Controllers make the “magic” happen
- Think CRDs as a way to teach kube-apiserver to store new objects
- Controllers process these objects as clients of the api-server
- Control Loop
- watch relevant objects via kube-apiserver
- compare .status and .spec
- take action to converge actual to desired state
- update managed object’s .status
- core control loops run in kube-controller-manager
- e.g. ReplicaSetController
Controllers

@meshstack
Control Loops in distributed systems
are hard to implement correctly.
- shared state is hard
- timing is hard
- things go wrong™
Using Level Triggered Logic helps
reducing complexity and improving
correctness.
Think: delta vs. full-state reconcillation
Controller Basics - Loops
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d

@meshstack
Ideal conditions:
- Controller observes original signal
without disruptions
- Controller will take correct action
(i.e. process user intent)

@meshstack
Signal is observed with disruptions (i.e.
network disruption, timing issues,
control plane crash...)
- Edge triggered loop misses
trigger (does not take action)
- Level triggered loop eventually
takes correct action

@meshstack
Signal is observed with disruptions (i.e.
network disruption, timing issues,
control plane crash...)
- Edge triggered loop accumulates
incorrect state
- Level triggered loop eventually
converges to correct state

@meshstack
Example: ReplicaSet

@meshstack
Build Controllers using client-go (but
can use your own logic as well):
- Controllers watch() objects they’re
interested in from kube-apiserver
- resync period: periodically re-process
all objects delivered by kube-apiserver
- Delta Fifo Queue = “smart” queue
- Tombstones for deleted objects
- Merges duplicated updates
Example: OpenShift ResourceQuota
kubectl get -o=yaml resourcequota
--namespace=managed-customer-demo-project
Controllers - Loops in Practice

@meshstack
Operators combine CRDs + Controllers to manage stateful Applications
- Example: Automated Master Failover for MariaDB
- … whatever “manual Ops” would do - you can automate with K8s
- Operator Framework helps with building and operating operators
Operators

Thank you
Johannes Rudolph
jrudolph@meshcloud.io
www.meshcloud.io

Extending Kubernetes

More Related Content

What's hot

Similar to Extending Kubernetes

Recently uploaded

Extending Kubernetes