Kubernetes is designed to be an extensible system. But what is the vision for Kubernetes Extensibility? Do you know the difference between webhooks and cloud providers, or between CRI, CSI, and CNI? In this talk we will explore what extension points exist, how they have evolved, and how to use them to make the system do new and interesting things. We’ll give our vision for how they will probably evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build with them.

1. Eric Tune & Tim Hockin
Google
Kubernetes
Extensibility

2. Kubernetes is a container
management system

3. Kubernetes is a container
management system platform

4. What is Kubernetes?
...an abstraction layer over
infrastructure
...a framework for declarative
APIs and distributed control
Infrastructure
Extensibility
API
Extensibility

5. Extensibility Goals
Infrastructure Extensibility API Extensibility
Support portability Support customization
Autonomy Autonomy
Scalable growth of project Encourage new uses

6. A major focus of the last 2 years of development
From infrastructure to APIs, we have over a dozen
extension points
We have WAY more material than time!
https://goo.gl/2qz8jW
Kubernetes & Extensibility

7. Infrastructure Extensibility

8. Networks are like snowflakes
There is no “one size fits all” for almost
anything networking related
We needed a way for users to customize how
Kubernetes consumes networking infra
Network Plugins

9. Old: built-in “plugins” (aka “send Tim a PR”)
New: CNI - Container Network Interface
● Started by CoreOS, now CNCF with community
● “exec” interface with stdin/stdout/env API
Widely used, also by other projects (e.g. Mesos)
Underpins the default impl in Kubernetes
Network Plugins (present)

10. Proposal open for a gRPC based API which
covers more than just interfaces and IPAM
Tighter coupling with Service API seems valuable
Proposals open for multi-IP and multi-network
Network Plugins (future)

11. Many storage technologies - physical and
virtual, block and file
● Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ...
Many vendors want their products to support
Kubernetes
Storage Plugins

12. Old: built-in “plugins” (aka “send Tim a PR”)
Old: Volume “flex” plugins via “exec”
New: CSI - Container Storage Interface
● Collaboration: Google, Mesosphere, Docker, Cloud Foundry
● gRPC spec, with Kubernetes-specific adaptors
● In development now, alpha in Kubernetes 1.10
Plan to transition most in-tree plugins to CSI
Storage Plugins (present)

13. GPUs and other “accelerator” hardware is
becoming very common
Part of the larger resource model in Kubernetes
gRPC based plugins
Beta in Kubernetes 1.10
Device Plugins

14. Docker was baked-in, but people wanted to try
new and interesting ideas
● rkt, Containerd, CRI-O
● Kata containers, Hyper.sh, gVisor
Making it a plugin made the code better: win-win!
CRI - gRPC based plugins
Container Runtimes

15. ● Stateful, daemon plugins
● Upgradeable in-cluster plugins
● Evolution: exec → RPC
● Evolution: loose spec → tight
● Containerized plugins FTW
Lessons Learned
gRPC Plugins
Runtimes (CRI)
Storage (CSI)
Devices
Key Management
Networking (proposed)

16. Controllers
observe
diff
act

17. Controllers
THE fundamental design pattern in Kubernetes
Examples: scheduler, kubelet, deployments,
kube-proxy, cloud providers, load balancers,
volume provisioners, auto-scalers, ...
Allows automation & extension of almost any
existing API

18. resource
resource
resource
Higher level
of
abstraction
Lower level
of
abstraction

19. Kubernetes is designed to leverage clouds
Built-in cloud-provider API (i.e. send me a PR) is
hooked into many core control loops
Now 8 implementations (and huge LOC count),
so moving out-of-tree
Cloud Providers

20. The API is a VIP (more or less) and virtual LB
We ship a default implementation (kube-proxy),
but that can be replaced
Controller: watch the API server for Services
and Endpoints, program $NETWORK
Services

21. But Wait, There’s More!
● Secret management (KMS)
● HTTP load-balancing (Ingress)
● NetworkPolicy
● DNS
● Scheduler extenders & whole schedulers
● ...and that’s JUST the infrastructure (i.e. boring) parts

22. API Extensibility

23. ● Add new types of resources to your cluster
● Add custom policy hooks
○ to custom and built-in APIs
● "APIs that add and modify APIs"
API Extensibility

24. ● In Mac Edge, Windows Edge, and EE 2.0
● Supports API Extensions.
● Certified Kubernetes
● Docker Stacks uses API Extensions
Kubernetes for Docker

25. Exploring Stacks
Follow along at
https://goo.gl/JT7v8Z

26. Exploring Stacks
https://goo.gl/JT7v8Z
$ cat docker-compose.yml
version: "3.3"
services:
redis:
image: redis:alpine
ports:
- 6379
networks:
- frontend
deploy:
replicas: 1
networks:
frontend:

27. Exploring Stacks
https://goo.gl/JT7v8Z
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service redis has one container running
Stack stackdemo is stable and running

28. Exploring Stacks
https://goo.gl/JT7v8Z
$ kubectl config current-context
docker-for-desktop

29. Exploring Stacks
https://goo.gl/JT7v8Z
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d
redis ClusterIP None <none> 55555/TCP 1s
redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s

30. Exploring Stacks
https://goo.gl/JT7v8Z
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d
redis ClusterIP None <none> 55555/TCP 1s
redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
redis 1 1 1 1 2s

31. compose
resource
service
resource
deployment
resource
Higher level
of
abstraction
Lower level
of
abstraction

32. Kubernetes API Server
Service API
compose
resource
Deployment API
service
resource
deployment
resource

33. Kubernetes APIs
Service API
Deployment API
service
resource
deployment
resource
something
custom
compose
resource

34. Kubernetes APIs
Service API
Deployment API
service
resource
deployment
resource
dockerd
hypothetical
/stacks
compose
resource

35. Kubernetes APIs
Service API
Deployment API
service
resource
deployment
resource
dockerd
hypothetical
/stacks
compose
resource
docker
cli

36. Exploring Stacks API
https://goo.gl/JT7v8Z
# last time...
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service web has one container running
- Service redis has one container running
Stack stackdemo is stable and running

37. Exploring Stacks API
https://goo.gl/JT7v8Z
№ last time...
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service web has one container running
- Service redis has one container running
Stack stackdemo is stable and running
$ kubectl get stacks
NAME AGE
stackdemo 39s

38. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl proxy -v 5
Starting to serve on 127.0.0.1:8001

39. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl proxy -v 5
Starting to serve on 127.0.0.1:8001
$

40. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl proxy -v 5
Starting to serve on 127.0.0.1:8001
$ kubectl get stacks -s localhost:8001

41. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl proxy -v 5
Starting to serve on 127.0.0.1:8001
I0613 10:13:27.322416 82905
proxy_server.go:138] Filter accepting
GET
/apis/compose.docker.com/v1beta2/name
spaces/default/stacks localhost
$ kubectl get stacks -s localhost:8001
NAME AGE
stackdemo 1m

42. Kubernetes APIs
Service API
Deployment API
service
resource
deployment
resource
kubectl
compose.docker.com
API
compose
resource

43. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl get apiservices.apiregistration.k8s.io
NAME AGE
v1. 29d
v1.apps 29d
...
v1beta2.compose.docker.com 29d
v2beta1.autoscaling 29d

44. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com
Name: v1beta2.compose.docker.com
...
API Version: apiregistration.k8s.io/v1beta1
Kind: APIService
Metadata:
...
Spec:
...
Service:
Name: compose-api
Namespace: docker
Status:
Conditions:
Message: all checks passed

45. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl get services -n docker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d

46. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl get services -n docker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
$ kubectl get deployments -n docker
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
compose 1 1 1 1 29d
compose-api 1 1 1 1 29d

47. Kubernetes APIs
Service API
API Registration
API
Kubernetes Cluster
Deployment API
Compose-API

48. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API

49. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API

50. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API

51. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose

52. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose
redis
docker
CLI

53. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose
redis
redis
redis
redis

54. ● Users
○ Already have a client installed
○ Already know how to find, trust it (TLS) and auth to the API
● Controllers
○ Can efficiently watch your resources
● Admins
○ Can separate your resources by Namespace
○ Can authorize and audit log access to your resources
Why Use an API Extension?

55. API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)
API
resource
Controller

56. API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)
API
resource
Controller

57. Extension API
Server (EAS)
API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)Extension API
Server (EAS)
API
resource
Controller

58. Extension API
Server (EAS)
API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)Extension API
Server (EAS)
API
resource
Controller

59. Extension API
Server (EAS)
Extension API
Server (EAS)Extension API
Server (EAS)
API
resource
Controller

60. API
resource
Controller
Custom Resource Definitions

61. EAS
Forked LoC: 0
Storage: provided
Components: 1
Popularity: 100s
Multiversioning: not yet
Customizability: good
CRD
Forked LoC: 5000*
Storage: you manage
Components: 3
Popularity: 10s
Multiversioning: yes
Customizability: better
* http://github.com/sample-apiserver

62. Extension Ecosystem
Devices 5 public plugins
Storage 10 public plugins
Networking >20 public plugins
Custom APIs >400 Github Projects with
custom APIs

63. Extension Ecosystem
● 4 Serverless frameworks
● 6 PaaSes
● 10 CI/CD systems
● 14 different database controllers
● 4 popular ML toolkits

64. Adding Types to the API
● Extension API Servers
● Custom Resource Definitions
Adding Policy to the API
● ValidatingAdmissionWebhooks
● MutatingAdmissionWebhooks
API Extensions

65. Admission:
After authn/z but before storing the change.
Affects mutations, not reads.
Webhooks:
The API Server calls your URL, synchronously
Run in cluster via service or outside, e.g.
serverless.
Admission Webhooks

66. Old thinking:
Better to make narrow specific interfaces, like
ImagePolicyWebhook, for specific use cases.
Can make easier to use. Overly general
extensions may limit future optimization.
Admission Webhooks

67. New thinking:
Many custom resoures. Cluster owners need to
write policy for core resources and for custom
resources written by 3rd parties. Need to
compose policies written by different parties.
Admission Webhooks

68. Composability.
Make all the changes before doing all the checks.
MutatingAdmissionWebhooks
- then-
ValidatingAdmissionWebhooks
Admission Webhooks

69. Kelsey Hightower:
- reject pods that set environment variables
https://github.com/kelseyhightower/denyenv-validating-admission-webhook
CRD Authors :
- add complex validation
Validating Admission Webhooks

70. Istio:
inject sidecar into all the pods
Service Catalog:
inject credentials into
Mutating Admission Webhooks

71. - Mutate the pod template of a deployment
- Install a flaky webhook matching all resources.
Bad Ideas

72. •Kubernetes for Docker:
• Super easy way to try Kubernetes
•API Extensions:
• Use them. Author them. On Docker. For Kubernetes.
•Try it:
• https://goo.gl/JT7v8Z
Conclusion

73. v
Questions?
Learn more: https://goo.gl/JT7v8Z
Thanks!