Slides for my talk at Infracoders VII. Original Google Slides: https://docs.google.com/presentation/d/e/2PACX-1vQnYbG32QDRcrXPKCfE4G2YlPSLtphMbnePc_xfjhRU7T9vuKgK1kI9LjYqo5GVCxbW8P7NnC02_Lnu/pub?start=false&loop=false&delayms=3000

1. Someone is 👀 you
Kubernetes APIs and Watch Events Explained
Oliver Moser

2. The Guy
2
- TU Wien PhD
- Braintribe, Inc.
- Cloud / Automation / SRE

3. The Talk
3
- Kubernetes APIs
- Watch Events
- Live Examples

4. Kubernetes APIs

5. Kubernetes
API
5
- Resource-based programmable
interface for container workloads
- HTTP based API
- Entrypoint into Kubernetes
- Lives in the kube-apiserver

6. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
Master Nodes
Components
Worker Nodes
Components

7. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
/api/v1/namespaces/default/pods/nginx

8. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
WATCH /api/v1/watch/pods
pod.nodeName == nil
pod.nodeName == nil
pod.nodeName == NodeB

9. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
WATCH /apis/apps/v1/watch/replicasets
WATCH /api/v1/watch/pods
replicaset-controller
WATCH /api/v1/watch/namespaces
namespace-controller
certificate-controller
cronjob-controller
endpoint-controller
node-lifecycle-controller
...

10. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
WATCH /api/v1/watch/pods
FILTER pod.spec.nodeName == Node A

11. Kubernetes Components
kube-apiserver
kubelet
kube-scheduler
kube-controller-manager
kube-proxy
WATCH /api/v1/watch/services
FILTER pod.spec.nodeName == Node A

12. Kubernetes API Structure
{paths: [
/api,
/api/v1,
/apis,
/apis/,
/apis/admission.certmanager.k8s.io,
/apis/admission.certmanager.k8s.io/v1beta1,
/apis/admissionregistration.k8s.io,
/apis/admissionregistration.k8s.io/v1beta1,
/apis/apiextensions.k8s.io,
/apis/apiextensions.k8s.io/v1beta1,
/apis/apiregistration.k8s.io,
/apis/apiregistration.k8s.io/v1,
/apis/apiregistration.k8s.io/v1beta1,
/apis/apps,
/apis/apps/v1,
/apis/apps/v1beta1,
/apis/apps/v1beta2,
/apis/authentication.k8s.io,
/apis/authentication.k8s.io/v1,
…
]}
> kubectl proxy &
> curl localhost:8001

13. Kubernetes API Structure
stolen from: https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/

14. Kubernetes API Structure
{
kind: APIGroup,
apiVersion: v1,
name: apps,
versions: [
{
groupVersion: apps/v1,
version: v1
},
{
groupVersion: apps/v1beta2,
version: v1beta2
},
{
groupVersion: apps/v1beta1,
version: v1beta1
}
],
preferredVersion: {
groupVersion: apps/v1,
version: v1
}
}
> kubectl proxy &
> curl localhost:8001/apis/apps

15. Kubernetes API Structure
{
kind: APIResourceList,
apiVersion: v1,
groupVersion: apps/v1,
resources: [
...
{
name: deployments,
singularName: ,
namespaced: true,
kind: Deployment,
verbs: [
create,
delete,
get,
list,
patch,
update,
watch
],
shortNames: [
deploy
],
...
}
> kubectl proxy &
> curl localhost:8001/apis/apps/v1

16. Kubernetes API Structure {
kind: DeploymentList,
apiVersion: apps/v1,
metadata: {
selfLink: /apis/apps/v1/deployments,
resourceVersion: 242895786
},
items: [
{
metadata: {
name: phoenix-dev-adx-cartridge,
namespace: adx,
selfLink: /apis/apps/v1/namespaces/...,
uid: 9972a409-ef77-11e9-8904-42010a9c00ae,
resourceVersion: 241706486,
generation: 1,
creationTimestamp: 2019-10-15T18:14:16Z,
labels: {
app: adx-cartridge,
initiative: phoenix,
runtime: phoenix-dev,
stage: dev,
workspace: adx
},
...
> kubectl proxy &
> curl localhost:8001/apis/apps/v1/deployments

17. /apis/
apps/v1/
namespaces/default/
deployments/nginx
Version
Group
metadata.namespace
Kind
metadata.name
spec.scope: Namespaced
/apis/apps/v1/namespaces/default/deployments/nginx
Kubernetes Resource Naming: Group Version Kind

18. - Each Kubernetes Object has a resourceVersion
- Used for optimistic locking/ concurrency control
Kubernetes Resource Versioning
metadata.resourceVersion: 1492311
spec.image: nginx:1.13.2
metadata.resourceVersion: 1492918
spec.image: nginx:1.13.3
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
resourceVersion:
1492311
spec:
containers:
- image:
nginx:1.13.3
name:
nginx

19. Kubernetes Watch Events

20. Watch Events: Incremental State Change Notifications
kube-apiserver kube-scheduler
GET /api/v1/pods?watch=true
metadata.resourceVersion: 1503831
status: {
phase: Pending,
qosClass: BestEffort
}
metadata.resourceVersion: 1503832
nodeName: nodeA
status: {
phase: Pending,
conditions: [
{
type: PodScheduled,
status: True,
…
}
],
}
metadata.resourceVersion: 1503851
nodeName: nodeA
status: {
phase: Running,
conditions: [
{
type: ContainersReady,
status: True,
…
}
],
}
MODIFIEDADDED MODIFIED
time

21. Watch Events: Type Definition
type WatchEvent struct {
// The type of the watch event; added, modified, deleted, or error.
// +optional
Type watch.EventType `json:type,omitempty description:the type of watch event; may be ADDED,
MODIFIED, DELETED, BOOKMARK or ERROR`
// For added or modified objects, this is the new object; for deleted objects,
// it's the state of the object immediately prior to its deletion.
// For errors, it's an api.Status.
// +optional
Object runtime.RawExtension `json:object,omitempty description:the object being watched;
will match the type of the resource endpoint or be a Status object if the type is ERROR`
}

22. Watch Events: Build up State
Type: ModifiedType: Added
time
Type: Modified Type: Deleted

23. Watch Events: Change Propagation
time
“delta
propagation”
“full-state
propagation”
{
"operation": "add",
"field": "replicas",
"value": 2
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 2,
...
}
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 1,
...
}
}
{
"operation": "del",
"field": "replicas",
"value": 1
}

24. Watch Events: Change Propagation
time
{
"operation": "add",
"field": "replicas",
"value": 2
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 2,
...
}
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 1,
...
}
}
{
"operation": "del",
"field": "replicas",
"value": 1
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 4,
...
}
}
{
"operation": "add",
"field": "replicas",
"value": 3
}
replicas:=4
replicas:=4

25. Watch Events: Change Propagation
time
{
"operation": "add",
"field": "replicas",
"value": 2
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 2,
...
}
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 1,
...
}
}
{
"operation": "del",
"field": "replicas",
"value": 1
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": "calico-typha",
"namespace": "kube-system"
...
},
"spec": {
"replicas": 4,
...
}
}
{
"operation": "add",
"field": "replicas",
"value": 3
}
lost update!
replicas:=5
replicas:=5

26. Live Examples

27. The End
27
Thank You

28. Watch Events: Edge Triggered vs Level Triggered
Signal Rise
Signal FallEdge
Triggered
Level
Triggered
Signal High
Signal Low

29. Watch Events: Edge Triggered vs Level Triggered
add 2
replicas
remove 1
replica
Edge
Triggered
Level
Triggered
replicas:=4
replicas:=3
replicas:=2 replicas:=3

30. Watch Events: Edge Triggered vs Level Triggered
add 2
replicas
remove 1
replica
Edge
Triggered
Level
Triggered
replicas:=3
replicas:=2
replicas:=1
replicas:=4
replicas:=3
lost update!