https://kccna18.sched.com/event/GrUI/custom-deployment-strategies-for-kubernetes-nail-islamov-atlassian Many tech companies are using continuous deployments (CD) to deliver changes to their users faster and more frequently. One of the challenges with automated deployments is making them safe by detecting and quickly rolling back in the event of a bad release. Standard CD practices include using canary and blue-green deployments; unfortunately, Kubernetes only supports the "rolling update" deployment strategy out of the box, which can only prevent trivial failures. Thanks to extensibility of Kubernetes, it is possible to build custom advanced deployment strategies while reusing Kubernetes core concepts. Nail Islamov will give an overview of how Deployment, ReplicaSet and Pod objects work together along with Service and Ingress, and will provide examples of implementing blue-green and canary deployments reusing these concepts by introducing extra CRD resources.

1. NAIL ISLAMOV | SENIOR DEVELOPER | @NILEBOX
Custom Deployment Strategies
for Kubernetes

2. Continuous delivery is an approach where
teams release products frequently and
predictably from source code repository to
production in an automated fashion.
CONTINUOUS DELIVERY

3. CI/CD PIPELINE
Build & Test
Deploy to
Staging
Acceptance
Tests
Deploy to
Production
Monitoring

4. CI/CD PIPELINE
Build & Test
Deploy to
Staging
Acceptance
Tests
Deploy to
Production
Monitoring

5. Deployment

6. Recreate
Kill all existing pods before creating new ones.
RollingUpdate
Gradually scale down the old ReplicaSets and
scale up the new one.
Deployment
strategies

7. Recreate
Kill all existing pods before creating new ones.
RollingUpdate
Gradually scale down the old ReplicaSets and
scale up the new one.
Deployment
strategies

8. ROLLING UPDATE
Deployment
ReplicaSet Pod
ReplicaSet Pod
version 2
version 1

9. ROLLING UPDATE
Deployment
ReplicaSet Pod
version 2

10. Continuous Deployment

11. v1
ROLLING UPDATE: TRAFFIC TIMELINE
v1
v1
v1
v1
v1
v1
v1
v1
v1 v2
v2
v2
v2
v2
v2
v2
v2
v2
v2

12. v1
ROLLING UPDATE: ROLLBACK
v1
v1 v1
v1
v1
v1 v1
v2
v2
v2
v2 v2 v2
v2
v2
v2

13. How do we detect issues in
production?
Metrics.

14. How do we reduce impact in
case of a bad release?
Custom deployment strategies.

15. Custom Deployment Strategies

16. Pod
BLUE-GREEN DEPLOYMENT
IngressService
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“blue”
“green”
version 1
version 2
Production: “blue"

17. Pod
BLUE-GREEN DEPLOYMENT
IngressService
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“blue”
“green”
version 1
version 2
Production: “blue"

18. Pod
BLUE-GREEN DEPLOYMENT
IngressService
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“blue”
“green”
version 1
version 2
Production: “green”

19. v1
BLUE-GREEN DEPLOYMENT: TRAFFIC TIMELINE
v1 v1v1 v3v2 v2 v2 v2 v2 v2

20. v1
BLUE-GREEN DEPLOYMENT: ROLLBACK
v1 v1v1 v1v2 v2 v1 v1 v1 v1

21. Canary release is a technique to reduce the
risk of introducing a new software version
in production by slowly rolling out the
change to a small subset of users before
rolling it out to the entire infrastructure and
making it available to everybody.
MARTINFOWLER.COM

22. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“stable”
“canary”
version 1
version 1
Ingress
Service
Service
90%
10%

23. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“canary”
version 1
version 2
“stable”
Ingress
Service
Service
90%
10%

24. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“canary”
version 2
version 2
“stable”
Ingress
Service
Service
90%
10%

25. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“stable”
“canary”
version 2
version 2
Ingress
Service
Service
90%
10%

26. v1
CANARY DEPLOYMENT: TRAFFIC TIMELINE
v1 v1 v1 v1 v1 v1 v1
v1
v1 v2
v2
v2 v2 v2 v2 v2 v2 v2
v2

27. v1
CANARY DEPLOYMENT: ROLLBACK
v1 v1
v1
v1 v1 v1 v1 v1v1 v1
v2
v2 v2
v2

28. Canary Deployment Controller

29. How do we automate the
deployment rollout?
Scripts in CI/CD tool.

30. EXAMPLE
kubectl apply -f deployment-canary.yaml
kubectl apply -f deployment-stable.yaml
kubectl apply -f service-canary.yaml
kubectl apply -f service-stable.yaml
kubectl apply -f ingress.yaml

31. EXAMPLE
kubectl apply -f deployment-canary.yaml
sleep 5m # wait for rollout to finish
# check if application is healthy
curl http://metrics:9090/my-metric
# proceed or rollback
if ...
kubectl apply -f deployment-stable.yaml

32. Can we do better?

33. How do we automate the
deployment rollout?
Scripts in CI/CD tool.
CRD controller?

34. Self-healing
Reconciliation loop will keep
retrying until reaching the
final state
Benefits of CRDs
Reusable
Building block that can be
used together with other
Kubernetes resources
Declarative
Describes the desired state,
not the steps to reach it

35. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“canary”
version 1
version 2
“stable”
Ingress
Service
Service
90%
10%
“stable”
“canary”

36. Pod
CANARY DEPLOYMENT
Deployment ReplicaSet Pod
PodReplicaSet PodDeployment
“canary”
version 1
version 2
“stable”
Ingress
Service
Service
90%
10%
“stable”
“canary”

37. CANARY DEPLOYMENT
Deployment
Deployment
“canary”
version 1
version 2
“stable”
Ingress
Service
Service
90%
10%
“stable”
“canary”

38. CANARY DEPLOYMENT
Deployment
Deployment
“canary”
version 1
version 2
“stable”
Ingress
Service
Service
90%
10%
CanaryDeployment
“stable”
“canary”

39. CANARY DEPLOYMENT
Ingress
Service
Service
90%
10%
CanaryDeployment
“stable”
“canary”

40. DECLARATIVE CONTINUOUS DEPLOYMENT
kubectl apply -f canarydeployment.yaml
kubectl apply -f service-canary.yaml
kubectl apply -f service-stable.yaml
kubectl apply -f ingress.yaml

41. How will CanaryDeployment
controller detect a bad
release?
Metrics.

42. Kubernetes Metrics APIs

43. Horizontal Pod Autoscaler is the primary
consumer of Kubernetes Metrics APIs at the
moment.
HORIZONTAL POD AUTOSCALER (HPA)

44. HPA CONTROLLER
Deployment HPA Controller
Resource Metrics API
External Metrics API
Custom Metrics API

45. CANARY DEPLOYMENT CONTROLLER
Deployment
Deployment
“stable”
“canary”
version 1
version 2
Canary Deployment
Controller
Resource Metrics API
External Metrics API
Custom Metrics API

46. RESOURCE METRICS API
Metrics for Pods and Nodes
• CPU
• Memory

47. CUSTOM METRICS API
Arbitrary metrics for any Kubernetes resource
• Pod
• Service
• Ingress

48. Stackdriver (GCP)
https://cloud.google.com/monitoring/custom-metrics/
CUSTOM METRICS API ADAPTERS
Azure Kubernetes Metrics Adapter
https://github.com/Azure/azure-k8s-metrics-adapter
Prometheus Adapter
https://github.com/DirectXMan12/k8s-prometheus-adapter
Datadog Cluster Agent
https://github.com/DataDog/datadog-agent/blob/master/docs/cluster-agent/CUSTOM_METRICS_SERVER.md
Custom Metrics Adapter Server Boilerplate
https://github.com/kubernetes-incubator/custom-metrics-apiserver

49. EXTERNAL METRICS API
Arbitrary metrics from outside of Kubernetes
cluster
• Amazon SQS queue size (CloudWatch)
• Google Cloud Pub/Sub undelivered
messages (Stackdriver)

50. CanaryDeployment CRD

51. DEPLOYMENT
apiVersion: apps/v1
kind: Deployment
metadata:
name: foo
spec:
replicas: 5
selector: ... # Pod selector
template: ... # Pod template

52. CANARY DEPLOYMENT
apiVersion: kanarini.nilebox.github.com/v1alpha1
kind: CanaryDeployment
metadata:
name: foo
spec:
selector: ... # Pod selector
template: ... # Pod template
tracks:
canary: ... # "canary" track settings
stable: ... # "stable" track settings

53. CANARY DEPLOYMENT
apiVersion: kanarini.nilebox.github.com/v1alpha1
kind: CanaryDeployment
metadata:
name: foo
spec:
selector: ... # Pod selector
template: ... # Pod template
tracks:
canary: ... # "canary" track settings
stable: ... # "stable" track settings

54. CANARY DEPLOYMENT
apiVersion: kanarini.nilebox.github.com/v1alpha1
kind: CanaryDeployment
metadata:
name: foo
spec:
selector: ... # Pod selector
template: ... # Pod template
tracks:
canary: ... # "canary" track settings
stable: ... # "stable" track settings

55. CANARY DEPLOYMENT
tracks:
canary:
replicas: 1
labels:
track: canary
metricsCheckDelaySeconds: 120
metrics: ... # List of metrics to check against
stable:
replicas: 5
labels:
track: stable

56. CANARY DEPLOYMENT
metricsCheckDelaySeconds: 120
metrics:
- type: Object
object:
describedObject:
kind: Service
name: "foo-canary"
metric:
name: "request_failure_rate:1m"
target:
type: Value
value: 0.1

57. CANARY DEPLOYMENT
metricsCheckDelaySeconds: 120
metrics:
- type: Object
object:
describedObject:
kind: Service
name: "foo-canary"
metric:
name: "request_failure_rate:1m"
target:
type: Value
value: 0.1

58. CANARY DEPLOYMENT
metricsCheckDelaySeconds: 120
metrics:
- type: Object
object:
describedObject:
kind: Service
name: "foo-canary"
metric:
name: "request_failure_rate:1m"
target:
type: Value
value: 0.1

59. CANARY DEPLOYMENT
metricsCheckDelaySeconds: 120
metrics:
- type: Object
object:
describedObject:
kind: Service
name: "foo-canary"
metric:
name: "request_failure_rate:1m"
target:
type: Value
value: 0.1

60. Demo
Kanarini CRD Controller

61. EXAMPLE
Deployment
Deployment
“stable”
“canary”
version 1
version 2
Canary Deployment
Controller
Prometheus
k8s Custom Metrics API
Prometheus Adapter

62. EXAMPLE
Service
“stable”
“canary”
Service
Prometheus
Prometheus Adapter
Grafana

63. Demo script is available at
https://github.com/nilebox/kanarini

64. Prometheus Adapter for Custom Metrics API
https://github.com/DirectXMan12/k8s-prometheus-adapter
LINKS
Prometheus Operator Quickstart
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
Kanarini (CanaryDeployment CRD Controller)
https://github.com/nilebox/kanarini
Heptio Contour (Ingress Controller)
https://github.com/heptio/contour

65. Key takeaways for CRDs
Use abstractions
Generic APIs are reusable.
Use the power of open source
Read existing code and share your own code.
Reuse existing resources
No need to reinvent the wheel.
Keep it simple
Solve a minimal subset of a problem at once.

66. NAIL ISLAMOV | SENIOR DEVELOPER | @NILEBOX
Thank you!