As our need for more computing resources accelerates, so do the ways in which computing evolves. The arrival of the cloud has enabled us to easily scale to suit our needs. But if we want to keep pace, we need an even more automated way to scale our infrastructure. In this session, we review auto-scaling with Kubernetes, how to set it up, and, most importantly, what to monitor in order to drive auto-scaling in your organization. This presentation is brought to you by AWS partner Datadog.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automatically scaling
Kubernetes workloads
Sean O’Connor
Director of Engineering, Infra
Datadog
sean.oconnor@datadoghq.com
@theSeanOC
S V C 2 1 5 - S

2. “I think there is a
world market for
maybe 5 computers.”
—Thomas Watson, IBM

3. – Wildly misquoted!
“I think there is a
world market for
maybe 5 computers.”
—Thomas Watson, IBM

4. – Wildly misquoted!
– Sent prospectus to 20
clients, expected 5 orders
“I think there is a
world market for
maybe 5 computers.”
—Thomas Watson, IBM

5. – Wildly misquoted!
– Sent prospectus to 20
clients, expected 5 orders
– Received 18 orders!
“I think there is a
world market for
maybe 5 computers.”
—Thomas Watson, IBM

6. Who am I?
Sean O’Connor,
Director of Engineering
sean.oconnor@datadoghq.com
@theSeanOC

7. What is Datadog?
Observability platform
● Metrics
● Distributed traces (APM)
● Log analytics
● Synthetics
http://datadoghq.com

8. Capacity planning is
hard

9. More capacity?
Build a new office!

10. More capacity?
Add another rack!

11. Virtualization
FTW!

12. Config management
FTW!

13. The cloud!

14. The answer is always
containers*

15. *except when it’s not

16. Kubernetes
Horizontal Pod
Autoscaling

17. Basic in-cluster
metrics
1.2

18. Basic in-cluster
metrics
1.2 1.6
Custom in-cluster
metrics
Multiple metrics

19. Basic in-cluster
metrics
1.2 1.6 1.10
Custom in-cluster
metrics
Multiple metrics
Any metric, including
external sources

20. Choosing the
right metrics

21. – Throughput
– Success
– Error
– Performance
Work metrics

22. – Utilization
– Saturation
– Error
– Availability
Resource metrics

23. Services Pods
Almost like...

24. – Code/config changes
– Alerts
– Scaling events
Events

26. 4 golden signals (LETS)

27. Latency
4 golden signals (LETS)

28. Latency Errors
4 golden signals (LETS)

29. Latency Errors
4 golden signals (LETS)
Traffic

30. Latency Errors
4 golden signals (LETS)
Traffic Saturation

31. How does the
Horizontal Pod
Autoscaler
(HPA) work?

32. External metrics
server
Getting external metrics
In Kubernetes

33. External metrics
server
Your metrics source
Getting external metrics
In Kubernetes e.g., Datadog

34. HPA External metrics
server
Your metrics source
Getting external metrics
In Kubernetes e.g., DatadogIn Kubernetes

35. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period

36. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period
●Polls and calculates current metric value

37. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period
●Polls and calculates current metric value
●Calculates desired pod value

38. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period
●Polls and calculates current metric value
●Calculates desired pod value
desired_pods = ceiling(
(current_metric/desired_metric) * current_pods
)

39. Calculating pods
Target metric value = 500 req/sec/pod
Current metric value = 1,000 req/sec/pod
Current number of pods = 2

40. Calculating pods
Target metric value = 500 req/sec/pod
Current metric value = 1,000 req/sec/pod
Current number of pods = 2
ceiling((1,000/500) * 2) = 4 pods

41. Calculating pods
Target metric value = 500 req/sec/pod
Current metric value = 250 req/sec/pod
Current number of pods = 4

42. Calculating pods
Target metric value = 500 req/sec/pod
Current metric value = 250 req/sec/pod
Current number of pods = 4
ceiling((250/500) * 4) = 2 pods

43. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period
●Polls and calculates current metric value
●Calculates desired pod value

44. Control loop
●Runs every
horizontal_pod_autoscaler_sync_period
●Polls and calculates current metric value
●Calculates desired pod value
●Executes scaling

45. In practice

46. In practice
External metrics
server
In Kubernetes

47. Metrics server service (YAML)
kind: Service
apiVersion: v1
metadata:
name: datadog-custom-metrics-server
spec:
selector:
app: datadog-cluster-agent
ports:
- protocol: TCP
port: 443
targetPort: 443

48. Datadog Cluster Agent Deployment (more YAML!)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: datadog-cluster-agent
spec:
template:
metadata:
labels:
app: datadog-cluster-agent
spec:
containers:
- image: datadog/cluster-agent:latest
name: datadog-cluster-agent
env:
- name: DD_EXTERNAL_METRICS_PROVIDER_ENABLED
value: 'true'

49. Expose it to Kubernetes (even more YAML!)
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.external.metrics.k8s.io
spec:
group: external.metrics.k8s.io
groupPriorityMinimum: 100
versionPriority: 100
service:
name: datadog-custom-metrics-server
version: v1beta1

50. In practice
External metrics
server
In Kubernetes
Your metrics source
e.g., Datadog

51. Metrics from Datadog

52. In practice
External metrics
server
In Kubernetes
Your metrics source
e.g., Datadog
HPA
In Kubernetes

53. Our example application (we love YAML!)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
template:
metadata:
name: my-app
spec:
containers:
- name: my-app
image: my-cool-app:latest

54. HPA (you love YAML too!)
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: my-hpa
spec:
minReplicas: 1
maxReplicas: 3
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
metrics:
- type: External
external:
metricName: nginx.net.request_per_s
targetAverageValue: 500

55. kubectl describe hpa my-hpa
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able
to get the target's current
scale
ScalingActive False FailedGetExternalMetric the HPA was unable to compute
the replica count: unable to
get external metric
nginx.net.request_per_s: no
metrics returned from external
metrics API

56. kubectl describe hpa my-hpa
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able
to get the target's current
scale
ScalingActive True ValidMetricFound the HPA was able to
successfully calculate a
replica count from external
metric nginx.net.request_per_s
ScalingLimited False DesiredWithinRange the desired count is within the
acceptable range

57. kubectl describe hpa my-hpa
Reference: Deployment/my-app
Metrics: ( current / target )
"nginx.net.request_per_s" (target average value): 500 / 500
Min replicas: 1
Max replicas: 3
Deployment pods: 2 current / 2 desired

58. Recap

59. Kubernetes is the
natural evolution
HPA is a great tool
Recap

60. Kubernetes is the
natural evolution
HPA is a great tool
Use the right metrics
Think of work vs.
resource metrics
Consider the 4 golden signals
Recap

61. Additional resources
●Autoscaling K8s blog post:
https://dtdg.co/autoscaling-k8s
●Datadog Cluster Agent:
https://dtdg.co/cluster-agent
●Kubernetes HPA docs:
https://dtdg.co/k8s-hpa-docs

62. Thank you
sean.oconnor@datadoghq.com
@theSeanOC