1. Autoscaling your
Nomad jobs
a
HashiTalks 2021

2. ➔Used to be a Molecular Biologist,
Used to be a Molecular Biologist,
➔Then became a Dev,
Then became a Dev,
➔Now an Ops.
Now an Ops.
➔Currently
Currently CTO @ Hot Potatoes
CTO @ Hot Potatoes
➔Today Proud Dad
Today Proud Dad

3. Scaling applications, the hardware way

4. Moving it all to the cloud

5. Nomad

6. my overengineered personal website

7. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 1
network {
port "http" {
to = 80
}
}
task "nginx" {
driver = "docker"
config {
image = "private.dkr.ecr.us-east-1.amazonaws.com/blog:11"
ports = ["http"]
}
}
Deploy the blog

8. Nomad status

9. ●
Introduced in/with Nomad 0.11
●
(Currently) independently release cycle
●
Gaining new functionality every release
●
Build in Functionality for horizontal and vertical scaling
●
But extendable by your own (community) plugins
Nomad autoscaler

10. ●
Makes decisions based on a checks
●
Checks are a combination of
– Data queried from an APM
– Defined STRATEGY
– Attempt to approach TARGET value
●
Multiple Checks can be combined
– Answer with the most resources will win!
●
ScaleOut and ScaleIn => ScaleOut
●
ScaleOut and ScaleNone => ScaleOut
●
ScaleIn and ScaleNone => ScaleNone
●
ScaleOut(10) and ScaleOut(9) => ScaleOut(10)
●
ScaleIn(3) and ScaleIn(4) => ScaleIn(4)
Auto-scaling TLDR

11. job "autoscaler" {
type = "service"
datacenters = ["aws"]
group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.2.1"
command = "nomad-autoscaler"
args = [
"agent",
"-config",
"${NOMAD_TASK_DIR}/config.hcl",
"-http-bind-address",
"0.0.0.0",
]
Deploy the autoscaler

12. nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
strategy "target-value" {
driver = "target-value"
}
Config for the autoscaler

13. Metrics
https://prometheus.io/

14. job "blog" {
datacenters = ["aws"]
type = "service"
group "hugo" {
count = 3
scaling {
enabled = true
min = 1
max = 20
policy {
cooldown = "20s"
check "avg_instance_sessions" {
source = "prometheus"
query = "scalar(avg(traefik_service_open_connections{service="blog@consulcatalog"}))"
strategy "target-value" {
target = 5
}
Enable autoscaling for the blog

15. Dashboards
https://grafana.com/oss/grafana/

16. Enable autoscaling

17. Observe scaling down event

18. Observe the autoscaler
agent: querying APM: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus
strategy=target-value target=local-nomad
agent: calculating new count: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus
strategy=target-value target=local-nomad
agent: next count outside limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus
strategy=target-value target=local-nomad from=3 to=0 min=1 max=10
agent: updated count to be within limits: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d
source=prometheus strategy=target-value target=local-nomad from=3 to=1 min=1 max=10
agent: scaling target: policy_id=248f6157-ca37-f868-a0ab-cabbc67fec1d source=prometheus
strategy=target-value target=local-nomad target_config="map[group:demo job_id:webapp]" from=3 to=1
reason="capping count to min value of 1"

19. Apply load

20. Remove load

21. Logs
https://grafana.com/oss/loki/

22. group "autoscaler" {
count = 1
task "autoscaler" {
driver = "docker"
config {
image = "hashicorp/nomad-autoscaler:0.2.1"
command = "nomad-autoscaler"
logging {
type = "loki"
config {
loki-url = 'http://loki.service.consul:3100/api/prom/push'
tag = "loki"
}
}
Direct to loki
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

23. task "promtail" {
driver = "docker"
lifecycle {
hook = "prestart"
sidecar = true
}
config {
image = "grafana/promtail:1.5.0"
args = [
"-config.file",
"local/promtail.yaml",
]
Promtail sidecar

24. scrape_configs:
- job_name: system
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
task: autoscaler
__path__: /alloc/logs/autoscaler*
pipeline_stages:
- match:
selector: '{task="autoscaler"}'
stages:
- regex:
expression: '.*policy_id=(?P<policy_id>[a-zA-Z0-9_-]+).*source=(?P<source>[a-zA-Z0-9_-]+).*strategy=(?P<strategy>[a-zA-Z0-9_-]+).*target=(?P<target>[a-
zA-Z0-9_-]+).*Group:(?P<group>[a-zA-Z0-9]+).*Job:(?P<job>[a-zA-Z0-9_-]+).*Namespace:(?P<namespace>[a-zA-Z0-9_-]+)'
Promtail sidecar
https://grafana.com/docs/loki/latest/clients/promtail/

25. Annotate your graphs

26. Correlate events with metrics

27. https://learn.hashicorp.com/tutorials/nomad/
autoscaler-vagrant-demo?in=nomad/ecosystem
Try it yourself

28. Moving it all to the cloud *

29. nomad {
address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "prometheus" {
driver = "prometheus"
config = {
address = "http://prometheus.service.consul:9090"
}
}
target "aws-asg" {
driver = "aws-asg"
config = {
aws_region = "{{ $x := env "attr.platform.aws.placement.availability-zone" }}{{ $length := len $x |subtract 1 }}{{ slice $x 0 $length}}"
}
}
Grow into your platform

30. scaling "cluster_policy" {
policy {
cooldown = "2m"
evaluation_interval = "1m"
check "cpu_allocated_percentage" {
source = "prometheus"
query = "scalar(sum(nomad_client_allocated_cpu{node_class="hashistack"}*100/(nomad_client_unallocated_cpu{node_class="hashistack"}
+nomad_client_allocated_cpu{node_class="hashistack"}))/count(nomad_client_allocated_cpu{node_class="hashistack"}))"
strategy "target-value" {
target = 70
}
}
target "aws-asg" {
dry-run = "false"
aws_asg_name = "${client_asg_name}"
node_class = "hashistack"
node_drain_deadline = "5m"
}
Grow into your platform

31. https://github.com/hashicorp/nomad-autoscaler/
tree/master/demo/remote
Try it yourself

32. Moving it all to the cloud – QED

33. bram@attachmentgenie.com
@attachmentgenie
slideshare.net/attachmentgenie
Thank You