Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven't nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably. We all love infrastructure as code, we automate everything ™ but how many of us can really say we could destroy and recreate our core infrastructure without human intervention. A Container Scheduler adds application life cycle management, scheduling and placement based on available resources and connectivity features to your cloud system. It takes away the responsibility from you to take care for these tasks. During this talk we will investigate how we can setup and monitor a cloud native container platform that scales using hashicorp's consul and nomad service discovery and container scheduling tools will define some best practices that will allow your platforms to heal itself.

1. Nomad Best Practices for
Reliable Deploys
20/02/2020 HashiTalks 2020
Bram Vogelaar
@attachmentgenie

2. Today’s Schedule

3. Today’s Schedule

4. ~$ whoami
●
I used to be a Molecular Biologist,
●
Then became a Dev,
●
Now an Ops.
●
Open Source Consultant @inuits.eu

5. Nomad

6. job "blog" {
datacenters = ["waldorf",”statler”]
type = "service"
group "hugo" {
task "nginx" {
driver = "docker"
config {
image = "private.dkr.ecr.us-east-1.amazonaws.com/blog:1.3"
port_map {
http = 80
}
}
Lets deploy our blog

7. $problems++?
●
Broken Hardware
●
Broken / unreliable Service or OS
●
Broken Code

8. group "hugo" {
Count = 2
task "nginx" {
1 === none

9. group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}
task "nginx" {
Force onto different hardware

10. group "hugo" {
count = 2
constraint {
operator = "distinct_hosts"
value = "true"
}
Spread {
attribute = "${node.datacenter}"
}
task "nginx" {
(Suggest) Different Location

11. Client {
Enabled = true
Meta {
"rack" = "his"
}
}
Based on custom meta-data

12. Spread {
attribute = "${meta.rack}"
target "his" {
percent = 50
}
target "her" {
percent = 50
}
}
Based on custom meta-data

13. Consul

14. service {
name = "blog"
port = "http"
check {
type = "tcp"
interval = "10s"
timeout = "2s"
}
}
Service Definition

15. Check {
type = "tcp"
interval = "10s"
timeout = "2s"
check_restart {
limit = 3
grace = "10s"
ignore_warnings = false
}
}
Stampeding herd

16. group "hugo" {
Restart {
interval = "10m"
attempts = 2
delay = "15s"
mode = "fail"
}
task "nginx" {
Restart failed jobs

17. group "hugo" {
Count = 2
reschedule {
delay = "30s"
delay_function = "constant" #constant, exponential, fibonacci
unlimited = true # or max_delay = “1h”
}
task "nginx" {
Reschedule a job

18. Testing your assumptions
●
https://github.com/ThomasObenaus/dummy-services
●
https://github.com/Shopify/toxiproxy

19. group "hugo" {
Count = 10
update {
max_parallel = 2
min_healthy_time = "30s"
healthy_deadline = "5m"
}
task "nginx" {
Updates

20. group "hugo" {
Count = 10
Update {
max_parallel = 1
canary = 10
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = false
}
task "nginx" {
Blue/Green Release

21. group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = true
}
task "nginx" {
Canary Release

22. service {
name = "blog"
tags = ["v2"]
}
$version++

23. group "hugo" {
Count = 5
Update {
max_parallel = 1
canary = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
auto_revert = true
auto_promote = false
}
task "nginx" {
Canary Release++

24. kind = "service-router"
name = "blog"
routes = [
{
match {
http {
header = [
{
name = "group"
exact = "test"
}, ] } }
destination {
service = "blog"
service_subset = "v2"
} },]
Consul to the rescue

25. Autoscaling Service Instances
●
Sherpa
●
Scale jobs based on Nomad resource
consumption and external metric
●
Highly available and fault tolerant
https://github.com/jrasell/sherpa

26. Autoscaling “Hardware”
●
Chemtrail
●
Scale node worker pool based on allocated
demand
●
Extensible, Aws build in
https://github.com/jrasell/chemtrail

27. INUITS bvba
Essensteenweg 31
2930 Brasschaat
Belgium
BE 0891.514.231
Contact:
+32.380.821.05
info@inuits.eu
inuits.eu
Bram Vogelaar
bram@inuits.eu
@attachmentgenie