The document describes a presentation about Nomad, an open-source cluster scheduler and workload orchestrator made by HashiCorp. It outlines the steps to deploy and configure a Nomad cluster across multiple datacenters and regions, including initializing a Nomad cluster, creating and running a sample job, extending the cluster to a new datacenter in France, and updating the job configuration. It also demonstrates monitoring the status of the Consul and Nomad services and sample applications running on the cluster.
2. Paris Container Day 2017 #ContainerDayFR
Architecte virtualisation
Lead DevOps
Cloud Builder
…
Je suis...
Bastien Cadiot (@bcadiot)
www.wescale.fr | @YesWeScale
HashiCorp User : Terraform, Consul, Vagrant
Et … utilisateur de Nomad depuis la version 0.1.0
Nomad, l’orchestration made in Hashicorp
2
3. #ContainerDayFRParis Container Day 2017
Au commencement...
En réalité...
Un monde de conteneurs
Nomad, l’orchestration made in Hashicorp
3
4. Paris Container Day 2017 #ContainerDayFR
Nos clientsNos clients
La démo n’est pas la réalité
Nomad, l’orchestration made in Hashicorp
4
5. Paris Container Day 2017 #ContainerDayFR
L’environnement de dev n’est pas la production
Nomad, l’orchestration made in Hashicorp
5
6. Paris Container Day 2017 #ContainerDayFR
Comprendre l’architecture n’est pas optionnel
Nomad, l’orchestration made in Hashicorp
6
7. Paris Container Day 2017 #ContainerDayFR
Une installation compliquée est rarement un bon
signe pour l’avenir
Nomad, l’orchestration made in Hashicorp
7
8. #ContainerDayFRParis Container Day 2017
● Rationaliser
● Simplifier
Pourquoi orchestrer ?
Rappel des principes
● Ordonnancer
● Prioriser
● Standardiser
● Densifier
● Piloter
● Sécuriser
Nomad, l’orchestration made in Hashicorp
8
9. Paris Container Day 2017 #ContainerDayFR
Trendy - Kubernetes Stable - Mesos Easy - Swarm
Quels sont les choix ?
Nomad, l’orchestration made in Hashicorp
9
10. Paris Container Day 2017 #ContainerDayFR
Nomad, l’orchestration made in Hashicorp
11. Paris Container Day 2017 #ContainerDayFR
Distribué Flexible Simple
● Multi-datacenter
● Multi-cloud
● Hautement disponible
● Any Runner
● Service et batch
● Isolation
● Service unique
● Configuration simple
● Librairies stables
Pourquoi Nomad ?
Nomad, l’orchestration made in Hashicorp
11
12. Paris Container Day 2017 #ContainerDayFR
Dessine-moi une infrastructure Nomad
Nomad, l’orchestration made in Hashicorp
12
13. #ContainerDayFRParis Container Day 2017
Etape 1 : Création d’un cluster Nomad
● Instanciation de 3 VM serveurs
● Installation de Consul et Nomad Server
● Instanciation de 3 VM clients
● Installation de Consul Client, Nomad
Client, et Docker
Etape 2 : Création d’une appli Nomad
● Création du fichier job
● Enregistrement du job sur le cluster
Nomad
● Vérification du lancement de
l’application
Nomad, l’orchestration made in Hashicorp
13
Init & Run
Besoin : Déployer une application sur une infra
de conteneurs
14. Paris Container Day 2017 #ContainerDayFR
{
"bootstrap_expect": 3,
"server": true,
"datacenter": "gce-west1",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"bind_addr": "10.1.2.3",
"client_addr": "0.0.0.0",
"advertise_addr": "8.7.6.5"
}
Etape 1 - Initialisation
Fichiers de configurations (Serveurs)
region = "europe"
datacenter = "gce-west1"
data_dir = "/var/nomad"
bind_addr = "0.0.0.0"
advertise {
rpc = "8.7.6.5"
http = "8.7.6.5"
serf = "8.7.6.5"
}
server {
enabled = true
bootstrap_expect = 3
}
consul {
address = "127.0.0.1:8500"
}
14
Nomad, l’orchestration made in Hashicorp
15. Paris Container Day 2017 #ContainerDayFR
Etape 1 - Démo
Consul
# consul members
Node Address Status Type Build Protocol DC
client-europe-1 192.158.29.140:8301 alive client 0.8.3 2 gce-west1
client-europe-2 130.211.85.225:8301 alive client 0.8.3 2 gce-west1
client-europe-3 35.187.10.89:8301 alive client 0.8.3 2 gce-west1
server-europe-1 104.199.99.190:8301 alive server 0.8.3 2 gce-west1
server-europe-2 104.199.75.126:8301 alive server 0.8.3 2 gce-west1
server-europe-3 146.148.13.182:8301 alive server 0.8.3 2 gce-west1
15
Nomad, l’orchestration made in Hashicorp
16. Paris Container Day 2017 #ContainerDayFR
Etape 1 - Démo
Nomad
# nomad server-members
Name Address Port Status Leader Protocol Build Datacenter Region
server-europe-1.europe 104.199.99.190 4648 alive false 2 0.5.6 gce-west1 europe
server-europe-2.europe 104.199.75.126 4648 alive true 2 0.5.6 gce-west1 europe
server-europe-3.europe 146.148.13.182 4648 alive false 2 0.5.6 gce-west1 europe
# nomad node-status
ID DC Name Class Drain Status
47e449e7 gce-west1 client-europe-3 <none> false ready
e53bb5ae gce-west1 client-europe-1 <none> false ready
f3e16921 gce-west1 client-europe-2 <none> false ready
16
Nomad, l’orchestration made in Hashicorp
17. Paris Container Day 2017 #ContainerDayFR
Etape 2 - Running
Fichier de configuration (Job)
job "pcd2017" {
region = "europe"
datacenters = ["gce-west1"]
type = "service"
group "webs" { … }
group "database" { … }
}
group "webs" {
count = 2
task "frontend" {
driver = "docker"
config {
image = "bcadiot/app-pcd2017:1.0"
}
service {
port = "http"
}
resources {
cpu = 200
memory = 64
network {
mbits = 100
port "http" {
static = 80
}
}
}
}
}
17
Nomad, l’orchestration made in Hashicorp
18. Paris Container Day 2017 #ContainerDayFR
Etape 2 - Démo
Job run
# nomad run -address=http://104.199.99.190:4646 app.nomad
==> Monitoring evaluation "db7e8098"
Evaluation triggered by job "pcd2017"
Allocation "16dc6ad2" created: node "e53bb5ae", group "webs"
Allocation "b606c351" created: node "f3e16921", group "database"
Allocation "f8c7662f" created: node "f3e16921", group "webs"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "db7e8098" finished with status "complete"
18
Nomad, l’orchestration made in Hashicorp
19. Paris Container Day 2017 #ContainerDayFR
Etape 2 - Démo
Job status
nomad status
-address=http://104.199.99.190:4646
pcd2017
ID = pcd2017
Name = pcd2017
Type = service
Priority = 50
Datacenters = gce-west1
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
database 0 0 1 0 0 0
webs 0 0 2 0 0 0
Allocations
ID Eval ID Node ID Task Group Desired Status Created At
16dc6ad2 db7e8098 e53bb5ae webs run running 06/12/17
23:25:05 CEST
b606c351 db7e8098 f3e16921 database run running 06/12/17
23:25:05 CEST
f8c7662f db7e8098 f3e16921 webs run running 06/12/17
23:25:05 CEST
19
Nomad, l’orchestration made in Hashicorp
20. #ContainerDayFRParis Container Day 2017
Etape 3 : Création d’un datacenter France
● Création d’un datacenter chez un cloud
provider Français (OVH)
● Instanciation de 3 VM serveurs
● Installation de Consul Server
● Jonction avec les serveurs Europe
● Instanciation de 1 VM client
● Installation de Consul Client, Nomad
Besoin : Mise à jour pour un besoin client
Etape 4 : Mise à jour d’une appli Nomad
● Modification du fichier job pour ajouter
les contraintes
● Mise à jour du job sur le cluster Nomad
● Vérification de l’application
Extend & Update
20
Nomad, l’orchestration made in Hashicorp
21. Paris Container Day 2017 #ContainerDayFR
Etape 3 - Extending
Fichier de configuration (Nomad Client)
region = "europe"
datacenter = "france"
data_dir = "/var/nomad"
bind_addr = "0.0.0.0"
advertise {
rpc = "5.6.7.8"
http = "5.6.7.8"
serf = "5.6.7.8"
}
client {
enabled = true
node_class = "data"
}
21
Nomad, l’orchestration made in Hashicorp
22. Paris Container Day 2017 #ContainerDayFR
Etape 3 - Démo
Consul
# consul members -wan
Node Address Status Type Build Protocol DC
server-europe-1.gce-west1 104.199.99.190:8302 alive server 0.8.3 2 gce-west1
server-europe-2.gce-west1 104.199.75.126:8302 alive server 0.8.3 2 gce-west1
server-europe-3.gce-west1 146.148.13.182:8302 alive server 0.8.3 2 gce-west1
server-ovh-france-1.localdomain.france 217.182.82.249:8302 alive server 0.8.3 2 france
server-ovh-france-2.localdomain.france 217.182.82.247:8302 alive server 0.8.3 2 france
server-ovh-france-3.localdomain.france 217.182.82.243:8302 alive server 0.8.3 2 france
22
Nomad, l’orchestration made in Hashicorp
23. Paris Container Day 2017 #ContainerDayFR
Etape 3 - Démo
Nomad
# nomad node-status
ID DC Name Class Drain Status
e9a30fd3 france client-ovh-france-1.localdomain data false ready
47e449e7 gce-west1 client-europe-3 <none> false ready
e53bb5ae gce-west1 client-europe-1 <none> false ready
f3e16921 gce-west1 client-europe-2 <none> false ready
23
Nomad, l’orchestration made in Hashicorp
24. Paris Container Day 2017 #ContainerDayFR
Etape 4 - Updating
Fichier de configuration (Job)
job "pcd2017" {
...
datacenters = ["gce-west1", "france"]
...
group "webs" { … }
group "database" { … }
}
group "database" {
count = 1
…
constraint {
attribute = "${node.class}"
value = "data"
}
constraint {
attribute = "${node.datacenter}"
value = "france"
}
...
}
24
Nomad, l’orchestration made in Hashicorp
25. Paris Container Day 2017 #ContainerDayFR
Etape 4 - Démo
Job run
# nomad run -address=http://104.199.99.190:4646 app.nomad
==> Monitoring evaluation "22cc22e9"
Evaluation triggered by job "pcd2017"
Allocation "f8c7662f" modified: node "f3e16921", group "webs"
Allocation "689de99d" created: node "e9a30fd3", group "database"
Allocation "16dc6ad2" modified: node "e53bb5ae", group "webs"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "22cc22e9" finished with status "complete"
25
Nomad, l’orchestration made in Hashicorp
26. Paris Container Day 2017 #ContainerDayFR
Etape 4 - Démo
Job status
nomad status
-address=http://104.199.99.190:4646
pcd2017
ID = pcd2017
Name = pcd2017
Type = service
Priority = 50
Datacenters = gce-west1,france
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost
database 0 0 1 0 1 0
webs 0 0 2 0 0 0
Allocations
ID Eval ID Node ID Task Group Desired Status Created At
689de99d 22cc22e9 e9a30fd3 database run running 06/12/17
23:32:32 CEST
16dc6ad2 22cc22e9 e53bb5ae webs run running 06/12/17
23:25:05 CEST
b606c351 db7e8098 f3e16921 database stop complete 06/12/17
23:25:05 CEST
f8c7662f 22cc22e9 f3e16921 webs run running 06/12/17
23:25:05 CEST
26
Nomad, l’orchestration made in Hashicorp
27. #ContainerDayFRParis Container Day 2017
Etape 5 : Création d’une région Nomad
située aux USA et liaison des deux
régions
● Création d’un VPC chez un cloud provider
Américain (AWS)
● Instanciation de 3 VM serveurs
● Installation de Consul Server, et Nomad
Server
● Jonction de la nouvelle région avec la
région Europe
● Instanciation de 3 VM clients
● Installation de Consul Client, Nomad
Client, et Docker
Besoin : Accompagner la montée en puissance
Scaling
27
Nomad, l’orchestration made in Hashicorp
28. Paris Container Day 2017 #ContainerDayFR
{
"bootstrap_expect": 3,
"server": true,
"datacenter": "aws-west2",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"bind_addr": "35.99.88.77",
"client_addr": "0.0.0.0",
"advertise_addr": "35.99.88.77"
}
Etape 5 - Scaling
Fichiers de configurations (Serveurs)
region = "us"
datacenter = "aws-west2"
data_dir = "/var/nomad"
bind_addr = "0.0.0.0"
advertise {
rpc = "35.99.88.77"
http = "35.99.88.77"
serf = "35.99.88.77"
}
server {
enabled = true
bootstrap_expect = 3
}
consul {
address = "127.0.0.1:8500"
}
28
Nomad, l’orchestration made in Hashicorp
29. Paris Container Day 2017 #ContainerDayFR
Etape 5 - Démo
Consul
# consul members -wan
Node Address Status Type Build Protocol DC
ip-10-59-1-10.aws-west2 54.218.254.37:8302 alive server 0.8.3 2 aws-west2
ip-10-59-1-201.aws-west2 54.191.239.249:8302 alive server 0.8.3 2 aws-west2
ip-10-59-1-217.aws-west2 54.202.180.132:8302 alive server 0.8.3 2 aws-west2
server-europe-1.gce-west1 104.199.99.190:8302 alive server 0.8.3 2 gce-west1
server-europe-2.gce-west1 104.199.75.126:8302 alive server 0.8.3 2 gce-west1
server-europe-3.gce-west1 146.148.13.182:8302 alive server 0.8.3 2 gce-west1
server-ovh-france-1.localdomain.france 217.182.82.249:8302 alive server 0.8.3 2 france
server-ovh-france-2.localdomain.france 217.182.82.247:8302 alive server 0.8.3 2 france
server-ovh-france-3.localdomain.france 217.182.82.243:8302 alive server 0.8.3 2 france
29
Nomad, l’orchestration made in Hashicorp
30. Paris Container Day 2017 #ContainerDayFR
Etape 5 - Démo
Nomad
# nomad server-join 104.199.75.126
Joined 1 servers successfully
# nomad server-members
Name Address Port Status Leader Protocol Build Datacenter Region
server-europe-1.europe 104.199.99.190 4648 alive false 2 0.5.6 gce-west1 europe
server-europe-2.europe 104.199.75.126 4648 alive true 2 0.5.6 gce-west1 europe
server-europe-3.europe 146.148.13.182 4648 alive false 2 0.5.6 gce-west1 europe
ip-10-59-1-10.us 54.218.254.37 4648 alive false 2 0.5.6 aws-west2 us
ip-10-59-1-201.us 54.191.239.249 4648 alive true 2 0.5.6 aws-west2 us
ip-10-59-1-217.us 54.202.180.132 4648 alive false 2 0.5.6 aws-west2 us
30
Nomad, l’orchestration made in Hashicorp
31. Paris Container Day 2017 #ContainerDayFR
Pour finir...
Sécurité Stockage
Réseau
Nomad, l’orchestration made in Hashicorp
31
32. #ContainerDayFRParis Container Day 2017
Questions ?
Sources :
https://github.com
/bcadiot/ParisContainerDay_2017
Twitter : @bcadiot
Merci !
32
Nomad, l’orchestration made in Hashicorp