Introduction to Nomad and its multi-cloud capability

1. www.hashicorp.comO @hashicorphello@hashicorp.com

2. nic@hashicorp.com
@sheriffjackson
NIC JACKSON

3. 3
AGENDA
HASHICORP
SCHEDULING
NOMAD
Overview
Fundamentals
Job Configuration
Scheduling
Demo

4. HASHICORP
OVERVIEW

5. 5
FOUNDED 2012 by Mitchell Hashimoto and Armon Dadgar
MISSION We enable organizations to provision, secure, and run any infrastructure for any application
INVESTORS Mayfield Fund, GGV Capital, Redpoint and True Ventures
KEY PRODUCTS Vagrant, Packer, Terraform, Vault, Nomad, Consul
COMPANY OVERVIEW

6. 6
OSS TO ENTERPRISE
SOFTWARE INNOVATORS TECHNOLOGY PARTNERS

7. 7
PRODUCT SUITE

8. 8
NOMAD
Nomad

9. SCHEDULING
OVERVIEW

10. Schedulers map a set of work to a
set of resources

11. 11
CPU SCHEDULER
11
CORE
CORE
CORE
CORE
CPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

12. 12
CPU SCHEDULER
12
CORE
CORECPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

13. 13
SCHEDULERS IN THE WILD
13
Type Work Resources
CPU Scheduler Threads Physical Cores
EC2 / Nova Virtual Machines Hypervisors
Hadoop YARN MapReduce Jobs Client Nodes
Cluster Scheduler Applications Machines

14. 14
SCHEDULER ADVANTAGES
14
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

15. 15
SCHEDULER ADVANTAGES
15
Bin Packing
Over-Subscription
Job Queueing
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

16. 16
SCHEDULER ADVANTAGES
16
Abstraction
API Contracts
Standardization
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

17. 17
SCHEDULER ADVANTAGES
17
Priorities
Resource Isolation
Pre-emption
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

18. 18
NOT A NEW CONCEPT
18

19. 19
BASED ON RESEARCH
19

20. NOMAD
OVERVIEW

21. 21
NOMAD DESIGN PRINCIPLES
21HashiCorp confidential do not distribute
Integrated scheduler and cluster manager
Distributed, shared state, optimistically concurrent
Agent-based, client/server
No dependencies

22. 22
NOMAD CHARACTERISTICS
22HashiCorp confidential do not distribute
Multi-datacenter and multi-region
Highly performant and highly available
Hybrid workloads with multiple schedulers and drivers
Seamlessly integrates with HashiCorp ecosystem

23. NOMAD
FUNDAMENTALS

24. 24
SINGLE REGION DEPLOYMENT
24
SERVER SERVER SERVER
CLIENT CLIENT CLIENTDC1 DC2 DC3
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
FORWARDING
RPC RPC RPC

25. 25
MULTI REGION DEPLOYMENT
25
SERVER SERVER SERVER
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
REGION B GOSSIP
REPLICATION REPLICATION
FORWARDING
REGION FORWARDING
 REGION A
SERVER
FOLLOWER
SERVER SERVER
LEADER FOLLOWER

26. 26
SERVER ARCHITECTURE
26
Omega Class Scheduler
Pluggable Logic
Internal Coordination and State
Multi-Region / Multi-Datacenter

27. 27
CLIENT ARCHITECTURE
27
Broad OS Support
Host Fingerprinting
Pluggable Drivers
Job restarts and lifecycle management

28. 28
CLIENT DRIVERS
28
Containerized
Docker
rkt
Windows Server Containers
Virtualized
Qemu / KVM
Hyper-V
Xen
Standalone
Java Jar
C#
Static Binaries

29. 29
CLIENT FINGERPRINTING
29
Type Examples
Operating System Kernel, OS, Version
Hardware CPU, Memory, Disk
Apps (Capabilities) Docker, Java, Consul
Environment AWS, GCE

30. NOMAD
JOB CONFIGURATION

31. 31
JOB FILE
31
Declarative
Scheduler, driver, and resource needs
Lifecycle behavior
Constraints
Versioned

32. 32
redis.nomad
JOB FILE
job "redis" {
datacenters = ["us-east-1"]
task "redis" {
driver = "docker"
config { image = "redis:v13" }
resources {
cpu = 500 # Mhz
memory = 256 # MB
network {
mbits = 10
dynamic_ports = ["redis"]
}
}
}
}

33. 33
redis.nomad
JOB FILE: TASK GROUPS
job "app" {
group "app" {
task "redis" {
# ...
}
task "app" {
# ...
}
}
}

34. 34
redis.nomad
JOB FILE: CONSTRAINTS
job "redis" {
constraint {
attribute = "${attr.kernel.version}"
operator = "version"
value = "> 3.19"
}
constraint {
attribute = "${attr.platform.aws.instance-type}"
value = "p2.16xlarge"
}
task "redis" {
# ...
}
}

35. 35
redis.nomad
JOB FILE: CONSUL SERVICE DISCOVERY
job "redis" {
task "redis" {
# ...
service {
port = “redis”
check {
type = “tcp”
interval = “10s”
}
}
}
}

36. 36
redis.nomad
JOB FILE: CONSUL CONFIGURATION
job "redis" {
task "redis" {
# ...
template {
data = <<EOH
bind_port: {{ env "NOMAD_PORT_db" }}
scratch_dir: {{ env "NOMAD_TASK_DIR" }}
service_key: {{ key "service/my-key" }}
EOH
destination = "local/file.yml"
}
}
}

37. 37
redis.nomad
JOB FILE: VAULT INTEGRATION
job "redis" {
task "redis" {
# ...
template {
data = <<EOH
{{ with secret "secret/credentials" }}
username: {{ .Data.username }}
password: {{ .Data.password }}
{{ end }}
EOH
destination = "local/file.yml"
}
}
}

38. 38
redis.nomad
JOB FILE: PARAMETERIZED
job "encode" {
type = "batch"
parameterized {
payload = "required"
meta_required = ["s3-input", "s3-output", ...]
}
# ...
task "ffmpeg" {
driver = "exec"
config {
command = "ffmpeg"
# When dispatched, the payload is written to a file that is then
# read by the created task upon startup
args = ["-config=${NOMAD_TASK_DIR}/config.json"]
# ...
}

39. 39
$ nomad job dispatch encode video-config.json
$
$ cat video-config.json
{
"s3-input": "https://s3-us-west-1.com/video-bucket/cb31dabb1",
"s3-output": "https://s3-us-west-1.com/video-bucket/a149adbe3",
"input-codec": "mp4",
"output-codec": "webm",
"quality": "1080p"
}
Text
JOB FILE: PARAMETERIZED

40. NOMAD
MULTI-CLOUD

41. Why Multi-Cloud?
• High Availability
• Redundancy
• Burstable Workload
• Cloud Migration
• Because we can

42. 42
CONSUL
NOMAD
SERVER
LEADER
SERVER
FOLLOWER
SERVER
LEADER
SERVER
FOLLOWER
SERVER
FOLLOWER
SERVER
FOLLOWER
NODE A NODE B
GOOGLE CLOUD
NATS CLOUD
MESSAGING
REPLICATION
FORWARDING
REPLICATION
FORWARDING
REPLICATION
FORWARDING
REPLICATION
FORWARDING
LOAD BALANCER
LOAD BALANCER
CONSUL
NOMAD
SERVER
LEADER
SERVER
FOLLOWER
SERVER
LEADER
SERVER
FOLLOWER
SERVER
FOLLOWER
SERVER
FOLLOWER
NODE A NODE B
AWS
REPLICATION
FORWARDING
REPLICATION
FORWARDING
REPLICATION
FORWARDING
REPLICATION
FORWARDING
LOAD BALANCER
REGION FORWARDING (VPN)
REGION FORWARDING (VPN)

43. NOMAD
SCHEDULING

44. 44
SCHEDULING
44
Schedulers process evaluations and generate allocation plans.
Placement is determined using the relevant scheduler.
Scheduling involves feasibility checking and ranking.
Feasibility filters out nodes missing necessary drivers and those failing
the specified constraints.
Ranking score feasible nodes to find the best fit (bin packing).

45. 45
SCHEDULER TYPES
45HashiCorp confidential do not distribute
Service
Long-running applications and services
Batch
Short-lived data processing jobs (benefit from fast placement)
System
Lower level jobs that run on all clients (logging, monitoring)

46. 46
$ nomad plan example.nomad
+ Job: "example"
+ Task Group: "cache" (1 create)
+ Task: "redis" (forces create)
Scheduler dry-run:
- All tasks successfully allocated.
$
Text
SCHEDULING: PLAN

47. 47
$ nomad plan example.nomad.java
+ Job: "example"
+ Task Group: "web" (1 create)
+ Task: "tomcat" (forces create)
Scheduler dry-run:
- WARNING: Failed to place all allocations.
Task Group "web" (failed to place 1 allocation):
* Constraint "missing drivers" filtered 2 nodes
$
Text
SCHEDULING: PLAN

48. 48
$ nomad run example.nomad
==> Monitoring evaluation "4b8b7779"
Evaluation triggered by job "example"
Allocation "38720b8e" created: node "ec2f0830", group "cache"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "4b8b7779" finished with status "complete"
$
Text
SCHEDULING: RUN

49. 49
$ nomad run -region=gcp events.nomad
==> Monitoring evaluation "e2a8dfe6"
!On branch master
Evaluation triggered by job "events"
!Your branch is up-to-date with 'origin/master'.
Allocation "6615b39f" modified: node "0d6a6103", group "pubsub"
!nothing to commit, working tree clean
Evaluation status changed: "pending" -> "complete"
!
==> Evaluation "e2a8dfe6" finished with status "complete"
$
Text
SCHEDULING: RUN DIFFERENT REGION

50. 50
$ nomad status example
ID = example
Name = example
Type = service
Priority = 50
Datacenters = us-west-1
Status = running
Summary
Task Group Queued Starting Running Failed Complete Lost
cache 0 0 1 0 0 0
Allocations
ID Eval ID Node ID Task Group Desired Status Created At
38720b8e 4b8b7779 ec2f0830 cache run running 04/26/17 ...
$
Text
SCHEDULING: STATUS

51. DEMO!

52. Nomad
Million Container
Challenge
1,000 Jobs
1,000 Tasks per Job
5,000 Hosts on GCE
1,000,000 Containers

53. 53
MILLION CONTAINER CHALLENGE
53

54. 54
MILLION CONTAINER CHALLENGE
54
– Bill Gates
640 KB ought to be enough for anybody.
“

55. 55
REAL WORLD SCALE
55
2nd Largest Hedge Fund
18K Cores
5 Hours
2,200 Containers/second

56. Q/A
AND HASHICONF

57. SEPTEMBER18-20
AUSTIN, TEXAS
www.hashiconf.com
#hashiconf

58. #hashiconf
Links:
https://www.nomadproject.io
https://github.com/nicholasjackson/terraform-nomad-multi-cloud