Downloaded 17 times
![32
redis.nomad
JOB FILE
job "redis" {
datacenters = ["us-east-1"]
task "redis" {
driver = "docker"
config { image = "redis:v13" }
resources {
cpu = 500 # Mhz
memory = 256 # MB
network {
mbits = 10
dynamic_ports = ["redis"]
}
}
}
}](https://image.slidesharecdn.com/slidesnicjackson-170704132024/85/Living-the-Nomadic-life-Nic-Jackson-32-320.jpg)
![38
redis.nomad
JOB FILE: PARAMETERIZED
job "encode" {
type = "batch"
parameterized {
payload = "required"
meta_required = ["s3-input", "s3-output", ...]
}
# ...
task "ffmpeg" {
driver = "exec"
config {
command = "ffmpeg"
# When dispatched, the payload is written to a file that is then
# read by the created task upon startup
args = ["-config=${NOMAD_TASK_DIR}/config.json"]
# ...
}](https://image.slidesharecdn.com/slidesnicjackson-170704132024/85/Living-the-Nomadic-life-Nic-Jackson-38-320.jpg)
Uploaded byParis Container Day
Living the Nomadic life - Nic Jackson
This document provides an overview and agenda for a presentation on Nomad, an open source cluster scheduler created by HashiCorp. The presentation will cover Nomad fundamentals including architecture, job configuration, and scheduling. It will also demonstrate Nomad's ability to schedule a million containers across thousands of hosts on Google Cloud Platform.
More Related Content
![Docker for Developers - php[tek] 2017](https://cdn.slidesharecdn.com/ss_thumbnails/dockerfordevelopers-trainingtek2017-170523133513-thumbnail.jpg?width=640&height=640&fit=bounds)
![Docker, cornerstone of cloud hybridation ? [Cloud Expo Europe 2016]](https://cdn.slidesharecdn.com/ss_thumbnails/dockercloudhybridationv2-161129152507-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Living the Nomadic life - Nic Jackson
Living the Nomadic life - Nic Jackson
- 1.
- 2.
- 3.
- 4.
- 5. 5 FOUNDED 2012 byMitchell Hashimoto and Armon Dadgar MISSION We enable organizations to provision, secure, and run any infrastructure for any application INVESTORS Mayfield Fund, GGV Capital, Redpoint and True Ventures KEY PRODUCTS Vagrant, Packer, Terraform, Vault, Nomad, Consul COMPANY OVERVIEW
- 6. 6 OSS TO ENTERPRISE SOFTWAREINNOVATORS TECHNOLOGY PARTNERS
- 7.
- 8.
- 9.
- 10. Schedulers map aset of work to a set of resources
- 11.
- 12.
- 13. 13 SCHEDULERS IN THEWILD 13 Type Work Resources CPU Scheduler Threads Physical Cores EC2 / Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Machines
- 14. 14 SCHEDULER ADVANTAGES 14 Higher ResourceUtilization Decouple Work from Resources Better Quality of Service
- 15. 15 SCHEDULER ADVANTAGES 15 Bin Packing Over-Subscription JobQueueing Higher Resource Utilization Decouple Work from Resources Better Quality of Service
- 16. 16 SCHEDULER ADVANTAGES 16 Abstraction API Contracts Standardization HigherResource Utilization Decouple Work from Resources Better Quality of Service
- 17. 17 SCHEDULER ADVANTAGES 17 Priorities Resource Isolation Pre-emption HigherResource Utilization Decouple Work from Resources Better Quality of Service
- 18. 18 NOT A NEWCONCEPT 18
- 19.
- 20.
- 21. 21 NOMAD DESIGN PRINCIPLES 21HashiCorpconfidential do not distribute Integrated scheduler and cluster manager Distributed, shared state, optimistically concurrent Agent-based, client/server No dependencies
- 22. 22 NOMAD CHARACTERISTICS 22HashiCorp confidentialdo not distribute Multi-datacenter and multi-region Highly performant and highly available Hybrid workloads with multiple schedulers and drivers Seamlessly integrates with HashiCorp ecosystem
- 23.
- 24. 24 SINGLE REGION DEPLOYMENT 24 SERVERSERVER SERVER CLIENT CLIENT CLIENTDC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC
- 25. 25 MULTI REGION DEPLOYMENT 25 SERVERSERVER SERVER FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION REGION B GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING REGION A SERVER FOLLOWER SERVER SERVER LEADER FOLLOWER
- 26. 26 SERVER ARCHITECTURE 26 Omega ClassScheduler Pluggable Logic Internal Coordination and State Multi-Region / Multi-Datacenter
- 27. 27 CLIENT ARCHITECTURE 27 Broad OSSupport Host Fingerprinting Pluggable Drivers Job restarts and lifecycle management
- 28. 28 CLIENT DRIVERS 28 Containerized Docker rkt Windows ServerContainers Virtualized Qemu / KVM Hyper-V Xen Standalone Java Jar C# Static Binaries
- 29. 29 CLIENT FINGERPRINTING 29 Type Examples OperatingSystem Kernel, OS, Version Hardware CPU, Memory, Disk Apps (Capabilities) Docker, Java, Consul Environment AWS, GCE
- 30.
- 31. 31 JOB FILE 31 Declarative Scheduler, driver,and resource needs Lifecycle behavior Constraints Versioned
- 32. 32 redis.nomad JOB FILE job "redis"{ datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:v13" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }
- 33. 33 redis.nomad JOB FILE: TASKGROUPS job "app" { group "app" { task "redis" { # ... } task "app" { # ... } } }
- 34. 34 redis.nomad JOB FILE: CONSTRAINTS job"redis" { constraint { attribute = "${attr.kernel.version}" operator = "version" value = "> 3.19" } constraint { attribute = "${attr.platform.aws.instance-type}" value = "p2.16xlarge" } task "redis" { # ... } }
- 35. 35 redis.nomad JOB FILE: CONSULSERVICE DISCOVERY job "redis" { task "redis" { # ... service { port = “redis” check { type = “tcp” interval = “10s” } } } }
- 36. 36 redis.nomad JOB FILE: CONSULCONFIGURATION job "redis" { task "redis" { # ... template { data = <<EOH bind_port: {{ env "NOMAD_PORT_db" }} scratch_dir: {{ env "NOMAD_TASK_DIR" }} service_key: {{ key "service/my-key" }} EOH destination = "local/file.yml" } } }
- 37. 37 redis.nomad JOB FILE: VAULTINTEGRATION job "redis" { task "redis" { # ... template { data = <<EOH {{ with secret "secret/credentials" }} username: {{ .Data.username }} password: {{ .Data.password }} {{ end }} EOH destination = "local/file.yml" } } }
- 38. 38 redis.nomad JOB FILE: PARAMETERIZED job"encode" { type = "batch" parameterized { payload = "required" meta_required = ["s3-input", "s3-output", ...] } # ... task "ffmpeg" { driver = "exec" config { command = "ffmpeg" # When dispatched, the payload is written to a file that is then # read by the created task upon startup args = ["-config=${NOMAD_TASK_DIR}/config.json"] # ... }
- 39. 39 $ nomad jobdispatch encode video-config.json $ $ cat video-config.json { "s3-input": "https://s3-us-west-1.com/video-bucket/cb31dabb1", "s3-output": "https://s3-us-west-1.com/video-bucket/a149adbe3", "input-codec": "mp4", "output-codec": "webm", "quality": "1080p" } Text JOB FILE: PARAMETERIZED
- 40.
- 41. Why Multi-Cloud? • HighAvailability • Redundancy • Burstable Workload • Cloud Migration • Because we can
- 42. 42 CONSUL NOMAD SERVER LEADER SERVER FOLLOWER SERVER LEADER SERVER FOLLOWER SERVER FOLLOWER SERVER FOLLOWER NODE A NODEB GOOGLE CLOUD NATS CLOUD MESSAGING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING LOAD BALANCER LOAD BALANCER CONSUL NOMAD SERVER LEADER SERVER FOLLOWER SERVER LEADER SERVER FOLLOWER SERVER FOLLOWER SERVER FOLLOWER NODE A NODE B AWS REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING LOAD BALANCER REGION FORWARDING (VPN) REGION FORWARDING (VPN)
- 43.
- 44. 44 SCHEDULING 44 Schedulers process evaluationsand generate allocation plans. Placement is determined using the relevant scheduler. Scheduling involves feasibility checking and ranking. Feasibility filters out nodes missing necessary drivers and those failing the specified constraints. Ranking score feasible nodes to find the best fit (bin packing).
- 45. 45 SCHEDULER TYPES 45HashiCorp confidentialdo not distribute Service Long-running applications and services Batch Short-lived data processing jobs (benefit from fast placement) System Lower level jobs that run on all clients (logging, monitoring)
- 46. 46 $ nomad planexample.nomad + Job: "example" + Task Group: "cache" (1 create) + Task: "redis" (forces create) Scheduler dry-run: - All tasks successfully allocated. $ Text SCHEDULING: PLAN
- 47. 47 $ nomad planexample.nomad.java + Job: "example" + Task Group: "web" (1 create) + Task: "tomcat" (forces create) Scheduler dry-run: - WARNING: Failed to place all allocations. Task Group "web" (failed to place 1 allocation): * Constraint "missing drivers" filtered 2 nodes $ Text SCHEDULING: PLAN
- 48. 48 $ nomad runexample.nomad ==> Monitoring evaluation "4b8b7779" Evaluation triggered by job "example" Allocation "38720b8e" created: node "ec2f0830", group "cache" Evaluation status changed: "pending" -> "complete" ==> Evaluation "4b8b7779" finished with status "complete" $ Text SCHEDULING: RUN
- 49. 49 $ nomad run-region=gcp events.nomad ==> Monitoring evaluation "e2a8dfe6" !On branch master Evaluation triggered by job "events" !Your branch is up-to-date with 'origin/master'. Allocation "6615b39f" modified: node "0d6a6103", group "pubsub" !nothing to commit, working tree clean Evaluation status changed: "pending" -> "complete" ! ==> Evaluation "e2a8dfe6" finished with status "complete" $ Text SCHEDULING: RUN DIFFERENT REGION
- 50. 50 $ nomad statusexample ID = example Name = example Type = service Priority = 50 Datacenters = us-west-1 Status = running Summary Task Group Queued Starting Running Failed Complete Lost cache 0 0 1 0 0 0 Allocations ID Eval ID Node ID Task Group Desired Status Created At 38720b8e 4b8b7779 ec2f0830 cache run running 04/26/17 ... $ Text SCHEDULING: STATUS
- 51.
- 52. Nomad Million Container Challenge 1,000 Jobs 1,000Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers
- 53.
- 54. 54 MILLION CONTAINER CHALLENGE 54 –Bill Gates 640 KB ought to be enough for anybody. “
- 55. 55 REAL WORLD SCALE 55 2ndLargest Hedge Fund 18K Cores 5 Hours 2,200 Containers/ second
- 56.
- 57.
- 58.