Armon Dadgar offers an overview of Nomad, an application scheduler designed for both long-running services and batch jobs. Along the way, Armon explores the benefits of using schedulers for empowering developers and increasing resource utilization and how schedulers enable new next-generation application architectures.

1. Nomad and next-generation
application architectures

2. Armon Dadgar
Founder and CTO
@armon

3. HashiCorpSuite
CONNECT
RUN
SECURE
PROVISION
Infrastructure & applications
Applications
Infrastructure & applications
Infrastructure
Consul
Nomad
Terraform
Vault
Packer
Vagrant
Consul Enterprise
Nomad Enterprise
Vault Enterprise
Terraform Enterprise
FOR TEAMS
OSS TOOL SUITE PRODUCT SUITE

4. Nomad
Cluster Manager
Scheduler

5. Nomad
Cluster Manager
Scheduler

6. Schedulers map a set of
work to a set of resources

7. CPU Scheduler
Web Server -Thread 1
CPU - Core 1
CPU - Core 2
Web Server -Thread 2
Redis -Thread 1
Kernel -Thread 1
Work (Input) Resources
CPU
Scheduler

8. Advantages
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

9. Schedulers in the Wild
Type Work Resources
CPU Scheduler Threads Physical Cores
AWS EC2 /
OpenStack Nova
Virtual Machines Hypervisors
Hadoop YARN MapReduce Jobs Client Nodes
Cluster Scheduler Applications Servers

10. Nomad
Cluster Scheduler
Easily Deploy Applications
Operationally Simple
Built for Scale
API for Next-Gen Patterns

11. job "redis" {
datacenters = ["us-east-1"]
task "redis" {
driver = "docker"
config { image = "redis:latest" }
resources {
cpu = 500 # Mhz
memory = 256 # MB
network {
mbits = 10
port “redis" {}
}
}
}
}
example.nomad

12. Declares what to run

13. Nomad determines where
and manages how to run

14. Nomad
Declarative Jobs
Infrastructure as Code
Consul Integration
Vault Integration
Composable vs Platform

15. Operationally Simple
&
Scalable

16. Nomad
Single Binary
No Dependencies
Highly Available
Multi-DC/Region Support

17. Built on Research & Experience
GOSSIP CONSENSUS
Mature Libraries Proven Design Patterns

18. Nomad
Inspired by Google Omega
Optimistic Concurrency
Service & Batch workloads
Pluggable Architecture

20. Next-Gen Patterns

21. MonolithMicro-Service
SOA Spectrum

22. MonolithMicro-Service
SOA Spectrum
Utility of Service

23. Monoliths have high
application complexity

24. Microservices have high
operational complexity

25. Abstractions allow us to
scale complexity

26. Frameworks :: Monoliths
Schedulers :: Services

27. Schedulers abstract details,
focus on service composition

28. Side Cars
• Sidecar or Co-Process Pattern
• Application that runs alongside “main” process
• Nomad “Task Group”
• Borg “Alloc”
• Kubernetes “Pod”

29. Side Cars
• Configuration (Consul-Template)
• Logging Agents (Splunk, CloudWatch)
• Telemetry Agents (Datadog)
• Service Mesh (Envoy, Linkerd)
• Load Balancing (HAProxy, Nginx, Fabio)

30. Nomad
Transparent Scheduling
API awareness
Dynamic Behavior

31. Queues

32. Queues
• Workers are online service doing batch work
• Workers provisioned in advance
• N+1 instances for high availability
• Typically idle or underutilized

33. Nomad Dispatch
• “Dispatch” a worker for each incoming event
• Consumer launched on-demand and terminates when done
• Publisher shielded from implementation detail
• Nomad job acts like a future, queues when busy
• Avoids underutilization

34. job “my-dispatch” {
datacenter = [“dc1”]
type = “batch”
parameterized {
meta_required = [“input”]
}
task “worker” {
driver = “docker”
config = {
image = “myworker:latest”
args = [“—input”, “${NOMAD_META_INPUT}”]
}
}
}
my-dispatch.job

35. Nomad
Server
Register
Dispatch Job

36. Nomad
Server
Register
Dispatch Job
Web Server
Dispatch

37. Nomad
Server
Register
Dispatch Job
Web Server
Dispatch
Worker 1
Schedule

38. Nomad
Server
Register
Dispatch Job
Web Server
Dispatch
Worker 1
Schedule
Worker 2
Worker N

39. Function-as-a-Service
• AWS Lambda
• Small Granularity
• Low Volume, Latency Insensitive => Nomad Dispatch
• High Volume, Latency Sensitive => Setup Overhead Prohibitive

40. FaaS / Serverless
• Process multiple events per worker
• Dynamically scale workers
• Queue messages to avoid dropping

41. Nomad
Server
Register
Dispatch Job
Controller

42. Nomad
Server
Register
Dispatch Job
Web Server
Push
Controller
Dispatch

43. Nomad
Server
Register
Dispatch Job
Web Server
Push
Worker 1
Schedule
Controller
Pull
Dispatch

44. Nomad
Server
Register
Dispatch Job
Web Server
Push
Worker 1
Schedule
Controller
Pull
Dispatch
Deep Message Queue

45. Nomad
Server
Register
Dispatch Job
Web Server
Push
Worker 1
Schedule
Worker 2
Worker NController
Pull
Dispatch

46. Big Data Processing
• Large scale batch workload
• Graph of processing steps
• Each phase dynamic size
• Programmatically setup/teardown workers
• Native Spark Integration!

47. Nomad
Server
Executor 1
Schedule
Executor 2
Executor N
Launch
Executors
Submit Job

48. Scheduler API blurs line between
Application and Infrastructure

49. Nomad
Cluster Scheduler
Easily Deploy Applications
Operationally Simple
Built for Scale
API for Next-Gen Patterns

50. Thanks!
@armon