Distributed Cluster Schedulers are becoming increasingly popular. They present a good abstraction for running workloads at a “warehouse-scale” on the public and private clouds by decoupling workload from compute, network and storage resources. In this talk, we will talk about the operational challenges of running a Cluster Scheduler to serve highly available services across multiple geographies and in a heterogeneous runtime environment. We will go into details of the needs from a cluster scheduler with respect to managing multiple runtime/virtualization platforms, provide observability, running maintenance on hardware and software, etc.

1. HASHICORP
Taming the modern public clouds with Nomad
Diptanu Gon Choudhury
@diptanu

4. HASHICORP
Evolution of compute infrastructure
1995 2000 2015

5. HASHICORP
Evolution of compute infrastructure

6. HASHICORP
Evolution of compute infrastructure
Global Public Cloud
AWS - US-West-2 AWS - US-East-1
GCP - US-Central-1
Private Clouds Private Clouds

7. HASHICORP
Challenges of the modern cloud
10s of 1000s of compute nodes to manage
Compute clusters are spread across the globe
Static and offline partitioning of clusters are no longer efficient

8. HASHICORP
Challenges of the modern cloud
Heterogenous API for accessing compute infrastructure
Heterogenous primitives for managing network, secrets, etc

9. HASHICORP
Evolution of application architecture
SOA and Micro Services are replacing monoliths
Distributed Systems are the new normal

10. HASHICORP
Challenges in running modern services
Orchestrated deployment and rollback strategies
More modes of failures

11. Operator
Datacenter
Skywalker Vader Leia Solo

12. Operator
Datacenter
PYTHON
PYTHON
GOLANG
GOLANG
GOLANG
Skywalker Vader Leia Solo

13. Operator
Datacenter
RUBY PYTHON
PYTHON
PYTHON
GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Vader Leia Solo

14. Operator
Datacenter
RUBY PYTHON
PYTHON
PYTHON
GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Vader Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.5
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Randomly kills applications

15. Operator
Datacenter
RUBY PYTHON
PYTHON
PYTHON
GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.5
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Randomly kills applications
FFVader

16. Operator
Datacenter
RUBY PYTHON
PYTHON
PYTHON
GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.5
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Randomly kills applications
FFVader
PYTHON
PYTHON
PYTHON

17. Operator
Datacenter
RUBY GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.5
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Randomly kills applications
Vader
PYTHON
PYTHON
PYTHON

18. Operator
Datacenter
RUBY GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.9
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Rebuilt on 04/20/2016
Vader
PYTHON
PYTHON
PYTHON

19. Operator
Datacenter
RUBY GOLANG
GOLANG
GOLANG
GOLANG
NODE
Skywalker Leia Solo
RUBY
VADER
LEIA
SOLO
192.168.1.4
192.168.1.9
192.168.1.7
192.168.1.253
88:45:13:B6:87:C4
94:CE:4F:C8:54:C3
CA:9A:3D:7F:8B:CB
72:30:9C:0D:1E:74
Rebuilt on 04/20/2016
Vader
PYTHON
PYTHON
PYTHON

20. This does not scale

21. HASHICORP
Cluster Schedulers to the rescue
Decouple Work from Resources
Better Quality of Service
Higher Resource Utilization

22. Nomad
HASHICORP
Multi-Datacenter
Multi-Region
Flexible Workloads
Job Priorities
Bin Packing
Large Scale
Operationally Simple

23. HASHICORP
Nomad as Cluster Scheduler
Bin Packing
Job Queueing
Over-Subscription
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

24. HASHICORP
Nomad as the Cluster Scheduler
Abstraction
API Contracts
Standardization
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

25. HASHICORP
Nomad as the Cluster Scheduler
Priorities
Resource Isolation
Pre-emption
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

26. HASHICORP
Job Specification
Declares what to run

27. HASHICORP
example.nomad
# Define our simple redis job
job "redis" {
# Run only in us-east-1
datacenters = ["us-east-1"]
# Define the single redis task using Docker
task "redis" {
driver = "docker"
config {
image = "redis:latest"
}
resources {
cpu = 500 # Mhz
memory = 256 # MB
network {
mbits = 10
dynamic_ports = ["redis"]
}
}
}
}

28. HASHICORP
Job Specification
Nomad determines where and
manages how to run

29. HASHICORP
Job Specification
Abstract work from resources

30. HASHICORP
Supports multiple Clouds, DCs and Regions
Resources across DCs are presented as single pool
Developers can target multiple datacenter in the same job file
Unified interface for developers across clouds

31. HASHICORP
Unified interface across hybrid clouds
AWS GCP Azure
On-Prem
DC
Nomad
Job Spec

32. HASHICORP
Single Region Architecture
SERVER SERVER SERVER
CLIENT CLIENT CLIENT
DC1 DC2 DC3
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
FORWARDING
RPC RPC RPC

33. HASHICORP
Multi Region Architecture
SERVER SERVER SERVER
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
REGION B GOSSIP
REPLICATION REPLICATION
FORWARDING
REGION FORWARDING
REGION A
SERVER
FOLLOWER
SERVER SERVER
LEADER FOLLOWER

34. Nomad
HASHICORP
Region is Isolation Domain
1-N Datacenters Per Region
Flexibility to do 1:1 (Consul)
Scheduling Boundary

35. Data Model
ALLOCATION
JOB
EVALUATION
NODE

36. Evaluation ~= State
Change

37. Evaluations
Create / Update / Delete Job
Node Up / Node Down
Allocation Failed

38. Evaluations
SCHEDULER
func(Evaluation) => []AllocationUpdates

39. Evaluations
SCHEDULER
func(Evaluation) => []AllocationUpdates
Service, Batch, System

40. HASHICORP
Scheduler Architecture
Concurrent and optimistic scheduling
Event Driven invocation of schedulers
No head of line blocking for different type of workloads

41. HASHICORP
Client Architecture
Broad OS Support
Host Fingerprinting
Pluggable Drivers

42. HASHICORP
Drivers
Execute Tasks
Provide Resource Isolation

43. HASHICORP
Containerized
Virtualized
Standalone
Docker
Qemu / KVM
Java Jar
Static Binaries
Rocket

44. HASHICORP
Containerized
Virtualized
Standalone
Docker
Rocket
Windows Server Containers
Qemu / KVM
Hyper-V
Xen
Java Jar
Static Binaries
C#

45. HASHICORP
Maintainance Primitives
First class support for doing maintenance on nodes
Drain allocations running on a node
nomad node-drain -enable 149cc920
Are you sure you want to enable drain mode for node "149cc920"? [y/N]

46. HASHICORP
Service Discovery Aware
Allows developers to define services exposed by a job
Keep services and checks synced

47. HASHICORP
example.nomad
job "redis" {
task "redis" {
………
service {
name = “binstore”
tags = [“env:staging”, “stack:beta”]
port = “http”
check {
name = “binstore-http”
type = “http”
path = “/status”
interval = “30s”
timeout = “2s”
}
}
…………
}
}

48. HASHICORP
System Job Scheduler
Runs a job on every node on the cluster
Great for running monitoring, logging, auditing software

49. HASHICORP
Log Management
Takes care of rotating logs of services
Log forwarding coming soon

51. HASHICORP
Thanks!
https://github.com/hashicorp/nomad
https://www.nomadproject.io/