Containers are the future for all microservice based apps. Where do you deploy them? How do you manage them? At Digital Ocean we went through growing pains of trying out 5 of the top major Docker container schedulers, Mesos, Kubernetes, Docker Swarm, Nomad and we even tried manual scheduling of containers. Let us walk you through how we chose different schedulers for different applications, and tips and tricks for choosing a scheduler to use.

1. Distributed scheduler hell

2. Story of ..
How we moved 100(s) of virtual machines, onto containers

3. What is a distributed scheduler
Your cloud provider: Digital Ocean, AWS, Google.
For containers: Kubernetes, Nomad, Mesos

4. Why do we care?
Microservices...

5. Vulcan
Distributed timeseries database.
open source https://github.com/digitalocean/vulcan

6. Requirements
4 3 Gbits a second of Network traffic
4 20 TB Storage
4 Sub 100ms Read times
4 100k writes a second

8. Single process

9. Hypervisor
Linux Scheduler

10. Kernel Provides
4 Virtual Memory
4 Process Isolation
4 Disk storage
4 Networking
4 CPU scheduling

11. Distributed Scheduler Provides
4 Container Deploy
4 Virtual Machine Deploy
4 Memory Quota
4 Disk storage
4 Networking
4 CPU scheduling
4 Scaling Instances

12. Mesos

15. Deploying our App On Mesos

17. Marathon Architecture

18. Kafka
Custom Scheduler

19. Cassandra
Pinning to specific Mesos nodes

20. Mesos Failure Modes
Network Partition FAIL

21. Counter Example: Nomad

22. Nomad Architecture

23. Kubernetes

25. Custom Json Language
{
"application": {
"name": "timeseries-ingestor",
"scale": 1,
"ingresses": {
"timeseries-ingestor-health": {
"scheme": "http",
"container_port": 8001
}
},
"containers": {
"timeseries-ingestor": {
"image": "docker.com/timeseries/ingestor",
"image_tag": "fcb24ca",
"ports": [
8001,
9090
],
"env": {
"CASSANDRA_ADDRESS_LIST": ""
},
"resources": {
"cpu": "4",
"memory": "4000"
}
}
},
"metrics": {
"port": 8001,
"path": "/debug/metrics"
}
},
"maintainer": "dummy@digitalocean.com"
}

26. Command Line Deploys
docc --contexts dev_env deploy timeseries-microservice1.json
Docc is our internal Kubernetes tool

27. Deployment tool with diffs to Kubernetes
"env": {
+++ "CASSANDRA_CQL_VERSION": "3.1.7",
--- "CASSANDRA_KEYSPACE": "staging_vulcan",
+++ "CASSANDRA_KEYSPACE": "staging_vulcan_123",
Simliar to KubeDiff

28. Final Architecture

30. Load balancing
Pushing 3 Gbs to kubernetes using Flannel

31. Metrics

33. Logging

35. Upsides to Distributed Schedulers

36. How to choice your abstractation

37. Alles hat ein Ende nur die Wurst hat
zwei
('everything has one end, only the sausage has two')