lthough containers are bringing a refreshing flexibility when deploying services in production, the management of those containers in such an environment still requires special care in order to keep the application up and running. In this regard, orchestration platforms like Docker, Kubernetes and Nomad have been trying to alleviate this responsibility, facilitating the task of deploying and maintaining the entire application stack in its desired state. This ensures that a service will be always running, tolerating machine failures, network erratic behavior or software updates and downtime. The purpose of this talk is to explain the mechanisms and architecture of the Docker Engine orchestration platform (using a framework called swarmkit) to tolerate failures of services and machines, from cluster state replication and leader-election to container re-scheduling logic when a host goes down.

1. Orchestrating Linux Containers
while Tolerating Failures
Container Camp, London
September 9, 2016
Nishant Totla, Core Team Software Engineer, Docker Inc.
@nishanttotla

2. What is a Linux Container?
• A Linux container is a self-contained execution environment with isolated
resources
• The technology has been around for a while - LXC, FreeBSD Jails, Solaris
Zones
• The Docker project, open-sourced in 2013 made containers extremely
popular and significantly easier to use

3. What is a Linux Container?
• A Linux container is a self-contained execution environment with isolated
resources
• The technology has been around for a while - LXC, FreeBSD Jails, Solaris
Zones
• The Docker project, open-sourced in 2013 made containers extremely
popular and significantly easier to use
Source: Docker blog (August ‘16)

4. Real-world Applications
• The growth of containers has brought a fresh perspective on deploying
containerized applications, particularly microservices
• This brings up interesting problems
• Infrastructure
• Deployment
• Security
• Networking
• Scaling
• Fault-tolerance
• New tools and abstractions needed
• Container Orchestration is the key!

5. Trends
Docker
Orchestration, etc.

6. Orchestration is Hard
• Distributed Systems are hard to get right
• Failures happen all the time!
• Machines go down
• Networks can fail or drop packets
• Latency can cause unexpected scenarios

7. Can we make it easier to orchestrate real-world containerized applications?

8. Can we make it easier to orchestrate real-world containerized applications?
à Swarm Mode in Docker 1.12

9. Swarm Mode in Docker 1.12
• Goals
• It should be simple to setup and use a distributed cluster
• It should be seamless to move from development to deployment

10. Swarm Mode in Docker 1.12
• Goals
• It should be simple to setup and use a distributed cluster
• It should be seamless to move from development to deployment
• Starting with 1.12, Docker now has inbuilt orchestration
• The easiest way to orchestrate Docker containers is now Docker
• Multiple Docker Engines can be brought together to form a Swarm cluster
• Really easy to manage applications on the cluster

11. The Docker approach to container orchestration
1. Swarm Topology
2. Managing Cluster State
3. Managing Application State
Demo

12. Part 1: Swarm Topology

13. Single Node Cluster
$ docker swarm init --advertise-addr 198.211.109.17
Engine

14. Adding another Node
$ docker swarm init --advertise-addr 198.211.109.17
$ docker swarm join 198.211.109.17:2377
Engine Engine

15. More Nodes
$ docker swarm init --advertise-addr 198.211.109.17
$ docker swarm join 198.211.109.17:2377
Engine Engine
Engine
Engine
Engine

16. Managers and Workers
• Swarm nodes can be workers or managers
• Worker nodes
• Run Tasks (containers)
• Manager nodes
• Run Tasks (containers)
• Accept specifications from the user
• Reconcile desired and and actual cluster state
• Workers can be promoted to managers
• Managers can be demoted to workers
Engine
Engine

17. Secure by Default with end to end Encryption
• Cryptographic node
identity
• Automatic encryption and
mutual auth (TLS)
• Automatic certificate
rotation
• External CA integration
• Encrypted communication
Manager
[TLS]
[CA]
Manager
[TLS]
[CA]
Manager
[TLS]
[CA]
Agent
[TLS]
Agent
[TLS]
Agent
[TLS]

18. Part 2: Managing Cluster State

19. The Raft Algorithm
• Raft is a popular distributed consensus algorithm
• Features
• Single leader performs all updates
• Leader election – elect a leader
• Log replication – replicate the log of cluster operations
• Consensus is maintained as long as a majority of nodes are working

20. Raft Consensus Layer
Manager
Internal Distributed State Store
Manager Manager
Raft consensus group
• Built on top of the etcd Raft library
• Strongly consistent - holds desired state
• Fast (in-memory reads, domain-specific indexing, batched operations)
• Secure

21. Raft Consensus Layer
• Swarm managers are crucial
• Manager nodes form a Raft consensus group
• No need to set up an external KV store
• As long as a majority of managers are running, the cluster will function
normally (should start with an odd number)
• All managers can accept API requests (HA, no single point of failure)

22. Worker to Worker Gossip
Engine Engine Engine EngineEngine
Gossip Network
• Different from Raft to keep Raft concise
• High volume, p2p network between workers
• Eventually consistent – networking, load balancing rules, etc.
• Secure: symmetric encryption with key rotation in Raft

23. Putting it All Together

24. Full Architecture
Engine Engine Engine EngineEngine
Gossip Network
Manager
Internal Distributed State Store
Manager Manager
Raft consensus group
gRPC

25. Part 3: Managing Application State

26. A Higher Level of Abstraction
• From containers to services
$ docker run $ docker service create
• Docker services
• Allow declarative specification
• Work seamlessly on any number of nodes
• Perform desired state reconciliation
• Can be scaled or updated easily

27. Services
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0
Engine Engine
Engine
Engine
Engine
mynet

28. Swarm Routing Mesh
• All nodes expose port
8080 for myapp
• Routing mesh ensures
traffic is routed
transparently to a
running container of the
service
Manager
Agent 1 Agent 2 Agent 3
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0
:8080 :8080 :8080
:8080
access
myapp.com:8080

29. Swarm Routing Mesh
• Every service gets
assigned a virtual IP (VIP)
• Built-in DNS-based service
discovery: service name
resolves to service VIP
• VIP round-robins between
container IPs using Linux
IPVS
Manager
Agent 1 Agent 2 Agent 3
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0
:8080 :8080 :8080
:8080
access
myapp.com:8080

30. Node Failure
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0
Engine Engine
Engine
Engine
Engine
mynet

31. Desired State ≠ Actual State
Engine
Engine
Engine
Engine
mynet
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0

32. Converge back to Desired State
Engine
Engine
Engine
Engine
mynet
$ docker service create
--replicas 3
--name frontend
--network mynet
--publish 8080:80
frontend_image:1.0

33. Service Scaling
Engine
Engine
Engine
Engine
mynet
$ docker service scale frontend=6

34. And more!
• Global services
• Configurable rolling updates
• Constraints
• Restart policies
• Resource aware scheduling

35. Behind the scenes
API
Orchestrator
Allocator
Scheduler
Dispatcher
Worker
Executor
Manager
node
Worker
node
Accepts command from client and creates service object
Reconciliation loop for service objects and updates tasks
Allocates IP address to tasks
Assigns nodes to tasks
Dispatches tasks and checks in on workers (heartbeat)
Connects to Dispatcher to check on assigned tasks
Executes tasks assigned to worker node
$ docker service create

36. Node Failure
• Manager failure
• Handled by the Raft algorithm, as long as consensus is maintained
• Worker failure
• Dispatcher detects a node is down (failed heartbeat)
• Orchestrator reconciles tasks
• Allocator, Scheduler, and Dispatcher redo the previous steps to set up new tasks

37. If Everything Fails
$ docker swarm init
--advertise-addr 198.211.109.17
--force-new-cluster
• The flag forces an existing node that was part of a lost quorum to restart
as a single node Manager without losing its data

38. To Conclude
• Swarm mode in Docker 1.12 makes it very easy to create and manage a
failure-tolerant cluster
• Networking and security are handled out of the box
• The service abstraction allows for a declarative specification of how the
application is expected to run
• Much more coming up in the next few releases!

39. Contributions welcome!
• Docker open-source on GitHub
• github.com/docker/swarmkit | github.com/docker/docker

40. Special thanks to my co-workers for helping out with this talk
- Nathan LeClaire,
- Alexandre Beslic,
- Aaron Lehmann,
- Drew Erny,
- Jana Radhakrishnan,
- Dongluo Chen

41. Nishant Totla
@nishanttotla
Thank You!

42. Constraints
Engine Engine
Engine
Engine
Engine $ docker daemon --label
com.example.storage=“ssd”
$ docker daemon --label
com.example.storage=“ssd”

43. Constraints
$ docker service create
--replicas 5
--name frontend
--network mynet
--publish 8080:80
--constraint engine.labels.com.example.storage==ssd
frontend_image:1.0
Engine Engine
Engine
Engine
Engine
mynet
$ docker daemon --label
com.example.storage=“ssd”
$ docker daemon --label
com.example.storage=“ssd”

44. Constraints
Engine Engine
Engine
Engine
Engine
mynet
$ docker daemon --label
com.example.storage=“ssd”
$ docker daemon --label
com.example.storage=“ssd”
$ docker service scale frontend=10