Talk from Docker SF Meetup #50 Abstract: Docker swarm mode enables users to manage their applications with service primitives. In this talk we demonstrate how to do service upgrades without impacting your application. The Healthcheck feature provides health indication for a container. Coming up in Docker 1.13 release, Docker Swarm can connect healthcheck result with load balancer to implement no-loss service upgrade. Speaker Biographies: Nishant Totla is a software engineer at Docker, and works on the core open source team. He is currently working on Docker SwarmKit and Docker Swarm. Prior to Docker, he was a PhD student at UC Berkeley, doing research on programming languages. In his spare time, he enjoys long-distance running, biking, and other outdoor activities. Nishant tweets at @nishanttotla. Dongluo Chen is a software engineer at Docker focusing on orchestration and container development. Before Docker he was software engineer manager at Microsoft Azure building and automating global data centers. He worked at France Telecom (Orange) and the Ohio State University as research scientist in networking area.

1. Using Docker Swarm Mode
to Deploy Service Without
Loss
Dong Chen, dong@docker.com
Nishant Totla, nishant.totla@docker.com

2. Overview
● Service and its access methods
○ Docker 1.10 added distributed DNS to docker engine
○ Docker 1.11 added load balancer support thru libnetwork
○ Docker 1.12 added Docker Swarm mode where service API is supported
○ Service may be accessed thru load balancer or individual instances returned by DNS name
● Load balancer entries and DNS entries are service records
● Problem: a container is added to load balancer and DNS record when it
starts. If requests are sent thru load balancer, they may be routed to the new
instances right away.
○ If the container takes time to initialize, some requests are lost. This affects availability.
○ The problem comes from the gaps between container/task status and network update.
● In Docker 1.13, several changes are made to connect these 2 components.

3. Docker Swarm Mode
● Clustering
○ Swarm is the representation of a cluster. Use ‘docker swarm init’ to initialize a cluster.
○ A node is an instance of the Docker Engine in the Swarm. A node can be a manager or a
worker, joining the cluster thru ‘docker node join …’
● Orchestration
○ A service represents deployment unit in Swarm mode. It consists of multiple tasks with same
Taskspec, controlled by ‘docker service …’
○ A task is the atomic scheduling unit of Swarm. For example a task may be to schedule a
Redis container to run on a node.
● Networking
○ A service can join different networks to gain access to other services or isolation from other
services.
○ A published port is a public access point for a service. It’s reachable from every node in the

4. Service deployment procedure
● A service deployment is a change in service which updates its tasks
○ Fresh deployment or service create
○ Scale up/down
○ task spec change, e.g., image, published ports, constraints, labels, ...
● Deployment procedure for image update
○ Orchestrator receives notification of service update
○ Orchestrator checks the difference between current state and desired state.
○ Orchestrator triggers update for all tasks of the service.
■ Concurrent update is rate limited by update-parallelism
■ update-delay provides gap between updates. User can interrupt deployment if service
has error or performance degrade.

5. Request loss problem
● Access a service with multiple running instances
○ Thru load balancer
○ Thru individual instances returned by DNS
● Docker 1.12 does NOT coordinate service record update (load balancer or
DNS) and container state
○ A container is added to load balancer/DNS when it starts
○ When a request is routed to a container before it’s ready, the request is lost.
● Docker 1.13 adds healthcheck aware service record update to support
deployment without loss
○ Change in libnetwork to separate container create from service record update
○ Change in docker daemon to update service record based on container state

6. Health check
● Docker 1.12 introduced
healthcheck to show container
health
○ CMD, Interval, timeout, retries
○ Containerd executes healthcheck
and report container’s liveness
○ Health change triggers events
● Docker 1.13 uses healthcheck
to update service record
○ Docker 1.13 also adds CLI
support for healthcheck
FROM alpine
RUN apk --update add curl
HEALTHCHECK --interval=3s --timeout=1s --retries=3
CMD curl -f http://127.0.0.1:5000/health/ || exit 1
EXPOSE 5000
COPY simpleweb /usr/bin
CMD sleep 5 && simpleweb

7. Load balancer update
● In Docker 1.12, container
create and add to load balancer
are coupled together
● Docker 1.13 adds
enableService(bool) primitive
○ Separation of containerd
operations and service records
○ Swarm can add/remove container
to service records at appropriate
time

8. Service update call sequence - docker 1.13
● Add container to
service record when
container is healthy
● Remove container
from load balancer
when container is
unhealthy or going
to shutdown
● No requests lost
during service
update