Sep 2017
MySQL on Docker:
Containerizing the Dolphin
Ashraf Sharif, Support Engineer
Presenter
ashraf@severalnines.com

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
About Severalnines and ClusterControl

Agenda
Copyright 2012 Severalnines AB
● Introduction
● Image
● Container
● Volume
● Network
● Orchestration
● MySQL Container Management

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Introduction

What is Docker?
Copyright 2012 Severalnines AB
● Lightweight application container platform.
● Each container has its own:
○ Filesystem
○ Process space
○ Network stack
● Use pre-built image.
● Advantages:
○ Rapid deployment
○ Greater density per host
○ Isolation
Docker Architecture

Docker Components
Copyright 2012 Severalnines AB
Docker Components

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Docker Image

Docker Image - Introduction
Copyright 2012 Severalnines AB
● A template for Docker container.
● Create an image:
○ Commit the container's changes
○ Build from Dockerfile
● Share the image:
○ Save as tarball.
○ Push to registry:
■ Public - Docker Hub, Docker Store
■ Private, hosted - Quay, ECR, GCR, Bintray,
Artifactory
■ Private, self-hosted - Your own registry
# Save image to tarball
$ docker save -o mysql.tar abc
# Start your own Docker registry service
$ docker run -d -p 5000:5000 --name registry
registry:2
# Push image to Docker Hub
$ docker push myimage/mysql:5.7

Docker Image - Commit a Container
Copyright 2012 Severalnines AB
Host
# Save image to tarball
$ docker run -d --name=abc ubuntu:16.04
# Commit the changes (in another window)
$ docker commit abc myimage/mysql:5.7
# Push image to Docker Hub
$ docker exec -it abc /bin/bash
root@abc:/# apt-get update
root@abc:/# DEBIAN_FRONTEND=noninteractive apt-get
install -y mysql-server
root@abc:/# vi /etc/mysql/my.cnf
# Ubuntu 16.04 host
$ apt-get update
$ DEBIAN_FRONTEND=noninteractive apt-get install -y
mysql-server
$ vi /etc/mysql/my.cnf
# Run MySQL server as a process
$ mysqld --socket=/var/lib/mysql/mysqld.sock
--pid-file=/var/lib/mysql/mysqld.pid
Docker
# Run MySQL server as a container
$ docker run -d myimage/mysql:5.7 mysqld
--socket=/var/lib/mysql/mysqld.sock
--pid-file=/var/lib/mysql/mysqld.pid

Docker Image - Using Dockerfile
Copyright 2012 Severalnines AB
Host
# Build as image
$ docker build -t myimage/mysql:5.7 .
# vi Dockerfile
FROM ubuntu:16.04
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y
mysql-server
COPY my.cnf /etc/mysql/my.cnf
EXPOSE 3306
# Ubuntu 16.04 host
$ apt-get update
$ DEBIAN_FRONTEND=noninteractive apt-get install -y
mysql-server
$ vi /etc/mysql/my.cnf
# Run MySQL server as a process
$ mysqld --socket=/var/lib/mysql/mysqld.sock
--pid-file=/var/lib/mysql/mysqld.pid
Docker
# Run MySQL server as a container
$ docker run -d myimage/mysql:5.7 mysqld
--socket=/var/lib/mysql/mysqld.sock
--pid-file=/var/lib/mysql/mysqld.pid
# Create MySQL config file
$ vi my.cnf

Docker Image - Build an Image
Copyright 2012 Severalnines AB
# Get image size
$ docker images
REPOSITORY TAG SIZE
myimage/mysql 5.7 529MB
# Show created layers
$ docker history myimage/mysql:5.7
CREATED BY SIZE
/bin/sh -c #(nop) COPY file:e8163ef656b6da... 101B
/bin/sh -c DEBIAN_FRONTEND=noninteractive ... 370MB
/bin/sh -c apt-get update 39.1MB
/bin/sh -c #(nop) CMD ["/bin/bash"] 0B
/bin/sh -c mkdir -p /run/systemd && echo '... 7B
/bin/sh -c sed -i 's/^#s*(deb.*universe... 2.76kB
/bin/sh -c rm -rf /var/lib/apt/lists/* 0B
/bin/sh -c set -xe && echo '#!/bin/sh' >... 745B
/bin/sh -c #(nop) ADD file:39d3593ea220e68... 120MB
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
724179e94999
f6016aab25f1
d51f459239fb
Image: myimage/mysql:5.7
ubuntu:16.04
ad74af05f5a2
Dockerfile

Docker Image - Best Practice for MySQL
Copyright 2012 Severalnines AB
# Get image size
$ docker images
REPOSITORY TAG SIZE
mysql 5.6 299MB
mysql 5.7 412MB
# tail entrypoint.sh
...
...
exec mysqld "$@"
● Extend an existing image available in the registry,
instead of building a new one from scratch.
● The smaller the image size, the faster the deployment.
(Hint: alpine).
● If using an entrypoint script, ensure mysqld command
runs with "exec", indicating it's the primary process
that holds PID 1 inside container.
● General guidelines:
https://docs.docker.com/engine/userguide/eng-image/
dockerfile_best-practices/
# vi Dockerfile
FROM mysql:5.7
# vi Dockerfile
FROM ubuntu:16.04
RUN apt-get install {all MySQL stuff}

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Docker Container

Docker Container - Run a Single
Container
Copyright 2012 Severalnines AB
# Run a MySQL container
$ docker run -d
--name my-test
--env MYSQL_ROOT_PASSWORD=mypassword
mysql
# Get the container layer size
$ docker ps -s
CONTAINER ID IMAGE NAMES SIZE
d51f459239fb mysql my-test 4B (virtual 412MB)
Image: docker.io/mysql:latest
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
d51f459239fb
0639788facc8
35369f9634e1
937bbdd4305a
my-test
Image layers
(RO)
412 MB
Container layers
(RW)
4 B

Docker Container - Run another
Container
Copyright 2012 Severalnines AB
# Run a MySQL container
$ docker run -d
--name my-test2
--env MYSQL_ROOT_PASSWORD=mypassword
mysql
# Get the container layer size
$ docker ps -s
CONTAINER ID IMAGE NAMES SIZE
d51f459239fb mysql my-test 4B (virtual 412MB)
e9de9ed50ced mysql my-test2 4B (virtual 412MB)
Image: docker.io/mysql
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
d51f459239fb
0639788facc8
35369f9634e1
937bbdd4305a
my-test
Image layers
(RO)
412 MB
Container
layers (RW)
4 B + 4 B
e9de9ed50ced
my-test2

Docker Container - Container Layer
Changes
Copyright 2012 Severalnines AB
# Operation 2 - Create a schema and table
$ docker exec -it my-test
mysql -uroot -pmypassword -e
'CREATE SCHEMA testdb; CREATE TABLE t1 (id INT PRIMARY
KEY AUTO_INCREMENT, data TEXT);'
Image: docker.io/mysql
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
d51f459239fb
0639788facc8
35369f9634e1
937bbdd4305a
my-test
c33a2b88395d
Copy-on-Write
(CoW)
416f2a97355f
Supported CoW file system & devices:
AUFS, Btrfs, Device Mapper, OverlayFS/Overlay2, ZFS, VFS
# Operation 1 - Update /etc/mysql/my.cnf
$ docker exec -it my-test vi /etc/mysql/my.cnf
1
2New
Branch

Docker Container - Application vs Data
Copyright 2012 Severalnines AB
Application Data
Stateless Stateful
Ephemeral Persistent
Independant Dependant
Frontend Backend
Web, API Database, File Storage

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Docker Volume

Docker Volume - Type
Copyright 2012 Severalnines AB
# Mount a named volume into container
-v mysql1-data:/var/lib/mysql
# Mount tmpfs into container
--tmpfs /tmp
● Bind mount:
○ A file or directory on the host machine is
mounted into a container.
○ You manage the directory’s contents.
● Named volume:
○ A new directory is created within Docker’s
storage directory.
○ Docker manages the directory’s contents.
○ Extensible via volume plugin.
● tmpfs:
○ Stored on host machine’s memory (or swap).
○ Linux only.
# Bind mount a directory into container
-v /data/mysql1/data:/var/lib/mysql
# Bind mount a file into container
-v /data/mysql1/my.cnf:/etc/mysql/my.cnf

Docker Volume - Persistent
Volumes
Copyright 2012 Severalnines AB
# Run with persistent named volume, using -v or --volume
$ docker run -d
--name mysql-local
--env MYSQL_ROOT_PASSWORD=mypassword
--volume local-datadir:/var/lib/mysql
mysql:5.7
Image: docker.io/mysql:5.7
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
0639788facc8
35369f9634e1
937bbdd4305a
mysql-local
416f2a92355f
/var/lib/mysql
local-datadir
# Run with persistent named volume, using --mount
$ docker run -d
--name mysql-local
--env MYSQL_ROOT_PASSWORD=mypassword
--mount source=local-datadir,target=/var/lib/mysql
mysql:5.7

Docker Volume - Changes on
Volume
Copyright 2012 Severalnines AB
# Run a container with port 3308
$ docker run -d
--name mysql-local
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v local-datadir:/var/lib/mysql
mysql
Image: docker.io/mysql:latest
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
0639788facc8
35369f9634e1
937bbdd4305a
mysql-local
416f2a92355f
/var/lib/mysql
local-datadir
# Run with persistent named volume, using --mount
$ mysql -uroot -p -h127.0.0.1 -P3308 < dump.sql
dump
.sql
3308

Docker Volume - Remote Volume
Copyright 2012 Severalnines AB
# Run a container with NFS bind mount
$ docker run -d
--name mysql-nfs
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v /nfs/mysql-nfs:/var/lib/mysql
mysql
Image: docker.io/mysql
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
0639788facc8
35369f9634e1
937bbdd4305a
mysql-nfs
416f2a92355f
/var/lib/mysql
192.168.1.100
/storage/docker
# Mount NFS with extra options for MySQL
$ mount 192.168.1.100:/storage/docker /nfs -o
noatime,nodiratime,hard,intr
3308
/nfs/mysql-nfs
Further Reading - Using NFS with MySQL:
https://dev.mysql.com/doc/refman/5.7/en/disk-issues.htm
l#disk-issues-nfs

Docker Volume - MySQL Persistency
Copyright 2012 Severalnines AB
Description Variable Name Default Value (5.7)
MySQL data directory datadir /var/lib/mysql
MySQL plugin directory plugin_dir {basedir}/lib/plugin
MySQL configuration directory config_dir /etc/my.cnf.d
MySQL binary log log_bin {datadir}/{hostname}-bin
Slow query log slow_query_log_file {datadir}/{hostname}-slow.log
Error log log_error {datadir}/{hostname}.err
General log general_log_file {datadir}/{hostname}.log

Docker Volume - Non-Persistent
Volume
Copyright 2012 Severalnines AB
# Run a container with NFS bind mount and tmpfs volumes
$ docker run -d
--name mysql-nfs-tmp
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v /nfs/mysql-nfs:/var/lib/mysql
--tmpfs /tmp:rw,size=1g,mode=177
mysql
Image: docker.io/mysql:latest
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
0639788facc8
35369f9634e1
937bbdd4305a
mysql-nfs-tmp
416f2a92355f
/var/lib/mysql
192.168.1.100
/storage/docker
3308
/nfs/mysql-nfs
/tmp

Docker Volume - Drivers
Copyright 2012 Severalnines AB
● Default is local.
● You can use other different volume driver (or plugin).
○ Install the plugin.
○ Use --volume-driver option.
● Volume drivers allow you to extend Docker's volume
capabilities:
○ Snapshot
○ Backup
○ Encryption
# Install Rexray EBS docker plugin
$ docker plugin install rexray/ebs EBS_ACCESSKEY=XXXX
EBS_SECRETKEY=YYYY
# Verify if the plugin is loaded
$ docker info -f '{{json .Plugins.Volume}}' | jq
[
"local",
"rexray"
]
# Run a container with volume plugin
$ docker run -d
--name=mysql-ebs
--volume-driver=rexray/ebs
-v ebs-datadir:/var/lib/mysql
mysql:5.7
# List volumes
$ docker volume ls
DRIVER VOLUME NAME
local local-datadir1
local local-datadir2
rexray ebs-datadir

Docker Volume - Drivers and Storage
Platforms
Copyright 2012 Severalnines AB
Further Reading - Volume Plugins List: https://docs.docker.com/engine/extend/legacy_plugins/#volume-plugins
Contiv
Fuxi Netshare
REX-ray
Horcrux
Convoy
OpenStorage
Amazon S3
Azure FS
Google PD
Amazon EBS DigitalOcean
BS
Flocker
EMC
FUSE
vSphere
iSCSI
CIFS NetApp
OpenStack
Cinder Minio CephFS
BeeGFS
NFS
SMB
Infinit

Docker Volume - Simple Benchmark
Copyright 2012 Severalnines AB
Local vs NFS vs NFS-tweaked vs EBS-standard

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Docker Network

Docker Network - Types
Copyright 2012 Severalnines AB
● Single-host:
○ Host
○ None
○ Bridge
■ Default bridge (docker0)
■ User-defined bridge
● Multi-host:
○ Default overlay
○ User-defined overlay
● Extensible via Docker network plugins.
# List all networks
$ docker network ls
NAME DRIVER SCOPE
bridge bridge local
db_multi overlay swarm
db_single bridge local
docker_gwbridge bridge local
host host local
ingress overlay swarm
none null local

Docker Network - Host Network
Copyright 2012 Severalnines AB
# Run a container on host network
$ docker run -d
--name mysql-host
--net host
-e MYSQL_ROOT_PASSWORD=mypassword
mysql:5.7
Container 1
eth0
eth0
Container 2
eth0
Docker
Host
# Display active TCP connections
$ netstat -tulpn | grep mysql
tcp6 0 0 :::3306 :::* LISTEN 24601/mysqld
● Container’s network interfaces will be identical with
the machine host.
● Only one host network per machine host.
● Container linking, --link mysql-test:mysql is not
supported.
● Port mapping, --publish 3307:3306 is not
supported.

Docker Network - Default Bridge
Network
Copyright 2012 Severalnines AB
# Run a container on docker0 bridge network
$ docker run -d
--name mysql-bridge
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
mysql:5.7
Container 1
eth0
eth0
Container 2
eth0
Docker
Host
# Display active TCP connections
$ netstat -tulpn | grep 3308
tcp6 0 0 :::3308 :::* LISTEN 24601/docker-proxy
● Packet forwarding by iptables to the bridge network.
● --publish is supported. Run multiple containers with
same image through different ports.
● --link is supported. Link with other containers to
expose environment variables.
● docker-proxy redirects connection to the correct
container through NAT.
docker0
(virtual Ethernet bridge)
vethxxx vethyyy

Docker Network - User-Defined
Bridge Network
Copyright 2012 Severalnines AB
# Run a container on user bridge network
$ docker run -d
--name mysql1
--net db_single
-p 3308:3306
--ip 192.168.10.10
--hostname mysql1
-e MYSQL_ROOT_PASSWORD=mypassword
mysql:5.7
Container 1
eth0
eth0
Container 2
eth0
Docker
Host
# Create a new bridge network
$ docker network create subnet=192.168.10.0/24 db_single
Similar to the default docker0 bridge, plus:
● --hostname is supported for sticky hostname.
● --ip is supported for sticky IP address.
● --link is not supported. Use embedded DNS to
resolve container's name in the same network.
db_single
(virtual Ethernet bridge)
vethxxx vethyyy
DNS

Docker Network - Plugins
Copyright 2012 Severalnines AB
● Similar to volume, Docker network can be extended
through plugins:
○ Multi-host networking (must use outside of
Swarm)
○ Name resolver and service discovery.
○ Encryption.
● Popular network plugins:
○ Contiv
○ Calico
○ Weave
○ Flannel
Further Reading - Multi-host Networking with Calico:
https://severalnines.com/blog/mysql-docker-multi-host-networkin
g-mysql-containers-part-2-calico

Docker Network - Default Overlay
Network
Copyright 2012 Severalnines AB
# Run a service on default overlay network
$ docker service create
--name mysql-service
--replicas 1
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v nfs-datadir:/var/lib/mysql
mysql:5.7
Container 1
eth0
eth0 eth1
Docker
Host
# Initialize Docker Swarm
$ docker swarm init
● Docker Swarm must be enabled.
● Each task (container) has:
○ eth0 - Interface to external network via
docker_gwbridge.
○ eth1 - Interface to overlay network via VXLAN.
docker_gw
bridge
vethxxx vethyyy
br0
External
Container 2
eth0 eth1
Docker
docker_gw
bridge
vethxxx vethyyy
br0
eth0
Host
VXLAN
udp/4789

Docker Network - User-Defined
Overlay Network
Copyright 2012 Severalnines AB
# Run a service on default overlay network
$ docker service create
--name mysql-service
--replicas 1
--net db_multi
-p 3308:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v nfs-datadir:/var/lib/mysql
mysql:5.7
Container 1
eth0
eth0 eth1
Docker
Host
# Create an overlay network
$ docker network create -d overlay db_multi
Similar with the default overlay, plus:
● Embedded DNS resolver:
○ {service_name} → Service VIP
○ tasks.{service_name} → Container's IP
docker_gw
bridge
vethxxx vethyyy
br0
External
Container 2
eth0 eth1
Docker
docker_gw
bridge
vethxxx vethyyy
br0
eth0
Host
VXLAN
udp/4789
DNS

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Docker Orchestration

Docker Orchestration - Container
Orchestration
Copyright 2012 Severalnines AB
● Simplify container's management at scale.
● Concept:
○ Multiple containers as one entity.
○ Structural YAML formatting.
● Official Docker comes with:
○ Docker Compose:
■ single-host, small scale, separate package.
○ Docker Swarm:
■ multi-host, big scale, out-of-the-box.
■ support compose formatting through
docker stack.

Docker Orchestration - Docker
Compose
Copyright 2012 Severalnines AB
Docker CLI
version: '2'
services:
mysql-prod:
image: mysql:5.7.16
command:
- --innodb_buffer_pool_size=2G
- --max_connections=50
ports:
- "3307:3306"
environment:
MYSQL_ROOT_PASSWORD: r00tP4ssw0rD
MYSQL_USER: myshop
MYSQL_PASSWORD: MyS3creTP4s5
MYSQL_DATABASE: myshop
hostname: mysql-prod
networks:
db-prod:
ipv4_address: 192.168.10.101
volumes:
- mysql-datadir:/var/lib/mysql
- mysql-tmp:/tmp
ulimits:
nofile:
soft: 16824
hard: 16824
mem_limit: 4g
memswap_limit: 4g
volumes:
mysql-datadir:
mysql-tmp:
networks:
db-prod:
driver: bridge
ipam:
driver: default
config:
- subnet: 192.168.10.0/24
# Production MySQL container - full command
$ docker network create
--subnet=192.168.10.0/24 db-prod
$ docker run -d
--name mysql-prod
--net db-production
--publish 3307:3306
--ip 192.168.10.101
--hostname mysql-prod
--memory 4g
--memory-swap 4g
--ulimit nofile=16824:16824
--volume mysql-datadir:/var/lib/mysql
--volume mysql-tmp:/tmp
--env MYSQL_ROOT_PASSWORD=r00tP4ssw0rD
--env MYSQL_USER=myshop
--env MYSQL_PASSWORD=MyS3creTP4s5
--env MYSQL_DATABASE=myshop
mysql:5.7.16
--innodb_buffer_pool_size=2G
--max_connections=50

Docker Orchestration - Run with
Docker Compose
Copyright 2012 Severalnines AB
# Bring up the production MySQL container in background
$ docker-compose up -d
Creating network "compose_db-prod" with driver "bridge"
Creating compose_mysql-prod_1
Image: docker.io/mysql:5.7.18
de70fa77eb2b
f6016aab25f1
0639788facc8
35369f9634e1
937bbdd4305a
ad74af05f5a2
0639788facc8
35369f9634e1
937bbdd4305a
mysql-prod
416f2a92355f
/var/lib/mysql
mysql-datadir
/tmp
mysql-tmp
3307
# Stop the container
$ docker-compose stop
# Destroy everything
$ docker-compose down
# Start the container
$ docker-compose start
Scaling is supported via docker-compose scale, however
conflicting options must be omitted:
hostname, ports, ipv4_address, volume

Docker Orchestration - Docker Swarm
Copyright 2012 Severalnines AB
● Multi-host deployment, through overlay network.
● Concepts:
○ Node
■ Manager - Dispatches tasks
■ Worker - Executes tasks by manager
○ Service
■ Replicated - Number of replicas
■ Global - One task per available node
○ Task
■ Container
○ Mesh routing
■ All Swarm nodes route to the running tasks
■ Ingress load balancing (round-robin only)
○ Scheduler
■ Resources availability
■ Label and constraints

Docker Orchestration - Simple Auto
Failover using Swarm
Copyright 2012 Severalnines AB
# Run a replica of MySQL service
$ docker service create
--name mysql-service
--replicas 1
--net db_multi
-p 3307:3306
-e MYSQL_ROOT_PASSWORD=mypassword
--mount
type=volume,source=ebs-datadir,destination=/var/lib/mysql
,volume-driver=rexray
--mount
type=volume,source=ebs-conf,destination=/etc/my.cnf.d,vol
ume-driver=rexray
mysql:5.7
mysql-
service
Docker
Host 1
ebs-datadir
mysql-
service
Docker
Host 2
3307
ebs-conf

Docker Orchestration - Galera
Cluster on Swarm
Copyright 2012 Severalnines AB
# Bootstrap-only container
$ docker service create
--name pxc-bs
--replicas 1
--net db-multi-hosts
-e MYSQL_ROOT_PASSWORD=mypassword123
-e CLUSTER_NAME=my_galera
-e CLUSTER_JOIN=
percona/percona-xtradb-cluster
Docker
Host 1
tasks.
pxc1
# 3-node Galera
$ for i in {1..3}; do docker service create
--name pxc${i}
--replicas 1
--net db-multi-hosts
-p 330${i}:3306
-e MYSQL_ROOT_PASSWORD=mypassword123
-e CLUSTER_NAME=my_galera
-e CLUSTER_JOIN=tasks.pxc-bs,tasks.pxc1,tasks.pxc2,tasks.pxc3
--mount source=pxc{i},target=/var/lib/mysql
percona/percona-xtradb-cluster;
sleep 1m;
done
tasks.
pxc-b
s
# Remove bootstrap-only container
$ docker service rm pxc-bs
Docker
Host 2
tasks.
pxc2
Docker
Host 3
tasks.
pxc3
HAproxy
3301
3302
3303
3301
3302
3303
3301
3302
3303
3306

Docker Orchestration - Other
Orchestration Tools
Copyright 2012 Severalnines AB

Docker Orchestration - Best Practice for
MySQL
Copyright 2012 Severalnines AB
● Add label to node and use placement constraints:
○ Avoid overcommit resources.
○ Manageability and predictability.
● Use Service's name and virtual IP address to distinguish
container's role.
● Understand the scheduling strategy:
○ Spread
○ Binpack
○ Random
○ Custom
● Rolling update:
○ Only perform update one container at a time.
○ Add delay before move to the next container.

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
MySQL Container Management

MySQL Container - Service Control
Copyright 2012 Severalnines AB
# Get container's log
$ docker logs -f mysql-staging
...
[Note] mysqld: Shutdown complete
# List processes inside container
$ docker exec -it mysql-staging ps -e
PID TTY TIME CMD
1 ? 00:00:00 mysqld
62 pts/0 00:00:00 bash
86 pts/0 00:00:00 ps
● mysqld must hold PID 1 inside container to receive the
signal.
● docker stop might be dangerous. Default is 10s grace
period. If still running, SIGKILL.
● Use docker kill with proper signal (SIGTERM). No
grace period.
● Verify the shutdown process through docker log.
● Stopping won’t delete the container.
# Stop MySQL container using kill
$ docker kill --signal=TERM mysql-staging
# Stop MySQL container using stop
$ docker stop mysql-staging

MySQL Container - Resource Control
Copyright 2012 Severalnines AB
# Limit memory and open file descriptors
$ docker run -d
--name mysql-staging
--memory 4g
--memory-swap 4g
--ulimit nofile=16824:16824
mysql:5.7.6
● By default, no resource constraints, use as much of a
given resource as the host’s kernel will allow.
● Important options!
○ --memory & --memory-swap
■ Must be equal to disable swap.
■ If --memory-swap is not set, container
swap is default to --memory * 2
○ --ulimit nofile={soft:hard}
■ Maximum number of files MySQL server
can open simultaneously.
● Some of the container resources can be controlled
dynamically using docker update:
○ Tune my.cnf accordingly.
# Increase container's memory.
$ docker update
--memory 6g
--memory-swap 6g
mysql-staging

MySQL Container - Resource Monitoring
Copyright 2012 Severalnines AB
# Monitor containers
$ docker stats --format "table
{{.Name}}t{{.CPUPerc}}t{{.MemUsage}}"
NAME CPU % MEM USAGE / LIMIT
minio1 0.00% 6.078MiB / 7.78GiB
mysql2 0.12% 194.1MiB / 7.78GiB
mysql-local 0.14% 220MiB / 7.78GiB
● Use docker stats.
● Inside container, free and top commands are not
accurate:
○ Memory - /sys/fs/cgroup/memory/memory.*
○ CPU - /sys/fs/cgroup/cpu/cpu.*
○ IO - /sys/fs/cgroup/blkio/blkio.*
● External open-source monitoring tools:
○ CLI: sysdig, dockviz
○ UI: Portainer, Shipyard, Rancher, Prometheus
Portainer

MySQL Container - Configuration
Management
Copyright 2012 Severalnines AB
# Bind mount my.cnf
$ docker run -d
--name mysql1
-e MYSQL_ROOT_PASSWORD=mypassword
-v /storage/mysql1/my.cnf:/etc/mysql/my.cnf
mysql:5.7.6
● Most MySQL configuration parameters can be changed
during runtime. (Hint: SET GLOBAL)
● You can either:
○ Use persistent storage:
■ Bind mount your my.cnf file.
■ Create a named volume for /etc/my.cnf.d
○ Append the configuration as a "CMD" (right after
image name), passes as flags to mysqld inside
container.
● Standard MySQL and InnoDB optimization applies.
# Named volume for MySQL configuration directory
$ docker run -d
--name mysql2
-e MYSQL_ROOT_PASSWORD=mypassword
-v mysql2-conf:/etc/my.cnf.d
mysql:5.7.6
# Append configuration options
$ docker run -d
--name mysql3
-e MYSQL_ROOT_PASSWORD=mypassword
mysql:5.7.6
--innodb_buffer_pool_size=1G
--max_connections=100

MySQL Container - Security
Copyright 2012 Severalnines AB
# Create a secret
$ echo 'MyP2s$w0rD' | docker secret mysql_root_passwd -
● Docker Secrets
○ Only for Swarm service.
○ Accessible under /run/secrets/ inside
container.
● Runtime privileges:
○ Unprivileged is default and recommended.
Container is not allowed to access other
devices.
○ Use --privileged only if you really need to
modify kernel parameters (sysctl, /proc,/sys)
or you want to run container inside
container.
# Create a service and use the secret as root password
$ docker service create
--name mysql-container
--publish 4406:3306
--secret mysql_root_passwd
-e MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql_root_passwd
-v local-datadir:/var/lib/mysql
mysql:5.7

MySQL Container - Backup
Copyright 2012 Severalnines AB
# mysqldump and store inside physical host
$ docker exec -it mysql-prod mysqldump -uroot -p
--single-transaction > /path/in/physical/host/dump.sql
● Logical backup:
○ Mysqldump
○ mysqlpump
● Xtrabackup (hot-backup):
○ Install inside machine host.
○ Install inside the MySQL container.
○ Use perconalab/percona-xtrabackup image.
● Snapshot:
○ MyISAM only - FTWRL + copy over
○ Stop container + copy over
# FTWRL
$ docker exec -it mysql-prod /bin/bash
root@mysql-prod:# mysql -uroot -p
mysql> FLUSH TABLE WITH READ LOCK;
# Copy volume to another location (another terminal)
$ cp /var/lib/docker/volumes/mysql-datadir/_data
/destination/in/physical/host
# Release lock (same terminal as FTWRL)
mysql> UNLOCK TABLES;
# Use perconalab/percona-xtrabackup image to backup
$ docker run --rm -it
--net db-prod
-v mysql-datadir:/var/lib/mysql
-v /tmp/backup:/xtrabackup_backupfiles
perconalab/percona-xtrabackup
--backup --host=mysql-prod --user=root
--password=mypassword

MySQL Container - Restore
Copyright 2012 Severalnines AB
# Restore a MySQL dump
$ docker exec -it mysql-staging mysql -uroot -p <
/path/in/physical/host/dump.sql
● Logical backup:
○ Fairly straightforward. Use standard procedure.
● Xtrabackup (hot-backup):
○ Have to be prepared first
○ Once prepared, simply starts a new container by
mounting the prepared directory as datadir.
● Snapshot & Copy:
○ Bind mount:
i. Direct mount the directory as a volume.
○ Named volume:
i. Create a volume.
ii. Copy the content to the volume's
directory.
iii. Start a new container by mounting the
volume.
# Run a test container from a copied datadir
(snapshot) or from a prepared xtrabackup directory.
$ docker run -d
--name mysql-restored
-v /tmp/backup:/var/lib/mysql
mysql:5.7.16
# Prepare the backup created by xtrabackup
$ docker run --rm -it
-v mysql-datadir:/var/lib/mysql
-v /tmp/backup:/xtrabackup_backupfiles
perconalab/percona-xtrabackup
--prepare --target-dir /xtrabackup_backupfiles

MySQL Container - Logical Upgrade
Copyright 2012 Severalnines AB
1. Pull the new image, M.
2. Start new container, B with new image, M.
3. Export MySQL on A.
4. Stop A.
5. Import into B.
6. Run mysql_upgrade inside B.
7. OK? Remove A.
8. Fallback? Stop B, start A.
# 1
$ docker pull mysql:5.7.18
# 2
$ docker run -d
--name=mysql-new
-e MYSQL_ROOT_PASSWORD=mypassword
-v local-datadir:/var/lib/mysql
mysql:5.7.18
# 3
$ docker exec -it mysql-old mysqldump -uroot -p > dump.sql
# 4
$ docker kill --signal=TERM mysql-old
# 5
$ docker exec -it mysql-new mysql -uroot -p < dump.sql
# 6
$ docker exec -it mysql-new mysql_upgrade -uroot -p
# 7
$ docker rm -f mysqld-old
# 8
$ docker stop mysqld-new
$ docker start mysqld-old

MySQL Container - In-Place Upgrade
Copyright 2012 Severalnines AB
# 1
$ docker pull mysql:5.7.18
# 2
$ docker exec -it mysql-old mysql -uroot -p -e 'SET GLOBAL
innodb_fast_shutdown = 0'
# 3
$ docker kill --signal=TERM mysqld-old
# 4
$ docker run -d
--name mysql-new
--publish 3307:3306
-e MYSQL_ROOT_PASSWORD=mypassword
-v local-datadir:/var/lib/mysql
mysql:5.7.18
# 5
$ docker exec -it mysql-new mysql_upgrade -uroot -p
# 6
$ docker rm -f mysqld-old
# 7
$ docker stop mysqld-new
$ docker start mysqld-old
1. Pull the new image, M.
2. Set innodb_fast_shutdown=0 inside container.
3. Stop the container, A.
4. Start a new container, B with new image, M.
5. Run mysql_upgrade inside B.
6. OK? Remove A.
7. Fallback? Stop B, start A.

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
What's Next?

Copyright 2017 Severalnines AB
● Visit our booth for a free T-shirt and other goodies. While
stock lasts!
● Follow our MySQL on Docker blog series at
severalnines.com/blog
● Attend the MySQL on Docker webinar, Wednesday Sept
27th, 2017:
○ 09:00 BST / 10:00 CEST
○ 09:00 PST (US) / 12:00 EST (US)
● Check out ClusterControl Docker image.
What's Next

Copyright 2017 Severalnines AB
Copyright 2017 Severalnines AB
Q & A