1.
Docker Tallinn @Pipedrive
Renno Reinurm
2016

2.
How application
performance
requirements impacted
the (r)evolution of the
Docker infrastructure at
Pipedrive.

3.
Visibility
Management
Interface
http://www.slideshare.net/rreinurm/living-with-mi
croservices-at-pipedrive

4.
Requirement 1
You shall not build/test/deploy
Docker container
over 5 minutes
Based on: xkcd.com

5.
Optimized Dockerfiles
First iteration, 14 layers:
FROM node
ENV SERVICE_NAME=company-statistics
ENV SERVICE_DESC="Company statistics"
ENV SERVICE_TAGS=statistics
ENV NODE_ENV=live
ENV SERVICE_CHECK_HTTP=/health
ENV SERVICE_CHECK_INTERVAL=10s
ENV SERVICE_CHECK_TIMEOUT=5s
ENV DOCKER=1
EXPOSE 8000
WORKDIR /src
USER nobody
COPY . /src/
CMD ["node", "."]
Optimized, 8 layers and ordered:
FROM node:6-slim
ENV SERVICE_NAME=company-statistics
SERVICE_DESC="Company statistics"
SERVICE_TAGS=statistics
NODE_ENV=live
SERVICE_CHECK_HTTP=/v1/companyStatistics/health
SERVICE_CHECK_INTERVAL=10s
SERVICE_CHECK_TIMEOUT=5s
DOCKER=1
EXPOSE 8000
WORKDIR /src
USER nobody
CMD ["node", "."]
COPY libraries/ /src/
COPY src/ /src/

6.
https://youtu.be/X_q2l8hotAc?t=365

7.
Deployment process optimizations
NB! https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
Replacement of Devicemapper to AUFS reduced deployment process time 10x.
There are still improvements possible:
● Handle Linux signals
● Parallel rolling updates

8.
Requirement 2
Consumers shall connect
only to healthy services

9.
Beware the service discovery corruption
● Always enable health checks
● Use unique health checks
SERVICE_CHECK_HTTP=/health
vs
SERVICE_CHECK_HTTP=/v1/companyStatistics/health

10.
Requirement 3
Container shall handle 10 000
connections and constant
high load.

11.
https://youtu.be/PivpCKEiQOQ

12.
Issues
● Linux kernel 3.13
● Fluentd logging agent
● Graylog logging driver
● Kernel sysctl parameters
● Swap usage
● PEBKAC
○ "net.ipv4.ip_forward" => 0
● WARNING: No memory limit support
● WARNING: No swap limit support
● WARNING: No kernel memory limit support
● WARNING: No oom kill disable support
● WARNING: No cpu cfs quota support
● WARNING: No cpu cfs period support

13.
Docker hosts tuning
● Linux kernel 4.x
● Removed Swap
● Dropped fluentd driver
● Docker Engine upgrade/patches
● Kernel performance tuning
http://www.devconf.cz/files/slides2015/friday/Performance%20Tuning%20of%2
0Docker%20and%20RHEL%20Atomic.pdf

14.
Service risk mitigation
● Number of nodes in cluster
○ If in doubt increase the number
● Spreading policies
● Multiple instances
● Memory limitations
● Healing policies
○ Autorestart
○ Reschedule

15.
Security mitigation
● TLS authentication
● Minimal & official Docker images
● Signed Docker images
● User Namespace Remapping
● PID limits
● Capabilities
● Seccomp profiles
● AppArmor/SELinux profiles
https://github.com/riyazdf/dockercon-workshop

16.
Recommendations for going
Live with Docker
● You still need to take care of OS
● Read Github issues
● Read from the source
● Keep it up to date
● (Performance) Test it

17.
Thank you!
Give me your feedback @rreinurm