Presentation from DockerCon EU '17 about how Aurea achieved over 50% cost reduction using Docker and about two major technical obstacles we had when dockerizing legacy applications.

1. Dockerizing Aurea
Lukasz Piatkowski, PhD
Chief Docker Architect
Matias Lespiau
Soft. Eng. Manager

2. Agenda
Dockerizing Aurea:
- Goals
- Results
- Technical challenges

4. Dockerization at Aurea
Goals
● #1 - Decrease computing expenses by
consolidation and simplification
● #2 - Improve Ops team productivity
through standardization

5. 1 app, 1 host, 2 CPUs
1GB RAM
4x average to peak
utilization
500 containers, 1 host,
128 CPUs, 2TB RAM
1.2x average to peak
utilization
Goal #1 - decrease computing expenses

6. Goal #2 - standardize Ops interfaces for operating products
Node app
PHP app
Python app
Java app
Application Runbooks
Dockerfiles Docker
compose
files
Docker CLI commands
Standard interface

7. Dockerization at Aurea
● 1 year results
○ Replaced 2000+ VMs with 1900+ containers
○ Decrease infrastructure costs from 13M to 6M (53%)
○ Utilization increased from 5% to 72%

8. Ok, what’s edgy about
our case?

9. Dockerization at Aurea
● Focus on the basics
○ Tried out swarm, ECS and plain docker with EE basic
license.
○ Teams using plain docker reached our main goal
faster
■ Simpler to onboard Ops and Eng
■ Avoids re engineering apps

10. Dockerization at Aurea
● Simplify instead of lift and shift
○ Centralize basic services like databases and reverse
proxies
○ Legacy HA setups added complexity and very little
value in return (less than 1% contribution to uptime)

11. Dense consolidation
● 2000 instances -> 7 docker hosts
● Benefits:
○ Higher utilization
○ Simpler to manage
● Don’t do this at home:
○ Issues when running more than 100 containers per
node
○ Interested? Hallway track!

12. Technical challenges

13. Copyright: Daniel Stori - @turnoff_us

14. TO DEL - Monitoring tips
● Fetching metrics is a heavy API call
○ Always make it once, store in DB and fetch from there
○ Always use container and host level monitoring in
production
● Useful tools
○ Reporters: cadvisor or telegraf
○ DB, alarms and graphs: prometheus, alertmanager
and grafana

15. Performance and
resource sharing

16. Performance & host sharing
● Learn performance debugging tools
○ Great talk by Brendan Gregg: https://goo.gl/UW2mHV

17. Performance & host sharing
● The biggest enemy: a noisy neighbour
● Fight him with resource limits:
○ CPU: --cpu-period, --cpu-quota, --cpus
○ Memory: --memory, --memory-swap (turn on accounting!)
○ IO: --device-[read|write]-bps, --device-[read|write]-iops

18. Performance - lessons learned
● Containers are not Virtual Machines
○ cgroups are not hypervisor
○ Remember the JVM
● Always set container’s resource limits
● Always label your containers
○ Owners info and container’s importance
● But how to make users comply?

19. Configuration
compliance

20. Docker enforcer
● A tool to run validation rules against
containers and stop ‘bad ones’
● https://github.com/piontec/docker-enforcer

21. Docker enforcer - rules
“Dear users,
We have created a nice big disk for your
containers’ data at /opt/big. Please use
this location for any docker volumes.
Admins”

22. Demo

23. Docker enforcer - rules
● “No worries!”
○ docker run -d -v /optbig:/my-mysql

24. Docker enforcer - rules
● “No worries!”
○ docker run -d -v /optbig:/my mysql
● Rules written in python, as a series of
lambdas applied to each container
rules = [{
"name": "uses valid dirs for docker volumes",
"rule": lambda c: False if not 'Binds' in c.params['HostConfig'] or c.params['HostConfig']['Binds'] is None
else any([not b.startswith("/opt/big") for b in c.params['HostConfig']['Binds']])
}]

25. Docker enforcer - rules
“Dear users,
Please always make sure your
containers are running with CPU and
memory resource limits.
Admins”

26. Demo

27. Docker enforcer - rules
● “No worries!”
○ docker run -d lets-mine-bitcoin:latest

28. Docker enforcer - rules
● “No worries!”
○ docker run -d lets-mine-bitcoin:latest
● Rule
{ "name": "must use less than 1 GiB of memory",
"rule": lambda c: c.params['HostConfig']['Memory'] == 0 or c.params['HostConfig']['Memory'] > 1074000000 },
{ "name": "must use less than 2 CPU cores",
"rule": lambda c: c.params['HostConfig']['CpuQuota'] == 0
or (int(c.params['HostConfig']['CpuQuota']) /
(100000 if c.params['HostConfig']['CpuPeriod'] == 0 else int(c.params['HostConfig']['CpuPeriod']))
) > 2}

29. Docker enforcer - rules
“Dear users,
Please be careful when you assign ports
to your containers. Don’t use ports
below 1024.
Admins”

30. Demo

31. Docker enforcer - rules
● “No worries!”
docker run -d -p 22:22 my-super-ssh

32. Docker enforcer - rules
● docker run -d -p 22:22 my-super-ssh
● rules = [{"name": "can't use ports below 1024 on the default IP",
"rule": lambda c: False if 'PortBindings' not in c.params['HostConfig']
or c.params['HostConfig']['PortBindings'] is None
else check_ports_on_default(c.params['HostConfig']['PortBindings'])
}]
def check_ports_on_default(bindings):
for pbs in bindings.values():
if pbs is None:
continue
if any(pb['HostPort'] != '' and int(pb['HostPort'].replace('/udp', '')) < 1024 for pb in pbs):
return True
return False

33. Docker enforcer - status API
● http://localhost:8888/
{
"last_full_check_run_time": "0:00:00.404284",
"detections":
[
{"id": "4c5..12b", "name": "/test", "violated_rule": "must have CPU limit", "count": 1,
"last_timestamp": "2017-09-17T13:55:04.851144" }
]}
● http://localhost:8888/config
{
"cache_params": true,
"disable_metrics": true, …
● http://localhost:8888/rules
rules = [...]

34. Docker enforcer - CLI call
$ docker run -d alpine true
docker: Error response from daemon: authorization
denied by plugin enforcer: must have CPU limit.
See 'docker run --help'.

35. Networking for legacy
applications

36. Networking - legacy
● Not HTTP+JSON microservices
○ Old friends: FTP, SMTP, SIP, …
● New requirements
○ Individual IPs for containers, but from different
subnets and preserving external (AWS VPC) IP
○ Exposing massive number of ports (SIP)

42. Networking - per container IPs
# interface
ip addr add 10.10.0.2/24 dev eth1
ip route add default via 10.10.0.1 dev eth1 table 101
ip route add 10.10.0.0/24 dev eth1 src 10.10.0.2 table 101
ip rule add from 10.10.0.2 lookup 101
# container
ip r a 172.17.0.0/16 dev docker0 tab 101
ip rule add from 172.17.0.20 lookup 101
iptables -t nat -I POSTROUTING -s 172.17.0.20 -j SNAT --to-source 10.10.0.2

43. Networking - port storm
● What will this command do?
○ docker run -d -p 2000-3000:2000-3000 alpine sh

44. Networking - port storm
● What will this command do?
○ docker run -d -p 2000-3000:2000-3000 alpine sh
● Don’t expose single ports at all, translate
whole IP in one go
○ docker run -d alpine sh
○ iptables -t nat -I PREROUTING -i eth1
-d 10.10.0.3 -j DNAT
--to-destination 172.17.0.20

45. [TO DEL] Dockerization
processDiscovery
Image
Validation
cutover

46. [TO DEL] Discovery

47. [TO DEL] Process management
Diagram - how signal handling works

48. [TO DEL] Image Quality
● Process management
○ Init / Process manager
○ Avoid process in containers getting in D state (check
for remote filesystem, etc)
○ Signal handling
○ Handle container main process restart/termination

49. [TO DEL] Building Docker
Images● Java
○ JVM limits CPU threads, memory

50. Recap
● Outcomes
○ Increase utilization from 5% to 72%
○ Decrease infrastructure costs from 13M to 6M
● Main challenges
○ Noisy neighbours
○ Configuration compliance
○ Networking

51. Roadmap
● Dockerize everything, our goal is to have 0 VMs out of our
CaaS platform
● New platform for stateless containers
○ Orchestration
○ Multi AZ on AWS Spot
● Invest in re-engeering non-dockerizable apps to make
them dockerizable and in dockerized app to make them
cloud enabled.

52. We’re hiring?
Hallway tracks:
● Docker Enforcer
● Running 100+ containers per node
Q & A?