This document discusses Docker and its use for the Douban App Engine (DAE). It covers: - The history of adopting Docker for DAE applications from 2014 to 2016. - How DAE uses Docker to build and deploy over 400 application images across different environments. - Techniques used to optimize the Docker build process and reduce image sizes. - Integrating Docker with the DAE monitoring, logging, and maintenance systems.

1. Docker
@tianweidut
2016.7.9

2. •
• DAE-Docker
•
Agenda

3. Docker
• DAE App
• App docker image
• DAE Python2.7, Python3, Golang
• online/qaci/sandbox/prelease/dev
• container
• Docker

4. • 2014 Q4: DAE Docker App
• 2015 Q1: App
• 2015 Q2: App Docker
• 2015 Q4: QACI/Prelease Docker
• 2016.06 - : Mesos + Marathon

5. (1)
• (400+ APP)
• web, service(pidl/thrift)
• DAE App cron, daemon, mq worker, remoteshell, app-admin
• QACI container docker image
• Prelease/Sandbox container docker image
• Docker pidlproxy, dpark, scribe-sink, syslog-ng, thrift-router, dnasmasq ...

6. (2)
•
• 100 200-400 docker image build
• container 370
• 400+
• 7500 +
• Prelease/Sandbox 200+

7. Douban App Engine For Docker

8. • container web service container
container
• container
• Image DAE
• Dev QACI Prelease Sandbox Online

9. DAE
• PaaS Python Golang 400+ App
• Web, PIDL Service, Thrift Service, Cron, Daemon, MQ, Runscript, Admin, Dpark, QACI, Prelease, Sandbox...
App
MySQL, Beansdb, Memcahce, MQ, MFS, CDN, Redis...
Proj
A
Proj
B
Proj
C
$ dae create
$ dae serve
$ dae test
$ dae deploy
App Auto Scale Load Balancer
Node Balance Elastic Pool
Prelease Sandbox Profile

10. DAE-Docker
Cron
MQ Worker
DaemonDaemonDaemon
DaemonMQ Worker
M
S
S zookeeper
Mesos Marathon+
MQ: Beanstalkd
New!
Gunicorn:
web._admin
websocket- runscript
- remoteshell
- dbshell
online
offline
http(s)
LVS
Nginx(lb)
Nginx
Gateway
douban
service
thrift
dae_thrift_
router
pidl
pidlproxy-
client
pidlproxy-
server
Router
Layer
unix domain socket
Node
Gate
Docker Container
Gunicorn- sync/async
- multi-instances
- master-worker
puppet:conf
mfs:permdir
volume
monitor/sa
agent
bridge/host
App
Node
DAE API (python/golang)
Auto Scale Distributed Trace Orchestration
waylifebeansdb/mysql/mc/mq/redis/graphite...
Bridge/Deploy Logs
Resource
Layer
dev
Vagrant
Container
DAE-SDK
Docker Compose
Container
App-Test
Jenkins
mysql
redis mc
scribe beansdb
Prelease
Sandbox
Golang
Gevent daes

11. DAE CI/CD
Github/Code
pull request
merged
review
test + pylint
prelease/sandbox
Make Archive Image Build Image Push
App Node1
App Node2
App NodeN
docker pull
Build Static
Jenkins:QACI
app.yaml chk
hook trigger
deploy
Auto Stage0 Stage Deploy Deploy
Rolling Update cfg update container restart selftest
Image Cached
Rollback
No
Yes
update conf irc/slack/email notify end
Regi
stry

12. Docker

13. 1. : CentOS 6.5 vs Gentoo
• Centos 6.5 (2014.9)
• Docker ( Gentoo)
• Docker 1.2 Ubuntu/Centos
• CentOS6.5 Gentoo
• CentOS 6.5, Linux 2.6 -> Docker Daemon crash, Kernal Panic
• DAE / -> Gentoo binpkg CentOS RPM
• CentOS 2.6 Python 2.6 -> Python2.7 hack link
• Gentoo
• Gentoo Linux kernel 3.18.10(overlayfs merged)

14. 2. Docker (1)
• Docker 1.2.0 / 1.3.2 : ->
• Docker 1.4.1: production ready
• zombie container (container 1 docker daemon container
docker run container name DAE container
)
• zombie container -> kill docker run process -> rename linkgraphdb ->
cleanup container files

15. 2. Docker (2)
• Docker 1.5.0 DAE
• : push/pull Docker Registry V1
• image pre-build pre-pull (
)
• Docker 1.7.0: Docker
• : docker daemon docker daemon 7-8G container
• + docker daemon

16. 2. Docker (3)
• Docker 1.8.1 6
• 1 : container /etc/hosts net=host container /etc/hosts
/etc/hosts container
• container /etc/hosts container hosts /var/
lib/container/${cid}/hosts `mount --bind` container namespace
net=host :
• volume /var/run/nscd lookup cache
• /var/lib/docker/${cid}/hosts inode
container

17. 2. Docker (4)
• Docker 1.8.1 6
• 2 : volume-from data container data container volume
container app
• data container ready
• 3: docker registry v2 push image buffering disk,
image partent image id
• Docker 1.10.1
• : `Removal In Progress` container container name
• docker rename
• Docker 1.10.3 + docker registry v2 (2016.2 )
push/pull

18. 3. Docker Registry v2
• distribution: docker >1.10 push/pull
•
• docker.douban : build push
• dev.docker.douban :
• distribution api, DAE app
Image
• nginx upstream round robin push
image `digest invalid: provided digest did not
match uploaded content` mfs
layer size
• nginx ip_hash
• distribution sentry
load balancer
distribution distribution distribution
mossefs: Imageredis:cache

19. 4. Docker (1): dm vs overlay
• DeviceMapper
• 2014.09 ~ 2015.06
•
• build image 400+ App 100 build 200-400
build
•
• 1. docker daemon : ` --storage-opt dm.blkdiscard=false --storage-opt
dm.mountopt=nodiscard` --> kernel crash rm image pool
• 2. /var/lib/docker/devicemapper/data `direct-lvm` -> image
build 180s -> 120s

20. 4. Docker (2): dm vs overlay
• ref: http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/

21. 4. Docker (3): dm vs overlay
• Overlay
• 2015.06 - overlay2
•
• 1. AUFS Linux upstream AUFS
• 2. Overlay AUFS Linux 3.18 COW
docker image
•
• /var/lib/docker/overlay lvm Ext4
• build 180s -> 120s -> 100s

22. 4. Docker (4): dm vs overlay
• Overlay
• inode ; build image , `too many links` --> docker 1.12
+ kernel 4.0 + overlay2
•
• 1. base image hard link kernal links
• 2. linux ext4: `#define EXT4_LINK_MAX 65000`
• docker overlay
overlay container
- low-id: image id
- uppper: container
- work: overlay
- merged: container , mount-bind
image
overlay image
- image
- Linux 7 copy
regular file hard link.

23. 5. DAE-Docker Image (1)

24. 5. DAE-Docker Image (2)
• DAE-Python App
• sys-req.txt: gentoo binpkg
• pip-req.txt : python pip requirements
sys-req.txt
pip-req.txt

25. 6. Docker Build (1)
• 400+ App 100 build image 200-400 build
•
• 1. App Image 4GB+, app 180s , `maintainers:
xxx@douban.com` 40s
• 2. Dockerfile layer, merge RUN: 180s -> 40s
• 3. Dockerfile Docker container run + commit : 40s -> 8s

26. 6. Docker Build (2)
• 95% layer diff
layer diff docker build
Gentoo Image portage + runtime
portage
emerge binpkg, python app
run-build Dockerfile volume Docker run + volume + commit Image
: 40s -> 8s Image : 4.0GB -> 1.7GB

27. 6. Docker Build (3)
•
• 1. gentoo_with_portage portage docker
container
• 2. dockerfile docker run + commit docker build docker run volume
gentoo_with_portage container container portage emerge
binpkg
•
• 1. Image
• 2. gentoo_with_portage container
• 3. Dockerfile
• 4. volume-from container `mount --bind` container
gentoo_with_portage volume list
- /usr/portage/
- /etc/portage/
- /var/lib/layman/
- /var/cache/edb/
- /usr/libexec/gcc/
- /usr/share/include/
- /usr/x86_64-pc-linux-gnu/
- /lib64/modules/
- /lib64/firmware/
- /usr/lib64/gcc/x86_64-pc-linux-gnu/

28. 7.
• :
•
• 1. puppet
• 2. ossetup app image /var/dae/app/xxx/
• 3. volume container
• - volume inode container

29. 8. moosefs fuse (1)
• : DAE app permdir mfs( fuse )
mfs volume container mfs-
mount ( mfs master ) container mfs
`Transport endpoint is not connected` container
•
• 1. docker volume mfs inode 1
mfsmount mfs /proc/self/mountinfo
container /etc/self/mountinfo

30. 8. moosefs fuse (2)
•
• 2. docker volume linux `mount --bind `
• 3. Linux shared subtrees mount namespace mount umount
•
• 1. /fuse mount point : `mount --bind /fuse
/fuse`
• 2. /fuse mount mode shared mount `mount --make-shared /fuse`
• 3. mfs mount /fuse/mfs
• 4. container `-v /fuse:/fuse:shared` , shared /fuse mount
namespace container mount namespace mount/umount , mfsmount
container

31. 9. container
• : cadvisor(
) -> dae-monitor-agent
•
• 1. docker ps/inspect/exec container
docker dameon
docker daemon
• 2. docker api
• cgroups: cpu/mem/network/blkio
• docker container
• /var/lib/docker/container/${cid}/hostconfig.json
• /var/run/docker/*/${cid}/state.json
daemon-agent container-agent oom-agent
monitor data upload
redis mc
graphite
DAE Monitor Aggregator
API Icinga2
sentry
email
- version/info
- daemon process
- image storage
- dm cpu/mem
- mem/cpu/blkio
- network
- image version
- fuse mount
dae-monitor-agent

32. 10. container log
• DAE Python:
• monkey patch: logger handler stdout/stderr -> ScribeHandler , container log-
driver=none
• , DAE-go
• container syslog-driver + syslog-ng + scribe/mfs
app-container
--log-driver=syslog
--log-opt syslog-address=tcp://127.0.0.1:9614
--log-opt tag=‘scribe@dae_applog/container’
--log-opt tag='/fuse/dfs/permdirs/daegrade/test.log'
syslog-ng-container
- 9614 syslog-ng
- tag `scribe@` python
scribe mod for syslog-ng, log
scribe-buffer
- tag `/fuse` log mfs
mfs
scribe-buffer
scribe-sink

33. 11. maintainance
• image container(exited/stopped/dead)
• image docker rmi -v -v volume-from
• docker daemon `-H 0.0.0.0:4243` tls
• Docker Daemon daemontools supervisor docker daemon
• docker daemon