Slides from my DockerCon EU 2017 Talk. Find the abstract below: "In this talk, we'll discover how Docker comes to the rescue of the Ops Team, while rebuilding from scratch our monitoring infrastructure. We'll start by quickly describing the challenges, to focus on why and how using docker saved the project. From fixing dependencies and isolation issues, implementing rolling upgrades and new features hot addition, to building a completely modular, scalable and resilient infrastructure, we'll talk about why CI/CD workflows, docker tooling and Docker Swarm were the key to success."

1. Docker to the rescue
of an Ops Team
Rachid
Zarouali
C.I.O Synolia
Twitter / Slack : Xinity
rachid.zarouali@synolia.com

2. Agenda
This talk is about:
● Monitoring (a bit)
● Private Registry
● CI/CD
● Security
● Docker experience

3. Once upon a time
An ops team starts a new project

4. Rebuild everything !
June 2014
A monitoring system from scratch
“Microservices” oriented
Replaceable parts

5. Some rules first !
Simple
Efficient
Extendable
Python based

6. Components
Collectd
Collectd proxy
Graphite
Grafana
Cabot (alerting)

7. Test your might !
Python 2.6 along with python 2.7
Different version of “some” libraries
Whisper backend (I/O storm)
Upstream repositories issues

8. Docker to the rescue
Save our project

9. Docker ? way too soon !
Barely no skills
Used only to do some testing
Pretty serious concerns

10. Ok let’s gamble !
Grow our docker fu
Write some Dockerfiles
Build images locally
Spawn PoC platform

11. …..
RUN echo "deb http://mirror.debian.ikoula.com/debian wheezy-backports main" >> /etc/apt/sources.list
RUN apt-get -qq update
RUN apt-get -qqy dist-upgrade
RUN apt-get -qqy --force-yes install vim python-cairo gunicorn supervisor (...)
RUN pip install whitenoise txamqp whisper==0.9.13 carbonate
RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==0.9.13
RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp"
graphite-web==0.9.13
ADD conf/nginx.conf /etc/nginx/nginx.conf
ADD conf/supervisord.conf /etc/supervisor/conf.d/grafana.conf
ADD initial_data.json /var/lib/graphite/webapp/graphite/initial_data.json
ADD conf/local_settings.py /var/lib/graphite/webapp/graphite/local_settings.py
ADD conf/carbon.conf /var/lib/graphite/conf/carbon.conf
ADD conf/storage-schemas.conf /var/lib/graphite/conf/storage-schemas.conf
RUN mkdir -p /var/lib/graphite/storage/whisper
RUN touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/index
RUN chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisper
RUN python /var/lib/graphite/webapp/graphite/manage.py syncdb --noinput --pythonpath=/var/lib/graphite/webapp/graphite
--settings=settings
RUN chmod 0664 /var/lib/graphite/storage/graphite.db
RUN chown -R www-data /var/lib/graphite/storage
…..

12. WHAT ???

13. Container = OS …. Wait !
Too many layers ( 121+ layer issue)
Build time …. (20 to 30 minutes at best)
Huge Images (800+ Mo)
Unnecessary tools and libs

14. Bye Bye !!!!

15. We can do better !
Apply best Practices (@abbyfuller)
Implement simple CI/CD
Dockerfile Linting
Build a private registry
Deal with security concerns

16. …
RUN echo "APT::Install-Recommends false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::Install-Suggests false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::AutoRemove::RecommendsImportant false;" >> /etc/apt/apt.conf.d/00recommends
&& echo "APT::AutoRemove::SuggestsImportant false;" >> /etc/apt/apt.conf.d/00recommends
ENV DEBIAN_FRONTEND noninteractive
ENV GRAPHITE_VERS 0.9.13
RUN apt-get -qqy update
&& apt-get -qqy install python-cairo gunicorn git python2.7-dev wget ca-certificates python-flup expect sqlite3 libcairo2
libcairo2-dev pkg-config nodejs sqlite3 memcached python-ldap make gcc libffi-dev
RUN wget https://bootstrap.pypa.io/get-pip.py
&& python get-pip.py
&& pip install --no-cache-dir --upgrade setuptools
&& pip install --no-cache-dir django django-admin-tools
&& pip install --no-cache-dir whitenoise txamqp whisper==${GRAPHITE_VERS} carbonate
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib"
carbon==${GRAPHITE_VERS}
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp"
graphite-web==${GRAPHITE_VERS}
RUN apt-get purge gcc make python2.7-dev libcairo2-dev libffi-dev python2.7-dev pkg-config -qqy
&& apt-get clean
&& apt-get autoremove -qqy
&& rm -rf /root/.cache /var/lib/apt/lists/* /tmp/* /var/tmp/*
...

17. Best practices :)
Few image layers ( < 20 )
Small image (~ 400Mo)
Lower footprint (100Mo)
Faster build time (~5Min)

18. We did it , we did it Yeah !

19. CI/CD Diagram

20. CI/CD Recipe
build:
image:
registry.synolia.com/synolia/dockerunitest:latest
publish:
docker:
repo: synomon_datastor
tag: $${BRANCH/master/latest}
file: Dockerfile
insecure: true
when:
repo: synolia/systeam-monitoring_datastor
branch: [develop, master]
notify:
hipchat:
from: "synoci"
room_id_or_name: "$$ROOM_ID"
auth_token: "$$AUTH_TOKEN"
notify: true
when:
success: false
failure: true
Dockerfile Linting
Build
Push
Notify (fail only)

21. Docker to the rescue II
The return of the hero moby

22. This isn’t over yet !
Docker udp issues
Tricky iptable filtering
Unstable data volume
Configuration management

23. Round 2: FIGHT !
Metric proxy (Collectd) on the
host!!
Simplify iptables rules
Mount directories (metrics)

24. A new path opens
To a brighter future

25. Epic loots !
No more dependency issues
Replaceable and movable parts
Greater security level
Clustering ready (Swarm/K8S)

26. Lessons learned
Caution when using udp IPv4
Config files out of the container
Don’t use env variables (security)
Use (abuse) automation

27. What’s next ?
Greater Docker challenges

28. Many rooms to grow
Reduce (even more) Image size
Sign Images (notary to the rescue)
Vulnerability scanning
Implement rolling upgrades

29. New docker based projects
Migrate Development platform (2015)
Swarm clustering (*)
Full scale Docker (*)
(*) Work In Progress

30. Thank You DockerCon !
PS: don’t forget to rate my talk :)