Docker in production, for real! The Yuzu startup, helped by Vixns, chose to have a docker infrastructure with Mesos/Marathon/Consul. From the development environment to our prod monitoring, we share our mistakes, successes, workflows and tools.

1. How we dockerized a startup?
#docker-marseille

2. Jonathan Martin
@john_iw2
Stéphane Cottin
@vixns

3. #architecture #hosting #opsdev #libre #r&d
#techtrends #IoT #artprojects

4. #startup #newyork #marseille #r&d #frenchtech
#ecommerce #bpi #shake #network #bigdata
#dockerlover #opensourcelover #hashtaglover

5. The first global cross-selling
network between eMerchants
“ “

6. Yuzu By Example

7. At the beginning

8. months to do a MVP
4

9. No time to reinvent the wheel

10. Symfony - Java - MongoDB

11. PRODUCTION
Web BigData
- 2 physical servers
- Multiple VMs
- Chef & Chef server
- 2 physical servers
- Stack Cloudera (Hadoop/Spark/...)

12. Six months later...

13. A lot of technos joined
Yuzu during the MVP
Welcome to nodejs, scala, elasticsearch, hbase, redis,
kafka, couchbase,...

14. They lived happily
ever after

15. It doesn’t work in production…
… that’s works fine in my local env
“
“

16. This Lib isn’t in the right version...
It’s slow… why??
Where are my logs...
“
“

17. Time to look around...

18. Be kind, rewind
Spring cleaning
Microservices / 12 factors
Resources isolation
Improved security
Continuous deployment
Start-up compliant workflow
Orchestration & supervision

19. AND
WINNERS ARE ...THE

20. Docker + Mesos + Marathon + Consul

21. {
"container": {
"type": "DOCKER",
"docker": {
"image": "redis:2.6.17",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 6379,
"hostPort": 0,
"protocol": "tcp"
}
]
}
},
"id": "redis",
"instances": 1,
"cpus": 0.5,
"mem": 1024,
"healthChecks": [
{
"protocol": "TCP",
"portIndex": 0
}
]
}
MARATHON
wants
JSON

23. Dynamic Configuration
Docker
Registrator
Consul
Consul-template

24. Logs, metrics, alerting...

25. Collect everything collectd / fluentd / rsyslog
Convert to async messages kafka
Filter and keep valuable data riemann / elasticsearch / influxdb
Generate realtime alerts riemann
Addictive dashboards kibana / grafana / riemann-dash

26. CHRONOS
CRON for mesos

28. Our workflow

29. Git flow
Jenkins DinD
Private Registry Docker
Doploy
Marathon & Chronos
Boot2Docker + Parallels

30. Don’t / Do

31. use tag "latest" on production
use explicit version tags for your
docker images
DON’T

32. use data containers for valuable data
storage layer may crash / change
easy to forget when cleaning unused
containers / images
DON’T

33. use shared storage (glusterfs, nfs, ...) for your codebase,
working dirs, config files or sessions
sloooooooooow
generate scary lock errors and timeouts
fscache crash full systems easily
DON’T

34. tag your custom images with the VCS commit hash
● makes your workflow better (same version tag in VCS
and images)
● prevent useless image rebuilds ( speedup deploys )
● easy way to know exactly which code version is running,
even if the tag was changed / deleted in the VCS
DO

35. avoid shared filesystems
● put all versioned data in containers
● use external object storage for user files (S3, ceph, swift ,...)
● use database / memcached / couchbase for sessions
● use templates to generate local config files (consul-template ,
confd)
DO

36. Use docker independant storage for critical data
● lvm is your friend (unless you use and saturate thin volume
metadata, use thin volumes with care)
● use storage plugins with docker >= 1.9.0 (convoy ?)
● redundant backups saves lives
DO

37. Mistaking
is
learning

38. - we tried to put the chicken in the egg
- generate config files to shared storage
- dns for discovery
- use mongo replicaset without sharding
- ...

39. Switch from registrator to mesos-consul
Remove the lasts SPOFs
Move user files to S3
Use the docker storage/network plugins
XDCC
Improve logs & metrics filtering and alerting

40. PHP/Symfony ReactJS Scala Go Clojure NodeJS Spark
Langages/technos
MongoDB Redis CouchBase Hdfs/HBase ElasticSearch
Databases
InfluxDB Sentry Statsd Grafana Riemann Collectd Kibana Fluentd
Monitoring
Gearman Kafka
Queue
Jenkins Doploy
Continuous Integration & Delivery
Route53 S3 CloudFront
AWS
Debian Docker Mesos Registrator Zookeeper Consul Marathon Chronos
SYSTEM

41. BigData

43. Mesos and the yellow elephant
- yarn, mesos, or both ? (myriad)
- hdfs/hbase on mesos, not so simple (still testing)
- kafka is awesome (we run it with docker)
- spark on mesos
- elasticsearch, born to cluster, also in docker

44. 2015
Other mesos/docker stories
IPROFS
A large scale
php/drupal
worldwide social
app for “Institut
français”
ARTE.TV
We are migrating
all their vod and
svod services to
mesos/docker
(java apps)
VIXNS
A mesos/docker
cluster is collecting
all logs and metrics
from hundreds of
servers

45. Conclusion

46. Questions?
@john_iw2 @vixns