[NYC Meetup] Docker at Nuxeo

2,383 views

Published on

In this talk, Damien describes the infrastructure Nuxeo has built around Docker containers, which is mainly based on CoreOS and Docker, and how it provides a way to generically run applications not only on a single host, but across a whole cluster of hosts. The resulting architecture can be used to implement a PaaS approach for any application.

Published in: Technology, Business

[NYC Meetup] Docker at Nuxeo

  1. 1. Docker at Nuxeo
  2. 2. Who we are
  3. 3. What is Nuxeo platform • A platform to develop you own content apps • Highly pluggable thru extension points • Model you content with Nuxeo Studio • Point and click UI to define your business model • with hot-reload
  4. 4. Nuxeo tech. stack • Java stack : Seam + JSF for Nuxeo DM and DAM • Rest API to build your JS Apps • Nuxeo Studio (Saas) to model your business • Thru data model (doc types, schemas) • Thru Automation chains
  5. 5. Nuxeo in the cloud ?
  6. 6. Goals • Customers ask to host their app • Provide quick access to test instances (onboarding) • Provide onDemand dev or CI instances
  7. 7. Goals • Customers ask to host their app • Provide quick access to test instances (onboarding) • Provide onDemand dev or CI instances This is the nuxeo.io promise
  8. 8. First attempt
  9. 9. Manual OnDemand • Based on AWS CloudFormation • Manually deployed • Works but : • not scalable • manual admin & monitoring • difficult upgrade process
  10. 10. Second attempt
  11. 11. AWS Automation • Provisionning may be automated • Working mockup (python, celery, boto) but… • Strong adherence to AWS • difficult to test • slow provisionning (one EC2 instance per Nuxeo instance) • upgrade process still difficult • scaling out is difficult • sensible to failures
  12. 12. This leads to our challenge
  13. 13. Current challenge • Build a scalable infrastructure • Quick provisioning (less than a minute) • Easy setup (no admin needed) • Easy upgrade process • Automated healing • Easy to test
  14. 14. How Docker solves some of these issues ?
  15. 15. Quick provisioning • We don’t have to start a server • No VM overhead • Start time is Nuxeo start time
  16. 16. Data free containers • Docker container doesn’t hold data • Upgrading is just using a new version of the image • Healing means killing and restarting Nuxeo v5.9.3 S3Postgres
  17. 17. Data free containers • Docker container doesn’t hold data • Upgrading is just using a new version of the image • Healing means killing and restarting S3Postgres
  18. 18. Data free containers • Docker container doesn’t hold data • Upgrading is just using a new version of the image • Healing means killing and restarting S3Postgres Nuxeo v5.9.4
  19. 19. Several containers per server • Scaling is just a matter of starting several containers • But means infinite CPU / RAM and IO Nuxeo.io server io container NXIO-0001 io container NXIO-0002 io container NXIO-0003 io container NXIO-XXXX …
  20. 20. Challenge status • Build a scalable infrastructure • Quick provisioning (less than a minute) • Easy setup (no admin needed) • Easy upgrade process • Automated healing • Easy to test
  21. 21. How to solve the other issues ? … with a little help from my friends
  22. 22. The infinite resource computer…
  23. 23. … is a cluster
  24. 24. Docker is not sufficient • Docker handles processes at the OS level • On the cluster level we need a companion • To load balance the processes • To check the statuses of the processes • We need a docker ps at the cluster level
  25. 25. Introducing CoreOS • A minimal Linux distribution (small footprint) • Designed to handle cluster • Providing cluster tooling • Docker • etcd • fleet https://coreos.com/
  26. 26. etcd : service discovery • etcd is • small distributed key value store • keys may have TTL • using Raft protocol to elect cluster leader • suitable for : • shared configuration • service discovery
  27. 27. fleet : clustered init system • fleet is : • an init system at the cluster level • compatible with systemd unit files • suitable for : • launching docker containers • distributing load across cluster • get the status of the jobs
  28. 28. fleet demo • Deploy our nuxeo container in the cluster
  29. 29. Challenge status • Build a scalable infrastructure • Quick provisioning (less than a minute) • Easy setup (no admin needed) • Easy upgrade process • Automated healing • Easy to test
  30. 30. What are the admin tasks ? • Launch the fleetctl command • Setup DNS and reverse proxies • Link container to external services (S3, Postgres etc…)
  31. 31. DNS and reverse proxy • docker start containers and link host port to container port • The cluster has to redirect calls to mycustomDNS.io to the running container meaning knowing • the ip of the host • the port on the host
  32. 32. Introducing Gogeta ! /services/NXIO-0001/location:{host:172.31.51.1,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001 Given this etcd configuration
  33. 33. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  34. 34. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  35. 35. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  36. 36. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  37. 37. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  38. 38. Cluster CoreOS 172.31.51.20 etcd io container NXIO-0098 systemd CoreOS 172.31.51.1 etcd io container NXIO-0001 Gogeta systemd … … … … Fleet … Gogeta … Introducing Gogeta Round robin load Balancer etcd ! /services/NXIO-0098/location:{host:172.31.51.20,port:49153}!                 /status/current:started!                       /expected:started! !! ! ! ! ! /alive:1! /domains/mydomain.nuxeo.io/type:iocontainer!                           /value:NXIO-0001
  39. 39. Introducing Gogeta • Easy to start as a docker container ! ! • Written in Go • Available at : http://github.com/nuxeo/gogeta /usr/bin/docker run --rm --name gogeta -p 7777:7777 nuxeo/gogeta
  40. 40. Application startup unit ! [Unit]! Description=Sample startup script! ! [Service]! ExecStartPre= Set starting status in etcd! ExecStart= Launch docker container! ExecStartPost= Wait container to be launched and set app location in etcd! ExecStop= Stop docker container! ExecStopPost= Set stopped status in etcd!
  41. 41. Linking services • Docker recommend to use ambassadors to link services • Service may be hosted on other servers or outside of the cluster • We’ve implemented dynamic ambassadors • http://coreos.com/blog/docker-dynamic-ambassador- powered-by-etcd/
  42. 42. Cluster CoreOS 172.31.51.20 etcd Postgres systemd CoreOS 172.31.51.1 etcd io-container NXIO-0001 systemd … … … … Fleet postgres-amb Dynamic ambassador etcd ! /services/postgres/location:{host:172.31.51.20,port:49159}! https://github.com/nuxeo/etcd-netfw
  43. 43. Cluster CoreOS 172.31.51.20 etcd Postgres systemd CoreOS 172.31.51.1 etcd io-container NXIO-0001 systemd … … … … Fleet postgres-amb Dynamic ambassador etcd ! /services/postgres/location:{host:172.31.51.20,port:49159}! https://github.com/nuxeo/etcd-netfw
  44. 44. Cluster CoreOS 172.31.51.20 etcd Postgres systemd CoreOS 172.31.51.1 etcd io-container NXIO-0001 systemd … … … … Fleet postgres-amb Dynamic ambassador etcd ! /services/postgres/location:{host:172.31.51.20,port:49159}! https://github.com/nuxeo/etcd-netfw
  45. 45. Dynamic ambassador • Very easy to start ! • Uses https://github.com/nuxeo/etcd-netfw docker run --rm --name postgres-amb -P nuxeo/service-amb postgres-service
  46. 46. Launching commands • A manager web app now launches commands : • sets up basic etcd values (like domain to service resolution) • launches fleet commands (start & stop apps) • polls statuses
  47. 47. Challenge status • Build a scalable infrastructure • Quick provisioning (less than a minute) • Easy setup (no admin needed) • Easy upgrade process • Automated healing • Easy to test
  48. 48. Let’s resume
  49. 49. From nuxeo.io… • We’ve built a cluster infrastructure that : • knows how to deploy Docker containers (with fleet) • Expose those containers thru domain name resolution (with gogeta) • Link dependent services to the container (with etcd-netfw)
  50. 50. From nuxeo.io… • Given that a container should represent an application • Running this apps means : • Launch it with runtime parameters • Link it to its dependent services • Then we can build a metamodel around it
  51. 51. … name: io-container! version: 1.0! params:! - env_tech_id: ! - description: The id of the environment to run! - type:string! - clid:! - description: a registered connect client ID! - type:string! ! buildimage:! path: "./docker"! ! dependencies:! - postgres.service! - s3.service! !
  52. 52. … to arken • to be able to run any application ! • it will take care of : • Submitting the model to the cluster • Building the image if needed • Launching the container • Registering everything in etcd arkenctl start iocontainer.ark —domain myapp.mycluster.io
  53. 53. Arken status • We are extracting things from http://github.com/nuxeo/nuxeo.io • http://github.com/arkenio • home of gogeta and etcd • provide cluster init instruction • arken and arkenctl tools needs to be done • new opensource projects (not in nuxeo repo) • pull requests are welcome
  54. 54. Conclusion • Docker helped us to enter the micro containers world • Lots of new features : multi-repo, automatic /etc/hosts etc… • CoreOS provided us cluster deployment • Arken wants to provide a higher deployment model
  55. 55. Time for questions @damienmetzler

×