Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a PaaS with Docker, Consul and Python

2,066 views

Published on

Here I tell about my experience in building http://tarantool.io

Published in: Technology
  • Be the first to comment

Building a PaaS with Docker, Consul and Python

  1. 1. Building a PaaS with Docker, Consul and Python by Konstantin Nazarov
  2. 2. I work in Tarantool And I've built a PaaS this way
  3. 3. I'll share how to build one • It is simple • The experience is highly portable • You can start small and grow iteratively • Fits your requirements • The tech stack is widely known
  4. 4. 1. docker host 2. docker host 3. orchestrator 4. consul 5. web UI
  5. 5. Why?
  6. 6. Why build your own PaaS? • If you develop your own product and sell it • Small initial investment to solve lifecycle problems • A way to fit your requirements exactly • Keep your operations lean • To enable fast experiments
  7. 7. When not to build it • When you run off-she-shelf software • If you have large monolithic services
  8. 8. What a PaaS should do • Run the code you give to it • Abstract away the OS
  9. 9. Let's build it
  10. 10. • We will build progressively • On each step there will be a working system Let's build it
  11. 11. A few building blocks
  12. 12. 1. Python • Because it is simple • There are bindings for almost all existing stuff
  13. 13. 2. Docker • Has remote HTTP API to run stuff • Has convenient packaging format
  14. 14. 2. Docker: the good • Simple • Well documented
  15. 15. 2. Docker: the bad • Bugs • Weak networking
  16. 16. 2. Docker: Alternatives • fabric (yes, as an RPC) • gearman • nomad
  17. 17. 3. Consul • Fault-tolerant key-value storage • Service registry with active checks • Easily deployed
  18. 18. 3. Consul: alternatives • etcd • zookeeper
  19. 19. MK1: Smart command-line client to Docker/Consul
  20. 20. Why? • quick to implement • very high value compared to effort
  21. 21. What it should do • run • inspect • upgrade • rm • ps
  22. 22. Example usage $ mypaas run git://gitserver/project.git v1.2 fddf3f $ mypaas upgrade fddf3f v1.3 $ mypaas rm fddf3f
  23. 23. How • Docker API is exposed on physical servers • Physical servers are registered in consul • CLI connects to a known consul host • Docker API is used to build app container "in place"
  24. 24. How to choose physical nodes • By maximum memory requirements • By conventional CPU units • By number of services running
  25. 25. Upgrading versions • Stop the running container • Start new container inheriting volumes from the old • On success, remove old container • On failure, restart old container
  26. 26. Result • A working PaaS
  27. 27. Improving MK1: health checks • Consul API can be used to register health checks • Consul can run commands in docker containers • The CLI can poll consul for service statuses
  28. 28. Wiring things together via network
  29. 29. Docker networking hell • There are overlay networks (UDP encapsulation) • And macvlan (adding new MAC addresses to eth) • And openvswitch with DPDK • And god knows what else (BGP routing anyone?)
  30. 30. Let's use plain bridges br0
  31. 31. Let's use plain bridges • Docker IPAM doesn't know about other nodes • IP conflicts are possible • So we have to use our own IPAM • Write allocated IPs to consul KV • Set IPs explicitly
  32. 32. IPAM
  33. 33. MK2: running as a service
  34. 34. Why? • Limiting access to production servers • Active monitoring of business logic • Concurrent access
  35. 35. How? • flask • flask-restful • gevent
  36. 36. Why these tools? • Everything fits in one app • Orchestration code is easier to write in async mode • Quick to implement • Your CLI becomes an HTTP API client
  37. 37. Separate state • Orchestrator itself should be stateless • Orchestrator should show the system overview • You probably need basic auth at this step
  38. 38. MK3: admin UI
  39. 39. Why? • Easier to manage • Easier to debug
  40. 40. Example
  41. 41. How? • flask templates • bootstrap
  42. 42. What to have here • Overview of physical servers and their state • Overview of your services and their state • CRUD
  43. 43. MK4: delayed tasks and active checks
  44. 44. Why? • Distributed cron • Data extraction • Backups
  45. 45. Tools • gevent • Docker exec API
  46. 46. How? • On server start, spawn a worker fiber • In the fiber, poll consul and run your code • Or start worker fibers on demand • Send notification emails upon completion
  47. 47. MK5: metrics and time series
  48. 48. Why? • Historical data matters for problem solving • See how well your new code is behaving over time
  49. 49. How? • install prometheus and hook it up to consul • use prometheus API to query aggregates
  50. 50. Exporting metrics to prometheus • Either add support to your service • Prometheus protocol is very simple! • Or collect via the orchestrator
  51. 51. Recap
  52. 52. Recap • docker-python - running stuff • python-consul - storing information about running stuff • flask - serving admin UI • flask-restful - providing HTTP API • bootstrap - making your web page less ugly • gevent - running delayed tasks and async code • prometheus - storing time series
  53. 53. Thanks! Konstantin Nazarov mail@kn.am Building orchestrators is not that hard. @racktear http://bit.ly/paas-bom

×