An Ensemble Core with Docker - Solving a Real Pain in the PaaS
•Download as PPTX, PDF•
0 likes•766 views
Docker by itself is only an engine powering containers. You need a containership to run it in production. CoreOS is a purpose-built containership that powers Docker conatiners, however, without higher-level orchestration managing hundreds or thousands of containers is not manageable. Ensemble is the answer for running containers at scale on top of CoreOS.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to An Ensemble Core with Docker - Solving a Real Pain in the PaaS
Similar to An Ensemble Core with Docker - Solving a Real Pain in the PaaS (20)
More from Erik Osterman
More from Erik Osterman (9)
Recently uploaded
Recently uploaded (20)
An Ensemble Core with Docker - Solving a Real Pain in the PaaS
- 1. Docker CoreOS Solving a Real Pain in the PaaS Ensemble +
- 2. THE PROBLEM I present to you...
- 3. Lot’s of Tools, Few Solutions ● No Master Blueprints or Distributions ● Docker is Not Enough ● Docker Hub is a Mess ● Fleet has Obvious Limitations ● Managing Systemd Units is Cumbersome ● Logging is a Real Pain ● Data Persistence is Necessary ● Kubernetes & Mesos are Overkill *
- 5. But You Run Fat Stacks... Redis, Memcache, Varnish, MongoDB, CouchDB, Storm, Jenkins, NodeJS, MySQL, Postgres, Apache, Nginx, Tomcat, Mongrel, Mongrel2, Sinatra, Riak, Graphite, Cassandra, Hadoop, ActiveMQ, Solr, RabbitMQ, Elastic Search, Logstash, Kibana, HAProxy, Postfix, and Graphite
- 6. What Not To Do ☒ Manually Create Unit Files ☒ Blindly Use Docker Hub Images ☒ Run Data Volumes without a Backup Plan ☒ Manual Configuration or Conf Management ☒ Modify CoreOS Outside of Cloud Init ☒ Execute Multiple Processes per Container ☒ Expose Lots of Public Ports
- 7. Docker Madness Ehh?? What am I looking at?68ce1cebbf07 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10 days 80/tcp tp_web-2 3f26d14fe567 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10 days tp_web_backups a77bcb74d75c 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10 days 80/tcp tp_web-1 98f15fac74ca 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10 days tp_mysql_backups 8019d38dd49b cloudposse/library:btsync ""/bin/sh -c '/usr/ 10 days ago Up 10 days 44444/tcp tp_btsync 652e815800d7 cloudposse/library:mysql "/entrypoint.sh mysq 10 days ago Up 10 days 3306/tcp tp_mysql 21a18d390135 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10 days vps_akira_backups 751f682f9606 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10 days 0.0.0.0:30000->22/tcp vps_osterman 3da29f6eda58 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10 days 0.0.0.0:30002->22/tcp vps_akira 02c
- 8. Fleet-ing Commands WTF Fleet is Not much better… osterman_web@1.service 3ea56e45.../167.114.0.38 active running osterman_web@2.service 3ea56e45.../167.114.0.38 active running osterman_web_announce.service 3ea56e45.../167.114.0.38 active exited osterman_web_backups.service 3ea56e45.../167.114.0.38 inactive dead registrator_skydns.service 3ea56e45.../167.114.0.38 active running tp_btsync.service 3ea56e45.../167.114.0.38 active running tp_mysql.service 3ea56e45.../167.114.0.38 active running tp_web@1.service 3ea56e45.../167.114.0.38 active running tp_web@2.service 3ea56e45.../167.114.0.38 active running vps_akira.service 3ea56e45.../167.114.0.38 active running vps_akira_backups.service 3ea56e45.../167.114.0.38 active running
- 9. Sample Apache Systemd Unit File [Unit] Description=Apache Web Server After=docker.service Requires=docker.service [Service] User=core EnvironmentFile=/etc/environment TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill ib_web-1 ExecStartPre=-/usr/bin/docker rm ib_web-1 ExecStartPre=/usr/bin/docker --debug=true pull registry.hub.docker.com/cloudposse/library:apache-php
- 10. (Continued) ExecStart=/usr/bin/docker run --name ib_web-1 --rm --dns=${COREOS_PRIVATE_IPV4} --dns-search=ib.cloudposse.local --volume /media/sdb/ib_web:/var/www/html -e "DB_USER=wordpress" -e "DB_PASS=XXXXXXXX" -e "DB_HOST=web.mysql.ib.cloudposse.local" -e "DB_NAME=wordpress" -e "SERVICE_NAME=ib/web" -e "SERVICE_ID=1" registry.hub.docker.com/cloudposse/library:apache-php ExecStop=-/usr/bin/docker kill ib_web-1 ExecStop=-/usr/bin/docker rm ib_web-1
- 11. The Missing Pieces ♟Fleet Orchestration ♟Systemd Unit Templates ♟Basic Building Blocks ♟Naming Conventions ♟Introspection ♟Overlay Networking
- 12. Low and behold! THE OPPORTUNITY
- 13. What if I told you…. There was a way you could quickly spin up docker containers for everything, running coreos in the cloud or bare metal, with multiple environments or stages, using simple commands on your laptop it would just work? =)
- 14. Well…. YOU CAN
- 16. The Blue Print ● Ensemble Services Distribution ✓ 60+ Essential Services (e.g nginx, varnish, mysql) ✓ HTTP Router with Vulcand ✓ RFC1918 Overlay Network ✓ Service Composition, Orchestration ✓ Library Unit Templates, Images ✓ DNS Based Service Discovery & Registration ✓ Logging to PaperTrail, DataDogHQ, LogEntries ✓ Remote VPN Access ✓ Backups & BitTorrent Data Replication
- 17. Ensemble Command Line Usage: bin/ensemble options [action] [service] [unit] Some of the Supported actions: build Compile the m4 templates for each service commit Execute "git commit" modified units for each service push Execute "git push" to master committed units for each service kickstart Shortcut for stop+destroy+start specified units restart Shortcut for stop+start specified units view Display the current unit definition template Display the template definition status Display status of all units in cluster stats Display realtime performance metrics of specified units in cluster journal Stream stdout logs for specified unit from each server in the cluster shell Attach to the container and execute a shell (/bin/bash) Actions can be combined using +
- 18. Command Line Examples # Build all configurations for all services ensemble --config deployment.yml build # Build and deploy only the “ib” service ensemble --config deployment.yml build+kickstart ib # Give me the operating status of the “ib” service ensemble --config deployment.yml status ib # Open up a shell inside the web container ensemble --config deployment.yml shell ib web@1 # Display real-time memory and network utilization of the “ib” service ensemble --config deployment.yml stats ib # Commit changed configurations and push them to master ensemble --config deployment.yml commit+push ib
- 19. Tons of Ready Made Templates datadog.service.m4, logentries.service.m4, logspout.service.m4, boundary_agent.service.m4, newrelic_sysmond.service.m4, nodejs.service.m4, apache.service.m4, openvpn.service.m4, docker_registry.service.m4 duplicity.service.m4, haproxy_with_discover.service.m4, os_swap.service.m4 jenkins_master.service.m4, jenkins_swarm.service.m4, kiwiirc.service.m4, memcache.service.m4, skydns.service.m4 mysqldump.service.m4, mysql_galera.service.m4, generic.timer.m4, mysql_galera_cluster.service.m4, mysql_galera_garbd.service.m4, mysql_galera_lb.service.m4, mysql_server.service.m4, btsync.service.m4, postfix.service.m4, redis.service.m4, registrator.service.m4, vps.service.m4, vulcand.service.m4, os_mount.service.m4
- 20. Ensemble Configuration (YAML) fleet_tunnel: "cluster.cloudposse.local" # Cluster to administer fleet_ssh_timeout: 20 # How long to wait before giving up template_path: "~/Dev/ensemble/units" # Where to find the templates template_engine: "m4" # How to rewrite the templates target_path: "~/Dev/cluster-configs/units" # Where to send the rendered units globals: # Global variables can be used anywhere in the YAML config cluster_name: "cluster-1" # Sample definition container_storage: "@(mount_dir)/sdb" # Where we’ll store persistent data (it will be backed up!) mount_dir: "/media" # Mount point on CoreOS for persistent storage services: # Services are composed of a collection of units - include: "services/core.yml" # CoreOS extensions deployed via units - include: "services/jenkins.yml" # Jenkins CI with Jenkins Swarm - include: "services/osterman.yml" # My own website and virtuals - include: "services/tp.yml" # TP Deployment - include: "services/isc_staging.yml" # Staging environment for ISC - include: "services/isc_prod.yml" # Production environemnt for ISC
- 21. Pretty Service Configurations # Osterman Associates name: "osterman" # The name of this service globals: # Local service globals # Set/Override New Globals mysql_user: "wordpress" mysql_pass: "XXXXXXXXXX" mysql_host: "mysql.@(service).@(local_domain)" mysql_port: 3306 mysql_database: "wordpress" mysql_root_pass: "XXXXXXXXXXXXXX" mysql_backups_volume: "@(container_storage)/@(service)_mysqldump:/backups" mysql_volume: "@(container_storage)/@(service)_mysql:/var/lib/mysql" web_volume: "@(container_storage)/@(service)_web:/var/www/hosts"
- 22. Minimal Unit Configurations # Database instance for wordpress - name: "mysql" # Name of this unit template: "mysql_server.service.m4" type: "service" define: "DOCKER_VOLUME": "@(mysql_volume)" "DOCKER_IMAGE": "@(docker_registry)/cloudposse/library:mysql" "MYSQL_USER": "@(mysql_user)" "MYSQL_PASS": "@(mysql_pass)" "MYSQL_HOST": "@(mysql_host)" "MYSQL_DATABASE": "@(mysql_database)"
- 23. Clustering Made Easy # Apache web service for wordpress multisite - name: "web@%i" template: "apache.service.m4" type: "service" instances: 5 # Define all the virtual hosts referencing another definition - name: "vhost@www.osterman.com" from: "vhost@%i" - name: "vhost@osterman.com" from: "vhost@%i"
- 24. Screencast ● Overview ● Sample Configurations (YAML, m4) ● Demonstration of COOL Commands ● Data Dog, Paper Trail, Boundary
- 25. BAM! That’s all I got... Ask Erik Osterman erik@cloudposse.com (415) 535-8615
Editor's Notes
- My objective objective is to convince you that running Docker on CoreOS is a real possibility, but it requires some extra tooling First I will cover about 20 slides to go over the problem and some details I assume that you’ve played with docker a little bit and know what CoreOS provides. I will cover what I have built and how it solves the problems The solution is simple, does not require any extra services to run and is built on top of existing tools
- The problem is that all these tools have sprung up and it’s not clear how to use them together. We’re in the early days of the wild-wild-west as it relates to docker. What we’re really doing is observing a transformation in how services are shipped. Before there were distributions like Redhat or Ubuntu, there were no good ways to make software work together. Just like there used to be no good way to make software work together on the same OS, now there is no good way to make services work together on the same cloud.
- By overkill, I mean overkill for smaller deployments and add a level of complexity not necessary.
- Services need to be aware of each other. Even a simple blog has multiple moving pieces. They need a way to do service discovery. Many ways of doing it. Dynamic DNS is one easy way.
- If only things were that easy. LAMP is dead. Long live the LAMP stack. Open Source has exploded & that’s a good thing. But now we gotta find out how to get this new fangled stack to run that defies abbreviation. Today we have to worry about getting all these components to play nicely with each other.
- Exposing ports is evil. They are a scarce commodity. There should only be 2 ports exposed - 80 and 443. I hate it when I see a charming image only to disappoint me by bundling supervisor and voiding the one-process-per-container model. This masks any underlying problems Prevents systemd from using the rules that it’s been configured to work with
- No good way to see what is running. Lots of commands but not stitched together. Docker shows you all the services that are running. This is unmanageable when you have hundreds or thousands of services. Docker won’t ensure your services are always running It’s hard to visualize how services are composed and related to each other. There needs to be a way to look just at how one group of services is performing You can link containers together, but right now that doesn’t work across hosts.
- It helps us schedule services ala systemd, but there a lot of gotchas You can’t easily tell if you forgot to start a service You can’t see if a service has been orphaned Order matters - You have to remember the order in which to start services or you reach a dead lock. Try doing that with hundreds of services. You can’t easily restart services when making changes to unit files without running a half dozen commands. You can’t easily tell at a glance what is failed vs dead You cannot drop to shell inside of a container to inspect what’s going on Does not handle journaling of global services
- Units require a lot of configuration Everything highlighted might need to change between different stages or environments. Leaves a lot of room for error. There’s a lot of redundancy Unit files are easily resuable for different images. It doesn’t make sense to hardcode the docker image. Docker names need to be unique.
- These are the missing pieces needed to tie everything together. Fleet only does scheduling. Scheduling is handled by systemd. Something needs to handle orchestration and the composition of microservices Docker as it stands today, is only really good at running containers not managing relationships between containers..
- Can’t there be a way to stitch all the tools available today to get a working containerized environment without going overkill with Apache Mesos and Kubernetes?
- ...all this without sacrificing good architecture & design. All this without having to invest all the time and effort to get the pieces to play together. That would be pretty cool, right? Keep it simple, stupid.
- Here are just some of the things I want to call out that it supports. Convention over configuration by design; seek to decrease the number of decisions that you need to make, gaining simplicity, but not necessarily losing flexibility due to it’s empirical design.
- I am now going to go over what can be done. I’m might brush over it a little bit quickly because I want to get to the demo. Everything in green are awesome extensions that implement original functionality. Where applicable, they are cluster-aware.
- Templates define imperative configurations.
- DNS Search also supported. So it could just say “mysql” and it will search the local domain
- Order matters! Just define the units in the order they need to be instantiated and it will respected
- Just specify how many of them you want to create.
- ping vpn.central.cloudposse.local ensemble --config ~/Dev/cluster-configs/deployment.yml build ensemble --config ~/Dev/cluster-configs/deployment.yml build ib ensemble --config ~/Dev/cluster-configs/deployment.yml build ib web_mysql ensemble --config ~/Dev/cluster-configs/deployment.yml status ib ensemble --config ~/Dev/cluster-configs/deployment.yml stats ib