Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Docker in production service discovery with consul - road to opscon 2015
1. Docker in production:
service discovery with Consul
Giovanni Toraldo - Lead developer @ ClouDesire.com
Road to OpsCon 2015 - Milano
2. About me
Open Source Enthusiast
with SuperCow Powers
PHP/Java/whatever
developer
writer of the OpenNebula
book
Lead developer at
ClouDesire.com
2
3. What is ClouDesire?
Application Marketplace to help software vendors to sell and provision
applications
● Web Applications:
○ provision VM
○ on multiple cloud providers
○ deploy/upgrade application and dependencies
○ application logging
○ resource monitoring
● With multi-tenant applications/SaaS:
○ expose REST hooks and API for billing lifecycle
● manage subscriptions, billing, pay-per-use, invoicing, payments.
3
4. ClouDesire platform components
Multitude of components of different stacks:
● Multiple Java/Tomcat REST backends
● Minor Ruby, Node REST backends
● AngularJS frontend
● PostgreSQL
● MongoDB
● Rsyslog + nxlog + Logstash + ElasticSearch
● ActiveMQ
4
5. Before Docker: platform deployments
Problems faced:
● Different stacks, different ways to handle deployments
○ Hundreds LOC of Chef recipes
○ Inevitable complexity
○ No trustworthy procedure to rollback
● Hard to switch between deps versions
○ Only the one available on official Ubuntu repositories
● Heterogeneous dev environments (vagrant?)
○ Ubuntu, Arch Linux, Mac
5
6. Before Docker: application packaging
How to grab applications from ClouDesire vendors?
● GIT repository (heroku)?
○ Not really its use-case, require understanding
○ Customization may require branching
● DEB/RPM packages?
○ Too much complexity
● Custom approach?
6
7. Before Docker: hand-made zip package
Requirements:
● Everyone should be able to do it
● even Windows users
● in a short time
Our solution, not elegant but effective:
● A ZIP archive with:
○ a folder for sources/artifacts
○ a folder for sql scripts
○ predefined environment variables to use
7
8. Before Docker: custom zip package
Application source code itself is not sufficient:
● Custom dependencies (e.g.: Alfresco)
● Custom dependencies versions
● Configuration for external resources
● Shameful hacks?
8
11. Docker: container lifecycle
● Pulls an image from a registry
● Creates a new container
○ Allocates a r/w filesystem
○ Allocates a network interface (on a bridge)
○ Sets up network (IP address, dns..)
● Launch a process into the container
● Captures and provides application output
● Container terminates when the process exits
11
12. Docker: why?
Enables software developers to:
● package an application
● with all dependencies
● runs it everywhere unchanged
● re-use images via composition
12
13. Docker: why?
Enables system administrators to:
● standardize application deployment
● ease scale-up & scale-down
● process separation/isolation
13
14. Docker adoption: platform deployment
● Different stacks, different ways to handle deployments
○ Docker Image for each component
○ Dozens LOC of Chef recipe for deploy / upgrades
○ Deployment complexity hidden into Dockerfile
○ Rollback is equal to upgrade
● Hard to switch between deps versions
○ Docker for external dependencies
○ Use them without thinking how to deploy / upgrade
● Heterogeneous dev environments (vagrant?)
○ Docker-compose for each module
14
15. Docker adoption: application packaging
How to grab applications from ClouDesire vendors?
● Custom ZIP package
○ Build a docker image for the application
○ Push your image to our registry.
Advantages:
● Easy to follow documentation
● Fast Try-Fail-Retry cycle while building
● Works-for-me is works everywhere
● Re-use community images for dependencies
15
16. Docker adoption: application packaging
● Custom dependencies (e.g.: Alfresco)
○ (Re-)Use multiple containers
● Custom dependencies versions
○ Multiple versions easily available
● Configuration for external resources
○ Environment variables
● Shameful hacks?
○ Hidden in the Dockerfile
16
17. Docker: it looks like something is missing
Moving to containers introduced a new layer of complexity:
● communication between containers:
○ Same host (with docker linking feature)
■ restart everything after redeploy
○ Multiple hosts
■ not a docker problem
● Scaling/hot-deploy ready
○ but load balancers are statically configured
17
20. Before Consul (and service discovery)
● Hardcoded IP:port configuration
○ Hand-made copy-paste
○ Not fault-tolerant
○ No autoscaling
○ No self-healing
● Configuration management (e.g. Puppet, Chef)
○ Slow to react to events
○ Ordered service starting
Above not really feasible when running dozens of containers
20
22. When service discovery is needed
● Monolithic architecture? No problem.
● Distributed architecture?
○ Is there at least one foo instance running?
○ At which address?
○ On which port?
FOO
BAR
Service
Registry
Client
22
23. Consul features
● Agent based
● Query interfaces
○ HTTP JSON API
○ DNS
● Health-Checking
○ No one want borked services
● Key-Value Store
○ Shared dynamic configuration
○ Feature toggle
○ Leader election
23
30. Register a new service
$ curl -X POST -d @service.json localhost:
8500/v1/agent/service/register
{
"ID": "redis1",
"Name": "redis",
"Tags": [
"master",
"v1"
],
"Address": "172.16.0.2",
"Port": 8000
}
30
31. Retrieve service details
$ dig redis.service.consul @localhost -p 8600 +tcp
redis.service.consul. 0 IN A 172.16.0.2
$ curl localhost:8500/v1/agent/services
{"consul":{"ID":"consul","Service":"consul","Tags":[],"
Address":"","Port":8300},"redis1":{"ID":"redis1","Service":"redis","
Tags":["master","v1"],"Address":"172.16.0.2","Port":8000}}%
31
32. Populate services from Docker
https://github.com/gliderlabs/registrator
● Discover containers using docker API
● Skip containers without published ports
● Register service when container goes up
○ Service name is the image name
○ Tags via environment variables
● Unregister service when container goes down
32
33. Running registrator
$ docker run -d
--name=registrator
--net=host
--volume=/var/run/docker.sock:/tmp/docker.sock
gliderlabs/registrator:latest
consul://localhost:8500
Note: exposing docker.sock in a container is a security
concern.
33
34. Enrich service metadata
● Registrator auto discovery can be enriched via
environment variables
$ docker run --name redis-0
-p 10000:6379
-e "SERVICE_NAME=db"
-e "SERVICE_TAGS=master"
-e "SERVICE_REGION=it" redis
34
35. Retrieve registrator services
$ dig consul.service.consul @localhost -p 8600 +tcp
consul.service.consul. 0 IN A 172.17.0.3
consul.service.consul. 0 IN A 172.17.0.2
$ curl localhost:8500/v1/agent/services
{"consul":{"ID":"consul","Service":"consul","Tags":[],"Address":"","Port":8300},"
viserion:consul:8500":{"ID":"viserion:consul:8500","Service":"consul-8500","Tags":
null,"Address":"127.0.1.1","Port":8500},"viserion:consul:8600":{"ID":"viserion:consul:
8600","Service":"consul-8600","Tags":null,"Address":"127.0.1.1","Port":8600}}%
35
36. Automatic reverse proxy for web services
$ consul-template
-consul 127.0.0.1:8500
-template "/tmp/template.ctmpl:/var/www/nginx.conf:service
nginx restart"
-retry 30s
upstream upstream-<%= @service_name %> {
least_conn;
{{range service "<%= @service_name %>"}}server {{.Address}}:
{{.Port}} max_fails=3 fail_timeout=60 weight=1;
{{else}}server 127.0.0.1:65535; # force a 502{{end}}
} 36
37. Example nginx.conf fragment
server {
listen 80 default_server;
location ~ ^/api/(.*)$ {
proxy_pass http://upstream-service-name/$1$is_args$args;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
37
38. Real-world architecture of everything
Container
A
Container
B
Registrator
Consul
agent
Consul
agent
Registrator
Backend Node A Backend Node B
Network
● Consul agent running on each node
● Registrator on each docker node
● Every node has 127.0.0.1 in /etc/resolv.conf
● Services discover dependencies via DNS
● Nginx endpoint generated by consul-template
Frontend Node(s)
Nginx
Consul
agent
Consul-
template 38
39. Consul: additional goodies
● Key-Value store for configurations
● DNS forwarding
● DNS caching
● WAN replication (Multi-DC)
● Atlas bootstrapping (https://atlas.hashicorp.
com/)
● Web UI (http://demo.consul.io/ui/)
39
41. Homework for the coming months
● Take a look at:
○ Docker swarm (1.0 released on nov 2015)
○ Kubernetes (1.0 release on july 2015)
○ Apache Mesos (0.25 release on oct 2015)
41