Microservices
with Docker Swarm and Consul
Agenda
- Something about Microservices
- Implement Microservices with Docker Swarm and Consul
“You shouldn't start a new project with
microservices, even if you're sure your
application will be big enough to make it
worthwhile”
(Martin Fowler)
Some Suggestions
Programing: Python, NodeJS, Golang*
Orchestration: Docker, Docker Swarm, Consul, Consul-Template, Registrator, CoreOS*
Databases & Caches: PosgreSQL, Cassanda, ElasticSearch, Redis
Load Balancer & API Gateway: Nginx
Queues: Kafka
CI/CD: Jenkins, Ansible
Monitors: Grafana + InfluxDB + (CollectD, cAdvisor)
Logs: Kibana + ElasticSearch + FluentD
Data Center: AWS EC2, AWS S3, AWS Float IP, Cloudfront
CDN & DNS & DDOS Protection: Cloudflare
Documents: Swagger, Markdown, mkdocs
Backend System Suggestions
An Example
Node Gateway will be used a API
Gateway and Load Balancer
Node Agent One, Node Agent Two
will be used to run services that are
able to scale.
Docker & Docker Swarm
Docker is a platform for developers and
sysadmins to develop, ship, and run
applications. Docker lets you quickly assemble
applications from components and eliminates
the friction that can come when shipping code.
Docker lets you get your code tested and
deployed into production as fast as possible.
Docker Swarm is native clustering for Docker. It
allows you create and access to a pool of
Docker hosts using the full suite of Docker tools.
Because Docker Swarm serves the standard
Docker API, any tool that already communicates
with a Docker daemon can use Swarm to
transparently scale to multiple hosts
#install
$ docker pull swarm
#run joiner
$ docker run -d swarm join  --
addr=172.20.20.11:2375 
token://acdb9dfa3ea6da0b0cfb2c819385fcd3
#run manager
$ docker run -d -p 12375:2375 
swarm manage 
token://acdb9dfa3ea6da0b0cfb2c819385fcd3
Consul and Consul-Template
Consul
- Service Discovery
- Key/Value Storage
- Failure Detection
- Multiple Datacenter
Consul-Template
- Listen/Query the updates from
Consul
- Updates Configuration File by the
templates provided
#run a consul agent
$ consul agent -data-dir /tmp/consul -node=agent-one 
-bind=172.20.20.11 -client=0.0.0.0 
-config-dir /etc/consul.d 
-retry-join 172.20.20.10
#run consul-template
exec consul-template 
-consul 127.0.0.1:8500 
-template "/build/gateway/consul-
template/nginx.ctmpl:/etc/nginx/sites-
available/default:service nginx reload" 
-retry 30s
Registrator
Registrator automatically registers and
deregisters services for any Docker container by
inspecting containers as they come online.
Registrator supports pluggable service
registries, which currently includes Consul, etcd
and SkyDNS 2.
#run registrator
$ docker run -d 
--name=registrator --net=host  --
volume=/var/run/docker.sock:/tmp/docker.sock 
gliderlabs/registrator:latest 
consul://localhost:8500
A Scenario
At the beginning, Service #1 only runs in Node 2.
After that Service #1 scales to Node 2
Via Docker API, Registrator will know the appearance of
Service #1 on Node 2.
Registrator will register Service #1 with Consul.
In the meanwhile, Consul-Template in listening Consul. So
Consul-Template will know the new service is up in Node 2.
Finally, Consul-Template will trigger a reload command to
Nginx add the new service into load balancer.
DEMOhttps://github.com/thanhson1085/microservices-swarm-consul
MICROSERVICES
DEVOPS
References
http://martinfowler.com
http://microservices.io/patterns/
https://www.sequoiacap.com/article/build-us-microservices/
https://sonnguyen.ws/microservices-with-docker-swarm-and-consul/

Microservices with docker swarm and consul

  • 1.
  • 3.
    Agenda - Something aboutMicroservices - Implement Microservices with Docker Swarm and Consul
  • 9.
    “You shouldn't starta new project with microservices, even if you're sure your application will be big enough to make it worthwhile” (Martin Fowler)
  • 11.
    Some Suggestions Programing: Python,NodeJS, Golang* Orchestration: Docker, Docker Swarm, Consul, Consul-Template, Registrator, CoreOS* Databases & Caches: PosgreSQL, Cassanda, ElasticSearch, Redis Load Balancer & API Gateway: Nginx Queues: Kafka CI/CD: Jenkins, Ansible Monitors: Grafana + InfluxDB + (CollectD, cAdvisor) Logs: Kibana + ElasticSearch + FluentD Data Center: AWS EC2, AWS S3, AWS Float IP, Cloudfront CDN & DNS & DDOS Protection: Cloudflare Documents: Swagger, Markdown, mkdocs
  • 12.
  • 13.
    An Example Node Gatewaywill be used a API Gateway and Load Balancer Node Agent One, Node Agent Two will be used to run services that are able to scale.
  • 14.
    Docker & DockerSwarm Docker is a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible. Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts #install $ docker pull swarm #run joiner $ docker run -d swarm join -- addr=172.20.20.11:2375 token://acdb9dfa3ea6da0b0cfb2c819385fcd3 #run manager $ docker run -d -p 12375:2375 swarm manage token://acdb9dfa3ea6da0b0cfb2c819385fcd3
  • 15.
    Consul and Consul-Template Consul -Service Discovery - Key/Value Storage - Failure Detection - Multiple Datacenter Consul-Template - Listen/Query the updates from Consul - Updates Configuration File by the templates provided #run a consul agent $ consul agent -data-dir /tmp/consul -node=agent-one -bind=172.20.20.11 -client=0.0.0.0 -config-dir /etc/consul.d -retry-join 172.20.20.10 #run consul-template exec consul-template -consul 127.0.0.1:8500 -template "/build/gateway/consul- template/nginx.ctmpl:/etc/nginx/sites- available/default:service nginx reload" -retry 30s
  • 16.
    Registrator Registrator automatically registersand deregisters services for any Docker container by inspecting containers as they come online. Registrator supports pluggable service registries, which currently includes Consul, etcd and SkyDNS 2. #run registrator $ docker run -d --name=registrator --net=host -- volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://localhost:8500
  • 17.
    A Scenario At thebeginning, Service #1 only runs in Node 2. After that Service #1 scales to Node 2 Via Docker API, Registrator will know the appearance of Service #1 on Node 2. Registrator will register Service #1 with Consul. In the meanwhile, Consul-Template in listening Consul. So Consul-Template will know the new service is up in Node 2. Finally, Consul-Template will trigger a reload command to Nginx add the new service into load balancer.
  • 18.
  • 19.
  • 20.