Microservices Blue/Green Deployment with Docker

2. Book
●
DevOps 개발관련 HOWTO 를 잘 설명하였고 이문서의 근간이 된 책
●
https://www.amazon.com/DevOps-2-0-Toolkit-Containerized-Microservices/dp/B01FEK2HFQ/ref=sr_1_7?
ie=UTF8&qid=1477957713&sr=8-7&keywords=devops+2.0

3. Motive
Blue/Green Deployment 는 Service Upgrade 시 Service Downtime 없이 이전 Version(Blue) 에서 새로운
Version(Green) 으로 Seamless 하게 Deploy 하는 방법.
대부분의 Microservice 들은 Proxy(예를들어 nginx) 에 연결되어 Service 를 제공.
Proxy 의 Blue Service 연결을 Green Service 연결로 바로 전환할수 있다면 가능.
그런데 효율적 Resource 사용을 위해 Microservice 들은 Docker Container 안에서 실행,
그리고 Docker Container 들은 Swarm Cluster 위에 실행.
Microservice Docker Container 를 자동 Scale Out 하기 위해 Swarm Cluster 기반으로 Docker Compose
사용.
Configuration Management Tool 은 Ansible 사용.
그외 필요한 Tool 들은 Docker Private Registry, Consul, Consul-Registrator, Consul-Template 등.
Proxy
(nginx)
Blue Service
(Collection Handler v1)
Green Service
(Collection Handler v2)

4. Architecture
Consul Docker
Consul-Template
Docker
Registry
Blue Service
Collection Handler
Green Service
Collection Handler
Docker-Compose
Consul Docker
Consul-Template
Docker-Compose
Consul Docker
Consul-Template
Docker-Compose
Swarm
Master
Consul
Registrator
Swarm
Node
Consul
Registrator
Blue Service
Collection Handler
Green Service
Collection Handler
Consul Docker
Consul-Template
Docker-Compose
Swarm
Node
Consul
Registrator
Consul-TemplateConsulDocker Docker-Compose
Swarm
Node
Consul
Registrator
Proxy
nginx
Blue Service
Collection Handler
Green Service
Collection Handler
Green Service
Collection Handler
Blue Service
Collection Handler
Swarm Master Swarm Slave Swarm Slave
Swarm Slave

5. Architecture 2
●
사각형모양중 각이 둥근 사각형은 Docker Container, 각이 진 사각형은 Library 혹은 Process 를 의미.
●
Swarm Cluster 는 크게 Swarm Master 와 여러 Swarm Slave 로 구분.
●
모든 Docker Container 는 Swarm Cluster 위에 Deploy 됨.
●
Serivce 들은 Docker Registry 에 등록됨.
●
Docker Compose 를 통해 Swarm Cluster 에 Service Deploy 시 Docker Registry 를 lookup 하여
Service Image 를 Pull 함.
●
Service Discovery Consul 을 통해 Blue 혹은 Green Service 들의 IP, Port 정보가 등록.
●
이때 Consul-Registrator 가 이 Service 들의 IP, Port 정보를 자동으로 Consul 에 등록 해줌.
●
Consul-Template 을 통해 등록된 Service 들의 IP, Port 정보를 얻어 Proxy nginx configuration 을 변경.
●
Proxy nginx 는 restart 없이 configuration reload 를 하여 downtime 없이 blue 에서 green service 로 자
동 switching 됨.

6. 설치 Pipeline
설치 순서는 다음과 같음:
1. Consul
2. Docker
3. Consul-Registrator
4. Swarm
5. Proxy nginx
6. Service Collection Handler

7. 설치 Pipeline: Consul
- name: Start Consul Servers
shell: "nohup {{ consul_binary }} agent -server -bootstrap-expect=1
-data-dir={{ consul_data_dir }}
-config-dir={{ consul_config_dir }}
-client=0.0.0.0
-bind={{ private_ip }}
-node={{ inventory_hostname }}
-ui -ui-dir={{ consul_ui_dir }}
-pid-file={{ consul_pid_file }}
> {{ consul_log_dir }}/consul.log 2>&1 &"
args:
executable: /bin/bash
when: (inventory_hostname == hostvars[groups['consul-server-hosts'][0]]['inventory_hostname'])
- name: Start Consul Agents
shell: "nohup {{ consul_binary }} agent -join={{ hostvars[groups['consul-server-hosts'][0]]['inventory_hostname'] }}
-data-dir={{ consul_data_dir }}
-config-dir={{ consul_config_dir }}
-client=0.0.0.0
-bind={{ private_ip }}
-node={{ inventory_hostname }}
-ui -ui-dir={{ consul_ui_dir }}
-pid-file={{ consul_pid_file }}
> {{ consul_log_dir }}/consul.log 2>&1 &"
args:
executable: /bin/bash
when: (inventory_hostname != hostvars[groups['consul-server-hosts'][0]]['inventory_hostname'])
Ansible Script 를 통해 Consul Server 와 Agent 들을 설치.

8. 설치 Pipeline: Docker
- name: Install Docker Engine
yum:
name: docker-engine
state: latest
- name: Set docker configuration for swarm
lineinfile:
dest: /usr/lib/systemd/system/docker.service
state: present
regexp: "^ExecStart"
line: "ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375
-H unix:///var/run/docker.sock
--insecure-registry {{ registry_url }}"
- name: Reload Daemon
shell: "systemctl daemon-reload"
- name: Restart Docker
service:
name: docker
enabled: yes
state: restarted
- name: Run Docker Registry
docker:
name: registry
image: registry:2
state: started
ports:
- "5000:5000"
volumes:
- "{{ registry_volume }}"
when: (inventory_hostname == hostvars[groups['docker-registry-host'][0]]['inventory_hostname'])
- name: Install Docker Compose
pip:
name: docker-compose
state: present
Docker Engine 설치후 dockerd line 을 변경.
Docker Registry 설치.
Docker-Compose 설치.

9. 설치 Pipeline: Consul-Registrator
- name: Run Consul Registrator Docker Container
shell: "docker run -d --name=registrator --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest
consul://localhost:8500"
args:
executable: /bin/bash
- name: Run Consul KV Registrator Docker Container
shell: "docker run -d --name=registrator-consul-kv --net=host --volume=/var/run/docker.sock:/tmp/docker.sock
gliderlabs/registrator:latest consulkv://localhost:8500/services"
args:
executable: /bin/bash
Docker Container 로 감싼 Service 의 IP, Port 를 자동 인식하여 Consul 에 등록할수 있도록 Consul-Registrator 설치.

10. 설치 Pipeline: Swarm
- name: Run swarm node
docker:
name: swarm-node
image: swarm
command: "join --advertise={{ private_ip }}:2375 consul://{{ private_ip }}:8500"
env:
SERVICE_NAME: swarm-node
when: inventory_hostname in groups['swarm-node-hosts']
- name: Run swarm master
shell: "docker run -d -p 4000:4000 --name swarm-master
--env SERVICE_NAME=swarm-master
swarm manage -H :4000 --advertise {{ private_ip }}:4000 consul://{{ private_ip }}:8500"
when: inventory_hostname in groups['swarm-master-host']
Swarm Master, Slave 설치.
전체 Swarm Cluster 의 정보를 얻으려면:
sudo docker -H <swarm-master>:4000 info;

11. 설치 Pipeline: Proxy nginx
- name: Run nginx container
docker:
image: nginx:alpine
name: nginx
state: running
ports: "{{ ports }}"
volumes: "{{ volumes }}"
- name: Copy conf file
copy:
src: "{{ item }}"
dest: "{{ nginx_conf_dir }}"
with_items:
- "{{ playbook_dir }}/roles/{{ role_name }}/files/{{ services_conf_file }}"
register: result
- name: Reload nginx container
shell: docker kill -s HUP nginx
when: result|changed
Proxy nginx 설치.
Configuration copy 후 restart 없이 nginx reload 함.

12. 설치 Pipeline: Service Collection Handler
Service Collection Handler Dockerfile:
FROM java:8-jre
ENV COLLECTION_HANDLER_HOME "{{ collection_handler_home }}"
RUN mkdir -p "$COLLECTION_HANDLER_HOME"
WORKDIR $COLLECTION_HANDLER_HOME
ENV VERSION "{{ collection_handler_version }}"
ENV REPOSITORY "{{ maven_repo_type }}"
ENV NEXUS_CONTEXT "{{ nexus_context }}"
ENV ZIP_URL http://$NEXUS_CONTEXT/service/local/artifact/maven/redirect?r=$REPOSITORY&g=crochet&a=crochet-collection-collection-
handler&v=$VERSION&c=stand-alone&e=zip
RUN set -x
&& curl -fSL "$ZIP_URL" -o crochet-collection-collection-handler-$VERSION-stand-alone.zip
&& unzip -oq *.zip
&& chmod a+x -R bin
&& rm -rf crochet-collection-collection-handler-$VERSION-stand-alone.zip
EXPOSE "{{ collection_handler_port }}"
Docker Container Build 후 Docker Registry 에 image 등록:
- name: Build Collection Handler Docker Image
shell: "docker build --tag {{ docker_name }}:{{ docker_version }} {{ docker_file_path }}"
args:
executable: /bin/bash
when: (inventory_hostname == hostvars[groups['collection-handler-hosts'][0]]['inventory_hostname'])
- name: Tag and Push Collection Handler Docker Image to Registry
shell: "{{ item }}"
args:
executable: /bin/bash
with_items:
- "docker tag {{ docker_name }}:{{ docker_version }} {{ registry_tag }}"
- "docker push {{ registry_tag }}"
when: (inventory_hostname == hostvars[groups['collection-handler-hosts'][0]]['inventory_hostname'])

13. 설치 Pipeline: Service Collection Handler 2
- name: Run and scale Collection Handler with Compose
shell: "{{ item }}"
with_items:
- "docker-compose -f {{ docker_file_path }}/docker-compose.yml
-p collection-handler-project
pull {{ container_name }}"
- "docker-compose -f {{ docker_file_path }}/docker-compose.yml
-p collection-handler-project
up -d {{ container_name }}"
- "export DOCKER_HOST=tcp://{{ swarm_master_host }}:4000 &&
docker-compose -f {{ docker_file_path }}/docker-compose.yml
-p collection-handler-project
scale {{ container_name }}=3"
when:
- run_compose == "true"
- inventory_hostname == hostvars[groups['collection-handler-hosts'][0]]['inventory_hostname']
register: output
- debug: var=output
Service Collection Handler 는 Docker Compose 를 통해 pull, up 된후 swarm cluster 에 3개의 container
가 deploy 됨.

14. Blue/Green Deployment: Blue
Nginx includes configuration:
location /event/v1 {
proxy_pass http://collection-handler/event/v1;
proxy_next_upstream error timeout invalid_header http_500;
}
Nginx includes configuration file 을 nginx 설정 directory 에 copy.
Nginx upstreams configuration consul template:
upstream collection-handler {
{{range service "collection-handler-blue" "any"}}
server {{.Address}}:{{.Port}};
{{end}}
}
Service Collection Handler blue 에 대한 Consul Template 으로 다음과 같은 실행을 통해 Consul 에서 Service
collection-handler-blue 의 IP, Port 정보를 얻어 Template file 을 치환한후 nginx 설정 directory 로 copy:
- name: Run Consul Template for Collection Handler Service and move it to nginx upstreams directory
shell: "{{ item }}"
args:
executable: /bin/bash
with_items:
- "{{ consul_template_binary }}
-consul localhost:8500
-template {{ consul_template_temp_dir }}/{{ upstreams_file }}.ctmpl:{{ consul_template_temp_dir }}/
{{ nginx_collection_handler_upstreams_base }}.conf
-once"
- "cp {{ consul_template_temp_dir }}/{{ nginx_collection_handler_upstreams_base }}.conf {{ nginx_upstreams_dir }}/
{{ nginx_collection_handler_upstreams_base }}.conf"
register: result
그리고 nginx restart 없이 configuration reload 실행:
- name: Reload nginx container
shell: docker kill -s HUP nginx
when:
- result|changed

15. Blue/Green Deployment: Green
Service Collection Handler blue Deployment 와 유사하게 Green Service Deployment 역시 nginx upstreams
configuration consul template 은 아래와 같음:
upstream collection-handler {
{{range service "collection-handler-green" "any"}}
server {{.Address}}:{{.Port}};
{{end}}
}
Service Collection Handler green 역시 마찬가지로 Consul 로 부터 IP, Port 정보를 얻어 Template file 을 치환한후
nginx 설정 upstreams configuration file 을 덮어쓰기함.
그리고 nginx restart 없이 configuration reload 실행함:
- name: Reload nginx container
shell: docker kill -s HUP nginx
when:
- result|changed
이로서 blue service 에서 downtime 없이 seamless 하게 green service 로 자동 switching 되었음.
Green service 가 원활히 돌아가는것을 확인한후 blue service docker container 를 stop 시키고 삭제함:
- name: Remove Collection Handler with Compose
shell: "{{ item }}"
with_items:
- "export DOCKER_HOST=tcp://{{ swarm_master_host }}:4000 &&
docker-compose -f {{ docker_file_path }}/docker-compose.yml
-p collection-handler-project
stop {{ container_name }}"
- "export DOCKER_HOST=tcp://{{ swarm_master_host }}:4000 &&
docker-compose -f {{ docker_file_path }}/docker-compose.yml
-p collection-handler-project
rm -f {{ container_name }}"
when:
- run_compose == "true"
- inventory_hostname == hostvars[groups['collection-handler-hosts'][0]]['inventory_hostname']
register: output
ignore_errors: true
- debug: var=output

16. 맺음말
●
Containerization, 예를들어 Docker 는 Microservice 에 있어서 필요요소.
●
복잡한 Microservices Deployment 에는 Service Discovery(예를들어 Consul), Resource Management(예를들어
Swarm), Configuration Management(예를들어 Ansible) 등이 필요함.
●
이러한 Tool 들은 이문서의 예와 같은 DevOps Practice 를 통해 잘 활용됨으로서 가치가 드러남. 결과적으로
CI/CD 를 적용할수 있음.