Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling Your App With Docker Swarm using Terraform, Packer on Openstack

An overview of building a production-ready fully-scalable Docker Swarm using Terraform & Packer on OpenStack.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Scaling Your App With Docker Swarm using Terraform, Packer on Openstack

  1. 1. Scaling With Docker Swarm using Packer, Terraform & OpenStack Bobby DeVeaux - March 9th 2017 https://joind.in/talk/55008
  2. 2. https://joind.in/talk/55008 @bobbyjason • Created my first website at 9 Years old in 1995 😮 • Started coding PHP in 2001 - 16 years ago • Developer, Team Leader, CTO, Director & Consultant • Been using AWS for over 5 years • Web Development, Message Queues, Automation, CI&CD • Previously worked at SkyBet & DVSA • Now a DevOps Consultant with UKCloud, Evangelising OpenStack • Contributor to Terraform • I ♥️ Docker, Terraform & Golang (or anything Hashicorp) • #twitter: @bobbyjason About Me ☁️ 2
  3. 3. https://joind.in/talk/55008 @bobbyjason • I’m here to spread the awareness of UKCloud & OpenStack • I want you to use Docker Swarm • I want you to love Terraform • I want to show you how to scale an app using all the above Why Am I Here? 3
  4. 4. https://joind.in/talk/55008 @bobbyjason UKCloud at-a-glance 4
  5. 5. https://joind.in/talk/55008 @bobbyjason • Docker for AWS • Docker for Azure • UKCloud offer Cloud Native Infrastructure using Openstack Why Openstack? 5
  6. 6. https://joind.in/talk/55008 @bobbyjason • Who’s using Docker yet? • Who’s using Docker Swarm? • Who’s using Terraform? • Who’s using Packer? • Who’s not played with any of them, and would love to? Hands Up 6
  7. 7. https://joind.in/talk/55008 @bobbyjason What Is Docker? 7 “Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries – anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.” - docker.com
  8. 8. https://joind.in/talk/55008 @bobbyjason • Define multi-container setup • docker-compose.yml • beats running multiple docker run commands • can specify multiple Dockerfiles • docker-compose up .. similar to vagrant up Docker Compose 8
  9. 9. https://joind.in/talk/55008 @bobbyjason docker-compose.yml 9 version: "2" services: php-fpm: image: php:7.0.8-fpm-alpine ports: - "9000:9000" nginx: image: nginx:alpine ports: - "80:80"
  10. 10. https://joind.in/talk/55008 @bobbyjason • Allows tagging - useful for versioning: • docker tag myapp:latest myapp:2.0.5 • Resides locally Docker Tag 10
  11. 11. https://joind.in/talk/55008 @bobbyjason • Image Repository - default for Docker • Alternatives include: • Amazon ECR • Openstack Glance • quay.io Docker Hub 11
  12. 12. https://joind.in/talk/55008 @bobbyjason • Login to selected image repository • Docker HUB: $ docker login -u username -p password • Amazon: $ eval `aws ecr get-login` Docker Login 12
  13. 13. https://joind.in/talk/55008 @bobbyjason • Push images & tags up to image repository • docker push myapp:2.0.5 • docker push latest Docker Push 13
  14. 14. https://joind.in/talk/55008 @bobbyjason What Is Docker Swarm? 14 “Docker Swarm provides native clustering capabilities to turn a group of Docker engines into a single, virtual Docker Engine. With these pooled resources, you can scale out your application as if it were running on a single, huge computer.” - docker.com
  15. 15. https://joind.in/talk/55008 @bobbyjason 15
  16. 16. https://joind.in/talk/55008 @bobbyjason • Docker 1.12 & now 1.13 has Docker Swarm natively, “Swarm Mode” • Cluster management integrated with Docker Engine • Decentralized design • Declarative service model • Scaling • Desired state reconciliation • Multi-host networking • Service discovery • Load balancing • Secure by default • Rolling updates Swarm Mode? 16
  17. 17. https://joind.in/talk/55008 @bobbyjason • docker swarm init —advertise-addr • Initialises the node for Swarm mode Docker Swarm Init 17
  18. 18. https://joind.in/talk/55008 @bobbyjason • docker service create --env APPLICATION_ENV=dev --update- delay 10s --replicas 1 --network mynet --name php-fpm bobbydvo/ukc_php-fpm:latest • docker service create --update-delay 10s --replicas 1 -p 80:80 -- network mynet --name web bobbydvo/ukc_nginx:latest • docker service update --image bobbydvo/ukc_nginx:1.81 web Docker Service Create 18
  19. 19. https://joind.in/talk/55008 @bobbyjason • docker service scale web=5 Docker Service Scale 19
  20. 20. https://joind.in/talk/55008 @bobbyjason • docker stack deploy —compose-file docker-compose.yml mystack Docker Stack Deploy (1.13 only) 20 version: "3" services: web: tty: true depends_on: - php-fpm image: bobbydvo/ukc_nginx:latest ports: - "80:80" deploy: mode: replicated replicas: 2 update_config: parallelism: 1 delay: 10s failure_action: continue monitor: 60s max_failure_ratio: 0.3 php-fpm: tty: true build: ./ image: bobbydvo/dummyapp_php-fpm:latest ports: - "9000:9000" environment: - APPLICATION_ENV=prod deploy: mode: replicated replicas: 1 update_config: parallelism: 1 delay: 10s failure_action: continue monitor: 60s max_failure_ratio: 0.3
  21. 21. https://joind.in/talk/55008 @bobbyjason • We’re about to embark onto the interesting stuff • Any questions? …and Pause. 21
  22. 22. https://joind.in/talk/55008 @bobbyjason • Packer • Terraform • Docker Swarm • Jenkins For Release How Do We Scale on Real Infrastructure 22
  23. 23. https://joind.in/talk/55008 @bobbyjason • Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. • Infrastructure as Code: Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used. • Execution Plans: Terraform has a "planning" step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure What Is Terraform? 23
  24. 24. https://joind.in/talk/55008 @bobbyjason • Terraform plan • Will show you exactly what it’s going to do and how many resources are going to be affected • No surprises. Terraform Plan 24
  25. 25. https://joind.in/talk/55008 @bobbyjason • terraform apply • Applies the changes as shown in terraform plan using it’s resource graph. It know’s which resources have dependant resources and which ones don’t • Parallelised building - using the resource graph it will make all the changes as quickly/efficiently as possible Terraform Apply 25
  26. 26. https://joind.in/talk/55008 @bobbyjason • Using the .tfstate file, it is fully aware of the resources. • Terraform destroy will literally remove all the infrastructure it built for you. WARNING: There is no ctrl+z! Terraform Destroy 26
  27. 27. https://joind.in/talk/55008 @bobbyjason Terraform Demo 27 provider "openstack" { } resource "openstack_networking_network_v2" "example_network2" { name = "example_network_2" admin_state_up = "true" } resource "openstack_networking_subnet_v2" "example_subnet2" { name = "example_subnet_2" network_id = "${openstack_networking_network_v2.example_network2.id}" cidr = "10.10.1.0/24" ip_version = 4 dns_nameservers = ["8.8.8.8", "8.8.4.4"] }
  28. 28. https://joind.in/talk/55008 @bobbyjason • How long do your builds & deployments in travis / Jenkins take? • What’s acceptable? • ‘Quick’ is relative, and depends on your requirements. • When I say quick deployments, I’m referring to efficient deployments using Foundation Images. Who Likes Quick Deployments? 28
  29. 29. https://joind.in/talk/55008 @bobbyjason • Ansible / Puppet / Chef means that lots of projects now build from the base box image, i.e. CentOS6 or Ubuntu 14.04 etc. • Do you want to be building this each build? Some of you are clever, and don’t. Some of you are clever, but didn’t consider an alternative, or didn’t know how. Maybe some of you don’t even use automated builds… • Using Packer and your provisioner of choice, you can export the artefact and store it as a Docker Container or Image in your cloud provider (Amazon AMI, OpenStack Glance, etc). Foundation Images 29
  30. 30. https://joind.in/talk/55008 @bobbyjason • Tool for creating identical machine images • Supports multiple platforms • Supports many provisioners (Ansible, Chef, Puppet, Bash.. etc.) • Can export image in multiple formats AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc. What Is Packer? 30
  31. 31. https://joind.in/talk/55008 @bobbyjason Packer template.json 31 { "variables": { "aws_access_key": "", "aws_secret_key": "" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-fce3c696", "instance_type": "t2.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }] }
  32. 32. https://joind.in/talk/55008 @bobbyjason Ansible Provisioning 32 "provisioners": [ { "type": "shell", "inline": [ "rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-7.noarch.rpm", "yum -y update", "yum -y install ansible", "ansible --version" ] },{ "type": "ansible-local", "playbook_file": "./ansible/playbook.yml", "role_paths": [ "./ansible/roles/init", "./ansible/roles/server", "./ansible/roles/mongodb", "./ansible/roles/php7", "./ansible/roles/nginx", "./ansible/roles/supervisord", "./ansible/roles/redis" ], "group_vars": "./ansible/common/group_vars" }
  33. 33. The cool bit; Putting it all together!
  34. 34. https://joind.in/talk/55008 @bobbyjason Use Packer To Build CoreOS Image 34 { "_comment": "Template file pulling from CoreOS", "builders": [ { "type": "openstack", "image_name": "CoreOS-Docker-Alpha-1-13", "source_image": "8e892f81-2197-464a-9b6b-1a5045735f5d", "flavor": "c46be6d1-979d-4489-8ffe-e421a3c83fdd", "ssh_keypair_name": "ukcloudos", "ssh_private_key_file": "/Users/bobby/.ssh/ukcloudos", "use_floating_ip": true, "floating_ip_pool": "internet", "ssh_username": "core", "ssh_pty" : true } ],
  35. 35. https://joind.in/talk/55008 @bobbyjason Use Packer To Build CoreOS Image 35 "provisioners": [ { "type": "shell", "inline": [ "sudo sh -c 'echo GROUP=alpha > /etc/coreos/update.conf'", "sudo systemctl restart update-engine", "sudo update_engine_client -update", "sudo sh -c 'mkdir /opt/'", "sudo sh -c 'mkdir /opt/bin'", "sudo sh -c 'curl -L https://github.com/docker/compose/releases/download/1.9.0/docke "sudo sh -c 'chmod +x /opt/bin/docker-compose'" ] },{ "type": "file", "source": "/Users/bobby/.ssh/ukcloudos", "destination": "/home/core/.ssh/key.pem" } ] }
  36. 36. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - Nginx 36 FROM nginx:alpine RUN apk update && apk upgrade && apk add --update curl wget bash tree RUN mkdir -p /srv/web ADD ./conf/default.conf /etc/nginx/conf.d/default.conf ADD ./conf/fastcgi_params /etc/nginx/fastcgi_params ADD ./index.php /srv/web CMD ["nginx", "-g", "daemon off;"]
  37. 37. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - Nginx 37 server { listen *:80; server_name _ localhost; root /srv/web; index index.php index.html index.htm; location / { try_files $uri $uri/ /index.php?$query_string; } location ~* .(jpg|jpeg|png|gif|ico|css|js|woff)$ { expires 365d; } location ~* .php$ { fastcgi_pass php-fpm:9000; fastcgi_index index.php; fastcgi_read_timeout 180; include fastcgi_params; } }
  38. 38. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - Nginx 38 fastcgi_param QUERY_STRING $query_string; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param REQUEST_URI $request_uri; fastcgi_param DOCUMENT_URI $document_uri; fastcgi_param DOCUMENT_ROOT $document_root; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param REQUEST_SCHEME $scheme; fastcgi_param HTTPS $https if_not_empty; fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx/$nginx_version; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; # PHP only, required if PHP was built with --enable-force-cgi-redirect fastcgi_param REDIRECT_STATUS 200;
  39. 39. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - php-fpm 39 FROM php:7.0.8-fpm-alpine RUN apk update && apk upgrade && apk add --update curl wget bash tree autoconf gcc g++ make libffi-dev openssl-dev RUN apk add supervisor RUN pecl install xdebug && docker-php-ext-enable xdebug && docker-php-ext-install opcache RUN docker-php-ext-install pdo_mysql RUN mkdir -p /var/log/supervisor COPY ./conf/php-dev.ini /usr/local/etc/php/ COPY ./conf/php-prod.ini /usr/local/etc/php/ COPY ./conf/php-prod.ini /usr/local/etc/php/php.ini COPY ./conf/envars-development.conf /usr/local/etc/php-fpm.d COPY ./conf/supervisord.conf /etc WORKDIR /srv COPY ./index.php /srv/web/ WORKDIR /srv RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]
  40. 40. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - php-fpm 40 <?php echo 'Welcome to your php-fpm Docker container. You should copy your application into the /srv folder and overwrite this file.'; var_dump($_SERVER); phpinfo(); [program:php-fpm] command=php-fpm --nodaemonize -c /usr/local/etc/php/php-%(ENV_APPLICATION_ENV)s.ini
  41. 41. https://joind.in/talk/55008 @bobbyjason Build Basic Docker Images - docker-compose.yml 41 version: "2" services: php-fpm: build: ./php-fpm image: bobbydvo/ukc_php-fpm:latest ports: - "9000:9000" environment: - APPLICATION_ENV=dev web: build: ./nginx image: bobbydvo/ukc_nginx:latest ports: - "80:80" $ docker-compose up —build
  42. 42. https://joind.in/talk/55008 @bobbyjason Build Docker Images Regularly 42 #!/bin/bash VERSION=1 CONTAINER=$1 BUILD_NUMBER=$2 if [[ $CONTAINER == 'all' ]]; then for CONTAINER in php-fpm nginx dynamodb consul; do docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUIL done exit fi docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER docker push bobbydvo/ukc_$CONTAINER:latest docker push bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
  43. 43. https://joind.in/talk/55008 @bobbyjason How NOT to Build Your Jenkins Job 43 echo $BUILD_NUMBER docker -v whoami sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm sudo ./build.sh $CONTAINER $BUILD_NUMBER
  44. 44. https://joind.in/talk/55008 @bobbyjason How to Build Your Jenkins Job 44 docker login -u bobbdvo -p Lr6n9hrGBLNxBm .docker/config.json: { "auths": { "https://index.docker.io/v1/": { "auth": “************************************” } }
  45. 45. https://joind.in/talk/55008 @bobbyjason How to Build Your Jenkins Job 45 echo $BUILD_NUMBER docker -v whoami #sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm sudo ./build.sh $CONTAINER $BUILD_NUMBER
  46. 46. https://joind.in/talk/55008 @bobbyjason How to Build Your Jenkins Job 46
  47. 47. https://joind.in/talk/55008 @bobbyjason Building DummyPHP Docker Image 47 FROM bobbydvo/ukc_php-fpm:latest WORKDIR /srv COPY . /srv/ WORKDIR /srv RUN composer install CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]
  48. 48. https://joind.in/talk/55008 @bobbyjason On Merge Jenkins Hook 48 set -e DUMMY_VERSION=$BUILD_VERSION NGINX_VERSION='latest' sudo docker-compose build sudo docker run -i bobbydvo/dummyapp_php-fpm /srv/vendor/bin/phpunit -c /srv/app/phpunit.xml # tag & push only if all the above succeeded (set -e) sudo docker tag bobbydvo/dummyapp_php-fpm:latest bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION sudo docker push bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION sudo docker push bobbydvo/dummyapp_php-fpm:latest ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null core@51.179 ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null core@51.179
  49. 49. https://joind.in/talk/55008 @bobbyjason Infrastructure Next 49
  50. 50. https://joind.in/talk/55008 @bobbyjason Terraform Build Infrastructure - Manager1 50 resource "openstack_compute_instance_v2" "swarm_manager" { name = "swarm_manager_0" count = 1 #coreos-docker-alpha image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"] user_data = "${data.template_file.cloudinit.rendered}" network { name = "${openstack_networking_network_v2.example_network1.name}" floating_ip = "${openstack_networking_floatingip_v2.example_floatip_manager.address}" }
  51. 51. https://joind.in/talk/55008 @bobbyjason Terraform - Cloudinit.sh 51 data "template_file" "cloudinit" { template = "${file("cloudinit.sh")}" vars { application_env = "dev" } }
  52. 52. https://joind.in/talk/55008 @bobbyjason Terraform - Cloudinit.sh 52 docker swarm init docker swarm join-token --quiet worker > /home/core/worker-token docker swarm join-token --quiet manager > /home/core/manager-token docker stack deploy --compose-file /home/core/docker-compose.yml mystack > /dev/nu
  53. 53. https://joind.in/talk/55008 @bobbyjason Terraform - Master X 53 resource "openstack_compute_instance_v2" "swarm_managerx" { name = "swarm_manager_${count.index+1}" count = 2 #coreos-docker-beta image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"] user_data = "${data.template_file.managerinit.rendered}" network { name = "${openstack_networking_network_v2.example_network1.name}" } }
  54. 54. https://joind.in/talk/55008 @bobbyjason Terraform - Managerinit.sh 54 data "template_file" "managerinit" { template = "${file("managerinit.sh")}" vars { swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4} } }
  55. 55. https://joind.in/talk/55008 @bobbyjason Terraform - ManagerInit.sh 55 # Copy Tokens from master1 => masterX sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o U # Copy docker-compose.yml file sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o U sudo docker swarm join --token $(cat /home/core/manager-token) ${swarm_manager}
  56. 56. https://joind.in/talk/55008 @bobbyjason Manager Cluster Demo 56
  57. 57. https://joind.in/talk/55008 @bobbyjason Adding Workers 57 resource "openstack_compute_instance_v2" "swarm_slave" { name = "swarm_slave_${count.index}" count = "${var.swarm_node_count}" #coreos-docker-beta image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10" flavor_id = "c46be6d1-979d-4489-8ffe-e421a3c83fdd" key_pair = "${openstack_compute_keypair_v2.test-keypair.name}" security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"] user_data = "${data.template_file.slaveinit.rendered}" network { name = "${openstack_networking_network_v2.example_network1.name}" } }
  58. 58. https://joind.in/talk/55008 @bobbyjason Adding Workers - slaveinit.sh 58 data "template_file" "slaveinit" { template = "${file("slaveinit.sh")}" vars { swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}" node_count = "${var.swarm_node_count + 3}" } }
  59. 59. https://joind.in/talk/55008 @bobbyjason Adding Workers - slaveinit.sh 59 sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null sudo docker swarm join --token $(cat /home/core/worker-token) ${swarm_manager} ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /ho ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /ho ## Forces redistribution across all nodes ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /ho ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /ho
  60. 60. https://joind.in/talk/55008 @bobbyjason Adding Workers - How Many? 60 variable "swarm_node_count" { default = 1 } variable "swarm_node_count" { default = 5 }
  61. 61. https://joind.in/talk/55008 @bobbyjason • Docker will bring up another container. Lets try. What If A Container Dies? 61
  62. 62. https://joind.in/talk/55008 @bobbyjason Final Demo + Questions 62
  63. 63. https://joind.in/talk/55008 @bobbyjason • Provisioned Docker Containers • Infrastructure as Code • Automated Deployments for CI / CD • Scalable Architecture • Openstack + UKCloud There you have it! 63
  64. 64. Thank you :-) Bobby DeVeaux @bobbyjason https://joind.in/talk/55008

×