Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Loft Your Web Platform into the Clouds with Immutable Infrastructure


Published on

Watch the full video here:
DrupalCon New Orleans - May 11, 2016
Steven Merrill, Phase2 Director of DevOps

Noticing that you automatically scaled up to double your normal footprint and back down after a wave of high traffic passed is a great feeling, especially if you only notice it on Monday morning when you look back at the weekend’s traffic in your monitoring system. Being able to use autoscaling systems to horizontally scale the web tier of your Drupal web platform is a great outcome, but getting to that point requires some planning and rigor in your build and automation processes.

The concept of immutable infrastructure revolves around building immutable deployment artifacts and then promoting them through your environments so that you can horizontally scale your web tier. These immutable deployment artifacts can be machine images or Docker images.

Phase2 has helped major media customers and sports leagues set up and maintain autoscaling infrastructures for their digital web platforms that include Drupal, and in this talk we will talk about the requirements for implementing an autoscaling environment in a public cloud like Amazon’s EC2, as well as discussing how this translates into the use of Docker and an application scheduler.

In this talk, we’ll go over the following topics:
- How to set up a build process to produce machine images (e.g. AMIs) or Docker images
- How to use autoscaling in practice to save money and be able to automatically handle traffic spikes
- Hear about our experiences implementing these solutions for major media and sports leagues
- How to handle horizontal autoscaling of Docker containers in production using Kubernetes

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Loft Your Web Platform into the Clouds with Immutable Infrastructure

  1. 1. Loft Your Web PlatformIntothe Clouds withImmutable Infrastructure Steven Merrill
  2. 2. © 2016 Phase2 OVERVIEW
  3. 3. 4© 2016 Phase2 TOPICS COVERED Immutability in software development Immutable Infrastructure Building immutable artifacts Using immutable artifacts with AWS autoscaling Using immutable artifacts with containers Real-world examples
  4. 4. © 2016 Phase2 MUTABILITY
  5. 5. 6© 2016 Phase2 im·mu·ta·ble adjective unable to be changed
  6. 6. 7© 2016 Phase2 IMMUTABILITY IN SOFTWARE DEVELOPMENT First-class immutable variables in language runtimes No need to synchronize read access in parallel systems Immutable data stores Elm’s time-travel debugging Mercurial’s immutable history / write-ahead log Immutable data structures in React.js
  8. 8. 9© 2016 Phase2 IMMUTABLE INFRASTRUCTURE? WHAT? Continuation of infrastructure as code Core tenet: build immutable artifacts and promote them As in development, try to limit mutation of deployed infra Unchanging artifacts (AMIs, containers) eliminate drift
  9. 9. 10© 2016 Phase2 AN EXAMPLE A client had autoscaling troubles Provisioned from scratch from a plain Ubuntu AMI Problems: Drift Time to provision / autoscaling timeouts Hard dependency on configuration management server
  10. 10. 11© 2016 Phase2 COUNTERPOINTS Most servers may not have read-only disks Container systems can allow read-only rootfs Will you really disable SSH? One theory: contaminate instances if you do
  11. 11. 12© 2016 Phase2 WHY? Immutable infra practices enable autoscaling Rallying cry: have cattle, not pets It can allow for easier rollback Possibility for canary or blue/green deployment From here on out, we’ll discuss autoscaling readiness
  12. 12. © 2016 Phase2 FOUNDATIONS: BUILD PROCESSES
  13. 13. 14© 2016 Phase2 BUILD THEORY A modern web app likely has several build steps Pull dependencies down Source compilation CSS/JS compilation and assembly Phase2 has grunt-drupal-tasks to automate this Build from drush make, npm theme tasks Has a package command to make a folder or tarball
  14. 14. 15© 2016 Phase2 VENDORING Consider vendoring your dependencies Great discussion about this in “Who Owns Your Availability?” episode of Arrested DevOps
  15. 15. 16© 2016 Phase2 BUILD TARGETS Deployment repository Tarball Machine image Amazon AMI / Google Private images Rkt / Docker tagged container image
  17. 17. 18© 2016 Phase2 BUILD PROCESS EVOLUTION Awful Build in-place on each of n web heads Better Build once in workspace, rsync to docroots Good Build once, store artifact, capistrano-style deploy Goal Build artifact, build image around artifact
  18. 18. 19© 2016 Phase2 BUILDING IMAGES Autoscaling-ready image builds have 2 components: Tagged code releases A combination of Linux packages on a filesystem Linux vulnerabilities happen fairly frequently Be ready to build for new code releases and package CVEs
  19. 19. © 2016 Phase2 AUTOSCALING
  20. 20. 21© 2016 Phase2 WHY AUTOSCALE? Reduces stress Automatic healing Cost benefits Performance benefits Automatic scaling based on metrics Manual adjustment up and down
  22. 22. 23© 2016 Phase2 DRUPAL SITE COMPONENTS Big 3 components of a running Drupal site Codebase on an app server Relational database Files directory Optional Object cache (?) Varnish / CDN
  23. 23. 24© 2016 Phase2 AUTOSCALING GROUPS IN AWS An autoscaling group is attached to one or more ELBs ELB health checks and timeouts control ASG actions You use CloudWatch alarms to trigger autoscaling Avg CPU > 70% for 5 min, avg CPU < 40% for 30 min A LaunchConfig ties together an AMI and instance size e.g. ami-beefbeef on c4.xlarge w/ 100 GB disks One ASG is set to one LaunchConfig at a time
  24. 24. 25© 2016 Phase2 AWS RECIPE FOR DRUPAL Jenkins instance building tarballs + AMIs Autoscaling group with public Drupal instances RDS for MySQL S3 bucket for files directory via amazons3.module ElastiCache for memcache via memcache.module CloudFront for page caching / distribution SES for email via smtp.module
  25. 25. 26© 2016 Phase2 EXAMPLE AMI AUTOSCALING SETUP Have one build process to create a tarball / build artifact Use Packer to start from a base AMI File provisioner copies code up Run Ansible to install packages Create new LaunchConfig Flip instances over one at a time
  26. 26. 27© 2016 Phase2 ELIDED PACKER EXAMPLE { "variables": { "aws_access_key" : "xxxxxxxxxxxxxxxxxxxx", "aws_secret_key" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "aws_instance_type" : "t2.micro", "east_aws_ami" : "ami-00000000", "west_aws_ami" : "ami-11111111", "aws_ssh_username" : "ec2-user", “east_aws_subnet" : "subnet-00000000”, “west_aws_subnet" : "subnet-11111111", "east_aws_vpc": "vpc-00000000", "west_aws_vpc": "vpc-11111111" },
  27. 27. 28© 2016 Phase2 ELIDED PACKER EXAMPLE "provisioners": [{ "type": "file", "source": "/path/to/my/z", "destination": "/opt/z" }, { "type" : "shell", "inline": [ "/usr/bin/ansible-playbook -ilocal /opt/playbook.yml” ]}], ],
  28. 28. 29© 2016 Phase2 ELIDED PACKER EXAMPLE "builders": [{ "type": "amazon-ebs", "name": "east-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "{{user `east_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-00000000"], "subnet_id" : "{{ user `east_aws_subnet` }}", "vpc_id": "{{user `east_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_east" },
  29. 29. 30© 2016 Phase2 ELIDED PACKER EXAMPLE { "type": "amazon-ebs", "name": "west-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "{{user `west_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-11111111"], "subnet_id" : "{{ user `west_aws_subnet` }}", "vpc_id": "{{user `west_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_west" }]}
  30. 30. 31© 2016 Phase2 S3 SHARP EDGES S3 files with amazons3/s3fs.module use memory_limit You may be tempted to try the s3fs daemon Instead, s3fs_cors/amazons3_cors modules
  31. 31. 32© 2016 Phase2 AUTOSCALING QUIRKS Schema updates Whitescreens on new/old as schema changes flip Add double capacity, let ELB health checks do the work Maintenance mode Separate editorial instance Build one image and switch behavior on ENV
  32. 32. © 2016 Phase2 CLIENT EXAMPLE
  33. 33. 34© 2016 Phase2 MLS DIGITAL - MLSSOCCER.COM AND CLUBS 21-site Drupal 7 platform League and all club sites Game day scaling patterns - huge traffic spikes No autoscaling, separate docroots Moved to multisite in prep Build tarballs from 500 MB to 40 MB Total opcache / APC savings of 1.6 GB
  34. 34. 35© 2016 Phase2 MOVING MLS TO AUTOSCALING Old setup provisioned from a base Ubuntu with Salt Timeout problems Moved to Packer AMI builds with all software & code An AMI can come up if everything else is down Systems of record for Drupal state are elsewhere MySQL database - Amazon RDS Memcache - Amazon ElastiCache Files - Amazon S3 via s3fs.module
  35. 35. 36© 2016 Phase2 SUCCESS From 4 pets to 2-5 cattle Major hosting cost reduction Later Drupal optimizations lowered instance size
  38. 38. AFTER TUNING
  39. 39. © 2016 Phase2 CONTAINERS
  40. 40. 41© 2016 Phase2 POTENTIAL DOCKER BENEFITS Increased density Ability to run heterogenous workloads Native CPU and disk performance Resource limits
  41. 41. 42© 2016 Phase2 DOCKER IN PRODUCTION Docker on single hosts is easy Docker in production requires a lot of coordination Overlay networking / network security Service discovery Intelligent scheduling and capacity management Routing / load balancing Secrets management Monitoring and performance management
  42. 42. 43© 2016 Phase2 DOCKER BUILDS Build considerations Dockerfile vs configuration management Dynamic templating Build tools docker build Packer Configuration management system
  43. 43. 44© 2016 Phase2 DOCKER Docker daemon allows for tagging Ideally, use tags from canonical repo and as branches p2/site:v1.0.0, p2/site:v1.1.0 p2/site:dev, p2/site:stage, p2/site:prod Build script docker build -t p2/site:v1.1.0 . docker tag p2/site:v1.1.0 p2/site:dev docker push p2/site:v1.1.0; docker push p2/site:dev
  44. 44. 45© 2016 Phase2 KUBERNETES Open-source container scheduler from Google “Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.” Based on Borg design principles Large pool of contributors and upstream/downstream users Red Hat (OpenShift 3) Deis (Helm) CoreOS (Tectonic)
  45. 45. 46© 2016 Phase2 KUBERNETES Requires you to bring overlay networking Features Service discovery, load balancing, cluster DNS Pod scaling and autoscaling, node draining Resource-aware scheduling and capacity limits REST API and CLI clients Secrets management YAML or JSON definition of k8s API objects
  46. 46. 47© 2016 Phase2 KUBERNETES OBJECTS Nodes Labels Pods DaemonSets Replication controllers Services Deployments
  47. 47. 48© 2016 Phase2 KUBERNETES POD YAML apiVersion: v1 kind: Pod metadata: name: lamp app: lamp spec: containers: - name: web image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: "2000m”
  48. 48. 49© 2016 Phase2 KUBERNETES RC YAML apiVersion: v1 kind: ReplicationController metadata: name: lamp spec: replicas: 5 selector: app: lamp template: metadata: labels: app: lamp spec: containers: - name: lamp image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: “2000m” ports: - containerPort: 8080 name: lamp protocol: TCP
  49. 49. 50© 2016 Phase2 KUBERNETES SERVICE YAML apiVersion: v1 kind: Service metadata: name: lamp spec: selector: app: lamp ports: - name: lamp port: 8080
  50. 50. © 2016 Phase2 SUMMARY
  51. 51. 52© 2016 Phase2 IMMUTABLE INFRASTRUCTURE Build immutable artifacts for deployment Limit configuration drift in your infrastructure Be ready to build new images for code releases or CVEs Autoscaling can help with cost, stress, and self-healing
  52. 52. © 2016 Phase2 QUESTIONS?
  53. 53. SoHow Was It?- Tell Us What YouThink Evaluate this session - Thanks!