Immutable Deployments with AWS CloudFormation and AWS Lambda

22,899 views

Published on

San Francisco AdvancedAWS Meetup, 2016-03-17

Published in: Technology

Immutable Deployments with AWS CloudFormation and AWS Lambda

  1. 1. (and AWS Lambda)
  2. 2. & aoepeople!
  3. 3. E-Commerce: Magento CMS: TYPO3 Portals: ZF, Symfony,… Mobile Searchperience: ElasticSearch 250+ people world-wide (in 8 locations) Global Enterprise Projects Infrastructure: AWS
  4. 4. CloudFormation Lambda “Immutable”
  5. 5. /i(m)ˈmyo͞ odəb(ə)l/ adjective unchanging over time or unable to be changed. “disposable” “ephemeral”
  6. 6. Pet Cattle
  7. 7. not disposable disposable
  8. 8. disposable not disposable disposable
  9. 9. Static Resources BuildVPC Build
  10. 10. Build Bucket IAM Setup VPC Static Resources Build Manual Setup separate CloudFormation Stacks use resources from underlying stacks as input parameters Lambda functions, Monitoring,…
  11. 11. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) VPC VPC Static Resources Static Resources Build BuildBuild Build Build X Build X+1 Build X Build X+1 Manual Setup different scopes (build, environment, account, global…)
  12. 12. Build Bucket IAM Setup Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev) IAM Setup VPC VPC VPC VPC Static Resources Static Resources Static Resources Static Resources Build BuildBuild Build Build Build Build Build Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1 Manual Setup Manual Setup Build Bucket Access
  13. 13. Private subnets Public subnets ElastiCache (Redis) with replication groups for cache and sessions RDS (multi-az) with DB subnet group Bastion server s3: media storage* Route 53: DNS configuration CloudFront distribution SSL Certificates Security group for Varnish servers Security group for Magento servers Security group for Load Balancer Static Resources
  14. 14. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy
  15. 15. Build Auto Scaling group Auto Scaling group Elastic Load Balancer Auto- Scaling Group Launch Configurati on Scaling Policy Auto Scaling group Auto Scaling group
  16. 16. +1 Baking AMIs
  17. 17. “Chef vs. Puppet?” http://fbrnc.net/blog/2015/11/ how-to-provision-an-ec2-instance “…Ansible!” “BASH!”
  18. 18. Keep it simple…!
  19. 19. ✔ ✔ (✔)
  20. 20. the “last mile” most underestimated CFN feature!
  21. 21. var r = require('cfn-response'); exports.handler = function (event, context) { […] var res = {}; if (event.RequestType == 'Create') { res.Password = randomPassword(20); } r.send(event, context, r.SUCCESS, res); };
  22. 22. "IndexerDb": { "Type": "AWS::RDS::DBInstance", "Properties": { [...] "MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]}, [...] } }, "GenerateDbPassword": { "Type": "Custom::PasswordGenerator", "Properties": { "ServiceToken": {"Ref": "PasswordGeneratorArn"} } },
  23. 23. • launch some ASGs (set DesiredCapacity) • create database passwords • tag all resources (incl. ElastiCache!) • restore database and media files • (one-time) install scripts (db migrations,…) • detect Varnish backends • wait for healthy backends in the ELBs • run infrastructure tests • cache warming • update Route53 records sets • delete old stacks • …
  24. 24. DesiredCapacity = Number of Instances in current ASG x 1.2 better safe than sorry…
  25. 25. DesiredCapacity = Number of Instances in current ASG x 1.2
  26. 26. DesiredCapacity = Number of Instances in current ASG x 1.2 ScalingPolicy takes care…
  27. 27. "CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } }, "FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },
  28. 28. Update Route53 record set
  29. 29. Update Route53 record set Delete old stacks
  30. 30. "UpdateR53": { "DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"], "Type": "Custom::Route53Update", "Properties": { "ServiceToken": {"Ref": "R53Updater"}, "Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]}, "HostedZoneId": {"Ref": "HostedZoneId"}, "AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]}, "AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]}, "Comment": "Updated via CloudFormation/Lambda" } }, "DeleteStacks": { "Condition": "DeleteOldStacks", "DependsOn": ["UpdateR53"], "Type": "Custom::StackDeleter", "Properties": { "ServiceToken": {"Ref": "StackDeleter"}, "TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"}, "ExceptStackName": {"Ref": "AWS::StackName"} } }
  31. 31. +1 Baking AMIs Tools+CI
  32. 32. Have fun filling out 47 form fields! :)
  33. 33. aoepeople/stackformation command-line tool (Symfony console, uses AWS SDK for PHP) integrates nicely into your CI (Jenknis,…)
  34. 34. pre-process Userdata Lambda JS “Template” superset of CFN template (JSON) Parameterslookup from other stacks, env vars,… Template Parameter Values create/ update Stack policies, Tags,…
  35. 35. Stack magento-stage-build-5 Stack magento-stage-build-6 Stack magento-prod-build-6 Stack magento-prod-build-5 CloudFormation Template merge & pre-process “CloudFormation+X” Template(s) + Dynamic Parameters + Stack Policies + Behavior + Tags … Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD} Blueprint magento-{env:ENVIRONMENT}-setup Stack magento-stage-setup Stack magento-prod-setup
  36. 36. blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}' enforce “immutability” by denying updates!
  37. 37. aoepeople/cfn-vpc aoepeople/cfn-lambdahelper aoepeople/cfn-amibaker via composer
  38. 38. so we can integrate this into our CI pipeline…
  39. 39.
  40. 40. aoepeople/awsinspector command-line tool (Symfony console, uses AWS SDK for PHP) Domain models for PHP
  41. 41. $repository = new AwsInspectorModelElbRepository(); $dns = $repository->findElbsByTags([ 'Environment' => 'deploy', 'Build' => 554, 'Type' => 'Frontend’ ])->getFirst()->getDNSName();
  42. 42. > bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build filter by tag Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) > • will take jump hosts into account (ProxyCommand) • auto-detects your local (encrypted) private keys • multiplexed ssh connections • run commands directly
  43. 43. logins, orders,… deployments, scaling activity,… JMeter response time, error rate,… CPU, load, network I/O,…
  44. 44. correlate metrics from various sources
  45. 45. GrafanaElasticSearch (Service) time-series database
  46. 46. leftovers from “disposed” resources
  47. 47. … “Ready for testing” (WaitCondition) setup terminate instances and wait • Create Security allowing access to all IPs and attach to UAT’s ELB environment (incl. restoring db+media snapshot from prod) environment* Delete self* *via Lambda custom resource Pushing samples in near real-time via custom JMeter backend listener “Testing Completed” (WaitCondition) JMeter test CloudFormation Stack GrafanaElasticSearch (Service) CloudWatch Lambda find relevant resources for the current deployment (via tags), collect metrics, and push them to ElasticSearch
  48. 48. CloudFormation Stack • Install JMeter • Download testcase from S3 • Signal “Ready” and wait • Create SG allowing access to all IPs and attach to UAT’s ELB Auto-Scaling Group of Load Generator Instances Run Stress Test2. Time timeout 3600 jmeter –n –t testcase.jmx Delete test environment* Delete self* 3. 4. *via Lambda custom resource … Spin up test environment (incl. restoring db+media snapshot from prod) 1. CloudFormation Stack setup terminate instances JMeter test Pushing samples in near real-time via custom JMeter backend listener “Ready for testing” (WaitCondition) “Testing Completed” (WaitCondition)
  49. 49. https://blogs.aws.amazon.com/application- management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS- CloudFormation-Stacks-with-Lambda-backed-Custom-Resou
  50. 50. Provisioning “Ready for baking” (WaitCondition) EC2 AMI Baker (Lambda) Node.js SDK AWS CLI “Baking Completed” (WaitCondition) shutdown (-> terminate) ec2.createImage aws ec2 wait image-available AMI Baker (Lambda) Node.js SDK ec2.deleteImage Custom CloudFormation Resource Time
  51. 51.
  52. 52. Follow me on twitter! My blog

×