Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Screens | AWS re:Invent 2014

5,623 views

Published on

Some of the best businesses today are deploying their code dozens of times a day. How? By making heavy use of automation, smart tools, and repeatable patterns to get process out of the way and keep the workflow moving. Come to this session to learn how you can do this too, using services such as AWS OpsWorks, AWS CloudFormation, Amazon Simple Workflow Service, and other tools. We'll discuss a number of different deployment patterns, and what aspects you need to focus on when working toward deployment automation yourself.

Published in: Technology

(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Screens | AWS re:Invent 2014

  1. 1. November 13, 2014 | Las Vegas NV Chris Munns, AWS Solutions Architect
  2. 2. @chrismunns •munns@amazon.com
  3. 3. https://secure.flickr.com/photos/psd/4389135567/
  4. 4. Historically, there was no automation for developers:
  5. 5. Historically, development needed to be tightly controlled:
  6. 6. ?
  7. 7. QA Staging Dev Prod
  8. 8. QA Staging Dev Prod
  9. 9. QA Staging Dev ProdUnit TestsIntegration TestsUsability TestsPerformance TestsAcceptance TestsSystem TestsRegression TestsMonitoringA/B Tests
  10. 10. QA Staging Dev Prod
  11. 11. QA Staging Dev ProdUnit TestsIntegration TestsUsability TestsPerformance TestsAcceptance TestsSystem TestsRegression TestsMonitoringA/B Tests
  12. 12. ?
  13. 13. Where do we begin? https://secure.flickr.com/photos/stevendepolo/5749192025/
  14. 14. If you are part of an Ops / “DevOps” / “DevTools” / “CoreInfra” / whatever team that has developers as an internal customer, it’s your job to help them:
  15. 15. DEVQAPROD BarkerYour environments should be as similar to each other as possible!
  16. 16. Make the process so easy a Cavemancould do it* *provided they have the appropriate access to!
  17. 17. Complexity of the process isn’t necessarily bad, so long as not everyone in the organization HAS to know how the sausage is made https://secure.flickr.com/photos/erix/2657100921
  18. 18. https://secure.flickr.com/photos/jasoneppink/499531891Make the results of change visible to everyone who causes or deals with change!
  19. 19. Aim to reduce the “works on my machine” failures inherent with developing on one OS and running production on another:
  20. 20. Docker is really changing how applications are being built and deployed! NEW!
  21. 21. “A Better Dev/Test Experience: Docker and AWS” on Medium! https://medium.com/aws-activate-startup-blog/a-better-dev-test-experience- docker-and-aws-291da5ab1238 http://bit.ly/1saojKw
  22. 22. Dramatically lowers the complexity in running developer environments. Let’s set it up: [munns@maclaptop ~]$ vagrant init chef/centos-6.5 [munns@maclaptop ~]$ vagrant up [munns@maclaptop ~]$ vagrant ssh Last login: Fri Mar 7 16:57:20 2014 from 10.0.2.2 [vagrant@zekaih ~]$ uname -a Linux zekaih.munnsdev.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
  23. 23. •Continuous integration •Continuous deployment
  24. 24. •Continuous integration
  25. 25. Continuous deploymentalwayscontinuous integration. Continuous integration doesn’tmean that the code gets deployed to production every commit! https://secure.flickr.com/photos/isherwoodchris/6917253693
  26. 26. CI tools:
  27. 27. CI bits
  28. 28. Green is good!
  29. 29. Deploying code https://secure.flickr.com/photos/simononly/15386966677
  30. 30. Convenience Control Higher-level services Do it yourself AWSElastic Beanstalk AWSOpsWorks AWSCloudFormation AWS CodeDeploy AWS application/infrastructure management tools
  31. 31. AWS application/infrastructure management tools Convenience Control Higher-level services Do it yourself AWSElastic Beanstalk AWSOpsWorks AWSCloudFormation AWS CodeDeploy NEW!!
  32. 32. You’re not still configuring your servers by hand, right?
  33. 33. Options: Deciding factors: How are you getting the bits from your code repository to your destination environments?
  34. 34. Simplest of all methods. Use a deployment tool to either do a repo sync, or copy the raw files from one environment to another: Pros: –Easy to get started with –No need for midprocess packaging steps Cons: –Rollbacks could become a challenge –Harder to do at large scale AWS services can make it really easy to deploy from a repository: –AWS CodeDeploy –AWS Elastic Beanstalk –AWS OpsWorks
  35. 35. Example with OpsWorks: [root@saarbrucken infrahelper]# ll /srv/www/infrahelper/ total 8 lrwxrwxrwx 1 deploy apache 44 Oct 21 20:43 current -> /srv/www/infrahelper/releases/20141021204316 drwxr-xr-x 7 deploy apache 4096 Oct 21 20:43 releases drwxrwx---9 deploy apache 4096 Oct 21 20:43 shared <----------------------DEPLOY HAPPENS---------------------> [root@saarbrucken infrahelper]# ll /srv/www/infrahelper/ total 8 lrwxrwxrwx 1 deploy apache 44 Nov 7 21:44 current -> /srv/www/infrahelper/releases/20141107214310
  36. 36. Bundle up your code, deploy the bundle: Pros: –Very atomic solution to deploying code –Easy to track versions and changes across environments Cons: –Potentially very large deployable assets Package will be largely unique to the OS/language you are using:
  37. 37. Example using FPM (Effing Package Management) [munns@somehost ~]$ gem install fpm ….. [munns@somehost ~]$ git clone https://github.com/teknogeek0/ReInvent2014-InfraHelper.git …... [munns@somehost ~]$ fpm -s dir -t rpm -n "InfraHelper" -v 1.0 --epoch 1 ReInvent2014-InfraHelper/=/opt/InfraHelper Created package {:path=>"InfraHelper-1.0-1.x86_64.rpm"} [munns@somehost ~]$ rpm -ivh InfraHelper-1.0-1.x86_64.rpm ….. [munns@somehost ~]$ rpm -qa InfraHelper InfraHelper-1.0-1.x86_64 [munns@somehost ~]$ ls -l /opt/InfraHelper/ total 32 drwxr-xr-x 2 root root 4096 Nov 4 23:07 flow -rw-r--r--1 root root 156 Nov 4 23:06 Gemfile -rwxr-xr-x 1 root root 661 Nov 4 23:06 IHQueueWatcher_control.rb -rw-r--r--1 root root 2765 Nov 4 23:06 infrahelper_utils.rb …..
  38. 38. Generate a new AMI with your updated code on it. Launch it: Pros: –The most atomic way possible •Won’t affect any currently running instances –Can pretty easily run two versions side by side Cons: –Bit more work involved –Really have to think about data persistence –Have to think about how rollbacks would happen Bunch of tools to help you build AMIs quick and easy:
  39. 39. # ./packer validate webimage.json Template validated successfully. # ./packer build webimage.json amazon-ebs output will be in this color. ==> amazon-ebs: Inspecting the source AMI... ==> amazon-ebs: Creating temporary keypair: packer 5459736e-26a7-5983-db5a-df145dafa7e7 …. Build 'amazon-ebs' finished. ==> Builds finished. The artifacts of successful builds are: --> amazon-ebs: AMIs were created: us-west-2: ami-5f9bd36f # cat webimage.json { "variables": { "aws_access_key": "", "aws_secret_key": "" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "ami-b5a7ea85", "instance_type": ”m3.medium", "ssh_username": "ec2-user", "ami_name": "webserver {{timestamp}}", }] }
  40. 40. HOST METRICSSERVICEMETRICSLOG ANALYSISBUILD METRICS
  41. 41. AWS Marketplace and Partners • You can find, research, and buy software • Simple pricing, aligns with Amazon EC2 usage model • Launch in minutes • AWS Marketplace billing integrated into your AWS account • Can also find SaaS offerings from partners! • 1900+ products across 25 categories Learn more at: aws.amazon.com/marketplace
  42. 42. You’ve picked a deployment method; now how are you going to go about acting upon it? https://secure.flickr.com/photos/wscullin/3770015991
  43. 43. How do we go about rolling out our code? What gotchas are there?
  44. 44. Replace code on all of the instances without changing them or taking removing traffic: Elastic Load Balancing (ELB) Web/App instances Amazon DynamoDB MySQL Amazon RDS Instance Amazon ElastiCache Cache Node users v1 v2 Amazon Route 53
  45. 45. • Go through existing tier updating the application in batches: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v1
  46. 46. • Go through existing tier updating the application in batches: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v2 v1
  47. 47. • Go through existing tier updating the application in batches: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v2 v1
  48. 48. • Go through existing tier updating the application in batches: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v2
  49. 49. • Go through existing tier updating the application in batches: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v2
  50. 50. ”WebAutoScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "AvailabilityZones" : { "Fn::GetAZs" : { "Ref" : "AWS::Region" } }, "LaunchConfigurationName" : { "Ref" : ”WebAutoScalingLaunchConfig" }, "MaxSize" : ”20", "MinSize" : ”10" }, "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MinInstancesInService" : “6", "MaxBatchSize" : “2", "PauseTime" : "PT5M" } } } CloudFormation—Auto Scaling with rolling updates:
  51. 51. ”WebAutoScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "AvailabilityZones" : { "Fn::GetAZs" : { "Ref" : "AWS::Region" } }, "LaunchConfigurationName" : { "Ref" : ”WebAutoScalingLaunchConfig" }, "MaxSize" : ”20", "MinSize" : ”10" }, "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MinInstancesInService" : “6", "MaxBatchSize" : “2", "PauseTime" : "PT5M" } } } Replace 2 at a time, pause for 5 minutes before doing the next batch CloudFormation—Auto Scaling with rolling updates:
  52. 52. Elastic Beanstalk—batch deployments and rolling configuration updates:
  53. 53. CodeDeploy—rolling deployments:
  54. 54. CodeDeploy—rolling deployments:
  55. 55. CodeDeploy—rolling deployments:
  56. 56. We stand up a duplicate part of our infrastructure and slowly cut traffic over to it: As we shift more traffic over, let Auto Scaling grow/shrink our instances of the new or old application: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users v1 Amazon Route 53
  57. 57. We stand up a duplicate part of our infrastructure and slowly cut traffic over to it: As we shift more traffic over, let Auto Scaling grow/shrink our instances of the new or old application: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users Amazon Route 53 v1 ELB v2
  58. 58. We stand up a duplicate part of our infrastructure and slowly cut traffic over to it: As we shift more traffic over, let Auto Scaling grow/shrink our instances of the new or old application: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users v1 ELB v2 Amazon Route 53
  59. 59. We stand up a duplicate part of our infrastructure and slowly cut traffic over to it: As we shift more traffic over, let Auto Scaling grow/shrink our instances of the new or old application: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users v1 ELB v2 Amazon Route 53
  60. 60. We stand up a duplicate part of our infrastructure and slowly cut traffic over to it: As we shift more traffic over, let Auto Scaling grow/shrink our instances of the new or old application: ELB Web/App instances DynamoDB MySQL RDS Instance ElastiCache Cache Node users v2 Amazon Route 53
  61. 61. Some words of caution! https://secure.flickr.com/photos/e-coli/3888542890
  62. 62. Schema changes tied to deployments are a blocker to moving fast: Unlink this from code deploys:
  63. 63. Be prepared for things to go wrong! https://secure.flickr.com/photos/akyamada/4071735996
  64. 64. "Resources" : { "myMainSite" : { "Type" : "AWS::Route53::RecordSetGroup", "Properties" : { "HostedZoneName" : {"Ref" : "HostedZoneName"}, "Comment" : "Main site with fail whale.", "RecordSets" : [ { "Name" : {"Ref" : "DNSRecordName"}, "Type" : "A", "SetIdentifier": "Primary", "Failover": "PRIMARY", "AliasTarget" : { "HostedZoneId" : {"Ref" : "ELBHostedZoneID"}, "DNSName" : {"Ref" : "ELBDnsName"}, "EvaluateTargetHealth" : "True" } }, { "Name": {"Ref" : "DNSRecordName"}, "Type": "A", "SetIdentifier" : "Secondary", "Failover": "SECONDARY", "AliasTarget": { "HostedZoneId": { "Fn::FindInMap" : [ "S3RegionWebEndpoints", { "Ref" : "S3BucketRegion" }, "HostedZoneId" ]}, "EvaluateTargetHealth": "False", "DNSName": { "Fn::FindInMap" : [ "S3RegionWebEndpoints", { "Ref" : "S3BucketRegion" }, "DNSEndpoint" ]} } } ] } } } }
  65. 65. "Resources" : { "myMainSite" : { "Type" : "AWS::Route53::RecordSetGroup", "Properties" : { "HostedZoneName" : {"Ref" : "HostedZoneName"}, "Comment" : "Main site with fail whale.", "RecordSets" : [ { "Name" : {"Ref" : "DNSRecordName"}, "Type" : "A", "SetIdentifier": "Primary", "Failover": "PRIMARY", "AliasTarget" : { "HostedZoneId" : {"Ref" : "ELBHostedZoneID"}, "DNSName" : {"Ref" : "ELBDnsName"}, "EvaluateTargetHealth" : "True" } }, { "Name": {"Ref" : "DNSRecordName"}, "Type": "A", "SetIdentifier" : "Secondary", "Failover": "SECONDARY", "AliasTarget": { "HostedZoneId": { "Fn::FindInMap" : [ "S3RegionWebEndpoints", { "Ref" : "S3BucketRegion" }, "HostedZoneId" ]}, "EvaluateTargetHealth": "False", "DNSName": { "Fn::FindInMap" : [ "S3RegionWebEndpoints", { "Ref" : "S3BucketRegion" }, "DNSEndpoint" ]} } } ] } } } } Example template: http://bit.ly/113cNaa
  66. 66. HOST METRICSSERVICEMETRICSLOG ANALYSISEXTERNAL SITE METRICS
  67. 67. When things break during a deploy you’ll need to decide how to react: How do you decide? Deployment pattern and method will be deciding factors:
  68. 68. Have a real-time communication method for the entire company: Share knowledge:
  69. 69. QA Staging Dev Prod
  70. 70. We need robots https://secure.flickr.com/photos/spenceyc/7481166880
  71. 71. Good news, we’ve got lots of “robots”! https://secure.flickr.com/photos/peyri/10207629
  72. 72. (NEW!) (NEW!) (SOON!) (NEW!)
  73. 73. https://secure.flickr.com/photos/jeffedoe/506027963
  74. 74. Code Repository
  75. 75. Code Repository CI Infra CI SaaS
  76. 76. Code Repository CI Infra CI SaaS
  77. 77. Code Repository CI Infra CI SaaS Code Bundler
  78. 78. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket
  79. 79. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket
  80. 80. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWSOpsWorks
  81. 81. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWSOpsWorks Dev Web/App Servers
  82. 82. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWSOpsWorks Dev Web/App Servers Dev/ QA Users
  83. 83. Deploy Object Amazon S3 Bucket AWSOpsWorks Prod Web/App Servers Users Deployment Interface
  84. 84. Automation good! https://secure.flickr.com/photos/macwagen/94975613
  85. 85. ?
  86. 86.
  87. 87. ? https://secure.flickr.com/photos/dullhunk/202872717/
  88. 88. http://bit.ly/awsevals

×