Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018

873 views

Published on

Continuous delivery (CD) enables teams to be more agile and quickens the pace of innovation. Too often, however, teams adopt CD without putting the right safety mechanisms in place. In this talk, we discuss opportunities for you to transform your software release process into a safer one. We explore various DevOps best practices, showcasing sample applications and code with AWS CodePipeline and AWS CodeDeploy. We discuss how to set up delivery pipelines with nonproduction testing stages, failure cases, rollbacks, redundancy, canary testing and blue/green deployments, and monitoring. We discuss continuous delivery practices for deploying to Amazon EC2, AWS Lambda, and containers such as Amazon ECS or AWS Fargate.

Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Advanced Continuous Delivery Best Practices Curtis Rissi Senior Solutions Architect AWS D E V 3 1 7 Felipe Almeida Senior Software Dev Engineer AWS Developer Tools
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What you’ll learn from the session • Some of the continuous deployment best practices of Amazon • How and why you should employ these practices yourself • How to achieve continuous deployment nirvana • Leaving positive feedback in the post-session survey feels good
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is continuous delivery? Source Build Pre- Test Deploy Post- Test
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Basic continuous delivery best practices • Versioned source • Automated build • Automated deployments • Deploy to > one instance • Unit tests • Integration tests • Continuous delivery • Operations dashboard Source Build Deploy to integration stack Integration tests Deploy to production
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tools used in this talk Monitoring Amazon CloudWatch Software development Amazon SNS AWS Lambda Continuous delivery AWS CodeDeploy Continuous deployment AWS CodePipeline
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MyApp CodeCommit Source Build CodeCommit Build DeployToInteg CodeDeploy Integration IntegTest End2EndTester DeployToProd CodeDeploy Production Source Build Deploy to integration stack Integration tests Deploy to production Model the release process in CodePipeline Pipeline Run Action Stage Pipeline Source change • starts a run; and • creates an artifact to be used by other actions Change 1
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Release and deploy process: Starting point MyApp CodeCommit Source Build Build Build DeployToInteg CodeDeploy Integration IntegTest End2EndTester DeployToProd CodeDeploy Production Want to dive deep into to do this with Containers and Serverless? Check the session catalog or YouTube for: DEV309-R - CI/CD for Serverless and Containerized Applications
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. An automated pipeline should be … • Defined as code • Checked into a version control repository, such as AWS CodeCommit • Able to allow for extensibility through other AWS services or third- party tools • Able to provide FAST feedback on the success and failure of pipeline executions
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Opportunities for automation • Continuous integration processes: Builds, integration tests, UI testing, and more • Health checks • Application tests • Synthetic user tests and application performance monitoring • Notifications and alerts • AWS CloudWatch Alarms and third-party tools, such as Splunk, Datadog • Amazon SNS, Slack, Pagerduty, and more
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Build and unit tests 1. Trigger pipeline on source change 2. Build and unit tests 3. Deploy to integration environment 4. Execute UI tests 5. Execute integration tests Tests Source MyAppSource CodeCommit Build BuildAndUnitTests CodeBuild IntegrationDeploy CodeDeploy TestOnChrome CodeBuild TestOnChrome CodeBuild IntegTest End2EndTester
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 2: Notify on failed build and test Change 1 CloudWatch Events (Failed Action) Tests Source MyAppSource CodeCommit Build BuildAndUnitTests CodeBuild IntegrationDeploy CodeDeploy TestOnChrome CodeBuild TestOnChrome CodeBuild IntegTest End2EndTester Change 2 Lambda Function NotifySlackOnPipelineActionFailure()
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Builds on top of our foundation of automation • Purpose built to verify that a service is working after a new deployment • Helps to avoid needing to do this manually Manage deployment health
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2 Rolling deployments – Success
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2 Rolling deployments – Fail
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add safety to rolling deployments 1. Validate each host’s health 2. Ensure a minimum percentage of the fleet is healthy 3. Rollback if the deployment failed
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Deployment validation – AppSpec.yml
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 Step 1: Working tests raises more issues Failed deployment
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4 failures – 60% healthy MHH 70%, 10 hosts: V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2 Step 2: Use minimum healthy hosts 1 failure – 90% healthy
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 2: Use minimum health hosts - CodeDeploy
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 3: Rollback when a deployment fails • CodeDeploy: Configured in deployment group
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Release and deploy: Deployment health DeployToProd CodeDeploy Production Synthetic traffic
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Blue/green deploys Failed deployment Blue group Green group V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Blue/green deploys Successful deployment Blue group Green group V1 V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2 Auto Scaling Auto Scaling
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lower deployment risk by segmenting • Minimize the impact of deployment failures • Potentially catch issues before real users do • Enables you to roll back more quickly, with less impact
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Segmentation overview 1. Break production into multiple segments 2. Deploy to a segment 3. Test a segment after a deployment 4. Repeat 2 & 3 until done
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Break production into multiple segments Typical segment types: • Region • Availability Zone • Sub-Zonal • Single host (Canary) US-EAST-1 US-EAST-1A US-EAST-1B
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. V2 V2 V2V2V1 V1V1 Step 1: Typical deployment segmentation Availability Zone based deployment Availability Zone based deployment Availability Zone based deployment V2 V2V2V1 V1V1 V2 V2V2V1 V1V1 Production fleet Post-deployment test Canary deployment V1 Region based deployment
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Use deployment groups as segments Create deployment groups per segment using • Tags • Auto Scaling groups
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy DeployToInteg CodeDeploy Integration IntegTest End2EndTester Step 2: Deploy to each segment 1. Deploy to smallest segment 2. Post-deployment tests 3. Deploy to one availability zone 4. Post-deployment tests 5. Deploy to remaining availability zones 6. Repeat for other regions Production – Region 2 CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 3: Test each segment A deployment is valid if • The test has gathered enough data to gain confidence • CloudWatch metrics • No service alarms have fired • CloudWatch alarms • The test has not timed out • Code
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add segment tests to your pipeline Extend CodePipeline with: • Test Actions • Lambda Invoke Actions • Custom Actions • Approval Actions 1 hour timeout 7 day timeout
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use CodePipeline approvals to trigger tests Source MyAppSource CodeCommit Deploy DeployToSegment CodeDeploy ValidateSegment Approval putApprovalResult Approval message DeployToSegment CodeDeploy SNS Topic
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use SNS to start an automated approval check
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Creating a post-deployment test Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval Lambda Function registerDeployTest() Lambda Function evaluateDeploy() CloudWatch events (1m) Change 1 Prod-us-east-1a CodeDeploy AlarmTimeUsage SNS Topic
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post-deployment test – registerDeployTest Lambda Function registerDeployTest() Lambda Function evaluateDeploy() CloudWatch events (1m) Change 1 Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval Prod-us-east-1a CodeDeploy AlarmTimeUsage SNS Topic
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. registerDeployTest function – (Node.js 4.3)
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. registerDeployTest function – (Node.js 4.3)
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. registerDeployTest function – (Node.js 4.3)
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post-deployment test – evaluateDeployTest Lambda Function registerDeployTest() Lambda Function evaluateDeploy() CloudWatch events (1m) Change 1 Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval Prod-us-east-1a CodeDeploy AlarmTimeUsage SNS Topic
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. approveValidation function (Node.js 4.3)
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Canary deployments – They’re different All production hosts • Participates in serving production traffic • Configured as a production instance • Participates in production metrics stream Canary hosts • Has its own metrics stream • Canary validations use the canary metric stream
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary: Segment production • Segment production to reduce impact of a bad change • Minimum segmentation • Region • Canary deployment per region • Larger service segmentation • Zonal • Sub-zonal • Test each segment before moving on
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cross-region deployments • New as of November 2018 • Allows you to deploy to multiple regions from a single pipeline • Enables you to achieve lower latency and greater availability
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cross-region deployment actions Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeployR1 CodeDeploy ValidateCanaryR1 Approval Prod-us-east-1a CodeDeploy CanaryDeployR2 CodeDeploy ValidateCanaryR2 Approval Prod-us-west-2a CodeDeploy CanaryDeployR3 CodeDeploy ValidateCanaryR3 Approval Prod-ap-southeast-2a CodeDeploy Deploy R2 Deploy R3
  51. 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  53. 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Block non-compliant pipelines • Introducing changes or even new pipelines can cause serious problems • Downtime risk, reputation, and financial loss • Leverage AWS Config to ensure pipeline compliance before allowing them to deploy to production environments
  54. 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adding safety with AWS Config rules Build AWS Config rules • These alert when pipelines are not configured up to company best practices Building a pipeline which blocks production pushes on non-compliant pipelines • Use approvals to pause production deployments • Lambda to automatically approve when pipeline is compliant
  55. 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Approval action-based pipeline safety check Source MyAppSource CodeCommit Build MyAppBuild Build Deploy safetyCheck Approval ProductionDeploy CodeDeploy Lambda Function safetyCheck Change 1 SNS Topic
  56. 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Config rules-based pipeline safety check
  57. 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Config rules-based pipeline safety check Source MyAppSource CodeCommit Build MyAppBuild Build DeployToProd MyApp CodeDeploy CloudWatch Event (event- based) Lambda Function disablePushtoProduction
  58. 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary: Safety and compliance checks in your pipelines • Use AWS Config rules and CloudWatch events to automatically remediate non-compliant pipelines • Add common action to all pipelines • Provide developers a best practices pipeline to start from with AWS CloudFormation templates or AWS Service Catalog • Implement with approval actions in CodePipeline
  59. 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  60. 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy safetyCheck Approval Release and deploy: Gates Synthetic traffic Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy
  61. 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What we’ve learned Goal: Make your pipeline safer … 1. Identify production issues quickly Continuous production testing Automate notifications 2. Safely deploy changes Manage deployment health Segment production 3. Improve latency and availability Deploy across regions 4. Automatically decide when to release changes Block non-compliant pipelines
  62. 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Release and deploy process: Ending point DeployToProd CodeDeploy Production Synthetic traffic CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy safetyCheck Approval CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy Production
  63. 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Code is available online • github.com/awslabs/aws-codepipeline-time-windows • github.com/awslabs/aws-codepipeline-synthetic-tests • github.com/awslabs/aws-codepipeline-block-production
  64. 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related sessions • DEV303 – Deploying and Managing .NET Pipelines and Microsoft Workloads • DEV310 – DevOps on AWS: Choosing the Right Software Deployment Technique • DEV313 – Infrastructure Continuous Deployment Using AWS CloudFormation • SVR307 – Application Lifecycle Management in a Serverless World
  65. 65. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Curtis Rissi Felipe Almeida
  66. 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  67. 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Author: Original 2016 slides written and prepared by Mark Mansour, Senior Manager, Continuous Delivery, AWS. This presentation, “DevOps on AWS: Advanced Continuous Delivery Techniques,” was originally given at re:Invent 2016 on Nov 30, 2016. 2017 slides updated by Curtis Bray, Manager, AWS CodePipeline for DEV324 presentation at re:Invent 2017. This presentation was updated for Re:Invent 2018 by Leo Zhadanovsky, Curtis Rissi and Felipe Madrial

×