AWS re:Invent 2016: DevOps on AWS: Advanced Continuous Delivery Techniques (DEV403)
•
6 likes•1,972 views
Continuous delivery makes teams more agile and quickens the pace of innovation. Too often, though, teams adopt continuous delivery without putting the right safety mechanisms in place. In this talk, we'll transform a simple but typical software release process into one that is safe. We'll use DevOps techniques like continuous integration, a variety of non-production testing stages, rollbacks, machine redundancy, Availability Zone redundancy, canary deployments, canary tests, and dashboards. We'll use AWS Lambda, AWS CloudFormation, AWS CodePipeline, AWS CodeDeploy, Amazon CloudWatch alarms and dashboards, and AWS Elastic Beanstalk.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to AWS re:Invent 2016: DevOps on AWS: Advanced Continuous Delivery Techniques (DEV403)
Similar to AWS re:Invent 2016: DevOps on AWS: Advanced Continuous Delivery Techniques (DEV403) (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
AWS re:Invent 2016: DevOps on AWS: Advanced Continuous Delivery Techniques (DEV403)
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mark Mansour, Senior Manager, Continuous Delivery November 30, 2016 DEV403 DevOps on AWS Advanced Continuous Delivery Techniques
- 2. What to expect from the session Make your pipeline safer by 1. Identifying production issues quickly 2. Deploying changes safely 3. Automatically deciding when to release changes
- 3. Techniques 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates
- 4. Starting Point: The release process is automated
- 5. Prerequisites • Versioned source • Automated build • Automated deployments • Deploy to > 1 instance • Unit tests • Integration tests • Continuous Delivery • Operations dashboard Source Build Deploy to Integration Stack Integration Tests Deploy to Production
- 6. Best practices with your tools • Focus in on best practices • Keep using your current tools where possible • Deployment tools • Continuous Integration and Continuous Delivery Tools • Extend your current tools when needed • This talk uses AWS tools
- 7. Tools used in this talk Monitoring Amazon CloudWatch Software Development Amazon SNS AWS Lambda Deployment AWS CodeDeploy AWS CodePipeline
- 8. MyApp CodeCommit Source Build CodeCommit Build DeployToInteg CodeDeploy Integration IntegTest End2EndTester DeployToProd CodeDeploy Production Source Build Deploy to Integration Stack Integration Tests Deploy to Production Model the release process in CodePipeline Pipeline Run Action Stage Pipeline Source change • starts a run; and • creates an artifact to be used by other actions. Change 1
- 9. Release and deploy process: Starting point MyApp CodeCommit Source Build Build Build DeployToInteg CodeDeploy Integration IntegTest End2EndTester DeployToProd CodeDeploy Production CodeDeploy
- 10. 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates Techniques
- 11. Be aware when a service is unavailable Problem: A service can stop working at any time for reasons inside or outside of its control. Consequence: Your service may be unavailable without your team knowing about it. 1 of 5 – Continuous production testing
- 12. Use synthetic traffic to simulate real users • Test all business critical functionality (UI and APIs) • Tests must run quickly • Measure client latencies • Check for reachability 1 of 5 – Continuous production testing
- 13. Synthetic Traffic How synthetic traffic flows CloudWatch Alarm 1 of 5 – Continuous production testing CloudWatch Events (1m)
- 14. CloudWatch Events (1m) Synthetic Traffic Synthetic traffic flow – why two metric streams? CloudWatch Alarm 1 of 5 – Continuous production testing
- 15. Building a Synthetic Traffic Test
- 16. Building a synthetic traffic test • Keep it simple • Build logic in Lambda (invoke with CloudWatch Events) • Capture data in CloudWatch metrics 1 of 5 – Continuous production testing
- 17. Lambda’s synthetic traffic blueprint 1 of 5 – Continuous production testing
- 18. Scheduling the synthetic traffic test 1 of 5 – Continuous production testing
- 19. Building a synthetic traffic test - Code 1 of 5 – Continuous production testing
- 20. Building a synthetic traffic test – Alarming 1 of 5 – Continuous production testing
- 21. Release and deploy process: Synthetic traffic DeployToProd CodeDeploy Production Synthetic Traffic CodeDeploy
- 22. 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates Techniques
- 23. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2 Rolling deployments – success Production Fleet ELB 2 of 5 – Manage deployment health
- 24. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2 Rolling deployments – fail Production Fleet ELB 2 of 5 – Manage deployment health
- 25. Check for deployment failures in production Problem: There are no automated tests to verify a service is working after a new deployment. Consequence: Each production deployment needs to be checked manually. 2 of 5 – Manage deployment health
- 26. Add safety to rolling deployments 1. Validate each host’s health 2. Ensure a minimum percentage of the fleet is healthy 3. Rollback if the deployment failed 2 of 5 – Manage deployment health
- 28. Step 1: Deployment Validation – AppSpec.yml 2 of 5 – Manage deployment health
- 29. V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 Step 1: Working tests raises more issues Production Fleet ELB 2 of 5 – Manage deployment health Failed Deployment
- 30. 4 failures – 60% healthy MHH 70%, 10 hosts: V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2 Step 2: Use minimum healthy hosts Production Fleet ELB 2 of 5 – Manage deployment health 1 failure – 90% healthy
- 31. Step 2: Use minimum health hosts - CodeDeploy 2 of 5 – Manage deployment health
- 32. Step 3: Rollback when a deployment fails • CodeDeploy: configured in deployment group 2 of 5 – Manage deployment health
- 33. Release and deploy: Deployment health DeployToProd CodeDeploy Production Synthetic Traffic CodeDeploy
- 34. 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates Techniques 3 of 5 - Segment production
- 35. Bad changes must not affect all customers Pipeline Problem: When a critical issue reaches production all hosts are affected. Consequence: Bad changes impact all customers. 3 of 5 - Segment production
- 36. Lower deployment risk by segmenting 1. Break production into multiple segments 2. Deploy to a segment 3. Test a segment after a deployment 4. Repeat 2 & 3 until done 3 of 5 - Segment production
- 38. Step 1: Break production into multiple segments Typical segment types: • Region • Availability Zone • Sub-Zonal • Single Host (Canary) 3 of 5 - Segment production US-EAST-1 US-EAST-1A US-EAST-1B
- 39. V2 V2 V2V2V1 V1V1 Step 1: Typical deployment segmentation Availability Zone based Deployment Availability Zone based Deployment Availability Zone based Deployment V2 V2V2V1 V1V1 V2 V2V2V1 V1V1 Production Fleet Post-deployment test 3 of 5 - Segment production Canary Deployment V1 Region based Deployment
- 40. Step 1: Use deployment groups as segments Create deployment groups per segment using: • Tags • Auto Scaling groups 3 of 5 - Segment production
- 41. Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy DeployToInteg CodeDeploy Integration IntegTest End2EndTester 1. Deploy to smallest segment 2. Post-deployment tests 3. Deploy to one Availability Zone 4. Post-deployment tests 5. Deploy to remaining Availability Zones Step 2: Deploy to each segment 3 of 5 - Segment production
- 42. Step 3: Test each segment A deployment is valid if: • The test has gathered enough data to gain confidence • CloudWatch metrics • No service alarms have fired • CloudWatch alarms • The test has not timed out • Code 3 of 5 - Segment production
- 43. Add segment tests to your pipeline Extend CodePipeline with: • Test Actions • Lambda Invoke Actions • Custom Actions • Approval Actions 3 of 5 - Segment production 1 hour timeout 7 day timeout
- 44. Use CodePipeline approvals to trigger tests Source MyAppSource CodeCommit Deploy DeployToSegment CodeDeploy SNS topicValidateSegment Approval putApprovalResult Approval message 3 of 5 - Segment production DeployToSegment CodeDeploy
- 45. Use SNS to start an automated approval check 3 of 5 - Segment production
- 46. Creating a post-deployment test Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval SNS topic Lambda Function registerDeployTest() Lambda Function evaluateDeploy() DynamoDB CloudWatch Events (1m) Change 1 Prod-us-east-1a CodeDeploy alarmtimeusage 3 of 5 - Segment production
- 47. Post-deployment test – registerDeployTest Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval SNS topic Lambda Function registerDeployTest() Lambda Function evaluateDeploy() DynamoDB CloudWatch Events (1m) Change 1 Prod-us-east-1a CodeDeploy alarmtimeusage 3 of 5 - Segment production
- 48. registerDeployTest function – (Node.js 4.3) 3 of 5 - Segment production
- 49. Post-deployment test – evaluateDeployTest Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval SNS topic Lambda Function registerDeployTest() Lambda Function evaluateDeploy() DynamoDB CloudWatch Events (1m) Change 1 Prod-us-east-1a CodeDeploy alarmtimeusage 3 of 5 - Segment production
- 50. approveValidation function (Node.js 4.3) 3 of 5 - Segment production
- 51. Canary Deployments – they’re different All production hosts: • Participates in serving production traffic • Configured as a production instance • Participates in production metrics stream Canary hosts: • Has its own metrics stream • Canary validations use the canary metric stream 3 of 5 - Segment production
- 52. Summary: Segment production • Segment production to reduce impact of a bad change • Minimum segmentation: • Region • Canary deployment per region • Larger service segmentation • Zonal • Sub-zonal • Test each segment before moving on 3 of 5 - Segment production
- 53. Release and deploy: Segment production Synthetic Traffic CodeDeploy Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy DeployToProd CodeDeploy Production
- 54. 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates Techniques 3 of 5 - Segment production
- 55. 4 of 5 – Halt promotions EC2 instance Change 2Change 3 Don’t change the system under test Source MyAppSource CodeCommit Build MyAppBuild Build DeployToProd MyApp CodeDeploy deploys Change 1
- 56. Don’t compound problems during an outage Pipeline Problem: The pipeline is unaware of the health of the infrastructure that it is deploying to Consequence: Production changes, usually deployments, can make it difficult for an operator to resolve a production event. 4 of 5 – Halt promotions
- 58. Source MyAppSource CodeCommit Build MyAppBuild Build DeployToProd MyApp CodeDeploy Change 1Change 2 Automatically stop deploying to production during an event CloudWatch Synthetic Traffic deploys checks CloudWatch Events (1m) triggers emitsdisables disableTransition() Alarm EC2 instance SNS 4 of 5 – Halt promotions
- 59. disableTransition function (Lambda Node.js 4.3) 4 of 5 – Halt promotions
- 60. Enable production deployments - CodePipeline 4 of 5 – Halt promotions
- 61. Summary: Halt promotions • Halt promotions to production when your production environment has “issues” • Automate by disabling stage transitions 4 of 5 – Halt promotions
- 62. Release and deploy: Halt promotions Synthetic Traffic CodeDeploy Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy
- 63. 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates 1. Continuous production testing 2. Manage deployment health 3. Segment production 4. Halt promotions 5. Gates Techniques 3 of 5 - Segment production
- 64. Do not deploy at sensitive times Problem: A bad change during sensitive times has a disproportionate affect on the business. Consequence: Issues during sensitive days risk reputation and financial loss. 5 of 5 - Gates
- 65. Adding safety with deployment black-days Deploy to production during normal conditions • Halt deployments during sensitive times Building a black-day calendar with CodePipeline: • Use Approvals to pause production deployments • Lambda to automatically approve when the time is right 5 of 5 - Gates
- 67. Black-day test Source MyAppSource CodeCommit Build MyAppBuild Build Deploy BlackDayCheck Approval ProductionDeploy CodeDeploy SNS topic Lambda Function registerDeployment Lambda Function processTimeWindows DynamoDB CloudWatch Events (1m) Change 1 5 of 5 - Gates
- 68. This looks familiar… Source MyAppSource CodeCommit Build MyAppBuild Build Deploy BlackDayCheck Approval ProductionDeploy CodeDeploy SNS topic Lambda Function registerDeployment Lambda Function processTimeWindows DynamoDB CloudWatch Events (1m) 5 of 5 - Gates
- 69. This looks familiar – post-deployment test Source MyAppSource CodeCommit Build MyAppBuild Build Deploy CanaryDeploy CodeDeploy ValidateCanary Approval SNS topic Lambda Function registerDeployTest() Lambda Function evaluateDeploy() DynamoDB CloudWatch Events (1m) Prod-us-east-1a CodeDeploy alarmtimeusage 3 of 5 - Segment production
- 70. What’s the difference? Source MyAppSource CodeCommit Build MyAppBuild Build Deploy BlackDayCheck Approval ProductionDeploy CodeDeploy SNS topic Lambda Function registerDeployment Lambda Function processTimeWindows DynamoDB CloudWatch Events (1m) 5 of 5 - Gates
- 71. Summary: Gates • Black-days provide centralized control • Add common action to all pipelines • Black-days are a type of gate • Implement with Approval actions in CodePipeline 5 of 5 - Gates
- 72. Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy CheckBlackDays Approval Release and deploy: Gates Synthetic Traffic CodeDeploy Production CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy
- 73. What we’ve learned Goal: Make your pipeline safer… 1. Identify production issues quickly • Continuous Production Testing 2. Safely deploy changes • Manage deployment health • Segment production 3. Automatically decide when to release changes • Halt promotions • Black-days and Gates
- 74. Release and deploy process: Ending point DeployToProd CodeDeploy Production CodeDeploy Synthetic Traffic CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy CheckBlackDays Approval CanaryDeploy CodeDeploy PostDeployTest Approval Deploy-AZ-1 CodeDeploy PostDeployTest Approval Deploy-AZ-2 CodeDeploy Deploy-AZ-3 CodeDeploy Production
- 75. Thank you!
- 76. Remember to complete your evaluations!
- 77. Code is available online • github.com/awslabs/aws-codepipeline-time-windows • github.com/awslabs/aws-codepipeline-synthetic-tests • github.com/awslabs/aws-codepipeline-block-production
- 78. Related Sessions • DEV303 – Deploying and Managing .NET Pipelines and Microsoft Workloads • DEV310 – DevOps on AWS: Choosing the Right Software Deployment Technique • DEV313 – Infrastructure Continuous Deployment Using AWS CloudFormation • SVR307 – Application Lifecycle Management in a Serverless World
- 79. Author: Slides written and prepared by Mark Mansour, Senior Manager, Continuous Delivery, AWS. This presentation, “DevOps on AWS: Advanced Continuous Delivery Techniques”, was originally given at re:Invent 2016 on Nov 30.