Continuous delivery makes teams more agile and quickens the pace of innovation. Too often, though, teams adopt continuous delivery without defining the meta components or putting the right safety mechanisms in place. In this session, we'll start from the meta components and transform a typical software release process into one that will scale and is safe. We'll use DevOps techniques like continuous integration, a variety of non-production testing stages, rollbacks, redundancy, canary deployments and synthetic tests. We'll use AWS services such as Lambda, CloudFormation, CodePipeline, CodeBuild, CodeDeploy and CloudWatch alarms and dashboards. Speakers: Brent Maxwell, Partner Solutions Architect, Amazon Web Services Daniel Zoltak, Solutions Architect, Amazon Web Services

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brent Maxwell
Partner Solutions Architect
Amazon Web Services
Level 400
Advanced Continuous Delivery on AWS
Deliver Code Faster, Reliably and Safely
Daniel Zoltak
Solutions Architect
Amazon Web Services
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2. What to Expect From this Session
Learn some advanced techniques to make your pipelines safer:
1. How to identify Production issues quickly
2. How to deploy changes safely
3. How to automatically decide when to release changes
Check out AWS Labs on GitHub
• github.com/awslabs/

3. Tools Used in this Session
Management
Deployment
Software Development
AWS
CodeCommit
AWS
CodePipeline
AWS
CodeDeploy
AWS
CodeBuild
Amazon
CloudWatch
AWS
CloudFormation
AWS
Lambda
Amazon
S3
Amazon
DynamoDB
Amazon
SNS
Amazon
Rekognition

4. Release Process Phases
Source Build
Pre
Test
Deploy
Post
Test
Continuous integration
Continuous delivery
Continuous deployment

5. Characteristics of a Good Pipeline
Fast FeedbackExtensibleAs Code

6. Deployment Services - Build
AWS
Elastic Beanstalk
Amazon
Elastic Container Service
AWS
CodeDeploy
Mutable server Immutable servers
App v1
OS v1
Out of service
App v1
OS v1
Out of service

7. Deployment Services - Running
AWS
Elastic Beanstalk
Amazon
Elastic Container Service
AWS
CodeDeploy
Mutable server Immutable servers
App v1
OS v1
In service
App v1
OS v1
In service

8. Deployment Services - Update app
AWS
Elastic Beanstalk
Amazon
Elastic Container Service
AWS
CodeDeploy
Mutable server Immutable servers
App v2
OS v1
In service
App v1
OS v1
In service
App v2
OS v1
Out of serviceOut of service In service Out of service

9. Deployment Services - Update OS
AWS
Elastic Beanstalk
Amazon
Elastic Container Service
AWS
CodeDeploy
Mutable server Immutable servers
App v2
OS v2
In service
App v2
OS v1
In service
App v2
OS v2
Out of serviceOut of service In service Out of service

10. Version Control
DevOps DR
Test
Prod
Dev
Application
Infrastructure
CloudFormation
Deploy infrastructure only
AMI
Lint /
Validate
Template
Unit Test
Infra Code
Check in
Infra
Code
Deploy
Infra
Generate
AMIs
Artifact Repository
DevOps Infrastructure Lifecycle Management

11. Version Control
Build/
Compile
Code
DevOps
Unit Test
App Code
DR
Test
Prod
Dev
Check
in
Code
tar, war, zip,
yum, rpm
Deploy
App
Package
App
Deploy application only
Lint /
Validate
Template
Unit Test
Infra Code
Check in
Infra
Code
Deploy
Infra
Generate
AMIs
Artifact Repository
DevOps Application Lifecycle Management
Application
Infrastructure
CloudFormation
Deploy infrastructure only
AMI

12. Pipeline Demo
We have 1 repo with 2 pipelines:
Application pipeline
…and with some inception…
Infrastructure pipeline

13. Tomcat Stencil
CodeCommit
repo
1. Clone
Infrastructure Pipeline Demo
Bootstrap
CloudFormation
template
SummitApp
CodeCommit
repo
2. Execute 3. Create
Infrastructure
CodePipeline
Invokes Artifacts
Infrastructure
CloudFormation
template
Summit
App Code
Infrastructure
Stack
4. Create
Application
Pipeline
Elastic Load
Balancing
AWS
CodeDeploy
AWS
CodeBuild
Infrastructure Stack
Amazon EC2
Developers
5. Infrastructure
template update
6. Triggers 7. Stack
Update
Infrastructure Update Demo
Amazon EC2

14. Infrastructure Pipeline Demo

15. Application Evolution
App code is more complicated than Infra code.
Therefore, App pipelines are more complicated than Infra pipelines.

16. Application Deployment Patterns
Segment ProductionContinuous
Production Testing
Halt Promotions
with Gates
Synthetic Tests Canary Deployments Deployment Windows
Pattern
How

17. Synthetic Traffic
Synthetic Traffic flow
1 of 3 – Continuous production testing
CloudWatch
Events (1m)
CloudWatch
Alarm

18. Synthetic Traffic
Synthetic Traffic Flow – Why Two Metrics?
CloudWatch
1 of 3 – Continuous production testing
CloudWatch
Events (1m)
CloudWatch
Alarm

19. Building a Synthetic Test
• Keep it simple
• Build logic in Lambda (invoke with CloudWatch Events)
• Capture data in CloudWatch metrics
1 of 3 – Continuous production testing

20. Segmenting Production
Typical Segment Types:
• Region
• Availability Zone
• Sub-Zonal
• Single Host (Canary)
AP-SOUTHEAST-2
AP-SOUTHEAST-2A AP-SOUTHEAST-2B
2 of 3 – Segment Production

21. V2 V2 V2V2V1 V1V1
Typical Deployment Segmentation
Availability Zone based
Deployment
Availability Zone based
Deployment
Availability Zone based
Deployment
V2 V2V2V1 V1V1 V2 V2V2V1 V1V1
Production Fleet
Post-deployment test
Canary
Deployment
V1
Region based Deployment
2 of 3 – Segment Production

22. Step 1: Configure Deployment Groups
Create deployment groups per segment using:
• Tags
• Auto Scaling groups
2 of 3 – Segment Production

23. Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
1. Deploy to smallest segment
2. Run post-deployment tests
3. Deploy to one availability zone
4. Run post-deployment tests
5. Deploy to remaining availability zones
Step 2: Deploy to Each Segment
2 of 3 – Segment Production

24. Step 3: Post-Deployment Test
A deployment is valid if:
• The test has gathered enough data to gain confidence
− CloudWatch metrics
• No service alarms have fired
− CloudWatch alarms
• The test has not timed out
− Code
2 of 3 – Segment Production

25. Canary Deployments
All production hosts:
• Participates in serving production traffic
• Configured as a production instance
• Participates in production metrics stream
A canary host:
• Has its own metrics stream
• Canary validations use the canary metric stream
2 of 3 – Segment Production

26. Halt Deployments at Sensitive Times
Problem:
A bad change during sensitive times has a disproportionate affect
on the business.
Consequence:
Issues during sensitive days risk reputation and financial loss.
3 of 3 – Halt Promotions with Gates

27. Adding Safety With Deployment Black-Days
Deploy to production during normal conditions
• Halt deployments during sensitive times
Building a black-day calendar with CodePipeline:
• Use Approvals to pause production deployments
• Lambda to automatically approve when the time is right
3 of 3 – Halt Promotions with Gates

28. CanaryDeploy
CodeDeploy
CanaryApproval
Manual Approval
CanaryDeployPostTest
Lambda
CanaryRekogTest
Lambda
CanaryDeploy
TimeWindowApproval
Manual Approval
DemoResourcesSource
CodeCommit
SourceApplication Pipeline Demo
SummitApp
CodeCommit
Repo
Code update
ApplicationBuild
CodeBuild
Build
ProdDeploy
CodeDeploy
ProductionDeployment

29. Application Pipeline Demo

30. Summary
In this session we explored:
1. How to improve the safety of infrastructure changes using
deployment pipelines
• Inception with Infra and App pipelines
2. How to improve safety with advanced CD concepts
and mechanisms:
• Deployment-attached Synthetic Tests
• Segmenting Production and Canary deployments
• Halting promotions with Gates like Black Days

31. Resources
• github.com/awslabs/aws-codepipeline-time-windows
• github.com/awslabs/aws-codepipeline-synthetic-tests
• github.com/awslabs/aws-codepipeline-block-production
Check out AWS Labs on GitHub
• github.com/awslabs/

32. How Do I Start?
> THINK BIG
> START SMALL
> ACT NOW!

33. [FIN, ACK]
Thank you!