At Amazon, continuous integration and continuous delivery (CI/CD) techniques enable collaboration, increase agility, and deliver a high-quality product faster. In this talk, we walk you through the practices we use for both the CI and the CD of software delivery. For CI, we showcase how we incorporate pull requests to increase team collaboration. We also demonstrate how to optimize CI workflows for speed with caching, code analysis, and integration testing. For CD, we share example safety mechanisms, including canary testing, rollbacks, and Availability Zone redundancy. We use the AWS developer tools that were designed based on the internal Amazon tooling: AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, AWS CodeDeploy, and AWS X-Ray.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jay Yeras
Partner Solutions Architect, Amazon Web Services
SRV320
Amazon CI/CD Practices for Software
Development Teams

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What have I done???

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What to expect
Overview of continuous integration (CI) and continuous delivery (CD)
• Why it matters
• CI/CD tooling & techniques
Make your pipeline safer by:
1. Automating tests to catch regressions early
2. Identifying production issues quickly
3. Deploying changes safely
4. Automatically deciding when to release changes

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continuous integration

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is continuous deployment?

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is continuous delivery?

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does CI/CD matter?

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does CI/CD matter?
• Find bugs earlier
• Fix bugs faster
• Deliver faster
• Deliver more often
• Unblock developers
• Grow skills faster
Quality
5x
Lower change failure rate
Source: 2017 State of DevOps Report (Puppet)

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does CI/CD matter?
• Find bugs earlier
• Fix bugs faster
• Deliver faster
• Deliver more often
• Unblock developers
• Grow skills faster
Delivery
440x
Faster from commit to deploy
Source: 2017 State of DevOps Report (Puppet)

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does CI/CD matter?
• Find bugs earlier
• Fix bugs faster
• Deliver faster
• Deliver more often
• Unblock developers
• Grow skills faster
Delivery
46x
More frequent deployments
Source: 2017 State of DevOps Report (Puppet)

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does CI/CD matter?
• Find bugs earlier
• Fix bugs faster
• Deliver faster
• Deliver more often
• Unblock developers
• Grow skills faster
Happy teams
44%
More time spent on new
features and code
Source: 2017 State of DevOps Report (Puppet)

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous production testing
3. Manage deployment health
4. Segment production
5. Halt promotions

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prerequisites
• Versioned source
• Automated build
• Automated deployments
• Deploy to > one instance
• Unit tests
• Integration tests
• Continuous delivery

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prerequisites
• Versioned source
• Automated build
• Automated deployments
• Deploy to > one instance
• Unit tests
• Integration tests
• Continuous delivery
Source
Build
Deploy to
integration stack
integration tests
Deploy to
production

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices with your tools
• Focus on best practices
• Keep using your current tools where possible
• Build and testing tools
• Deployment tools
• Continuous integration and continuous delivery tools
• Extend your current tools when needed
• This talk uses AWS tools

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tools used
Monitoring
Amazon CloudWatch
Software development
Amazon SNS
AWS Lambda
Build & test
AWS CodeBuild
Deployment
AWS CodeDeploy
AWS CodePipeline

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model the release process in AWS CodePipeline
Source
Build
Deploy to
integration stack
Integration tests
Deploy to
production
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Production
Deploy to Prod
CodeDeploy
Integration
Deploy to Integ
CodeDeploy
IntegTest
End2EndTester

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model the release process in AWS CodePipeline
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Production
Deploy to Prod
CodeDeploy
Integration
Deploy to Integ
CodeDeploy
IntegTest
End2EndTester
Pipeline
Stage
Action

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model the release process in AWS CodePipeline
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Production
Deploy to Prod
CodeDeploy
Integration
Deploy to Integ
CodeDeploy
IntegTest
End2EndTester
Change 1 Source change
• Starts a run
• Creates an artifact to be
used by other actions
Pipeline run

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Starting point
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Production
Deploy to Prod
CodeDeploy
Integration
Deploy to Integ
CodeDeploy
IntegTest
End2EndTester
AWS
CodeDeploy

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous service testing
3. Manage deployment health
4. Segment production
5. Halt promotions

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Be aware when a code change fails tests
Problem
A code change can fail automated tests without developer
knowledge
Consequence
Your pipeline becomes blocked and changes stop flowing to
production

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add notifications to your automated tests
1. Run unit tests during build process
2. Deploy to integration environment
3. Run integration and UI tests
4. Notify developer chat room if the build or tests fail

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configure AWS CodeBuild

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Build and unit tests
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Tests
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnChrome
CodeBuild
IntegTest
End2EndTester
1. Trigger pipeline on source change
2. Build and unit tests
3. Deploy to integration environment
4. Execute UI tests
5. Execute integration tests

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Notify on failed build and test
Source
MyAppSource
CodeCommit
Build
Build
CodeBuild
Tests
IntegrationDeploy
CodeDeploy
TestOnChrome
CodeBuild
TestOnChrome
CodeBuild
IntegTest
End2EndTester
Change 1
CloudWatch
Events
(failed action)
Lambda function
NotifyChimeOnPipelineActionFailure()
Amazon Chime

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous service testing
3. Manage deployment health
4. Segment production
5. Halt promotions

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Be aware when a service is unavailable
Problem
A service can stop working at any time for reasons inside or
outside of its control
Consequence
Your service may be unavailable without your team knowing
about it

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use synthetic traffic to simulate real users
• Test all business-critical functionality (UI and APIs)
• Tests must run quickly
• Measure client latencies
• Check for reachability

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Synthetic traffic
How synthetic traffic flows
CloudWatch
Alarm
CloudWatch
Events (1m)

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CloudWatch
Events (1m)
Synthetic traffic
Synthetic traffic flow: Why two metric streams?
CloudWatch
Alarm

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a synthetic traffic test

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a synthetic traffic test
• Keep it simple
• Build logic in Lambda (invoke with CloudWatch Events)
• Capture data in CloudWatch metrics

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda synthetic traffic blueprint

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scheduling the synthetic traffic test

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a synthetic traffic test: Code

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a synthetic traffic test: Alarming

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Synthetic traffic
DeployToProd
CodeDeploy
Production
Synthetic traffic
AWS
CodeDeploy

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous service testing
3. Manage deployment health
4. Segment production
5. Halt promotions

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments: Success
Production fleet
Elastic Load Balancing

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2 V2 V2 V2 V2
Rolling deployments: Fail
Production fleet
Elastic Load Balancing

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Check for deployment failures in production
Problem
There are no automated tests to verify that a service is working
after a new deployment
Consequence
Each production deployment needs to be checked manually

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add safety to rolling deployments
1. Validate each host’s health
2. Ensure that a minimum percentage of the fleet is healthy
3. Rollback if the deployment failed

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configure AWS CodeDeploy

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Deployment validation – AppSpec.yaml

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V1V1 V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2 V2V2
Step 1: Working tests raises more issues
Production fleet
Elastic Load Balancing
Failed deployment

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
4 failures – 60% healthy
MHH 70%, 10 hosts:
V1V2 V1V1 V1 V1 V1 V1 V1 V1 V1V2 V2 V2 V2V2 V2 V2 V2 V2
Step 2: Use minimum healthy hosts
Production fleet
Elastic Load Balancing
1 failure – 90% healthy

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 2: Use minimum health hosts: AWS CodeDeploy

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Rollback when a deployment fails
• AWS CodeDeploy: Configured in deployment group

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy: Deployment health
DeployToProd
AWS CodeDeploy
Production
Synthetic traffic
AWS CodeDeploy
3 of 5 – Manage deployment health

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous service testing
3. Manage deployment health
4. Segment production
5. Halt promotions

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bad changes must not affect all customers
Pipeline Problem
When a critical issue reaches production, all hosts are affected
Consequence
Bad changes impact all customers

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lower deployment risk by segmenting
1. Break production into multiple segments
2. Deploy to a segment
3. Test a segment after a deployment
4. Repeat 2 & 3 until done

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Break production into multiple segments
Typical segment types
• Region
• Availability Zone
• Subzonal
• Single host (canary)
US-EAST-1
US-EAST-1A US-EAST-1B

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V2 V2 V2V2V1 V1V1
Step 1: Typical deployment segmentation
Availability Zone-based
deployment
Availability Zone-based
deployment
Availability Zone-based
deployment
V2 V2V2V1 V1V1 V2 V2V2V1 V1V1
Production fleet
Post-deployment test
Canary
deployment
V1
Region-based deployment

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 1: Use deployment groups as segments
Create deployment groups per segment using:
• Tags
• Auto Scaling groups

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToInteg
CodeDeploy
Integration
IntegTest
End2EndTester
Step 2: Deploy to each segment
1. Deploy to smallest segment
2. Post-deployment tests
3. Deploy to one Availability Zone
4. Post-deployment tests
5. Deploy to remaining Availability
Zones

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Step 3: Test each segment
A deployment is valid if:
• The test has gathered enough data to gain confidence
• CloudWatch metrics
• No service alarms have fired
• CloudWatch alarms
• The test has not timed out
• Code

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Add segment tests to your pipeline
Extend AWS CodePipeline with:
• Test actions
• Lambda Invoke actions
• Custom actions
• Approval actions
1-hour timeout
7-day timeout

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use AWS CodePipeline approvals to trigger tests
Source
MyAppSource
CodeCommit
Deploy
DeployToSegment
CodeDeploy
ValidateSegment
Approval
putApprovalResult
Approval
message
DeployToSegment
CodeDeploy
SNS topic

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use SNS to start an automated approval check

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Creating a post-deployment test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
registerDeployTest()
Lambda function
evaluateDeploy()
Amazon
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy AlarmTimeUsage
SNS topic

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – registerDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
registerDeployTest()
Lambda function
evaluateDeploy()
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy AlarmTimeUsage
SNS topic

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
registerDeployTest function – (Node.js 4.3)

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Post-deployment test – evaluateDeployTest
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
Deploy
CanaryDeploy
CodeDeploy
ValidateCanary
Approval
Lambda function
registerDeployTest()
Lambda function
evaluateDeploy()
DynamoDB
CloudWatch
Events (1m)
Change 1
Prod-us-east-1a
CodeDeploy AlarmTimeUsage
SNS topic

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
approveValidation function (Node.js 4.3)

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Canary deployments – they’re different
All production hosts
• Participates in serving production traffic
• Configured as a production instance
• Participates in production metrics stream
Canary hosts
• Has its own metrics stream
• Canary validations use the canary metric stream

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy: Segment production
Synthetic traffic
AWS CodeDeploy
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
DeployToProd
CodeDeploy
Production

70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Techniques
1. Automated testing
2. Continuous service testing
3. Manage deployment health
4. Segment production
5. Halt promotions

71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EC2 instance
Change 2Change 3
Don’t change the system under test
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
DeployToProd
MyApp
CodeDeploy
Deploys
Change 1

72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Don’t compound problems during an outage
Pipeline problem
The pipeline is unaware of the health of the infrastructure it is
deploying to
Consequence
Production changes, usually deployments, can make it difficult for
an operator to resolve a production event

73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build promotion blockers

74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source
MyAppSource
CodeCommit
Build
MyAppBuild
CodeBuild
DeployToProd
MyApp
CodeDeploy
Change 1Change 2
Auto-stop deploying to PRD during an event
CloudWatch
Synthetic
traffic
Deploys
Checks
CloudWatch
Events (1m)
Triggers
EmitsDisables
disableTransition() CloudWatch
alarm
EC2 instance
SNS

75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
disableTransition function (Lambda Node.js 4.3)

76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable production deployments – AWS CodePipeline

77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy: Halt promotions
Synthetic traffic
AWS CodeDeploy
Production
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy

78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continuous delivery summary
Goal: Make your pipeline safer
1. Automated testing and notifications
• Keep pipeline unblocked
2. Identify production issues quickly
• Continuous production testing
3. Safely deploy changes
• Manage deployment health
• Segment production
4. Automatically decide when to release changes
• Halt promotions

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Release and deploy process: Ending point
DeployToProd
CodeDeploy
Production
AWS CodeDeploy
Synthetic traffic
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
CanaryDeploy
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-1
CodeDeploy
PostDeployTest
Approval
Deploy-AZ-2
CodeDeploy
Deploy-AZ-3
CodeDeploy
Production

80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Code is available online
• https://github.com/aws-samples/aws-codebuild-samples
• https://github.com/awslabs/aws-codepipeline-synthetic-tests
• https://github.com/awslabs/aws-codepipeline-block-production

81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
Acknowledgements
• Original 2016 slides written and prepared by Mark
Mansour, Senior Manager, Continuous Delivery, AWS
• This presentation, “DevOps on AWS: Advanced
Continuous Delivery Techniques,” was originally given
at re:Invent 2016 on Nov 30, 2016
• 2017 slides updated by Curtis Bray, Manager, AWS
CodePipeline for DEV324 presentation at re:Invent
2017
• 2018 slides updated by Curtis Rissi, Sr. Solutions
Architect to incorporate “Continuous Integration Best
Practices for Software Development Teams” by Clare
Liguori, AWS Senior Software Engineer, to cover the full
CI/CD process