The document discusses AWS CodePipeline and continuous delivery. It covers how Amazon evolved their own release processes, an overview of common release process terminology and phases, a tour of AWS CodePipeline features and concepts, how to extend AWS CodePipeline using custom actions or Lambda functions, and examples of building custom actions and job workers.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mark Mansour, AWS Continuous Delivery Teams
Fast and safe release
automation with
AWS CodePipeline

2. What to Expect from the Session
1. What we learned as we evolved our release processes
2. Overview of release process terminology
3. A tour of AWS CodePipeline
4. Look under the hood of AWS CodePipeline
5. Extending AWS CodePipeline

3. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 1: What we learned as we evolved our
release processes

4. Amazon started as a monolithic application

5. We slowed down, so we broke apart.

6. We ran a study to find out where we spent our time.

7. We were just waiting.
WaitWrite
Code WaitBuild
Code WaitDeploy
to Test
Deploy
to
Prod
Weeks
Mins Days Mins Days Mins Days Mins

9. Some obvious and some surprising findings
Faster Safer

10. Visualizing process was key to improving it.

11. Simplification and Standardization are needed

12. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 2: Overview of release processes

13. Release Processes have four major phases
Source Build Test Production
• Check-in
source code
such as .java
files.
• Peer review
new code
• Compile code
• Unit tests
• Style
Checkers –
FindBugs,
CheckStyle
• Code Metrics
– Cobertura,
EMMA
• Packaging -
docker
• Integration
tests with
other systems
• Load testings
• UI Tests
• Penetration
testing
• Incremental
rollout to
production
environments

14. Release Processes levels
Source Build Test Production
Continuous Integration
Continuous Delivery
Continuous Deployment

15. A real pipeline of a simple service
Build and Unit
Test
DeploymentsValidation
With increase confidence we increase the blast radius
Does it compile
and pass unit
tests?
Does it
integrate in
an isolated
stack?
Does it
integrate
against
prod?
Does it
integrate
in
production
region 1?
Does it
integrate in
production
region 2?
Deploy
to prod

16. A real pipeline with staggered deployments

17. A real pipeline with thorough validation

18. A build service is not enough
• Our release processes emphasize safety, so we have more steps
• Many CI systems hide the release process, making failures hard to
find
• CI systems don’t provide needed modeling primitives
• Serial and parallel execution
• Easily add a new step to your process
• Pause for manual approvals
• Multiple deployment actions
• Multiple source actions
• CI systems don’t allow multiple changes concurrently through the
release process

19. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 3: A tour of AWS CodePipeline

20. We have a strong partner list, and its growing
Source Build Test Deploy
*beta

21. We’re continuing to integrate AWS products
Source Invoke Logic . Deploy
AWS CodeDeploy
AWS Elastic Beanstalk
AWS OpsWorks
More to come…
AWS CodeCommit
Amazon S3
AWS Lambda

22. CodePipeline concepts on the dashboard page

23. CodePipeline concepts on the pipeline page
PipelineStage
Action
Pipeline Run
Source change
• starts a run; and
• creates an artifact to be used by other actions.
Manual Approval

24. CI is great start. CD with CodePipepline is
better.
• Visualizes your release process so they can be understood
• Allows powerful modeling of your release process
• Serial and parallel execution
• Easily add a new step to your process
• Pause for manual approvals
• Multiple deployment steps
• Allows multiple changes to be processed concurrently

25. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Partner: Apica

26. Why Load Test?

29. Catch Problems that UAT and Automated
Testing Often Miss
Changes in App Performance
Concurrency related bugs
Code interplay from multiple rolled up commits

30. Even Smaller Pre-Deploy
Load Tests Have Value

38. Test Anywhere from 1 to Millions of
Concurrent Users

41. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 4: Look under the hood of AWS
CodePipeline

42. 8. Retrieve build artifact
EC2 instance
CodePipeline
App Pipeline
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
RailsApp
Elastic Beanstalk
Source Artifact
S3
Build Artifact
S3
5. Get source artifact
1. Get Changes
6. Store build artifact
3. Poll for Job
4. Acknowledge Job
7. Put Success
9. Deploy build artifact
Elastic Beanstalk
Web container
Rails App

43. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 5: Extending AWS CodePipeline

44. Extend AWS CodePipeline Using Custom Actions
Update tickets Provision resources
Update dashboards
Mobile testing
Send notifications Security scan

45. How would we send a message to slack?
CodePipeline
App Pipeline
Source
Source
GitHub
Build
JenkinsForReinvent
Jenkins
Deploy
RailsApp
Elastic Beanstalk

46. AWS CodePipeline extension options
Per account based extension – for customers
• Option 1: AWS Lambda function
• Option 2: Custom Actions
Global Extensions – for AWS partners
• Option 3: Third Party Actions

47. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 5: Extending AWS CodePipeline
Option 1: AWS Lambda function

48. Extend AWS CodePipeline with Lambda
Push message to slack when our Pipeline run completes
1. Add in a Lambda Invoke stage
2. Select a “send message to slack” function
3. Run the pipeline

51. Lambda example – include libs and handler
var AWS = require('aws-sdk');
var https = require('https');
exports.handler = function(event, context) {
var cp = new AWS.CodePipeline();
…
};

52. Lambda example – setup HTTP config
var httpParams = {
hostname: 'slack.com',
path: '/api/chat.postMessage?token=MYTOKEN&
text=Hello&channel=%23testing',
method: 'GET'
};

53. Lambda example – send message to Slack
// Send Message to Slack
var req = https.request(httpParams, function(response) {
response.on('data', function(c) {});
response.on('end', sendResultToCodePipeline);
response.resume();
});
req.end();

54. Lambda example – notify AWS CodePipeline
var sendResultToCodePipeline = function () {
var jobId = event["CodePipeline.job"].id;
cp.putJobSuccessResult({ jobId: jobId }, function(err,
data) {
if(err) { context.fail(err); }
else { context.succeed("Passed"); }
});
};

55. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved
Part 5: Extending AWS CodePipeline
Option 2: Custom Action

56. Custom Actions and Job Workers collaborate
Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker

57. Creating a custom action and job worker takes
3 easy steps
1. Register you custom action in CodePipeline
2. Write your Custom Action
• Integrate with an external service
• Write stand alone Custom Action
3. Deploy custom action

58. CodePipeline
App Pipeline
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
Action
Custom Action
RailsApp
Elastic Beanstalk
Deploy
RailsApp
Elastic Beanstalk
The Job Worker drives the
interaction between
AWS CodePipeline and Slack
Messaging App
Elastic Beanstalk
Worker Container
Job Worker 3. Send Message
1. Poll for Job
2. Acknowledge Job
4. Put Success

59. Make your Custom Action available to users
1. Register you custom action in CodePipeline
2. Write your Custom Action
• Integrate with an external service
• Combine Custom Action and processing task
3. Deploy custom action

60. CodePipeline
App Pipeline
Source
Source
GitHub
Build
JenkinsOnEc2
Jenkins
Deploy
Action
Custom Action
RailsApp
Elastic Beanstalk
RegisterCustomAction.json
{
"category": "Deploy",
"provider": "Slack-Notifier",
"version": "2",
"settings": {
"entityUrlTemplate": "https://codepipeline-demo.slack.com/messages/general/",
"executionUrlTemplate": "https://codepipeline-demo.slack.com/archives/general/{ExternalExecutionId}"
},
"inputArtifactDetails": {
"maximumCount": 0,
"minimumCount": 0
},
"outputArtifactDetails": {
"maximumCount": 0,
"minimumCount": 0
}
}
Unique identifier information
Files to consume during the action
Files to produce during the action

61. Use the AWS CLI to register the Custom Action
$ aws codepipeline
create-custom-action-type
--cli-input-json
file://lib/custom_action/RegisterCustomAction.json

62. Write the code to talk to AWS CodePipeline and
Slack
1. Register you custom action in CodePipeline
2. Write your Job Worker
• Integrate with an external service. e.g. Slack
• Combine with processing task.
3. Deploy Job Worker

63. Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker
cp = Aws::CodePipeline::Client.new(region: 'us-east-1')
poll_results = cp.poll_for_jobs({
action_type_id: {
category: 'Deploy',
owner: 'Custom',
provider: 'Slack-Notifier',
version: '2'
},
max_batch_size: 1
})
Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker
Job Worker – get work

64. job = poll_results.jobs.first
cp.acknowledge_job(job_id: job.id, nonce: job.nonce)
Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker
Job Worker – ack work

65. job = poll_results.jobs.first
cp.acknowledge_job(job_id: job.id, nonce: job.nonce)
# perform our important logic
execution_id = send_message_to_slack()
Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker
Job Worker – call Slack

66. cp.put_job_success_result({
job_id: job.id,
execution_details: {
summary: 'Success',
external_execution_id: execution_id,
percent_complete: 100
}
})
Stage
Action
Custom Action
1. Poll for Job
2. Acknowledge Job
3. Put Success
EC2 instance
Job Worker
Job Worker – signal
completion

67. Deploy Job Worker code to compute instance
1. Register you custom action in CodePipeline
2. Write your Job Worker
• Integrate with an external service
• Combine Job Worker and processing task
3. Deploy Job Worker

68. Recap creation of a custom action and job
worker
1. Register you custom action in CodePipeline
2. Write your Custom Action
• Integrate with an external service
• Combine Custom Action and processing task
3. Deploy custom action

69. What extension method should I use?
Lambda Custom Action
Short running tasks are easy to build Can perform any type of workload
Long running tasks need more work Control over links displayed in console
Node.js, Python and Java support Any language support
Runs on AWS Can run on-premise
No servers to provision or manage Requires compute resources

70. What did we cover today?
• The benefits of moving to Continuous Delivery
• We can get our software out in front of our users much
more rapidly
• By moving faster we can actually ensure better quality
• CodePipeline allows for integration with almost any
service or tool you can think of!
• Plus visualization of what’s going on!

71. How you can try AWS CodePipeline
• Use your AWS account to create a free pipeline
• We have examples and a tutorial
• There is thorough documentation too
• We provide support in the forums
• More CodePipeline code in awslabs on github.com

72. Resources to learn more:
• Continuous integration: https://aws.amazon.com/devops/continuous-integration/
• Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/
• CodePipeline
• https://aws.amazon.com/codepipeline/
• https://aws.amazon.com/documentation/codepipeline/
• CodeDeploy
• https://aws.amazon.com/codedeploy/
• CodeCommit
• https://aws.amazon.com/codecommit/
• Code Services Starter Kit: http://bit.ly/AWSCodeStarterKit
But wait, there’s more!

73. Questions?
markmans@amazon.com

74. Thank you!
markmans@amazon.com

75. Images
Haystack rock - https://commons.wikimedia.org/wiki/File:Haystack_rock_00022.jpg
Heatpipe tunnel copenhagen 2009 -
https://commons.wikimedia.org/wiki/File:Heatpipe_tunnel_copenhagen_2009.jpg
Lewis Hine, Boy studying -
https://commons.wikimedia.org/wiki/File:Lewis_Hine,_Boy_studying,_ca._1924.jpg
Cells – https://pixabay.com/en/stem-cell-sphere-163711/