Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Releasing Software Quickly and Reliably with AWS CodePipline

1,764 views

Published on

Learn how to model your software release process by wiring together different steps into an automated workflow using AWS CodePipeline.

Published in: Technology
  • Be the first to comment

Releasing Software Quickly and Reliably with AWS CodePipline

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Matt Yanchyshyn Sr Manager, AWS Solutions Architecture Fast and safe release automation with AWS CodePipeline
  2. 2. What to Expect from the Session 1. What we learned as we evolved our release processes 2. Overview of release process terminology 3. A tour of AWS CodePipeline 4. Look under the hood of AWS CodePipeline 5. Extending AWS CodePipeline
  3. 3. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 1: What we learned as we evolved our release processes
  4. 4. Amazon started as a monolithic application
  5. 5. We slowed down, so we broke apart.
  6. 6. We ran a study to find out where we spent our time.
  7. 7. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod Weeks Mins Days Mins Days Mins Days Mins
  8. 8. Some obvious and some surprising findings Faster Safer
  9. 9. Visualizing process was key to improving it.
  10. 10. Simplification and Standardization are needed
  11. 11. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 2: Overview of release processes
  12. 12. Release Processes have four major phases Source Build Test Production • Check-in source code such as .java files. • Peer review new code • Compile code • Unit tests • Style Checkers – FindBugs, CheckStyle • Code Metrics – Cobertura, EMMA • Packaging - docker • Integration tests with other systems • Load testings • UI Tests • Penetration testing • Incremental rollout to production environments
  13. 13. Release Processes levels Source Build Test Production Continuous Integration Continuous Delivery Continuous Deployment
  14. 14. A real pipeline of a simple service Build and Unit Test DeploymentsValidation With increase confidence we increase the blast radius Does it compile and pass unit tests? Does it integrate in an isolated stack? Does it integrate against prod? Does it integrate in production region 1? Does it integrate in production region 2? Deploy to prod
  15. 15. A real pipeline with staggered deployments
  16. 16. A real pipeline with thorough validation
  17. 17. A build service is not enough • Our release processes emphasize safety, so we have more steps • Many CI systems hide the release process, making failures hard to find • CI systems don’t provide needed modeling primitives • Serial and parallel execution • Easily add a new step to your process • Pause for manual approvals • Multiple deployment actions • Multiple source actions • CI systems don’t allow multiple changes concurrently through the release process
  18. 18. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 3: A tour of AWS CodePipeline
  19. 19. We have a strong partner list, and its growing Source Build Test Deploy *beta
  20. 20. We’re continuing to integrate AWS products Source Invoke Logic . Deploy AWS CodeDeploy AWS Elastic Beanstalk AWS OpsWorks More to come… AWS CodeCommit Amazon S3 AWS Lambda
  21. 21. CodePipeline concepts on the dashboard page
  22. 22. CodePipeline concepts on the pipeline page PipelineStage Action Pipeline Run Source change • starts a run; and • creates an artifact to be used by other actions. Manual Approval
  23. 23. CI is great start. CD with CodePipepline is better. • Visualizes your release process so they can be understood • Allows powerful modeling of your release process • Serial and parallel execution • Easily add a new step to your process • Pause for manual approvals • Multiple deployment steps • Allows multiple changes to be processed concurrently
  24. 24. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 4: Look under the hood of AWS CodePipeline
  25. 25. 8. Retrieve build artifact EC2 instance CodePipeline App Pipeline Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy RailsApp Elastic Beanstalk Source Artifact S3 Build Artifact S3 5. Get source artifact 1. Get Changes 6. Store build artifact 3. Poll for Job 4. Acknowledge Job 7. Put Success 9. Deploy build artifact Elastic Beanstalk Web container Rails App
  26. 26. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 5: Extending AWS CodePipeline
  27. 27. Extend AWS CodePipeline Using Custom Actions Update tickets Provision resources Update dashboards Mobile testing Send notifications Security scan
  28. 28. How would we send a message to slack? CodePipeline App Pipeline Source Source GitHub Build JenkinsForReinvent Jenkins Deploy RailsApp Elastic Beanstalk
  29. 29. AWS CodePipeline extension options Per account based extension – for customers • Option 1: AWS Lambda function • Option 2: Custom Actions Global Extensions – for AWS partners • Option 3: Third Party Actions
  30. 30. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 5: Extending AWS CodePipeline Option 1: AWS Lambda function
  31. 31. Extend AWS CodePipeline with Lambda Push message to slack when our Pipeline run completes 1. Add in a Lambda Invoke stage 2. Select a “send message to slack” function 3. Run the pipeline
  32. 32. Lambda example – include libs and handler var AWS = require('aws-sdk'); var https = require('https'); exports.handler = function(event, context) { var cp = new AWS.CodePipeline(); … };
  33. 33. Lambda example – setup HTTP config var httpParams = { hostname: 'slack.com', path: '/api/chat.postMessage?token=MYTOKEN& text=Hello&channel=%23testing', method: 'GET' };
  34. 34. Lambda example – send message to Slack // Send Message to Slack var req = https.request(httpParams, function(response) { response.on('data', function(c) {}); response.on('end', sendResultToCodePipeline); response.resume(); }); req.end();
  35. 35. Lambda example – notify AWS CodePipeline var sendResultToCodePipeline = function () { var jobId = event["CodePipeline.job"].id; cp.putJobSuccessResult({ jobId: jobId }, function(err, data) { if(err) { context.fail(err); } else { context.succeed("Passed"); } }); };
  36. 36. ©2016, Amazon Web Services, Inc. or its affiliates. All rights reserved Part 5: Extending AWS CodePipeline Option 2: Custom Action
  37. 37. Custom Actions and Job Workers collaborate Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker
  38. 38. Creating a custom action and job worker takes 3 easy steps 1. Register you custom action in CodePipeline 2. Write your Custom Action • Integrate with an external service • Write stand alone Custom Action 3. Deploy custom action
  39. 39. CodePipeline App Pipeline Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy Action Custom Action RailsApp Elastic Beanstalk Deploy RailsApp Elastic Beanstalk The Job Worker drives the interaction between AWS CodePipeline and Slack Messaging App Elastic Beanstalk Worker Container Job Worker 3. Send Message 1. Poll for Job 2. Acknowledge Job 4. Put Success
  40. 40. Make your Custom Action available to users 1. Register you custom action in CodePipeline 2. Write your Custom Action • Integrate with an external service • Combine Custom Action and processing task 3. Deploy custom action
  41. 41. CodePipeline App Pipeline Source Source GitHub Build JenkinsOnEc2 Jenkins Deploy Action Custom Action RailsApp Elastic Beanstalk RegisterCustomAction.json { "category": "Deploy", "provider": "Slack-Notifier", "version": "2", "settings": { "entityUrlTemplate": "https://codepipeline-demo.slack.com/messages/general/", "executionUrlTemplate": "https://codepipeline-demo.slack.com/archives/general/{ExternalExecutionId}" }, "inputArtifactDetails": { "maximumCount": 0, "minimumCount": 0 }, "outputArtifactDetails": { "maximumCount": 0, "minimumCount": 0 } } Unique identifier information Files to consume during the action Files to produce during the action
  42. 42. Use the AWS CLI to register the Custom Action $ aws codepipeline create-custom-action-type --cli-input-json file://lib/custom_action/RegisterCustomAction.json
  43. 43. Write the code to talk to AWS CodePipeline and Slack 1. Register you custom action in CodePipeline 2. Write your Job Worker • Integrate with an external service. e.g. Slack • Combine with processing task. 3. Deploy Job Worker
  44. 44. Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker cp = Aws::CodePipeline::Client.new(region: 'us-east-1') poll_results = cp.poll_for_jobs({ action_type_id: { category: 'Deploy', owner: 'Custom', provider: 'Slack-Notifier', version: '2' }, max_batch_size: 1 }) Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker Job Worker – get work
  45. 45. job = poll_results.jobs.first cp.acknowledge_job(job_id: job.id, nonce: job.nonce) Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker Job Worker – ack work
  46. 46. job = poll_results.jobs.first cp.acknowledge_job(job_id: job.id, nonce: job.nonce) # perform our important logic execution_id = send_message_to_slack() Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker Job Worker – call Slack
  47. 47. cp.put_job_success_result({ job_id: job.id, execution_details: { summary: 'Success', external_execution_id: execution_id, percent_complete: 100 } }) Stage Action Custom Action 1. Poll for Job 2. Acknowledge Job 3. Put Success EC2 instance Job Worker Job Worker – signal completion
  48. 48. Deploy Job Worker code to compute instance 1. Register you custom action in CodePipeline 2. Write your Job Worker • Integrate with an external service • Combine Job Worker and processing task 3. Deploy Job Worker
  49. 49. Recap creation of a custom action and job worker 1. Register you custom action in CodePipeline 2. Write your Custom Action • Integrate with an external service • Combine Custom Action and processing task 3. Deploy custom action
  50. 50. What extension method should I use? Lambda Custom Action Short running tasks are easy to build Can perform any type of workload Long running tasks need more work Control over links displayed in console Node.js, Python and Java support Any language support Runs on AWS Can run on-premise No servers to provision or manage Requires compute resources
  51. 51. What did we cover today? • The benefits of moving to Continuous Delivery • We can get our software out in front of our users much more rapidly • By moving faster we can actually ensure better quality • CodePipeline allows for integration with almost any service or tool you can think of! • Plus visualization of what’s going on!
  52. 52. How you can try AWS CodePipeline • Use your AWS account to create a free pipeline • We have examples and a tutorial • There is thorough documentation too • We provide support in the forums • More CodePipeline code in awslabs on github.com
  53. 53. Resources to learn more: • Continuous integration: https://aws.amazon.com/devops/continuous-integration/ • Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/ • CodePipeline • https://aws.amazon.com/codepipeline/ • https://aws.amazon.com/documentation/codepipeline/ • CodeDeploy • https://aws.amazon.com/codedeploy/ • CodeCommit • https://aws.amazon.com/codecommit/ • Code Services Starter Kit: http://bit.ly/AWSCodeStarterKit But wait, there’s more!
  54. 54. Questions? markmans@amazon.com
  55. 55. Thank you! markmans@amazon.com
  56. 56. Images Haystack rock - https://commons.wikimedia.org/wiki/File:Haystack_rock_00022.jpg Heatpipe tunnel copenhagen 2009 - https://commons.wikimedia.org/wiki/File:Heatpipe_tunnel_copenhagen_2009.jpg Lewis Hine, Boy studying - https://commons.wikimedia.org/wiki/File:Lewis_Hine,_Boy_studying,_ca._1924.jpg Cells – https://pixabay.com/en/stem-cell-sphere-163711/

×