AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS

3,284 views
3,106 views

Published on

With AWS companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100% API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session we'll talk about some key concepts and design patterns for Continuous Deployment and Continuous Integration, two elements of lean development of applications and infrastructures.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,284
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
42
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • All canonical software life cycles were designed by people who build installed software
  • The world’s handmade and vintage marketplace.
  • The world’s handmade and vintage marketplace.
  • Deploying code is the very first thing engineers learn to do at Etsy.
  • GUIs / algos
  • +24% on magic wand sales !
  • No authorization required, Detect problems quickly, gain confidence
  • Pour illustrer le launch early
  • AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS

    1. 1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Continuous Integration and Deployment on AWS Martin Elwin Manager, Solutions Architecture
    2. 2. CONTINUOUS INTEGRATION
    3. 3. DEVELOPER
    4. 4. SOURCE CODE REPOSITORY
    5. 5. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER
    6. 6. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER
    7. 7. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER PICK TASKS
    8. 8. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER SUBMIT CODE
    9. 9. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER CODE FETCH
    10. 10. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER BUILD OUTPUT
    11. 11. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER DOCS BINARIES & PACKAGES
    12. 12. SOURCE CODE REPOSITORY DNS CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER BUILDS
    13. 13. PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION
    14. 14. ON-DEMAND PAY AS YOU GO ELASTIC
    15. 15. = PROGRAMMABLE PLATFORM
    16. 16. … ec2.runInstances(new RunInstancesRequest() .withImageId("ami-4b814f22") .withInstanceType("m1.small") .withMinCount(1) .withMaxCount(1) .withKeyName("mykey") .withSecurityGroups("httpssh")); … JAVA SDK
    17. 17. … var runInstanceRequest = new RunInstancesRequest() { ImageId = "ami-4b814f22", InstanceType = "m1.small", MinCount = 1, MaxCount = 1, KeyName = "mykey" }; runInstanceRequest.SecurityGroup.Add("httpssh"); ec2.RunInstances(runInstanceRequest); … .NET SDK
    18. 18. … ec2.run_instances( 'ami-4b814f22', key_name='mykey', instance_type='m1.small', min_count=1, max_count=1, security_groups=['httpssh']) … PYTHON SDK
    19. 19. … ec2.instances.create( :image_id => 'ami-4b814f22', :instance_type => 'm1.small', :count => 1, :security_groups => 'httpssh', :key_pair => ec2.key_pairs['mykey']) … RUBY SDK
    20. 20. … $ec2 -> run_instances( 'ami-4b814f22', 1, 1, array( 'InstanceType' => 'm1.small', 'KeyName' => 'mykey', 'SecurityGroup' => 'httpssh') ); … PHP SDK
    21. 21. … var params = { ImageId: 'ami-4b814f22', InstanceType: 'm1.small', MinCount: 1, MaxCount: 1, SecurityGroups: ['httpssh'] }; ec2.runInstances(params, function(err, res) { … }); … JAVASCRIPT SDK
    22. 22. aws ec2 run-instances --image-id ami-4b814f22 --min-count 1 --max-count 1 --key-name mykey --security-groups httpssh UNIFIED CLI
    23. 23. New-EC2Instance -ImageId ami-4b814f22 -MinCount 1 -MaxCount 1 -KeyName mykey -SecurityGroupId sg-9cf9e5d9 -InstanceType m1.small POWERSHELL CLI
    24. 24. https://ec2.amazonaws.com/ ?Action=RunInstances &ImageId=ami-4b814f22 &MaxCount=1 &MinCount=1 &SecurityGroup.1=httpssh &KeyName=mykey &<AUTHPARAMS> AWS REST API
    25. 25. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
    26. 26. AWS CLOUDFORMATION STACK-BASED DEPLOYMENT SERVICE
    27. 27. AWS CLOUDFORMATION TEMPLATE
    28. 28. { "Description" : "Create a small instance", "Resources" : { "MyInstance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-4b814f22", "KeyName" : "mykey", "InstanceType" : "m1.small", "SecurityGroups" : ["httpssh"] } } } }
    29. 29. CLOUDFORMATION TEMPLATE DECLARATIVE DEFINITION Create it programmatically KNOWN CONFIGURATION Store stack configuration in source control PARAMETER DRIVEN Dynamic and user-driven templates COLLABORATION Share templates with ease as just files
    30. 30. APPLICATION VERSIONS + INFRASTRUCTURE VERSIONS
    31. 31. AWS CLOUDFORMATION TEMPLATE
    32. 32. Continuous Deployment at Viaplay
    33. 33. • Viaplay is Nordic’s leading online video service for TV, sport, film and kids' content. • The service is available on PC, Mac, Android and iOS mobile phones, tablets, the Viaplay box, LG Smart TV, Samsung Smart TV, Philips Smart TV, Panasonic Smart TV, Xbox 360, PlayStation®3 and PlayStation®4.
    34. 34. Preparations for the Olympics • New content / video / subscriber management platform • API:s completely rebuilt in Node.js in a service oriented architecture • All clients rebuilt (Web, Smartphones, Smart TV:s, Gaming consoles, Set Top Box) • Moved hosting of API:s to AWS
    35. 35. System Setup • Almost everything described in CloudFormation templates: Network (VPC), Load Balancers, Cache clusters, DNS entries, Autoscaling groups • The same templates for all different environments • Every application has its own CloudFormation stack with DNS entries, Elastic Load Balancers and an Auto Scaling Group
    36. 36. Auto Scaling • An ASG contains a variable number of instances running the same AMI • To enable auto scaling, the AMI must start automatically • Application servers starts with our custom AMI containing everything except the application build
    37. 37. Auto Scaling • Build and environment are set in tags and UserData. UserData is run at instance startup, tags are readable from the instance but more important clearly visible in the console. • When an EC2 instance is launched it fetches the build from S3, installs and starts it • As soon as the ELB health check is OK, the instance gets traffic
    38. 38. Deploy • Autoscaling has already made us create AMIs that starts the application at instance startup • Deploy is done by changing parameters to the stack • Setting new UserData on the ASG/EC2 launch config requires exchange of instances. CloudFormation does that for us with the Rolling Update policy
    39. 39. Lessons learnt • Rolling update takes time and is practically unstoppable during update. We are currently rewriting this so that we start a new ASG that completely replaces the old one • To be able to do a change in a stack, your IAM user needs to have the rights to change all resources in the template. All developers must be able to deploy their code, but we do not want all developers to be Power Users.
    40. 40. Custom Admin Application • Our admin application enables CloudFormation updates without giving Power User rights to all developers • We can keep track of who has done what in an easy way • The application also makes sure that deploys are done in exactly the same way every time • Built on top of the AWS Javascript SDK
    41. 41. Custom Admin Application • Our admin application enables CloudFormation updates without giving Power User rights to all developers • We can keep track of who has done what in an easy way • The application also makes sure that deploys are done in exactly the same way every time • Built on top of the AWS Javascript SDK
    42. 42. Deploy From a Web Interface • Build OK from Jenkins -> copied to S3 builds bucket • Deploy tool lists all files from the builds bucket
    43. 43. Deploy From a Web Interface • Write a short description of the deploy for the release email
    44. 44. Deploy From a Web Interface • The admin tool updates the CloudFormation stack with the new build parameter • The deploy notifier application sends an email with information on all commits done between the current and the new build
    45. 45. Deploy From a Web Interface • If everything looks good on stage - click ”thumbs up” to copy the build to the production bucket, else ”thumbs down” to reject the build
    46. 46. Ready for the Olympics • Everything deployed during the winter • Last user migrated in January • Opening ceremony at Friday Feb 7
    47. 47. Ready for the Olympics • 39 deploys during the week starting Feb 3 • 11 deploys in 6 different applications on the day of the opening ceremony • All went fine
    48. 48. Thanks! johan.waernulf@viaplay.com
    49. 49. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION
    50. 50. KEY = ITERATION
    51. 51. ITERATION = MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
    52. 52. 11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean number of hosts simultaneously receiving a deployment 30,000 Max number of hosts simultaneously receiving a deployment DEPLOYMENTS AT AMAZON.COM
    53. 53. SOFTWARE DEPLOY ≠ PRODUCT LAUNCH
    54. 54. 1.8 BILLION PAGE VIEWS $109 MILLION IN SALES 5.5 MILLION ITEMS SOLD
    55. 55. 30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES
    56. 56. "Production is truly the only place you can validate your code."
    57. 57. RUN-TIME CONFIGURATION OF ACTIVE FEATURES
    58. 58. AWS OPSWORKS INTEGRATED APPLICATION MANAGEMENT
    59. 59. 14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES/SEC
    60. 60. A / B TESTING
    61. 61. DATA-DRIVEN ARCHITECTURES
    62. 62. METRICS @ETSY
    63. 63. CONTINUOUS INTEGRATION CONTINUOUS DEPLOYMENT
    64. 64. CONTINUOUS DEPLOYMENT = CONTINUOUS EXPERIMENTATION
    65. 65. CONTINUOUS DEPLOYMENT = CONTINUOUS IMPROVEMENT
    66. 66. « Want to increase innovation? Lower the cost of failure » Joi Ito
    67. 67. SPEED AND AGILITY Experiment Often Fail quickly at a low cost More Innovation Experiment Infrequently Failure is expensive Less Innovation “ON-PREMISES”
    68. 68. wow so deploy much innovate such agility very continuous
    69. 69. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Thanks! Continuous Integration and Deployment on AWS Martin Elwin Manager, Solutions Architecture

    ×