Your SlideShare is downloading. ×
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or i...
CONTINUOUS
INTEGRATION
DEVELOPER
SOURCE CODE
REPOSITORY
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
PICK
TASKS
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
SUBMIT
CODE
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
CODE
FETCH
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
BUILD OUTPUT
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
DOCS
BINARIES
& PACKAGES
SOURCE CODE
REPOSITORY
DNS
CONTINUOUS
INTEGRATION SERVER
PROJECT
MANAGEMENT SERVER
BUILDS
PAIN POINTS
• UNIT TESTS INCOMPLETE
• MOCK MAINTENANCE
• EXPENSIVE TEST ENVIRONMENT
• TEST ENVIRONMENT ≠ PRODUCTION
ON-DEMAND
PAY AS YOU GO
ELASTIC
= PROGRAMMABLE PLATFORM
…
ec2.runInstances(new RunInstancesRequest()
.withImageId("ami-4b814f22")
.withInstanceType("m1.small")
.withMinCount(1)
....
…
var runInstanceRequest = new
RunInstancesRequest()
{
ImageId = "ami-4b814f22",
InstanceType = "m1.small",
MinCount = 1,
...
…
ec2.run_instances(
'ami-4b814f22',
key_name='mykey',
instance_type='m1.small',
min_count=1,
max_count=1,
security_groups...
…
ec2.instances.create(
:image_id => 'ami-4b814f22',
:instance_type => 'm1.small',
:count => 1,
:security_groups => 'https...
…
$ec2 -> run_instances(
'ami-4b814f22', 1, 1,
array(
'InstanceType' => 'm1.small',
'KeyName' => 'mykey',
'SecurityGroup' ...
…
var params = {
ImageId: 'ami-4b814f22',
InstanceType: 'm1.small',
MinCount: 1,
MaxCount: 1,
SecurityGroups: ['httpssh']
...
aws ec2 run-instances
--image-id ami-4b814f22
--min-count 1
--max-count 1
--key-name mykey
--security-groups httpssh
UNIFI...
New-EC2Instance
-ImageId ami-4b814f22
-MinCount 1
-MaxCount 1
-KeyName mykey
-SecurityGroupId sg-9cf9e5d9
-InstanceType m1...
https://ec2.amazonaws.com/
?Action=RunInstances
&ImageId=ami-4b814f22
&MaxCount=1
&MinCount=1
&SecurityGroup.1=httpssh
&Ke...
IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT
AWS
CLOUDFORMATION
STACK-BASED
DEPLOYMENT SERVICE
AWS CLOUDFORMATION
TEMPLATE
{
"Description" : "Create a small instance",
"Resources" : {
"MyInstance" : {
"Type" : "AWS::EC2::Instance",
"Properties" ...
CLOUDFORMATION
TEMPLATE
DECLARATIVE
DEFINITION
Create it programmatically
KNOWN
CONFIGURATION
Store stack configuration in...
APPLICATION
VERSIONS
+
INFRASTRUCTURE
VERSIONS
AWS
CLOUDFORMATION
TEMPLATE
Continuous Deployment at Viaplay
• Viaplay is Nordic’s leading online video service for TV, sport,
film and kids' content.
• The service is available on PC...
Preparations for the Olympics
• New content / video / subscriber management platform
• API:s completely rebuilt in Node.js...
System Setup
• Almost everything described in CloudFormation
templates: Network (VPC), Load Balancers, Cache clusters,
DNS...
Auto Scaling
• An ASG contains a variable number of instances running the
same AMI
• To enable auto scaling, the AMI must ...
Auto Scaling
• Build and environment are set in tags and UserData. UserData
is run at instance startup, tags are readable ...
Deploy
• Autoscaling has already made us create AMIs that starts the
application at instance startup
• Deploy is done by c...
Lessons learnt
• Rolling update takes time and is practically unstoppable
during update. We are currently rewriting this s...
Custom Admin Application
• Our admin application enables CloudFormation updates
without giving Power User rights to all de...
Custom Admin Application
• Our admin application enables CloudFormation updates
without giving Power User rights to all de...
Deploy From a Web Interface
• Build OK from Jenkins -> copied to S3 builds bucket
• Deploy tool lists all files from the b...
Deploy From a Web Interface
• Write a short description of the deploy for the release email
Deploy From a Web Interface
• The admin tool updates the CloudFormation stack with the
new build parameter
• The deploy no...
Deploy From a Web Interface
• If everything looks good on stage - click ”thumbs up” to copy
the build to the production bu...
Ready for the Olympics
• Everything deployed during the winter
• Last user migrated in January
• Opening ceremony at Frida...
Ready for the Olympics
• 39 deploys during the week starting Feb 3
• 11 deploys in 6 different applications on the day of ...
Thanks!
johan.waernulf@viaplay.com
CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES
CONSTANTLY INTEGRATING INTO
PRODUCTION
KEY = ITERATION
ITERATION
=
MODIFY THE SYSTEM TO BETTER
MEET THE EXPECTATIONS OF
YOUR USERS
11.6s
Mean time
between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean number of
hosts...
SOFTWARE DEPLOY
≠
PRODUCT LAUNCH
1.8 BILLION PAGE VIEWS
$109 MILLION IN SALES
5.5 MILLION ITEMS SOLD
30 DEPLOYS PER DAY
1 DEPLOY EVERY 20 MINUTES
"Production is truly the only place
you can validate your code."
RUN-TIME CONFIGURATION OF ACTIVE FEATURES
AWS OPSWORKS
INTEGRATED APPLICATION
MANAGEMENT
14 BILLION REQUESTS/MONTH
50 000 DATABASE UPDATES/SEC
A / B TESTING
DATA-DRIVEN
ARCHITECTURES
METRICS @ETSY
CONTINUOUS
INTEGRATION
CONTINUOUS
DEPLOYMENT
CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
EXPERIMENTATION
CONTINUOUS
DEPLOYMENT
=
CONTINUOUS
IMPROVEMENT
« Want to increase innovation?
Lower the cost of failure »
Joi Ito
SPEED AND AGILITY
Experiment
Often
Fail quickly at
a low cost
More
Innovation
Experiment
Infrequently
Failure is
expensive...
wow
so deploy
much innovate
such agility
very continuous
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or i...
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS
Upcoming SlideShare
Loading in...5
×

AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS

2,648

Published on

With AWS companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100% API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session we'll talk about some key concepts and design patterns for Continuous Deployment and Continuous Integration, two elements of lean development of applications and infrastructures.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,648
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
42
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • All canonical software life cycles were designed by people who build installed software
  • The world’s handmade and vintage marketplace.
  • The world’s handmade and vintage marketplace.
  • Deploying code is the very first thing engineers learn to do at Etsy.
  • GUIs / algos
  • +24% on magic wand sales !
  • No authorization required, Detect problems quickly, gain confidence
  • Pour illustrer le launch early
  • Transcript of "AWS Summit Stockholm 2014 – T4 – Continuous integration on AWS"

    1. 1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Continuous Integration and Deployment on AWS Martin Elwin Manager, Solutions Architecture
    2. 2. CONTINUOUS INTEGRATION
    3. 3. DEVELOPER
    4. 4. SOURCE CODE REPOSITORY
    5. 5. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER
    6. 6. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER
    7. 7. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER PICK TASKS
    8. 8. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER SUBMIT CODE
    9. 9. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER CODE FETCH
    10. 10. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER BUILD OUTPUT
    11. 11. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER DOCS BINARIES & PACKAGES
    12. 12. SOURCE CODE REPOSITORY DNS CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER BUILDS
    13. 13. PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION
    14. 14. ON-DEMAND PAY AS YOU GO ELASTIC
    15. 15. = PROGRAMMABLE PLATFORM
    16. 16. … ec2.runInstances(new RunInstancesRequest() .withImageId("ami-4b814f22") .withInstanceType("m1.small") .withMinCount(1) .withMaxCount(1) .withKeyName("mykey") .withSecurityGroups("httpssh")); … JAVA SDK
    17. 17. … var runInstanceRequest = new RunInstancesRequest() { ImageId = "ami-4b814f22", InstanceType = "m1.small", MinCount = 1, MaxCount = 1, KeyName = "mykey" }; runInstanceRequest.SecurityGroup.Add("httpssh"); ec2.RunInstances(runInstanceRequest); … .NET SDK
    18. 18. … ec2.run_instances( 'ami-4b814f22', key_name='mykey', instance_type='m1.small', min_count=1, max_count=1, security_groups=['httpssh']) … PYTHON SDK
    19. 19. … ec2.instances.create( :image_id => 'ami-4b814f22', :instance_type => 'm1.small', :count => 1, :security_groups => 'httpssh', :key_pair => ec2.key_pairs['mykey']) … RUBY SDK
    20. 20. … $ec2 -> run_instances( 'ami-4b814f22', 1, 1, array( 'InstanceType' => 'm1.small', 'KeyName' => 'mykey', 'SecurityGroup' => 'httpssh') ); … PHP SDK
    21. 21. … var params = { ImageId: 'ami-4b814f22', InstanceType: 'm1.small', MinCount: 1, MaxCount: 1, SecurityGroups: ['httpssh'] }; ec2.runInstances(params, function(err, res) { … }); … JAVASCRIPT SDK
    22. 22. aws ec2 run-instances --image-id ami-4b814f22 --min-count 1 --max-count 1 --key-name mykey --security-groups httpssh UNIFIED CLI
    23. 23. New-EC2Instance -ImageId ami-4b814f22 -MinCount 1 -MaxCount 1 -KeyName mykey -SecurityGroupId sg-9cf9e5d9 -InstanceType m1.small POWERSHELL CLI
    24. 24. https://ec2.amazonaws.com/ ?Action=RunInstances &ImageId=ami-4b814f22 &MaxCount=1 &MinCount=1 &SecurityGroup.1=httpssh &KeyName=mykey &<AUTHPARAMS> AWS REST API
    25. 25. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
    26. 26. AWS CLOUDFORMATION STACK-BASED DEPLOYMENT SERVICE
    27. 27. AWS CLOUDFORMATION TEMPLATE
    28. 28. { "Description" : "Create a small instance", "Resources" : { "MyInstance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-4b814f22", "KeyName" : "mykey", "InstanceType" : "m1.small", "SecurityGroups" : ["httpssh"] } } } }
    29. 29. CLOUDFORMATION TEMPLATE DECLARATIVE DEFINITION Create it programmatically KNOWN CONFIGURATION Store stack configuration in source control PARAMETER DRIVEN Dynamic and user-driven templates COLLABORATION Share templates with ease as just files
    30. 30. APPLICATION VERSIONS + INFRASTRUCTURE VERSIONS
    31. 31. AWS CLOUDFORMATION TEMPLATE
    32. 32. Continuous Deployment at Viaplay
    33. 33. • Viaplay is Nordic’s leading online video service for TV, sport, film and kids' content. • The service is available on PC, Mac, Android and iOS mobile phones, tablets, the Viaplay box, LG Smart TV, Samsung Smart TV, Philips Smart TV, Panasonic Smart TV, Xbox 360, PlayStation®3 and PlayStation®4.
    34. 34. Preparations for the Olympics • New content / video / subscriber management platform • API:s completely rebuilt in Node.js in a service oriented architecture • All clients rebuilt (Web, Smartphones, Smart TV:s, Gaming consoles, Set Top Box) • Moved hosting of API:s to AWS
    35. 35. System Setup • Almost everything described in CloudFormation templates: Network (VPC), Load Balancers, Cache clusters, DNS entries, Autoscaling groups • The same templates for all different environments • Every application has its own CloudFormation stack with DNS entries, Elastic Load Balancers and an Auto Scaling Group
    36. 36. Auto Scaling • An ASG contains a variable number of instances running the same AMI • To enable auto scaling, the AMI must start automatically • Application servers starts with our custom AMI containing everything except the application build
    37. 37. Auto Scaling • Build and environment are set in tags and UserData. UserData is run at instance startup, tags are readable from the instance but more important clearly visible in the console. • When an EC2 instance is launched it fetches the build from S3, installs and starts it • As soon as the ELB health check is OK, the instance gets traffic
    38. 38. Deploy • Autoscaling has already made us create AMIs that starts the application at instance startup • Deploy is done by changing parameters to the stack • Setting new UserData on the ASG/EC2 launch config requires exchange of instances. CloudFormation does that for us with the Rolling Update policy
    39. 39. Lessons learnt • Rolling update takes time and is practically unstoppable during update. We are currently rewriting this so that we start a new ASG that completely replaces the old one • To be able to do a change in a stack, your IAM user needs to have the rights to change all resources in the template. All developers must be able to deploy their code, but we do not want all developers to be Power Users.
    40. 40. Custom Admin Application • Our admin application enables CloudFormation updates without giving Power User rights to all developers • We can keep track of who has done what in an easy way • The application also makes sure that deploys are done in exactly the same way every time • Built on top of the AWS Javascript SDK
    41. 41. Custom Admin Application • Our admin application enables CloudFormation updates without giving Power User rights to all developers • We can keep track of who has done what in an easy way • The application also makes sure that deploys are done in exactly the same way every time • Built on top of the AWS Javascript SDK
    42. 42. Deploy From a Web Interface • Build OK from Jenkins -> copied to S3 builds bucket • Deploy tool lists all files from the builds bucket
    43. 43. Deploy From a Web Interface • Write a short description of the deploy for the release email
    44. 44. Deploy From a Web Interface • The admin tool updates the CloudFormation stack with the new build parameter • The deploy notifier application sends an email with information on all commits done between the current and the new build
    45. 45. Deploy From a Web Interface • If everything looks good on stage - click ”thumbs up” to copy the build to the production bucket, else ”thumbs down” to reject the build
    46. 46. Ready for the Olympics • Everything deployed during the winter • Last user migrated in January • Opening ceremony at Friday Feb 7
    47. 47. Ready for the Olympics • 39 deploys during the week starting Feb 3 • 11 deploys in 6 different applications on the day of the opening ceremony • All went fine
    48. 48. Thanks! johan.waernulf@viaplay.com
    49. 49. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION
    50. 50. KEY = ITERATION
    51. 51. ITERATION = MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
    52. 52. 11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean number of hosts simultaneously receiving a deployment 30,000 Max number of hosts simultaneously receiving a deployment DEPLOYMENTS AT AMAZON.COM
    53. 53. SOFTWARE DEPLOY ≠ PRODUCT LAUNCH
    54. 54. 1.8 BILLION PAGE VIEWS $109 MILLION IN SALES 5.5 MILLION ITEMS SOLD
    55. 55. 30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES
    56. 56. "Production is truly the only place you can validate your code."
    57. 57. RUN-TIME CONFIGURATION OF ACTIVE FEATURES
    58. 58. AWS OPSWORKS INTEGRATED APPLICATION MANAGEMENT
    59. 59. 14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES/SEC
    60. 60. A / B TESTING
    61. 61. DATA-DRIVEN ARCHITECTURES
    62. 62. METRICS @ETSY
    63. 63. CONTINUOUS INTEGRATION CONTINUOUS DEPLOYMENT
    64. 64. CONTINUOUS DEPLOYMENT = CONTINUOUS EXPERIMENTATION
    65. 65. CONTINUOUS DEPLOYMENT = CONTINUOUS IMPROVEMENT
    66. 66. « Want to increase innovation? Lower the cost of failure » Joi Ito
    67. 67. SPEED AND AGILITY Experiment Often Fail quickly at a low cost More Innovation Experiment Infrequently Failure is expensive Less Innovation “ON-PREMISES”
    68. 68. wow so deploy much innovate such agility very continuous
    69. 69. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Thanks! Continuous Integration and Deployment on AWS Martin Elwin Manager, Solutions Architecture

    ×