Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
DevOps, Continuous Integration &
Deployment on AWS...
DevOps
What is DevOps?
• « DevOps is the practice of operations and
development engineers participating together in
the entire se...
Continuous Integration
What is Continuous Integration?
• Changes to code automatically deployed to
mainline branch
– After passing unit and mock ...
SOURCE CODE
REPOSITORY
PROJECT MANAGEMENT
SERVER
CONTINUOUS
INTEGRATION SERVER
DEVELOPER
PICK
TASKS
SUBMIT
CODE
SCHEDULE
B...
SOURCE CODE
REPOSITORY
DNS
CONTINUOUS
INTEGRATION SERVER
PROJECT
MANAGEMENT SERVER
BUILDS
AWS code services
AWS CodeCommit
Coming soon
AWS CodePipeline
Coming soon
AWS CodeDeploy
Launched Nov 2014
Cloud software development lifecycle
10/13/14 10
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
Am...
Source Control
Private Git repositories hosted on Amazon S3
• Full Git compatibility (use with existing tools)
• All the b...
Same Git experience
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli
Cloning into 'aws-cli'...
...
Continuous Delivery
Customizable release automation, with integrated build and test
• Model and visualize custom release w...
Deployment
Coordinate software updates to fleets of EC2 instances
• Rolling updates for no downtime
• Deployment health ch...
9/25/14 Slides not intended for 16
Cloud software development lifecycle
10/13/14 17
MonitorProvisionDeployTestBuildCode
AWS Elastic Beanstalk
AWS OpsWorks
Cl...
CODECOMMIT
DNS
CODEPIPELINE
PROJECT
MANAGEMENT SERVER
BUILDS
PAIN POINTS
• UNIT TESTS INCOMPLETE
• MOCK TESTS MAINTENANCE
• EXPENSIVE TEST ENVIRONMENT
• TEST ENVIRONMENT ≠
PRODUCTION
...
ON-DEMAND
PAY AS YOU GO
ELASTIC
=
PROGRAMMABLE PLATFORM
IF YOU CAN PROGRAM IT
YOU CAN AUTOMATE IT
A lot of options…
• Configuration Management Systems
– Puppet
– Chef
– Saltstack
• Deployment Frameworks
– CodeDeploy
– AW...
Bake an Image Configure dynamically
Time consuming
configuration (startup time)
Static configurations (less
change managem...
Bake an Image Configure dynamically
Continuous deployment
(latest code)
Environment specific (dev-
test-prod)
Bootstrapping
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
AWS CLOUDFORMATION
STACK-BASED DEPLOYMENT SERVICE
CLOUDFORMATION
TEMPLATE
Headers
Parameters
Mappings
Resources
Outputs
{
"Description" : "Create RDS with username and password",
"Resources" : {
"MyDB" : {
"Type" : "AWS::RDS::DBInstance",
"Pr...
"AWS::CloudFormation::Init" : { "config" : {
"packages" : {
"yum" : {
"mysql" : [],
"mysql-server" : [],
"httpd" : [],
"ph...
{
"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2
KeyPair to enable SSH
access to the instance",
"...
CLOUDFORMATION
TEMPLATE
PROCEDURAL
DEFINITION
Create it programmatically
KNOWN
CONFIGURATION
Store stack configuration in
...
APPLICATION VERSIONS
+
INFRASTRUCTURE VERSIONS
CLOUDFORMATION
TEMPLATE
CONTINUOUS
DEPLOYMENT
SMALL, FREQUENT CHANGES CONSTANTLY
INTEGRATING INTO PRODUCTION.
KEY = ITERATION
ITERATION
=
MODIFY THE SYSTEM TO BETTER MEET
THE EXPECTATIONS OF YOUR USERS
11.6s
Mean time
between
deployments
(weekday)
1,079
Max number of
deployments in a
single hour
10,000
Mean number of
hosts...
SOFTWARE DEPLOY
≠
PRODUCT LAUNCH
DATA-DRIVEN
ARCHITECTURES
METRICS @ETSY
METRICS @OBAMA FOR AMERICA
Metrics and Monitoring Options
CloudWatch
… and many more
CONTINUOUS
INTEGRATION
CONTINUOUS
DEPLOYMENT
CONTINUOUS DEPLOYMENT
=
CONTINUOUS EXPERIMENTATION
CONTINUOUS DEPLOYMENT
=
CONTINUOUS IMPROVEMENT
INNOVATE
SPEED AND AGILITY
Experiment
Often
Fail quickly at
a low cost
More
Innovation
Experiment
Infrequently
Failure is
expensive...
At a glance
What is NaNoWriMo?
• National
• Novel
• Writing
• Month
We are a community of novelists.
Every November, hundr...
It all started in 1999
Our founder, Chris Baty,
challenged 20 over-caffeinated
friends to write 50,000 words each
in a sin...
Explosive growth, and no plan to manage it
NaNoWriMo now
NaNoWriMo Main: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6
Camp NaNoWriMo: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6
The Yo...
November in detail
• Nov 1: ~500k sessions as people begin
• All month: ~50k forum posts per day
• Nov 30: Last chance to ...
Annual Site Crash
• Every year from 2003 to 2011 the site would crash on
November 1.
• In 2012 we migrated the sites to Am...
Outrage!
They wanted the site to crash; it was
considered a rite of passage. Oh well.
Keeping NaNoWriMo Online
• Proxy servers (Varnish) on Debian 7.5 EC2
instances
• Ruby on Rails on EC2 (Nginx)
• Worker ser...
The Year in Instances
Next Steps: Goals
 To increase uptime for all services
 To make server management less time-consuming
 To reliably and ...
Next Steps: Goodbye Drupal
What: Porting Drupal sites to Ruby 2.2, Rails 4.2
Why: The old Drupal-based sites are slow and ...
Next Steps: Emailing via Amazon SES
What: Migrating outgoing emails from PHPlist/Sendmail to
SES
Why: Event-driven emails ...
Next Steps: Load Balancing
What: Converting proxy servers to Elastic Load Balancers
Why: We will benefit from better AWS d...
Next Steps: CodeDeploy
What: Using CodeDeploy for deployment process
Why: Currently deployment relies on many custom shell...
Next Steps: Autoscaling
What: Implement an Auto Scaling process
Why: In the past we’ve had to predict our server needs
ahe...
Next Steps: Amazon SNS Notifications
What: Increased use of Amazon SNS service
Why: Last year SNS event messaging helped u...
Next Steps: On the Horizon
What: Amazon CloudSearch; AWS Lambda
Why: We are interested in easy-to-manage alternatives to
S...
Potential future architecture
Internet
Load Balancer
ELB EC2 instances
Code servers
EC2 instances
Databases
RDS DB instanc...
SAN FRANCISCO
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission
Upcoming SlideShare
Loading in …5
×

DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission

10,423 views

Published on

Organizations around the globe are leveraging the cloud to accomplish world-changing missions. This session will address how AWS can help organizations put more money toward their mission and scale outreach and operations to achieve more with less. Hear some of AWS’s most advanced customers on how their organizations handle DevOps, continuous integration and deployment. Learn how these practices allow them to rapidly develop, iterate, test and deploy highly-scalable web applications and core operational systems on AWS. The discussion will focus on best practices, lessons learned, and the specific technologies and services they use.

Published in: Technology

DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into Your Mission

  1. 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved DevOps, Continuous Integration & Deployment on AWS Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh Dave Beck, Technical Director, National Novel Writing Month @DaveB_NaNoWriMo
  2. 2. DevOps
  3. 3. What is DevOps? • « DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support » - theagileadmin.com
  4. 4. Continuous Integration
  5. 5. What is Continuous Integration? • Changes to code automatically deployed to mainline branch – After passing unit and mock tests • Makes changes to code, and deployments iterative, not monolithic • Bugs are detected quickly • Helps automate deployments • Allows rapid development and deployment
  6. 6. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER DEVELOPER PICK TASKS SUBMIT CODE SCHEDULE BUILD RECURRENT BUILDS CODE FETCHCODE QUALITY TESTS TEST RESULTS BUILD OUTPUT DOCS BINARIES & PACKAGES DEV FACING NOTIFICATIONS CLOUDFORMATION AMIS or CONTAINERS
  7. 7. SOURCE CODE REPOSITORY DNS CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER BUILDS
  8. 8. AWS code services AWS CodeCommit Coming soon AWS CodePipeline Coming soon AWS CodeDeploy Launched Nov 2014
  9. 9. Cloud software development lifecycle 10/13/14 10 MonitorProvisionDeployTestBuildCode AWS Elastic Beanstalk AWS OpsWorks Amazon CloudWatch AWS CloudFormation ?
  10. 10. Source Control Private Git repositories hosted on Amazon S3 • Full Git compatibility (use with existing tools) • All the benefits of the cloud (scalable, durable, reliable, low pay as you go pricing) • No size limits on repositories (store binary files) • Online code tools with browse, edit, diff 10/13/14 11
  11. 11. Same Git experience $ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli Cloning into 'aws-cli'... Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done. Resolving deltas: 100% (9900/9900), done. Checking connectivity... done. $ nano README.rst $ git commit -am 'updated README' [master 4fa0318] updated README 1 file changed, 1 insertion(+) $ git push Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli 4dacd6d..4fa0318 master -> master
  12. 12. Continuous Delivery Customizable release automation, with integrated build and test • Model and visualize custom release workflow (source  build  beta  gamma  prod) • Automate builds, tests, and deployments • Enforce custom rules, approvals, and gates • Integrate with third-party and custom tools 10/13/14 13
  13. 13. Deployment Coordinate software updates to fleets of EC2 instances • Rolling updates for no downtime • Deployment health checks and easy rollback • Auto Scaling integration • Works with any application • Reuse existing setup tools (Bash, Powershell, Chef, Puppet…) 10/13/14 15
  14. 14. 9/25/14 Slides not intended for 16
  15. 15. Cloud software development lifecycle 10/13/14 17 MonitorProvisionDeployTestBuildCode AWS Elastic Beanstalk AWS OpsWorks CloudWatchCloudFormationCodeDeploy CodeCommit CodePipeline
  16. 16. CODECOMMIT DNS CODEPIPELINE PROJECT MANAGEMENT SERVER BUILDS
  17. 17. PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES
  18. 18. ON-DEMAND PAY AS YOU GO ELASTIC
  19. 19. = PROGRAMMABLE PLATFORM
  20. 20. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
  21. 21. A lot of options… • Configuration Management Systems – Puppet – Chef – Saltstack • Deployment Frameworks – CodeDeploy – AWS Elastic Beanstalk – AWS OpsWorks – Ansible – Fabric – Capistrano • Infrastructure Management – CloudFormation • Containers – Amazon EC2 Container Service
  22. 22. Bake an Image Configure dynamically Time consuming configuration (startup time) Static configurations (less change management) Bootstrapping
  23. 23. Bake an Image Configure dynamically Continuous deployment (latest code) Environment specific (dev- test-prod) Bootstrapping
  24. 24. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved AWS CLOUDFORMATION STACK-BASED DEPLOYMENT SERVICE
  25. 25. CLOUDFORMATION TEMPLATE
  26. 26. Headers Parameters Mappings Resources Outputs
  27. 27. { "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }
  28. 28. "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }
  29. 29. { "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }
  30. 30. CLOUDFORMATION TEMPLATE PROCEDURAL DEFINITION Create it programmatically KNOWN CONFIGURATION Store stack configuration in source control PARAMETER DRIVEN Dynamic and user-driven templates COLLABORATION Share templates with ease as just files
  31. 31. APPLICATION VERSIONS + INFRASTRUCTURE VERSIONS
  32. 32. CLOUDFORMATION TEMPLATE
  33. 33. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION.
  34. 34. KEY = ITERATION
  35. 35. ITERATION = MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
  36. 36. 11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean number of hosts simultaneously receiving a deployment 30,000 Max number of hosts simultaneously receiving a deployment DEPLOYMENTS AT AMAZON.COM (in 2011)
  37. 37. SOFTWARE DEPLOY ≠ PRODUCT LAUNCH
  38. 38. DATA-DRIVEN ARCHITECTURES
  39. 39. METRICS @ETSY
  40. 40. METRICS @OBAMA FOR AMERICA
  41. 41. Metrics and Monitoring Options CloudWatch … and many more
  42. 42. CONTINUOUS INTEGRATION CONTINUOUS DEPLOYMENT
  43. 43. CONTINUOUS DEPLOYMENT = CONTINUOUS EXPERIMENTATION
  44. 44. CONTINUOUS DEPLOYMENT = CONTINUOUS IMPROVEMENT
  45. 45. INNOVATE
  46. 46. SPEED AND AGILITY Experiment Often Fail quickly at a low cost More Innovation Experiment Infrequently Failure is expensive Less Innovation “ON-PREMISE”
  47. 47. At a glance What is NaNoWriMo? • National • Novel • Writing • Month We are a community of novelists. Every November, hundreds of thousands of writers gather virtually and in-person to write 50,000 words each. (The Great Gatsby, by comparison, is 47,094 words.) Last year, 180,000 started new novels, and 40,000 finished.
  48. 48. It all started in 1999 Our founder, Chris Baty, challenged 20 over-caffeinated friends to write 50,000 words each in a single month. Six of them finished.
  49. 49. Explosive growth, and no plan to manage it
  50. 50. NaNoWriMo now NaNoWriMo Main: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6 Camp NaNoWriMo: Ruby 1.9.3, Rails 3.2.2, MySQL 5.6 The Young Writers Program: Drupal 5, MySQL 5.6 NaNoWriMo Store: Drupal 5, MySQL 5.6
  51. 51. November in detail • Nov 1: ~500k sessions as people begin • All month: ~50k forum posts per day • Nov 30: Last chance to post word counts
  52. 52. Annual Site Crash • Every year from 2003 to 2011 the site would crash on November 1. • In 2012 we migrated the sites to Amazon Web Services. • The result: No more outages! • Naturally, this is how our participants reacted…
  53. 53. Outrage! They wanted the site to crash; it was considered a rite of passage. Oh well.
  54. 54. Keeping NaNoWriMo Online • Proxy servers (Varnish) on Debian 7.5 EC2 instances • Ruby on Rails on EC2 (Nginx) • Worker servers running Resque on EC2 • Support servers running Resque, Redis, Sphinx • MySQL on RDS instances • Domains and subdomains on Route 53 • Image assets on S3 instances via EBS • Deployment via Capistrano, Puppet
  55. 55. The Year in Instances
  56. 56. Next Steps: Goals  To increase uptime for all services  To make server management less time-consuming  To reliably and quickly respond to user-driven events  To reduce EC2 costs through autoscaling  To create a faster and more dynamic email system  To improve site search
  57. 57. Next Steps: Goodbye Drupal What: Porting Drupal sites to Ruby 2.2, Rails 4.2 Why: The old Drupal-based sites are slow and use more than their share of server resources.
  58. 58. Next Steps: Emailing via Amazon SES What: Migrating outgoing emails from PHPlist/Sendmail to SES Why: Event-driven emails on the sites are backgrounded to support servers, which has proved slow and unreliable. Mass emails rely on an old and very sluggish application which has limited layout tools and metrics.
  59. 59. Next Steps: Load Balancing What: Converting proxy servers to Elastic Load Balancers Why: We will benefit from better AWS dashboard tools, metrics, and notifications. It is part of our autoscaling strategy.
  60. 60. Next Steps: CodeDeploy What: Using CodeDeploy for deployment process Why: Currently deployment relies on many custom shell scripts and tends to be unreliable. We intend to move to stateless instance spin-ups as part of our autoscaling strategy.
  61. 61. Next Steps: Autoscaling What: Implement an Auto Scaling process Why: In the past we’ve had to predict our server needs ahead of time. If we underestimated, the sites would crash; if we overestimated, we weren’t using our small budget effectively. And the planning process consumed too much staff time.
  62. 62. Next Steps: Amazon SNS Notifications What: Increased use of Amazon SNS service Why: Last year SNS event messaging helped us respond to emerging crises very quickly. With our expanded AWS presence, we will link event notifications to all of the services we use.
  63. 63. Next Steps: On the Horizon What: Amazon CloudSearch; AWS Lambda Why: We are interested in easy-to-manage alternatives to Sphinx – CloudSearch is one option. Lambda looks intriguing for a variety of possible applications.
  64. 64. Potential future architecture Internet Load Balancer ELB EC2 instances Code servers EC2 instances Databases RDS DB instances Search CloudSearch instance Outgoing email Amazon SES service
  65. 65. SAN FRANCISCO

×