Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013

  • 10,124 views
Uploaded on

With AWS, companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100 percent API driven enables …

With AWS, companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100 percent API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session, we talk about some key concepts and design patterns for continuous deployment and continuous integration, two elements of lean development of applications and infrastructures.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
10,124
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
173
Comments
0
Likes
16

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13th, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 2. Who Am I? I work for AWS I worked for the DNC 2009-2012 I was embedded in the DevOps Team @ OFA AWS does not endorse political candidates
  • 3. Who Am I? JP, DevOps for Mozilla Foundation Previous gig DevOps at OFA 2012 Before that, Ops at Threadless @jdotp Mozilla Foundation does endorse animated cats
  • 4. CONTINUOUS INTEGRATION
  • 5. What is Continuous Integration? Changes to code automatically deployed to mainline branch • After passing unit and mock tests Makes changes to code and deployments iterative, not monolithic Bugs are detected quickly Allows rapid development Helps automate deployments
  • 6. DEVELOPER
  • 7. SOURCE CODE REPOSITORY
  • 8. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER
  • 9. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER
  • 10. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER PICK TASKS
  • 11. SOURCE CODE REPOSITORY SUBMIT CODE CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER
  • 12. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER SCHEDULE BUILD PROJECT MANAGEMENT SERVER
  • 13. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER RECURRENT BUILDS PROJECT MANAGEMENT SERVER
  • 14. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER CODE FETCH PROJECT MANAGEMENT SERVER
  • 15. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER CODE QUALITY TESTS PROJECT MANAGEMENT SERVER TEST RESULTS
  • 16. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER BUILD OUTPUT PROJECT MANAGEMENT SERVER
  • 17. SOURCE CODE REPOSITORY DOCS CONTINUOUS INTEGRATION SERVER BINARIES & PACKAGES PROJECT MANAGEMENT SERVER DEV FACING NOTIFICATIONS
  • 18. SOURCE CODE REPOSITORY BUILDS CONTINUOUS INTEGRATION SERVER DNS PROJECT MANAGEMENT SERVER
  • 19. PAIN POINTS • • • • • UNIT TESTS INCOMPLETE MOCK TESTS MAINTENANCE EXPENSIVE TEST ENVIRONMENT TEST ENVIRONMENT ≠ PRODUCTION DEPLOYMENT CYCLES
  • 20. ON-DEMAND ELASTIC PAY AS YOU GO
  • 21. = PROGRAMMABLE PLATFORM
  • 22. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
  • 23. A lot of options… Configuration Management Systems • Puppet • Chef • Saltstack Deployment Frameworks • • • • • AWS Elastic Beanstalk AWS OpsWorks Ansible Fabric Capistrano Infrastructure Management • AWS CloudFormation
  • 24. Bootstrapping Bake an AMI Time consuming configuration (startup time) Static configurations (less change management) Configure dynamically
  • 25. Bootstrapping Bake an AMI Configure dynamically Continuous deployment (latest code) Environment specific (devtest-prod)
  • 26. Obama for America awsofa.info
  • 27. Web-Scale Applications
  • 28. 500k+ IOPS DB Systems
  • 29. Services API
  • 30. Typical Charts
  • 31. How?
  • 32. Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 33. Infrastructure, Configuration Management & Monitoring Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 34. Configuration Management: Puppet In mid-2011, we looked at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with stateless, horizontally scalable infrastructure How did we do this?
  • 35. Bootstrapping Puppet with CloudInit CloudInit is built into Ubuntu and Amazon Linux • Allows you to pass bootstrap parameters in Amazon EC2 user-data field, in YAML format
  • 36. Bootstrapping Puppet with CloudInit Don’t store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better – avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLI’s S3 client
  • 37. Bootstrapping Puppet with CloudInit Built-in puppet support Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on • Use security groups and/or NACLs for network-level security In nodes.pp, use regex to match node names
  • 38. Puppet Tips Use a base class to define your standard install
  • 39. Puppet Tips Use runstages Don’t store credentials in puppet, store them in private Amazon S3 buckets • Use AWS IAM to secure the credentials bucket/folders within that bucket
  • 40. Puppet Tips Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories • Can be either public or private repos, depending on your needs • Amazon S3 Private RPM Repos: http://git.io/YAcsbg • Amazon S3 Private Debian Repos: http://git.io/ecCjWQ
  • 41. Puppet Tips By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk
  • 42. Scaling the Puppet Masters Use an Auto Scaling group for puppet masters • Min size => 2, use multiple Availability Zones Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data Auto-sign must be on
  • 43. One thing that is difficult to prepare for…
  • 44. They had this built for the previous 3 months, a on the East Coast.
  • 45. They had this built for the previous 3 months, a on the East Coast. We built this part in 9 hours to be safe. AWS + Puppet + Netflix Asgard + WAN Optimization Software + DevOps = Cross-Continent FaultTolerance On-Demand
  • 46. ARC205 Thursday @ 3:00PM Lando 4303 Deploying the ‘League of Legends’ Data Pipeline with Chef
  • 47. Mozilla Foundation
  • 48. Webmaker.org circa 2012 • Included Apps, non-SOA: Thimble, Popcorn, Goggles • ~20 pushes of new software in 2012 • Operations and Development interacted mostly through bugzilla tickets for deploys • Hosting in physical datacenter at Mozilla Webmaker.org circa early 2013 • Deciding to go 12-factor, SOA in app layer • Weekly pushes of Popcorn on train model • Operations and Development interacted mostly through bugzilla tickets for deploys
  • 49. April 2013 Webmaker begins rebuilding entire platform SOA, 12-factor in node.js exclusively Moving apps into AWS and DevOps / CI
  • 50. Since April 2013…. Openbadges, Webmaker combine for: Pushes Per Day (Staging and Prod) Pushes Per Day to Staging / Prod 1339 Pushes
  • 51. Who? • • • • • ~30 Paid Developers Hundreds of Students Thousands of Contributors One DevOps / Internet Jedi Multiple Teams How? • Puppet, Jenkins, Fabric • Tight feedback loops: Newrelic, Opsview • Culture Shift • Staging Envs • Brave devs iterate, keeping work in-context • Visible Ops • Cross-train developers in operations
  • 52. What changed?
  • 53. 1) Know How You Were Doing Before 2) Know What Changed When 3) Know How You Are Now Doing = The confidence to try more things and try them faster, with minimum viable planning.
  • 54. Deployment Pipeline
  • 55. ARC312 Friday @ 1:30PM Lando 4206 SmugMug’s Zero-Downtime Migration to AWS
  • 56. AWS CLOUDFORMATION STACK-BASED DEPLOYMENT SERVICE
  • 57. AWS CLOUDFORMATION TEMPLATE
  • 58. { } "Description" : "Create RDS with username and password", "Resources" : { } "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } }
  • 59. "AWS::CloudFormation::Init" : { } "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" }
  • 60. { } "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } },
  • 61. PROCEDURAL DEFINITION KNOWN CONFIGURATION Create it programmatically Store stack configuration in source control CLOUDFORMATION TEMPLATE PARAMETER DRIVEN Dynamic and user-driven templates COLLABORATION Share templates with ease as just files
  • 62. APPLICATION VERSIONS + INFRASTRUCTURE VERSIONS
  • 63. AWS CLOUDFORMATION TEMPLATE
  • 64. ARC203 Wednesday @ 4:15 Lando 4303 How Adobe Deploys Refreshing the Entire Stack Every Time
  • 65. Mars Rover Landing by the #’s NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy) FUN FUN FUN
  • 66. NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy) NASA says we can’t use their live stream setup It’s 6 days before the landing It’s the same week as the Olympics Available technical resources from JPL: Brett and Khawaja NOT FUN NOT FUN NOT FUN Mars Rover Landing by the #’s
  • 67. The sticky wicket: manifest.f4m size = 4kb New every 4 sec caching difficult HD Video stream size = 4mb never changes easy to cache
  • 68. The plan Design a solution around our limits • • • • Max connections to origin = 6 Max streams per cache node = 20 Local Latency = critical US-WEST-1 capacity reserved for S3 static images Test the snot out of it Hang on!
  • 69. ok, so one of them is a rocket scientist..
  • 70. “The 42 pack”
  • 71. LOAD TESTING
  • 72. Benchmarking
  • 73. US-East Cache Node Performance 25.3 Gbps
  • 74. Impact on US-East FMS Origin Servers Only ~42Mbps
  • 75. ARC303 Friday @ 1:30PM Delfino 4003 Unmeltable Infrastructure at Scale: Using Apache Kafka, Twitter Storm and Elastic Search on AWS
  • 76. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION
  • 77. KEY = ITERATION
  • 78. ITERATION = MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
  • 79. DEPLOYMENTS AT AMAZON.COM 11.6s 1,079 10,000 30,000 Mean time between deployments (weekday) Max number of deployments in a single hour Mean number of hosts simultaneously receiving a deployment Max number of hosts simultaneously receiving a deployment
  • 80. SOFTWARE DEPLOY ≠ PRODUCT LAUNCH
  • 81. 1.5 BILLION PAGE VIEWS $83 MILLION IN TRANSACTIONS 4.2 MILLION ITEMS SOLD OCTOBER 2012
  • 82. 30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES
  • 83. "Production is truly the only place you can validate your code."
  • 84. AWS OPSWORKS INTEGRATED APPLICATION MANAGEMENT DMG 304 Thursday @ 3:00PM Murano 3206 AWS OPSWORKS UNDER THE HOOD
  • 85. 14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES / SEC NO CACHE
  • 86. DATA-DRIVEN ARCHITECTURES
  • 87. METRICS @ETSY
  • 88. METRICS @OBAMA FOR AMERICA
  • 89. COST-ORIENTED ARCHITECTURES
  • 90. PHP+APACHE+VARNISH NGINX+NODEJS
  • 91. CONTINUOUS DEPLOYMENT CONTINUOUS INTEGRATION
  • 92. CONTINUOUS DEPLOYMENT = CONTINUOUS EXPERIMENTATION
  • 93. CONTINUOUS DEPLOYMENT = CONTINUOUS IMPROVEMENT
  • 94. INNOVATE
  • 95. « Want to increase innovation? Lower the cost of failure » Joi Ito
  • 96. SPEED AND AGILITY “ON-PREMISES” Experiment Infrequently Experiment Often Failure is expensive Fail quickly at a low cost Less Innovation More Innovation
  • 97. AWS re:Invent Pub Crawl Join the AWS Startup Team this evening at the AWS Pub Crawl When: Wednesday November 13, 5:30pm - 7:30pm Where: Canaletto at The Venetian, 2nd Floor Who Will Be There: Startups, The AWS Startup Team, Startup Launch Companies and AWS re:Invent Hackathon winners
  • 98. Startup Spotlight Sessions with Dr. Werner Vogels Thurs. Nov 14, Marcello Room 4406 SPOT 203 - Fireside Chats – Startup Founders, 1:30-2:30pm – Eliot Horowitz, CTO of MongoDB – Jeff Lawson, CEO of Twilio – Valentino Volonghi, Chief Architect of AdRoll SPOT 204 - Fireside Chats – Startup Influencers, 3:00-4:00pm – Albert Wegner, Managing Partner at Union Square Ventures – David Cohen, Founder and CEO of TechStars SPOT 101 - Startup Launches, 4:15-5:15pm – 5 companies powered by AWS launching at AWS re:Invent 2013
  • 99. Please give us your feedback on this presentation ARC 307 As a thank you, we will select prize winners daily for completed surveys!