Downloaded 66 times
![build.gradle
apply plugin: 'nebula'
group = 'netflix'
artifacts {
archives package
archives javadocJar
}
dependencies {
compile(
[group: 'netflix', name: 'resourceregistry',
version: 'latest.release'],
[group: 'netflix', name: 'platform',
version: 'latest.release'],
...](https://image.slidesharecdn.com/building-cloud-tools-for-netflix-codemash2012-120116001727-phpapp02/85/Building-cloud-tools-for-netflix-code-mash2012-13-320.jpg)
Uploaded byCarl Quinn
Building cloud-tools-for-netflix-code mash2012
Netflix moved to AWS to leverage its scale and features. They refactored monolithic Java apps into microservices and baked application machine images for faster deployment. They automated builds with Jenkins and deployed services using auto scaling groups, load balancers, and fast rollbacks for optimism. They invented concepts like Applications and Clusters to organize AWS resources for their needs.
More Related Content
![[AWS Dev Day] 실습워크샵 | 모두를 위한 컴퓨터 비전 딥러닝 툴킷, GluonCV 따라하기](https://cdn.slidesharecdn.com/ss_thumbnails/gluoncv-190930065523-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AWS Dev Day] 기조연설 – Olivier Klein AWS 신기술 부문 책임자, 정성권 삼성전자 수석](https://cdn.slidesharecdn.com/ss_thumbnails/awsdevdaykeynote-190930084015-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Building cloud-tools-for-netflix-code mash2012
Building cloud-tools-for-netflix-code mash2012
- 1. Building for theCloud @ Netflix @cquinn #netflixcloud (@joesondow)
- 2. Who Am I? ManagerEngineering Tools team @ Netflix Automated build tools and systems Base machine image definition and maintenance Application machine image baking Netflix Application Console: deploy and manage
- 3. Why Cloud? WhyAWS? We can’t build datacenters fast enough We want to use Clouds, not build them Leverage AWS Scale “the biggest public cloud” Leverage AWS Features “two years ahead of the rest”
- 4. Legacy Website Mostly oneMonolithic Java Web App Single Big Oracle DB Dedicated Data Center Machines
- 5. Old School Build& Deploy Datacenter LB Push traffic control WebCM app bundle: war + config Website Website Website Website Dat Website Website aba se u Website Website pda tes Website Website Website Custom Perl web app: Oracle build, deploy & manage Manually configured: Linux, Apache, Tomcat
- 6. Path to AWS Buildingnewly refactored Java Web Services Baking (assembling) ec2 Amazon Machine Images Deploying and Managing Service Clusters
- 7.
- 8. Step One: Componentize Ant targets Workstation sync test build release source checkin library jars Perforce jars used in app builds
- 9. Step Two: Automate Ant targets Hudson sync test build release app bundles source, jars checkin library jars Perforce jars used in app builds
- 10. Step Three: FullPipeline app bundles Artifactory Ivy snapshot / release libraries libraries / apps Jenkins resolve test publish sync compile build report source some Groovy Perforce Ant targets Gradle coming
- 11. Step Three: FullPipeline app bundles Artifactory Ivy snapshot / release libraries libraries / apps Jenkins resolve test publish sync compile build report source some Groovy Perforce Ant targets Gradle coming
- 12. build.xml <project name="helloworld"> <import file="../../../Tools/build/webapplication.xml"/> </project> ivy.xml <info organisation="netflix" module="helloworld"> <publications> <artifact name="helloworld" type="package" e:classifier="package" ext="tgz"/> <artifact name="helloworld" type="javadoc" e:classifier="javadoc" ext="jar"/> </publications> <dependencies> <dependency org="netflix" name="resourceregistry" rev="latest.${input.status}" conf="compile"/> <dependency org="netflix" name="platform" rev="latest.${input.status}" conf="compile" /> ...
- 13. build.gradle apply plugin: 'nebula' group= 'netflix' artifacts { archives package archives javadocJar } dependencies { compile( [group: 'netflix', name: 'resourceregistry', version: 'latest.release'], [group: 'netflix', name: 'platform', version: 'latest.release'], ...
- 14. Step Three: FullPipeline app bundles Artifactory Ivy snapshot / release libraries libraries / apps Jenkins resolve test publish sync compile build report source some Groovy Perforce Ant targets Gradle coming
- 15. Step Three: FullPipeline app bundles Artifactory Ivy snapshot / release libraries libraries / apps Jenkins resolve test publish sync compile build report source some Groovy Perforce Ant targets Gradle coming
- 16. App Artifacts app bundles Jenkins Master Slave Slave Slave Slave Slave Slave Slave AWS us-west-1 VPC
- 17. App Artifacts app bundles Yum Jenkins Master app bundle RPMs Slave Slave Slave Slave Slave Slave Slave AWS us-west-1 VPC
- 18. App Artifacts app bundles Artifactory app bundle artifacts Yum Jenkins Master app bundle RPMs Slave Slave Slave Slave Slave Slave Slave AWS us-west-1 VPC
- 19. App Artifacts app bundles Artifactory app bundle artifacts Yum Jenkins Master app bundle RPMs Slave Slave Slave Slave Slave Slave Slave AWS us-west-1 VPC
- 20.
- 21.
- 22. Why bake? Traditional: •launch OS •installpackages •install app
- 23. Why bake? Traditional: •launch OS Generic AMI •install packages Instance •install app
- 24. Why bake? Traditional: •launch OS Generic AMI •install packages Instance •install app
- 25. Why bake? Traditional: •launch OS Generic AMI •install packages Instance •install app
- 26. Why bake? Traditional: •launch OS Generic AMI •install packages Instance •install app Netflix: •launch OS+app
- 27. Why bake? Traditional: •launch OS Generic AMI •install packages Instance •install app Netflix: •launch OS+app App AMI Instance
- 28. Base Image Baking S3 / EBS foundation AMI Linux: CentOS, Fedora, Ubuntu Yum / Apt install AWS
- 29. Base Image Baking S3 / EBS foundation AMI Linux: CentOS, Fedora, Ubuntu mount Yum / Apt RPMs: Apache, Java... Bakery install AWS ec2 slave instances
- 30. Base Image Baking S3 / EBS foundation AMI base Linux: CentOS, Fedora, Ubuntu AMI snapshot mount Yum / Apt RPMs: Apache, Java... Bakery install AWS ec2 slave instances
- 31. Base Image Baking S3 / EBS foundation AMI base Linux: CentOS, Fedora, Ubuntu AMI snapshot mount Ready Yum / for Apt RPMs: Apache, Java... Bakery app install bake AWS ec2 slave instances
- 32. App Image Baking S3 / EBS base AMI Linux, Apache, Java, Tomcat Jenkins / Yum / install Artifactory AWS
- 33. App Image Baking S3 / EBS base AMI Linux, Apache, Java, Tomcat mount Jenkins / Yum / app bundle Bakery install Artifactory AWS ec2 slave instances
- 34. App Image Baking S3 / EBS base AMI app Linux, Apache, Java, Tomcat AMI snapshot mount Jenkins / Yum / app bundle Bakery install Artifactory AWS ec2 slave instances
- 35. App Image Baking S3 / EBS base AMI app Linux, Apache, Java, Tomcat AMI snapshot mount Jenkins / Ready to launch Yum / app bundle Bakery install Artifactory AWS ec2 slave instances
- 36.
- 37.
- 38.
- 39. Cloud deployment model Auto Scaling Group
- 40. Cloud deployment model Auto Scaling Group Launch Configuration
- 41. Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Configuration
- 42. Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Configuration Amazon Machine Image
- 43. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Group Launch Configuration Amazon Machine Image
- 44. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
- 45. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
- 46. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
- 47. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
- 48.
- 49. Cloud deployment model Search
- 50. Cloud deployment model API Search
- 51. Cloud deployment model Ratings API Search
- 52. Cloud deployment model StreamingStarts Ratings API Search
- 53. Cloud deployment model StreamingStarts Ratings API Search Autocomplete
- 54. Cloud deployment model Sign Up Streaming Starts Ratings API Search Autocomplete
- 55. Cloud deployment model Sign Up Streaming Starts Ratings Application Application Application API Search Autocomplete Application Application Application
- 56.
- 57. Inventing the Application Problem: Application is not an Amazon concept
- 58. Inventing the Application Problem: Application is not an Amazon concept Solution: Create an Application domain in SimpleDB Enforce naming conventions on Amazon objects
- 59.
- 60.
- 61. Fast Rollback Optimism causesoutages Production traffic is unique
- 62. Fast Rollback Optimism causesoutages Production traffic is unique Keep old version running
- 63. Fast Rollback Optimism causesoutages Production traffic is unique Keep old version running Switch traffic to new version
- 64. Fast Rollback Optimism causesoutages Production traffic is unique Keep old version running Switch traffic to new version Monitor results
- 65. Fast Rollback Optimism causesoutages Production traffic is unique Keep old version running Switch traffic to new version Monitor results Revert traffic quickly
- 66.
- 67. Fast Rollback api-frontend api-usprod-v007
- 68. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 69. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 70. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 71. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 72. Fast Rollback api-frontend api-usprod-v008
- 73.
- 74. Fast Rollback api-frontend api-usprod-v007
- 75. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 76. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 77. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
- 78. Fast Rollback api-frontend api-usprod-v007
- 79.
- 80. Inventing the Cluster Problem: Two ASGs with one function but different names
- 81. Inventing the Cluster Problem: Two ASGs with one function but different names Solution: Append version number in reserved format Parse ASG name to determine long-term “cluster”
- 82. Thank you @cquinn (@joesondow) http://www.slideshare.net/carleq http://techblog.netflix.com
- 83. Thank you Questions? @cquinn (@joesondow) http://www.slideshare.net/carleq http://techblog.netflix.com
Editor's Notes
- #2 Abstract: Netflix has moved its production services from being completely datacenter-based, to being primarily cloud-based in a little over a year. This talk will briefly explain why we did it and why we chose Amazon's AWS. The talk will then delve into the evolution of our build and deployment architecture, starting with how our original monolithic DC-based SQL-tied webapp was built and deployed, and then detailing the evolution of our continuous integration systems which helped prepare us for the cloud move. And finally, it will cover our current full build pipeline and deployment system which keeps us agile in the cloud now by providing quick turnaround, and rapid roll-out and recovery.\n\n
- #3 I manage the Engineering Tools team at Netflix. We are all about creating tools and systems for our software engineers to use to build, test and deploy their apps to the cloud. (and DC too :)\n\n
- #4 Q: Why did we choose to move to the cloud?\nA: We found that as our rate of growth was growing, we were unable to predict how fast we would need new capacity, and our lead times were too long to keep up if things really took off. We also liked the idea of providing a cloud-like infrastructure to enable our individual engineering teams to be as agile and productive as possible. Since we also knew that we didn&#x2019;t want to become a cloud provider, we decided to leverage a 3rd party cloud solution.\n\nQ: Why did we choose Amazon AWS?\nA: We didn&#x2019;t want to be the big fish in a little pond&#x2014;we wanted to be a medium fish in a big pond. AWS is the largest public cloud around and is the only place we could be smaller than a big fish. Also, we didn&#x2019;t want to re-invent a lot of infrastructure, so we are making use of a number of AWS services like auto-scaling, simpleDB, elastic load balancing, etc. AWS is at least two years ahead of the closest competition in this area.\n\n
- #5 A little background about where we were coming from. We were originally one big monolithic Java webapp on a couple hundred machines in a local DC with a single big Oracle database behind it. All these machines were individual dedicated hardware boxes. (IBM Power6 with RedHat + JDK 5) Our IT department managed the DC, hardware, networking, security, machine setup and provisioning.\n\nIn the last year before the cloud move, we began refactoring the big webapp into middle-tier services, and IT introduced VMWare to manage new machine, but machine configuration was still very manual.\n\n
- #6 A rough picture of our old world:\n\nWe had a bi-weekly push cycle, with a fixed calendar for the year. Alternating weeks were database pushes, then code pushes.\n\nOur &#x201C;WebCM&#x201D; tool was a big mess of Perl that did everything for everyone. Builds, database pushes, netscaler control, code pushes, etc. Very brittle.\n\nNew app additions required code change to this tool, and often ran into the brittle areas.\n\n
- #7 What did we need to do to get to the cloud?\n\nEach individual product team was responsible for refactoring their functionality into a webservice, often times including shared libraries (jars).\n\nMy team was responsible for creating a set of easy to use tools to simplify and automate the build of the applications and shared libraries.\n\nWe also were responsible for building the base machine image, and creating the architecture for automating the assembly (aka baking) of the individual application images.\n\nAnd, we also were responsible for building the web-based tool which is used to deploy and manage the application clusters.\n
- #8 Let&#x2019;s step back and take a look at the evolution of our build system, from the old ad-hoc Ant scripts to where we are today.\n
- #9 Step one: Collect underpants. Pretty much.\n\nThe first step that we made, even pre-cloud, was to carve out common code into libraries that could be shared across apps. This enabled the middle-tier teams migrate out of the big webapp, and still share code with them.\n\nAt the same time, we collected all of the common Ant scripts into a single central package that could be used to build different kinds of libraries and apps.\n\nAlas, most of these libraries were built on developer workstations, and our mechanism for library sharing was to check the jars back into SCM (Perforce).\n\n
- #10 Step two: ?\n\nThe next step was to get this all on a continuous integration server. We picked Hudson because it was very feature rich, extensible, and had an active community. Now we at least had visibility into what was going on, and less worry about dirty builds published from developer machines.\n\nWe also began to add a lot of build metadata to the manifests so that we could retroactively figure out where things came from even before our whole pipeline had traceability.\n\nAnd, we were also able to publish application artifacts on the Jenkins Job/build pages via archive artifacts. This helped us transition the DC deployment tool to use Jenkins for all of its builds.\n\n
- #11 Step three: Profit!\n\nThis is where we stopped checking library jars into SCM and started publishing them to Artifactory. This also gave us access to the build metadata and allowed us to add Ivy to Ant to abstract the build and runtime jars into a dynamic dependency graph. So each project only had to know about its immediate dependencies.\n\nWe were then also able to start treating application artifacts like library artifacts and publish them to Artifactory as well.\n\n
- #12 Step three: Profit!\n\nThis is where we stopped checking library jars into SCM and started publishing them to Artifactory. This also gave us access to the build metadata and allowed us to add Ivy to Ant to abstract the build and runtime jars into a dynamic dependency graph. So each project only had to know about its immediate dependencies.\n\nWe were then also able to start treating application artifacts like library artifacts and publish them to Artifactory as well.\n\n
- #13 Step three: Profit!\n\nThis is where we stopped checking library jars into SCM and started publishing them to Artifactory. This also gave us access to the build metadata and allowed us to add Ivy to Ant to abstract the build and runtime jars into a dynamic dependency graph. So each project only had to know about its immediate dependencies.\n\nWe were then also able to start treating application artifacts like library artifacts and publish them to Artifactory as well.\n\n
- #14 Here is most of a typical project&#x2019;s Ant and Ivy files. You can see the Ant code simply pulls in one of the standard framework entry points like, library, webapplication, etc. \n\nThen the Ivy file specifies what needs to get built and what are the dependencies. We have some extra Groovy code added to our Ant scripts that can drive Ant targets based on the Ivy artifact definitions. This helps make the build definition declarative and yet flexible.\n\nYes, XML makes your eyes bleed, and there is a lot of redundancy here. But at least it&#x2019;s small and manageable.\n\n
- #15 Since we were already using Ant and Ivy, and had already incorporated groovy, it is/was a fairly small step to migrate our common build framework to Gradle.\n\nNow we have something lean and mean and clean. Nebula is our &#x201C;Netflix Build Language&#x201D; and is really just a Gradle plugin that provides a bunch of default configuration, customizes standard Gradle Java and other tasks, and defines some additional tasks.\n\n
- #16 OK, back to the build pipeline again. We have a vague &#x201C;app bundles&#x201D; output in this diagram.\nLet&#x2019;s delve into how we manage the application bundle artifacts in more detail.\n\n\n
- #17 Here is a simplified picture of what the flow looks like at the tail end of the build for getting the application artifacts ready for baking.\n\nThe first generation approach is to treat them like system packages and just package them up into RPMs, and post those to our Yum repo. (We actually scp them into an incoming dir which is then scanned by a daemon which adds the new RPMs to the repo and builds its index). This is problematic due to rpmbuild being RedHat/CentOS/Fedora specific and very brittle even across those distros and versions. It also means that we have to use build slaves with version matching the final production OS version, or we have to be really careful writing spec files.\n\nOur newer approach is to treat app bundles like the other build artifacts. Wrap all the the app files up into a simple archive, like .zip or .tgz, include a BOM file and just publish it to an app repo in Artifactory. We&#x2019;ll then let the bakery unpack the bundles when it assembles the image.\n\n
- #18 Here is a simplified picture of what the flow looks like at the tail end of the build for getting the application artifacts ready for baking.\n\nThe first generation approach is to treat them like system packages and just package them up into RPMs, and post those to our Yum repo. (We actually scp them into an incoming dir which is then scanned by a daemon which adds the new RPMs to the repo and builds its index). This is problematic due to rpmbuild being RedHat/CentOS/Fedora specific and very brittle even across those distros and versions. It also means that we have to use build slaves with version matching the final production OS version, or we have to be really careful writing spec files.\n\nOur newer approach is to treat app bundles like the other build artifacts. Wrap all the the app files up into a simple archive, like .zip or .tgz, include a BOM file and just publish it to an app repo in Artifactory. We&#x2019;ll then let the bakery unpack the bundles when it assembles the image.\n\n
- #19 Here is a simplified picture of what the flow looks like at the tail end of the build for getting the application artifacts ready for baking.\n\nThe first generation approach is to treat them like system packages and just package them up into RPMs, and post those to our Yum repo. (We actually scp them into an incoming dir which is then scanned by a daemon which adds the new RPMs to the repo and builds its index). This is problematic due to rpmbuild being RedHat/CentOS/Fedora specific and very brittle even across those distros and versions. It also means that we have to use build slaves with version matching the final production OS version, or we have to be really careful writing spec files.\n\nOur newer approach is to treat app bundles like the other build artifacts. Wrap all the the app files up into a simple archive, like .zip or .tgz, include a BOM file and just publish it to an app repo in Artifactory. We&#x2019;ll then let the bakery unpack the bundles when it assembles the image.\n\n
- #20 No, not the Qwikster kind of baked.\n\nBaking is where we pre-assemble complete and ready-to-launch machine images for each app.\n\n
- #21 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #22 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #23 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #24 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #25 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #26 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #27 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #28 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #29 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #30 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #31 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #32 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #33 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #34 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #35 So, why do we bake custom images instead of just using Puppet or Chef to deploy packages dynamically to launched generic machines?\n\nWe like to front-load the full machine assembly to build time, instead of waiting until deployment time. We do this because:\n&#x2022; More Reliable: less systems that can fail at deploy time right when we need them most.\n&#x2022; Faster Launch: means quicker reaction to load increases, e.g. autoscaling up can be more precise.\n&#x2022; Single image: produces exactly homogeneous clusters. No file/package version skew across machines in a cluster\n\n
- #36 The first step of the baking process is to create the &#x201C;base&#x201D; image that we will use for baking all app images. This is done once every week or two.\n\nWe start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages:\n&#x2022; Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc.\n\nThe end result is a beefed-up OS image that is ready to go, and just needs an app added.\n\n
- #37 The first step of the baking process is to create the &#x201C;base&#x201D; image that we will use for baking all app images. This is done once every week or two.\n\nWe start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages:\n&#x2022; Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc.\n\nThe end result is a beefed-up OS image that is ready to go, and just needs an app added.\n\n
- #38 The first step of the baking process is to create the &#x201C;base&#x201D; image that we will use for baking all app images. This is done once every week or two.\n\nWe start with a standard Linux distro as a foundation (CentOS now, Ubuntu on the way), and add in our favorite, our custom and customized packages:\n&#x2022; Apache, Java (JDK 6 and 7), Tomcat, Perl, Python, provisioning and startup scripts, log management tools, monitoring agents, etc.\n\nThe end result is a beefed-up OS image that is ready to go, and just needs an app added.\n\n
- #39 App images are baked in a high-speed bakery cluster, and each AWS region (Virginia, California, Oregon, Ireland) has a cluster with a handful of bakery instances.\n\nBase images are pre-mounted as EBS volumes, with a pool of multiple mounts per bakery instance.\nFor each app build the bakery does these things:\n&#x2022; One base volume is grabbed from the pool\n&#x2022; An app bundle is pushed or pulled to the bakery and then provisioned onto the image volume.\n&#x2022; Currently pushing RPMs to YUM repo & pulling them from bakery with yum install.\n&#x2022; The new way is to publish tarballs to Artifactory and pull them from bakery with wget and \n&#x2022; The EBS volume is then snapshotted, and the EBS snapshot is registered as an AMI.\n&#x2022; The volume is detached and a new clean base reattached in its place\n\nLater, unused and obsolete AMIs get cleaned up by the Janitor monkey \n\n
- #40 App images are baked in a high-speed bakery cluster, and each AWS region (Virginia, California, Oregon, Ireland) has a cluster with a handful of bakery instances.\n\nBase images are pre-mounted as EBS volumes, with a pool of multiple mounts per bakery instance.\nFor each app build the bakery does these things:\n&#x2022; One base volume is grabbed from the pool\n&#x2022; An app bundle is pushed or pulled to the bakery and then provisioned onto the image volume.\n&#x2022; Currently pushing RPMs to YUM repo & pulling them from bakery with yum install.\n&#x2022; The new way is to publish tarballs to Artifactory and pull them from bakery with wget and \n&#x2022; The EBS volume is then snapshotted, and the EBS snapshot is registered as an AMI.\n&#x2022; The volume is detached and a new clean base reattached in its place\n\nLater, unused and obsolete AMIs get cleaned up by the Janitor monkey \n\n
- #41 App images are baked in a high-speed bakery cluster, and each AWS region (Virginia, California, Oregon, Ireland) has a cluster with a handful of bakery instances.\n\nBase images are pre-mounted as EBS volumes, with a pool of multiple mounts per bakery instance.\nFor each app build the bakery does these things:\n&#x2022; One base volume is grabbed from the pool\n&#x2022; An app bundle is pushed or pulled to the bakery and then provisioned onto the image volume.\n&#x2022; Currently pushing RPMs to YUM repo & pulling them from bakery with yum install.\n&#x2022; The new way is to publish tarballs to Artifactory and pull them from bakery with wget and \n&#x2022; The EBS volume is then snapshotted, and the EBS snapshot is registered as an AMI.\n&#x2022; The volume is detached and a new clean base reattached in its place\n\nLater, unused and obsolete AMIs get cleaned up by the Janitor monkey \n\n
- #42 \n
- #43 \n
- #44 Now that we have all these pre-baked application machine images, what do we do with them?\n\nWe use the standard AWS architecture, plus add just a couple of light abstractions on top to help us manage things.\n\n
- #45 Now that we have all these pre-baked application machine images, what do we do with them?\n\nWe use the standard AWS architecture, plus add just a couple of light abstractions on top to help us manage things.\n\n
- #46 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #47 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #48 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #49 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #50 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #51 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #52 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #53 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #54 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #55 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #56 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #57 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #58 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #59 Here is an overview of the AWS architectural model, and the way that we use it to create applications and clusters. Each of the items here exists as an entity in the AWS model and can be created, named, listed and modified using the Amazon AWS APIs.\n\nThe heart of our cluster is the Auto Scaling Group (ASG), which is where we define the image to launch, how many instances we want, and what security and network routing they should have. The Elastic Load Balancer (ELB) is used for front-end traffic only, and thus not needed for middle-tier services.\n\n
- #60 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #61 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #62 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #63 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #64 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #65 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #66 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #67 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #68 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #69 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #70 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #71 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #72 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #73 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #74 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #75 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #76 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #77 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #78 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #79 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #80 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #81 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #82 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #83 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #84 Every application that we deploy always has this same model, so we maintain an many-to-one association between these entities and each application.\n
- #85 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #86 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #87 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #88 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #89 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #90 But there is no built-in support for this aggregating entity in AWS, so we just define our own domain in SimpleDB to keep track of them.\n
- #91 But we need more than one ASG for each app, and this is why.\n
- #92 But we need more than one ASG for each app, and this is why.\n
- #93 But we need more than one ASG for each app, and this is why.\n
- #94 But we need more than one ASG for each app, and this is why.\n
- #95 But we need more than one ASG for each app, and this is why.\n
- #96 But we need more than one ASG for each app, and this is why.\n
- #97 But we need more than one ASG for each app, and this is why.\n
- #98 But we need more than one ASG for each app, and this is why.\n
- #99 But we need more than one ASG for each app, and this is why.\n
- #100 But we need more than one ASG for each app, and this is why.\n
- #101 But we need more than one ASG for each app, and this is why.\n
- #102 But we need more than one ASG for each app, and this is why.\n
- #103 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #104 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #105 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #106 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #107 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #108 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #109 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #110 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #111 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #112 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #113 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #114 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #115 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #116 First scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, everything looks good so we switch all traffic over to it. After letting it run through one peak load (overnight usually) we know that it is good and we can terminate the version 007 ASG.\n\n
- #117 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #118 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #119 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #120 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #121 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #122 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #123 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #124 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #125 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #126 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #127 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #128 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #129 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #130 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #131 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #132 Different scenario: Version 007 is running smoothly in production. We push version 008 out, monitor it, things look fair so we switch all traffic over to it. Things start to unravel, and now we can see that 008 wasn&#x2019;t so great after all. So we just switch traffic back to 007, and then terminate 008. (or maybe leave some 008 machines around for forensic purposes.)\n\n\n
- #133 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #134 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #135 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #136 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #137 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #138 The cluster is another case where we have an abstract entity that doesn&#x2019;t exist in AWS. In this case, we just use naming patterns to associate related ASGs into logical clusters.\n
- #139 \n
- #140 \n
- #141 \n
- #142 \n
- #143 \n
- #144 \n
- #145 \n
- #146 \n
- #147 \n
- #148 \n
- #149 \n
- #150 \n
- #151 \n
- #152 \n
- #153 \n
- #154 \n
- #155 \n
- #156 \n
- #157 \n
- #158 \n
- #159 \n
- #160 \n
- #161 \n
- #162 \n
- #163 \n
- #164 \n
- #165 \n
- #166 \n
- #167 \n
- #168 \n
- #169 \n
- #170 \n
- #171 \n
- #172 \n
- #173 \n
- #174 \n
- #175 \n