Continuous Deployment & Delivery + Culture Hacks @ QCON 2012

Jesse RobbinsCofounder, Opscode@jesserobbinsjesse@opscode.comContinuous Delivery+ Hacking CultureQCON 2012

Operations at Web Scale is the ability to consistently create and deploy reliable software to an unreliable platform that scales horizontally.http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html

Google, Amazon, Microsoft, Yahoobuilt their own automation &deployment tools

but it was “secret sauce”

Cloud Infrastructure Automation

Continuous Delivery

This man isJohn Allspaw http://www.ﬂickr.com/photos/norby/7446208116/

This is Allspaw’s Monster RAAAWR!!! I’m SCARY!Image Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

I can haz cuddle?Images Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Continuous Deployment means: Faster Time to Value Higher Availability Happier Teams More Cool Stuff

Continuously Deploying code before you even write it.

meanwhile... back at the office...

Tools* are not enough(* even really great tools like Chef!)

Conway’s law:“Organizations which design systems ...are constrained to produce designswhich are copies of the communicationstructures of these organizations...”

Brook’s said:“Quality is strongly affected byorganization structure.”

Choose: Discourage change in the interests of stability or Allow change to happen as often as it needs toSlide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-ﬂickr

Common Attributes of Web Scale Cultures Infrastructure Application Dev / Ops as Code as Services as Teams‣ Full Stack Automation ‣ Service Orientation ‣ Shared Metrics /‣ Commodity Hardware ‣ Versioned APIs Monitoring‣ Reliability in software ‣ Software Resiliency ‣ Incident Management stack (Design for Failure) ‣ Service Owners On-call‣ Datacenter APIs ‣ Database/Storage ‣ Tight integration‣ Core Infra Services Abstraction ‣ Continuous Integration ‣ Infrastructure as ‣ Complexity pushed up ‣ Continuous Product the stack Deployment ‣ App as Customer ‣ Deep Instrumentation ‣ SRE/SRO ‣ GameDay

This isn’t new‣ Theory of Constraints‣ Lean / JIT‣ Six Sigma‣ Toyota Production System‣ Agile‣ etc... 26

...but we can’t do it that way because...elephants cannot fly just byflapping their ears harder... http://www.flickr.com/photos/garymacfadyen/6860004327/

To fly you must have wings, surface area, and a high power to weight ratio...elephants cannot fly by flappingtheir ears harder... http://www.ﬂickr.com/photos/lhirlimann/4872199920/

http://thisisindexed.com/2008/12/might-as-well-own-this-meltdown/

Common Attributes of Web Scale CulturesInfrastructure Application Dev / Ops as Code as Services as Teams‣ Full Stack Automation ‣ Service Orientation ‣ Shared Metrics /‣ Commodity Hardware ‣ Versioned APIs Monitoring‣ Reliability in software ‣ Software Resiliency ‣ Incident Management stack (Design for Failure) ‣ Service Owners On-call‣ Datacenter APIs ‣ Database/Storage ‣ Tight integration‣ Core Infra Services Abstraction ‣ Continuous Integration ‣ Infrastructure as ‣ Complexity pushed up & Deployment Product the stack ‣ SRE/SRO ‣ App as Customer ‣ Deep Instrumentation ‣ GameDay

Infrastructure as Code: Enable the reconstruction of the businessfrom nothing but a source code repository, an application data backup, and bare resources.

Unprecedented Growth …must change to get us here! Virtual Nodes Physical Hardware The1980 things that got us here… 1990 2000 2010+ Mainframe Client/Server Datacenter Cloud1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Managing Complexity Then To Add a New Server… • 2x Web Server Configurations • 2 Web Server RestartsAdd 1 server Frontend Web • 4x Database Configurations20+ Changes Servers • 8x Firewall Configurations • DNS Service Application Servers • Network Configuration • Deployer Database • 8x Monitoring Changes The Bottom Line… 20+ Changes 12+ New Infrastructure Dependencies 4+ Hours

Managing Complexity Later We added: • Load Balancers • MemCache • Search Appliances • Lots of VM’s • More Scale Exponential Increase In • Configuration Changes • Infrastructure Dependencies • Skills Needed • Greater Risk

Managing Complexity Today How Do we Manage This at Cloud Scale? • Thousands of infrastructure dependencies and configurations needed for each change. • Huge Amounts of Time • Increased Cost of Correction of Manual Errors • Huge Need for Talent • Risk of Critical Skills Shortage

How you get to Continuous Deployment Full Infrastructure Continuous Automation Application Configuration Management Deployment CommonDiscovery and Management Automation Tasks: Visibility Scripts, OS Compliance, Updates & Patches

Continuous Deployment: Version ControlKeep every relevant artifact in versioncontrol. ‣ Infrastructure ‣ Operations ‣ Applications ‣ Tests ‣ Documentation

Continuous Deployment: Code ReviewReview your code before deployment. ‣ Gate what gets pushed ‣ Code reviews ‣ Partial pre-testing with continuous integration

Continuous Deployment: ProvisioningAutomate the deployment infrastructure. ‣ Vagrant ‣ Bare-metal provisioning ‣ Virtualization ‣ Private cloud ‣ Public cloud ‣ knife

Continuous Deployment: Continuous IntegrationEvery commit gets tested. ‣ Build code ‣ Build infrastructure ‣ Deploy code and test ‣ TestKitchen ‣ Gauntlet ‣ Release to next stage ‣ Rinse, Repeat

Step 1 – SCM and DevelopersApplication Software Devs Conﬁguration Management (SCM)Infrastructure Devs

Step 2 – Introducing the Build Stage Build Changes in SCM llin g triggers builds Pu and tests Tag Payload NApplication Software Devs Conﬁguration Management (SCM) Payload 3 Payload 2 Payload 1Infrastructure Devs

Step 3 – Introducing Chef Server and the CD Process Latest Codebase and Build Create Data (#) Upload Cookbook Build Update DEV DEV Autodeploy to Chef ng ServerApplication Devs Infrastructure Devs Pul l i localhost Promote Tag Payload Request Portal QA N 1, 2, … N Software Autodeploy Promote Conﬁguration N Management PROD (SCM) Payload 3 ….. ….. Payload 2 2 …. Payload Builds 1 1

Common Attributes of Web Scale Cultures Infrastructure Application Dev / Ops as Code as Services as Teams‣ Full Stack Automation ‣ Service Orientation ‣ Shared Metrics /‣ Commodity Hardware ‣ Versioned APIs Monitoring‣ Reliability in software ‣ Software Resiliency ‣ Incident Management stack (Design for Failure) ‣ Service Owners On-call‣ Datacenter APIs ‣ Database/Storage ‣ Tight integration‣ Core Infra Services Abstraction ‣ Continuous Integration ‣ Infrastructure as ‣ Complexity pushed up & Deployment Product the stack ‣ SRE/SRO ‣ App as Customer ‣ Deep Instrumentation ‣ GameDay

Jesse RobbinsCofounder, Opscode@jesserobbinsjesse@opscode.com

http://www.scoop.it	93
https://twitter.com	18
http://pinterest.com	2

Continuous Deployment & Delivery + Culture Hacks @ QCON 2012

by Jesse Robbins on Dec 27, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 113

Statistics

Continuous Deployment & Delivery + Culture Hacks @ QCON 2012 Presentation Transcript