Operations as a Strategic Weapon (Part 2)

Operations is a Strategic Weapon (Part Deux)Saturday, September 24, 2011

Are you an ... aaSSaturday, September 24, 2011-If its not it should be-Lets take a look at GW.com.. yep-they sell GW as a service GreenWidgits_aaS-Obviously you better have kick “ass” green widgets; Pun intended....-But today that is not enough....-----------------------------------------What does amazon and netﬂix get that barnes and noble and blockbuster dont ?_aaS (circle the "S")-Over the next 45 minutes I am going to try and convince you that the principles behind Devops are the best solution toaddress the "S" in your aaS

Saturday, September 24, 2011-Most of you are probably familiar with “Software” TD-I am going to tell you a story about “Infrastructure TD”

Saturday, September 24, 2011-How many ppl know who these guys are?-The dudes who invented Facebook.. rght?-However, this was not there ﬁrst venture.-Green vs Red Widgits compnay story.-After many arguments the both decide to split up. Cameron to make green widgets and Tylerdecides to make red widgets.-They both go to their father each ask for a 1m dollars and the father asks how much moneyare you going to make. They both say 10 million. -Father calculates the ROR to be 900% and gives em each 1m.10-1 = 9/1 = 900%-However one of them lied.... The red widgets only return 233%10-1-2 = 7/3 = 233%

Green vs Red Widgits 1 Million 1 Million 2 Million No TD 10 Million Proﬁt 10 Million Proﬁt 10 - 3 = 7/3 10 - 1 = 9/1 233% ROR 900% RORSaturday, September 24, 2011-Father is not pleased with Rehaus.

It Gets Worse... Technical Vicious Toxic Terminal Debt Cycle OperationsSaturday, September 24, 2011- It gets even worse. Rehause starts getting what Isreal Ghat of the Cutter Group call the vicious cycle of TD.- You wind up fixing a lot of things that you didn’t fix in the first place..- this pulls more resources from delivery good service and compound effect is that you are spending more and more resources that you should have gotten right the firs time..- but ever worse the effect of customer satisfaction starts loosing more business and the “V” cycle is out of control.- TD->VTD->Toxic operations->Terminal- I call this are you running a business or building a business?- Toxic operations. Amertrade/etrade story

Tale of Two StartupsSaturday, September 24, 2011- Jesse Robbins my ex boss and CEO of Opscode/Chef did a great post on O’rielly rdar a few years ago called the Tale of Two Startups.- The chart looked like this ﬁrst 4 weeks.- First chart legacy (I call it the non devops startup/project)- Second Chart is the (secret sauce startup ... #devops)- I played around with this using “R” to be cool and I came up with a=140% ROR and 700% ROR

The Meat to Math Ratio http://radar.oreilly.com/2011/08/meat-to-math-ratio.htmlSaturday, September 24, 2011- Alistar Croll has a great post called the meat to math ratio.- Amazon had $12.95B in Q410 revenues and 33,700 employees, revenue per employee of $384,273.- For Barnes & Noble: Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee of $54,571.- Netﬂix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000.- Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees i- Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K.

Saturday, September 24, 2011•I got into a Twttier argument that went like this...• See “Cloud Gone Wrong”

Clouds Gone Wild Busine s s Dev AWS business s3put s3 ssh ideas ! apache/php Rightscale memcached mysqlSaturday, September 24, 2011-So letʼs start the story in the begining... Biz guy got a great idea for a service.-They used a classic web2.0 app architecture: apache/php, memcached & mysql.Development done on a single server-Production ran on EC2 and they used rightscale server templates-Release done by pushing code and assets to s3 buckets and then running a parallel SSHscripts to distribute them-This approach seemed to work... They got up to a few hundred nodes pretty fast.. businesswas cooking

apache/php p arty, 3rd s3 memcached & are AWS mysql w mi ddle Rightscale yum Business apache/php puppet memcached mysql apache/php s3 AWS s3 memcached AWS mysql Rightscale Rightscale s3put s3put ssh ssh Business2 Business3Saturday, September 24, 2011-First service was such a huge success they decided to launch other sevices-So they ”Copy and pasted” the whole architecture and lifecycle to launch the new businesses-Each new group pushed assets to s3, scripted the distribution, and hacked the rightscripts andtemplates-Things were obviously getting more complicated, so they did what they were supposed to doand added centralized tooling like puppet and yum-They thought they were doing things the cloud way and that all would be ﬁne

“As-is” operations server templates TAR RPM restarts TAR TAR code, EC2 content pupp deploys et RS app RS deployment devs deployment S3 yum repo Provision-time builds reconfigs TAR EXE TAR AUTO server TOOLS code templates TAR RPM middleware devs platform CONTROL PROVISION RELEASE system engSaturday, September 24, 2011First we got the team on the whiteboard to map out the “as is” picture. This is a, believe it ornot, a simplified version of that.Some of the highlights...-First youʼll notice that different groups had their own path to production... different methods ofcontrol, provisioning, and release.-Each group and role seemed to have a different way to editing or storing config-There were differing ways of packaging software... sometimes it might be a .tar.gz other timesit might be an RPM.-Shockingly... Some things were even being built directly on production servers.-There was no authoritative source of information is maintained about nodes, applicationtopology, software versions, etc....-Everything was being stored in S3 buckets... which is great because itʼs so easy to use... butitʼs unversioned and people would just upload whatever was newest for them. But that stuffwas never really tested in unison.. so old stuff wasnʼt working with new stuff... and its wasunclear what was different or why it was different.-We can go on... but you should get the point that they had the right cloud, the right tools, andlots of smart people, but it all got dangerously out of control very quicklyOther:-Changes hit all customers at once => Puppet configs in unversioned S3 buckets-Buggy node classification causing provisioning problems = > complex/long node classifierscript-”Dead boxes” after provisioning => rightscript/puppet ordering problems-”my box got clobbered!” => puppet, is it supposed to be on or off?-new environment setup was taking longer and longer => from days to weeks because of

Tec hnic alSaturday, September 24, 2011

Infrastructure Deveopment Life Cycle Everything starts here SVN hud agent cfg yum pup son repo pet packages active users run packages directory deck commands nag splunk ios resource model sys cfg right instance create scale events new node Node a log data CONTROL PROVISION RELEASESaturday, September 24, 2011I wonʼt go into too much detail about the tooling that was put into place to support all of this buthere are some highlights..-took a loosely coupled toolchain approach... using mostly open source tools-This became their standard stack of “operations middleware”. Of course, we are all used tothe notion of application middleware... but to an online service, the management infrastructureis just as much a part of the service as the application itself.-This operations middleware stack is a ﬁrst class citizen along with application stack and it allgoes through the same SDLC... everything is versioned, built, deployed, and packaged via thesame process-Once in place, this middleware provides a single path for releasing, provisioning, andcontrolling anything that goes into an environment.Other:Management infrastructure based on “swap-able” sets of integrated tools- Organized into three rough categories: Control, Provisioning, Monitoring - Control tools support routine and ad hoc procedures executed as commands/scripts - Provisioning tools support package delivery and post install customization - Monitoring tools actively check health and collect log dataWhen you think of middleware, you think of where your app code works.. but in the serviceworld, you have an operations middleware that is just as important. All the provisioning andmanagement stuff... itʼs just as important. Itʼs one and the same.Solve the information problem... where is the system of record? easy in cloud to get basicnode data... that comes from the compute service... but what about everything else you needto manage your infrastructure?

Meet your competitionSaturday, September 24, 2011

•Betting on better experience •Attacking / chased by the majors •Real money at stakeSaturday, September 24, 2011One last quick tale for you all....So what does it look like when you start with the end in mind...Let’s look a hot startup called Wealthfront-Its a ﬁnancial services company... but all you have to know that it manages real people’sretirement funds. It’s not a game or a toy.-They are being chased by the majors so they have to move fast and can’t screw up-To make this work, they subscribe to the theories of these two gentlemen, steven blank anderic reiss, and are using the Lean Startup methodology to run their business... and all youreally need to know about that right now is that it all hinges on getting new ideas in front ofyour customers as fast as possible and getting feedback as fast as possible.Customer discovery -> Customer Validation -> Customer Creation -> Company Building

Saturday, September 24, 2011So what was immediately identiﬁed as a major impediment to that fast cycle?The traditional development and deployment cycle. Even the 1 - 4 weeks cycles that are socommon is too long to wait.

Saturday, September 24, 2011So they rewired it in classic Continuous Deployment fashion...Get rid of the idea of batching up change for a release... They are always deploying as soonas a a piece of code is readyDon’t have staging... that’s a waste of time... production is the real environment anywayDon’t have an explicit QA stage... built extensive automated testing in all across the lifecycleand made quality be everyone’s responsibilityThey are always deploying from trunk... so no need to worry about patching and mergingbranchesoh and do they whole thing from check-in to running in production in 5 - 10 minutesare they crazy? They don’t think so. They have extreme conﬁdence in this system and arekicking ass with it.

go read: http://eng.wealthfront.comSaturday, September 24, 2011The results are striking...They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificingquality!A release can get from check-in in to production in under 10 minutes but it’s got to pass over6k tests and then there is a phased rollout with ongoing and extensive system and trafficanalysis post deployment.The result is that their customers are happy. Their partners are happy. And more importantlytheir team is happy...no burnout and releases are non-events. Want to go on vacation? Noproblem, any developer can step in and take over your task or fix problem because ofextensive test coverage and confidence in their quality systems.Operations is their strategic weapon

go read: http://codeascraft.etsy.comSaturday, September 24, 2011The results are striking...They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificingquality!A release can get from check-in in to production in under 10 minutes but it’s got to pass over6k tests and then there is a phased rollout with ongoing and extensive system and trafficanalysis post deployment.The result is that their customers are happy. Their partners are happy. And more importantlytheir team is happy...no burnout and releases are non-events. Want to go on vacation? Noproblem, any developer can step in and take over your task or fix problem because ofextensive test coverage and confidence in their quality systems.Operations is their strategic weapon

95% 5% Necessary Liability Strategic WeaponSaturday, September 24, 2011

DevOps Cafe subscribe in iTunesSaturday, September 24, 2011I also do a podcast with the famous cloud and IT management guru john willis.Interview based series where we talk to all kinds of movers and shakers across thedevelopment and operation spectrum.Between DTO and doing the devops cafe content I get to talk to a lot of companies and seewhat’s working and what isn’t working.

Let’s Talk.... @botchagalupe john@dtosolutions.comSaturday, September 24, 2011We do this stuff all day long for a lot of large and cutting edge clients... and we love talkingabout DevOps so drop me a line anytime if you want to talk

Let’s Talk.... @botchagalupe dev2ops.org john@dtosolutions.comSaturday, September 24, 2011We do this stuff all day long for a lot of large and cutting edge clients... and we love talkingabout DevOps so drop me a line anytime if you want to talk

Extra SlidesSaturday, September 24, 2011

C A M SSaturday, September 24, 2011First get you mind around what you are looking for... Simple framework for categorizingDevOps problems and solutions.

Culture Automation Measurement SharingSaturday, September 24, 2011

Operations as a Strategic Weapon (Part 2)

by botchagalupe

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Operations as a Strategic Weapon (Part 2) Document Transcript