Dev Ops @ Envato

Dev Ops @ Envato ... or how we found out there’s a name and a supportgroup for all the stuff we already did to let a team of 8deploy heaps of times a day to a Ruby on Rails app that has scaled up to around 20 million requests a week without an ops team.

John Barton @johnbartonjohn@envato.com

Envato?http://envato.com

Stock Marketplaces & Tutorial Network

Marketplaces

Like iStockPhoto but for other creative niches or eBay for digital goods

Tuts+ Network

Big Blog Network for education in creativeﬁelds.... and linkbait (ducks)

Big trafﬁc and interesting init’s own right, but that’s for @ryanfaceryan to talk about

2 things recently made us think we might be doingsomething a bit different

Neal Ford @ YOW! “Rails can Scale”

200k+ daily requests vs 2m+ daily requests

Dev Ops Melbourne“Continuous Deployment”

1 Deployment per week or fortnight vs 5 Deployments per Day

The Marketplace

August 2006

One Marketplace

FlashDen

Rails 0.13bNo UsersNo Trafﬁc

March 2011

9 Marketplaces

ActiveDen (nee FlashDen) AudioJungle ThemeForest VideoHive GraphicRiver CodeCanyon 3D Ocean Tuts+ Marketplace PhotoDune

Rails 2.3.11 687,830 Users20 Million Requests Weekly

One Codebase and OneProduction Environment

... and well gosh darn it we just deploy to that site whenever we darned well feel like it

As much a product of timeand circumstances as good decisions and hard work

The “Golden Age”

3 Developers30 minute feedback cycledeploy, discuss on forums, deploy again

Decision 1:Preserve everything goodabout the the startup days for as long as we can

Commodity Hosting

Notice I’m not saying“Cloud Computing”

You canttrust “thecloud” but you cantrust “the cloud to be “the cloud”.

Decision 2: Conservative Platformchoice so we don’t have to sweat the details

Corporate AntiPatterns

We’ve all been doing dev long enough to see thisstuff screwed up over and over again

Decision 3:Don’t do all that stuff

So what do we actually do?

“Culture of respect &trust, good attitude toward failure...”Ted Dzubia

“How about "culture of stop fucking up"?” Ted Dzubiahttp://teddziuba.com/2011/03/devops-scam.html

Ultimate Responsibility

"The fault, dearBrutus, is not in our QA or Ops, But in ourselves."

Test Driven Development vs. QA Team

Test Driven Infrastructure vs. Ops Team

Both as a team and asindividuals we own ourwork from when we are asked to do it...

... until is is demonstrablyerror free and performant in production

Everyone is in the (paid) on call roster

Everyone takes a turn atLevel 2 Customer Support

Want those jobs to be easier? Stop fucking up.

Process

LEAN / TPS Principles...without the process

You cannot write code anyfaster than you can deploy it to production

Long running projects?

A. B. C.Always Be Cmerging (the c is silent)

Dark LaunchFeature FlagsPrivate Beta

Community

We do trip up running this fast

But through years of openness with our usersvia our forums and owning up to our mistakes

... we’ve ended up with a (relatively) sympathetic community

Time Zones both help and hurt

Trafﬁc peaks during US day means that if things gowrong we’re usually asleep

But it makes it very easy todeploy during our business hours

Dev vs. Ops

Our Solution:Don’t have Ops

Outsource commodity platform bits: virtualisation/cloud, haverackspace take care of db/ mailserver

Ensure the dev team hasthe skills to take care of the rest

Keep the stack as Vanilla as possible

Virtualised servers in our own sandbox.Cloud Flexibility - Cloud Shit-ness = WIN

Automate Conﬁguration Management

Don’t modify live boxes,tear down and build again from scratch

Take advantage of an individuals talents, but don’t rely upon themie. don’t accidentally create an ops guy

Performance & Scaling

Not as big a deal as everyone thinks

Shared-nothing loadbalanced app servers + outof request queue workers not rocket surgery

Measure, deploy, measureagain and then tweak or rollback New Relic FTW

There is no code faster than no code.

Caveats

We rely on our growthfollowing a steady exponent for this approach to be viable

We made a conscious tradeoff between deliverytime and production snafus

We’ve traded off platform innovation for product innovation (no MongoDB, etc)

Auditors aren’t big fans ofone team with access to everything, but you can mitigate these problems

Question?

http://notes.envato.com	424
http://kb.dtosolutions.com	8
http://www.linkedin.com	2
http://feeds.feedburner.com	1
http://static.slidesharecdn.com	1
http://feeds2.feedburner.com	1

Dev Ops @ Envato

by John Barton on Mar 22, 2011

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 437

Statistics

Dev Ops @ Envato Presentation Transcript