These are the slides from my talk at the Surge Conference in 2010, in Baltimore: http://omniti.com/surge/2010/speakers/john-allspaw

1. Go or No-Go
Operability and Contingency Planning
John Allspaw, Etsy.com

4. Etsy as of now
Total Members: over 5.7 million
Total Sellers: over 400,000
Items Currently Listed: 6.5 million
Page Views per month: 775 million
Total $ sold (gross merchandise sales)
2010 = $179.4 million (through August)

5. New Features

6. Delivering OperableGoSoftware
Arch Review Development/Ops or No-Go Launch*
Feedback Loop

7. CONTINUOUS DEPLOYMENT
!=
deploying new features without
coordination and planning

8. Operability Review
Contingency Checklist

9. Not An Innovative Idea
http://en.wikipedia.org/wiki/Launch_status_check

10. 10 minute get-together
• Product
• Development
• Operations
• Design
• Community
• Support

11. Consensus

12. Informally Codifies “OK”
Dev “We all understand/agree/
Ops accept that we are OK here!”
Product
Community
Support
Buggy Stable Perfect!
Sloppy Finished Enough For Launch
Unfinished

13. Yes
or
No

14. Has the feature been tested enough to
deploy to production?
Is there any final functional QA still needed?

15. Is communication (blog post/forums/etc)
about the feature ready to go out with the
feature?

16. Does everyone know:
when it will go live, and
who will push the feature?

17. Has the feature been in production for staff
(or some other specific subset of the users)
already?
If not, could it have been?

18. Is it possible to dark launch this feature?
Will this feature be dark launched?
(or, has it already?)

19. Is it possible to turn up this feature on a
percentage basis?
If so: will we?

20. Does it involve any new infrastructure?
If so: are those pieces in monitoring and
metrics collection?
(this answer can’t be “no” before launch)

21. Do we have on/off switches for this feature?
If so: are those switches documented?
(this answer can’t be “no” before launch)

22. Are all the leads (Dev, Ops, Product,
Community, Support, etc.) available for the
launch and in communication?
(this answer can’t be “no” before launch)

23. Is there a single and easy place for users to
report bugs or concerns about the feature?
(forum topic, etc.)

24. Have all leads agreed upon a post-launch
“it’s all DONE” time to declare the launch
was successful?

25. Have we done a Contingency Checklist™
and everyone reviewed it?
(this answer can’t be “no” before launch)

26. Contingency Checklist

27. “What could possibly go wrong?”
“When it does go wrong,
WTF will we do?!”

28. NOTE:
This is worked outBEFORE launch, normally by product and
development, involving others where needed.
(when we have saner heads)

29. Issue Onsite Messaging
Likelihood Forums
Comment(s) Blog
Impact on Users PR
Engineering Response

30. Comment Impact on Engineering Onsite
Issue Likelihood Forums Blog PR
(s) Users Response Messaging

31. Example: Coffee!
AWESOME NEW FEATURE
• add coffee (like a tag) to your profile
• others can favorite coffees
• page showing all coffee favorites
• bulk-add coffees to your profile
• search people by their coffee

32. Issue
What could possibly go wrong with the feature launched in
production?
Example:
“The Coffees-You’ve-Favorited page is too expensive.”

33. Likelihood
How likely is this issue going to come up?
Example:
“Low to mid.”

34. Comment(s)
Any extra info about this issue here.
Example:
“Because of how we paginate coffee favorites page, they are
somewhat harder than normal favorites. If we do have to turn
this off, we’re saying that we need to re-design it, or it needs
to stay off until the initial burst of traffic from the launch.”

35. Impact
How much is this going to impact the experience of the feature, if
it does become a concern?
Example:
“High”

36. Engineering Response
What will we do to mitigate the issue (i.e. can we gracefully
degrade?)
Example:
“Set disable_coffee_favorites_page = 1”

37. Onsite Messaging
What is the messaging to the community in the forums/blog/etc.,
if this needs graceful degradation?
Example:
“‘The Coffee Favorites page is currently unavailable.’ Or, in the
forums: “We’re working through some issues with displaying
Coffee Favorites, we’ll let you know the status as time goes
on.’”

38. PR
Is the issue so severe that we need PR involved?
Example:
“The CEO sends a press release, apologizing to Folger’s,
Peet’s, and Starbucks with a witty yet calming voice of
explanation and a humble request for patience.”

39. * afterwards....

40. *successful launch...
Metrics?
Are we there yet?
OMG! Who to call if it breaks later?

41. * non-successful launch...
Metrics?
What’d we miss?
Post Mortem?
Ramp down?

43. Photos
http://www.flickr.com/photos/jliba/3783269078/
http://www.flickr.com/photos/mybloodyself/2072928376/
http://www.flickr.com/photos/jacy/360020853/
http://www.flickr.com/photos/f-l-e-x/2319852529/
http://www.flickr.com/photos/16230215@N08/3023061528/
http://www.flickr.com/photos/proimos/4199675334/
http://www.flickr.com/photos/askal_bosch/2579320395/