Ross Snyder, Etsy, SXSW Lean Startup 2013

1. Continuous Deployment at
:
A Tale of Two Approaches
Ross Snyder
ross@etsy.com
@beamrider9 March 9, 2013

2. A quick primer on

3. is:
The global marketplace we make together.

4. is:
The premier destination for handmade goods,
vintage items, and craft supplies.
simplertimestoys lacklusterco norwesterseaglass

5. quick facts: (as of March 2013)
• 22+ million members
• 800,000+ active shops
• 18+ million items currently for sale
• 20 cents to list item, 3.5% transaction fee
• 400+ employees (majority in Brooklyn)

6. Since opening its doors in June 2005, Etsy
has grown virtually non-stop.
$1,000
$800
$600
$400
$200
$0
2005 2006 2007 2008 2009 2010 2011 2012
Gross Merchandise Sales ($MM)

7. A nice problem to have:
“Our site is so
successful, how can we
move fast enough to
keep up with demand?”

8. CONTINUOUS
DEPLOYMENT

9. :
The Early Years
(2005 - 2008)

10. : The Early Years
1. Spend significant time writing code

11. : The Early Years
1. Weeks writing code
2. Painful source control merge

12. : The Early Years
1. Weeks writing code
2. Painful merge
3. Hand off to someone else to deploy

13. : The Early Years
1. Weeks writing code
2. Painful merge
3. Hand off to deployers
4. Deploy, site goes down

14. : The Early Years
1. Weeks writing code 4. Deploy, site down
2. Painful merge
3. Hand off to deployers
5. Roll back deploy

15. : The Early Years
1. Weeks writing code 4. Deploy, site down
2. Painful merge 5. Roll back deploy
3. Hand off to deployers
6. Spend hours (days?) fixing bugs

16. : The Early Years
1. Weeks writing code 4. Deploy, site down
2. Painful merge 5. Roll back deploy
3. Hand off to deployers 6. Fix bugs
7. Go back to step 2

17. : The Early Years
WATERFALL!

18. : The Early Years
Pros:
Early Etsy engineers
used this release
cycle to bootstrap the
marketplace from
nothing.
Forever grateful.

19. : The Early Years
Cons:
• Large changesets
• Infrequent deploys
• Weak confidence in deploy success
• Significant time spent deploying
• Low ability to experiment/iterate/react
• Developer stress/unhappiness

20. : The Early Years
By late 2008, Etsy is still a startup, but has the
deploy process of a much bulkier company.
Popularity is on the verge of outpacing capacity.

21. :
Today

22. : Today
1. Small changesets, deployed frequently

23. : Today
1. Small changesets
2. Engineers deploy the site

24. : Today
And not just engineers, but also:
• Designers
• Product Folks
• Upper Management
• Board Members
• Dogs

25. : Today
1. Small changesets
2. Engineers deploy
3. Deploys are fast and near-effortless

26. : Today
1. Small changesets
2. Engineers deploy
3. Deploys are fast
4. Most changes behind config flags
(safer deploys)

27. : Today
1. Small changesets 4. Changes behind flags
2. Engineers deploy
3. Deploys are fast
5. Graphs/metrics to assess deploy

28. : Today
1. Small changesets 4. Changes behind flags
2. Engineers deploy 5. Copious graphs/metrics
3. Deploys are fast
6. If issues, fix immediately & roll forward

29. : Today
This isn’t license to break stuff, quickly.
Engineer-driven QA and solid unit testing
are integral parts of the process.

30. : Today
1. Small changesets 4. Changes behind flags
2. Engineers deploy 5. Copious graphs/metrics
3. Deploys are fast 6. Fix fast & roll forward
7. Repeat 25+ times per day, every day

31. Then:
1. Weeks writing code
2. Painful merge
3. Hand off to deployers
4. Deploy, site down
5. Roll back deploy
6. Fix bugs, go to step 2
Now:
1. Small changesets
2. Engineers deploy
3. Deploys are fast
4. Changes behind flags
5. Copious graphs/metrics
6. Fix fast & roll forward

32. Etsy Deploy Stats: 2012
• Deployed to production 6,419 times
• On average, 535/month, 25/day
• Additional 3,851 config-only deploys
• 196 different people deployed to prod
• Nov/Dec 2012: deployed 752 times

33. Why does it work?

34. Continuous Deployment Math
• N = # of deploys
• P = probability of site degradation
• S = average severity of degradation
• T = time to detect/resolve
Expected
= N*P*S*T
Downtime

35. Continuous Deployment Math
N = # of deploys S = avg. severity of degradation
P = prob. of degradation T = time to detect/resolve
Before: Now:
•N=1 • N = 250 ↑↑↑↑
• P = 0.5 • P = 0.1 ↓
• S = 0.7 • S = 0.05 ↓↓
• T = 100 • T = 5 ↓↓↓
E.D. = 35 E.D. = 6.25
(all numbers completely arbitrary)

36. Big Takeaway
Etsy circa 2013 (400+ employees)
acts, in some ways, more like a startup than
Etsy circa 2008 (40+ employees).

37. Continuous Deployment makes possible:
“Continuous Experimentation”
http://etsy.me/continuous-experimentation

38. Continuous Experimentation
1. Small changes
2. Run experiment (A/B test)
3. Analyze data
4. Re-examine assumptions
Repeat continuously in pursuit
of larger goals.

39. Heard since 2010:
“Neat experiment, but
this will never scale.”
As of 2013, Etsy has
100+ engineers -
still going strong.

40. Some Etsy Customizations
Deploying is a first-class feature. Inability to
deploy is a P1 incident (same as site down).

41. Some Etsy Customizations
We continuously deploy not just the main
Etsy website, but as much as possible:
• Internal admin site
• API
• Big data
• Search
• Blog
• Deployinator itself

42. Some Etsy Customizations
In the rare case we can’t continuously
deploy, we create alternative tools:
• Database schema changes
• PCI-DSS environment (credit cards)
We do continuously deploy as much of our
payment processing as is safe & legal (98%).

43. Some Etsy Customizations
Keeping deploys fast is paramount and worth
the investment in manpower & hardware.

44. Some Etsy Customizations
Continuous deployment is all about moving
forward, sometimes at the expense of the past.
Our solution: engineering-wide bug rotation,
one day a month, every engineer participates.

45. Fun Fact:
Continuous Deployment is a fantastic
recruitment tool for attracting engineers
who like to move fast and get stuff done.

46. Learn more:
http://codeascraft.etsy.com/
Etsy open source (Deployinator, StatsD)
http://etsy.github.com/
Join the fun:
http://www.etsy.com/careers