Scaling :
What Went Wrong,
What Went Right
Ross Snyder
ross@etsy.com
@beamrider9 Sept. 30, 2011
1

Etsy is the world’s handmade marketplace.
(vintage and supplies, too)
2

Etsy was founded in mid-2005
and is constantly growing.
Gross Merchandise Sales ($MM)
3

From humble beginnings...
June Four employees, one web*,
2005: one db, founder’s apartment
* until getting slashdotted by a link
from Boing Boing in Aug. 2005
4

... to today’s handmade juggernaut.
Sept. 250+ employees, multiple
2011: offices, billions of pageviews
(NYC Mayor Mike Bloomberg visited
Etsy in June 2011)
5

How’d we get here?
6

Answer: with some difficulty.
“There is no education like adversity.” - Benjamin Disraeli
7

A few disclaimers
8

Hindsight is 20/20
9

“History is written by the victors”
10

Etsy thrives today
because of what
its early employees
accomplished
11

Your narrator wasn’t present for most
of the events covered in this talk
12

Etsy Architecture: 2007
13

Etsy Architecture: 2007
Operating System:
Database:
Webserver:
Languages:
14

Etsy Architecture: 2007
Most business logic in
Postgres stored procedures
15

Etsy Architecture: 2007
Front end / database interaction = stored
procedure calls wrapped with PHP functions
16

Etsy Architecture: 2007
Some database partitioning by feature,
but still with a large central DB
17

Etsy Architecture: 2007
Site uptime = not great
18

Etsy Architecture: 2007
“How do we scale?”
19

Etsy Architecture: 2007
“Let’s write some middleware!”
(runners up: “Let’s rewrite the site in Java!”
and “Let’s rewrite the site in Python!”)
20

Conway’s Law:
“Any organization that designs
a system (defined broadly) will
produce a design whose
structure is a copy of the
organization's communication
structure.”
- Melvin Conway, 1968
21

Etsy Engineering: 2007
Dev DBA Ops
22

Etsy Engineering: 2007
Dev DBA Ops
Devs write code
23

Etsy Engineering: 2007
Dev DBA Ops
DBAs write SQL
24

Etsy Engineering: 2007
Dev DBA Ops
Ops deploys code & touches prod
25

SILOS
26

Etsy’s big bet: “Sprouter”
(the Stored Procedure Router)
27

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
Runs on each webserver,
listens on port 8010
28

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
Maps name/arguments to a Postgres
stored procedure, calls it, returns results
29

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
Caches things
30

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
Supports sharding (in theory)
31

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
Devs write PHP, DBAs write SQL,
meet somewhere in the middle
32

SILOS
33

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
The hope: easier to scale Sprouter
than to scale the database itself
34

Sprouter
Web Sprouter DB
(PHP) (Python) (Postgres)
(scaling the db when everything’s in
stored procedures = somewhere between
hard and impossible)
35

Sprouter: Timeline
Fall ’07: Idea first discussed
Spring ’08: Alpha version debuts
Fall ’08: Released in production
36

Sprouter: Timeline
Fall ’07: Idea first discussed
Spring ’08: Alpha version debuts
Fall ’08: Released in production
Spring ’09: Sprouter deprecated
37

What happened?
38

Sprouter: “Good” Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Forcibly centralizes database access
39

Sprouter: “Good” Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Hides data store implementation
from caller
40

Sprouter: “Good” Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Opens the door for
“clever” automatic caching
41

Sprouter: “Good” Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Prevents developers from writing SQL (?)
42

43

Sprouter: Not-As-Good Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Creates substantial developer friction
44

Sprouter: Not-As-Good Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Homegrown daemon + dependencies
for Ops to maintain
45

Sprouter: Not-As-Good Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Lack of community support / provability
46

Sprouter: Not-As-Good Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Complex synchronization required to deploy
(due to tight coupling with Postgres)
47

Sprouter: Not-As-Good Parts
Web Sprouter DB
(PHP) (Python) (Postgres)
Database remains single point of failure
(sharding features never fully formed)
48

Sprouter: Summary
Extra barriers to development
49

Sprouter: Summary
Extra barriers to development
+ Negligible (negative?) effect on site reliability
50

Sprouter: Summary
Extra barriers to development
+ Negligible (negative?) effect on site reliability
+ Deploys even more painful
51

Sprouter: Summary
Extra barriers to development
+ Negligible (negative?) effect on site reliability
+ Deploys even more painful
+ Requires extra Ops/Dev resources
52

Sprouter: Summary
Extra barriers to development
+ Negligible (negative?) effect on site reliability
+ Deploys even more painful
+ Requires extra Ops/Dev resources
=
53

How did attitudes change so quickly?
54

Sprouter: Timeline
Fall ’07: Idea first discussed
Spring ’08: Alpha version debuts
Fall ’08: Released in production
Spring ’09: Sprouter deprecated
55

The Great Etsy Culture Shift
56

The Great Etsy Culture Shift
Just as Sprouter went live, many of its
strongest proponents departed Etsy
57

The Great Etsy Culture Shift
Taking with them...
58

The Great Etsy Culture Shift
Devotion to Postgres stored
procedures / types
59

The Great Etsy Culture Shift
Fear of developers writing SQL
60

The Great Etsy Culture Shift
Fear of developers touching prod
61

The Great Etsy Culture Shift
Infrequent / large deploys to production
62

The Great Etsy Culture Shift
“Not developed here”
63

The Great Etsy Culture Shift
Then Now
Fall
’08
64

DevOps
65

DevOps
Silos = bad
66

DevOps
Trust, cooperation, transparency,
shared responsibility = good
67

DevOps
“We’re all in this together”
68

The Way Forward: Part 1
Stabilize the site
69

The Way Forward: Part 1
Stabilize the site
Improve metrics & monitoring
70

The Way Forward: Part 1
Stabilize the site
StatsD
http://github.com/etsy/statsd
71

The Way Forward: Part 1
Stabilize the site
Upgrade database hardware
vertically as far as possible
72

The Way Forward: Part 1
Stabilize the site
Give developers production access to
help troubleshoot problems
73

The Way Forward: Part 2
Continuous Deployment
74

The Way Forward: Part 2
Continuous Deployment
Any engineer can deploy to prod
(generally happens 25+ times per day)
75

The Way Forward: Part 2
Continuous Deployment
Deployinator
http://github.com/etsy/deployinator
76

The Way Forward: Part 2
Continuous Deployment
One button that deploys the site
77

The Way Forward: Part 2
Continuous Deployment
Small changesets, deployed frequently
78

The Way Forward: Part 2
Continuous Deployment
Requires solid tests,
good communication
79

The Way Forward: Part 2
Continuous Deployment
Distributed developer-driven QA
80

The Way Forward: Part 3
Circumvent Sprouter
81

The Way Forward: Part 3
Circumvent Sprouter
Object-Relational Mapping (ORM)
82

The Way Forward: Part 3
Circumvent Sprouter
aka “The Vietnam of Computer Science”
(Google it)
83

The Way Forward: Part 3
Circumvent Sprouter
Front-end PHP talks directly to database
via ORM (also written in PHP)
84

The Way Forward: Part 3
Circumvent Sprouter
ORM can cache where appropriate
(as can front end)
85

The Way Forward: Part 4
Database Sharding
86

The Way Forward: Part 4
Database Sharding
Etsy has a lot of DNA from flickr -
including their DB sharding scheme
87

The Way Forward: Part 4
Database Sharding
Based on MySQL
88

The Way Forward: Part 4
Database Sharding
Battle-tested, well-known
89

The Way Forward: Part 4
Database Sharding
Scales horizontally to infinity
(or close enough)
90

The Way Forward: Part 4
Database Sharding
No single points of failure
(master-master replication)
91

The Way Forward: Part 4
Database Sharding
Gradually phase out Sprouter,
phase in ORM / sharded data
92

Sprouter: Timeline
Fall ’07: Idea first discussed
Spring ’08: Alpha version debuts
Fall ’08: Released in production
Spring ’09: Sprouter deprecated
93

Sprouter: Timeline
Fall ’07: Idea first discussed
Spring ’08: Alpha version debuts
Fall ’08: Released in production
Spring ’09: Sprouter deprecated
Spring ’11: Sprouter turned off
94

95

Lessons Learned
96

Etsy Architecture: 2007
Operating System:
Database:
Webserver:
Languages:
97

Etsy Architecture: 2011
Operating System:
Database:
Webserver:
Languages:
98

Open & trusting > closed & afraid
(DevOps DevOps DevOps)
99

Front end/database interaction is too critical
to take chances on novel/untested solutions
100

Side corollary: If you’re doing something
“clever”, you’re probably doing it wrong
101

The architectural decisions you make today
will have large impact long after you’re gone
102

No architectural hole is so deep that proven
scaling strategies don’t exist for digging out
103

Acknowledgement
We are probably making decisions today that
will be the subject of a similar talk in 2015
104

Learn More:
http://codeascraft.etsy.com/
@codeascraft
105

Etsy is hiring!
http://www.etsy.com/careers
@etsy
106