This document discusses how Etsy enables rapid experimentation through continuous deployment and metrics-driven development. Key points include deploying small code changes frequently using feature flags and ramp-ups, measuring everything with tools like StatsD, optimizing for learning by running many experiments, and gaining confidence to change through quantitative metrics. The goal is to make failures cheap and iterate quickly to find product-market fit.

1. Engineering Change:
Enable rapid experimentation
Kellan Elliott-McCrea
@kellan
CTO, Etsy
Thursday, May 17, 12

2. * A marketplace for handmade and vintage goods.
In April 2012
* 875,000 artisans and creative independents
* $63.6 million USD in sales
* 1.32 billion page views
Thursday, May 17, 12

3. Engineering Change:
* Continuous Deployment
* Metrics driven development
Thursday, May 17, 12

4. Engineering Change:
* Make mistakes quickly
* Learn from mistakes
Thursday, May 17, 12

5. Engineering Change:
* Continuous Deployment
* Metrics driven development
Thursday, May 17, 12

6. Continuous Deployment:
a technique for
rapid iteration
Thursday, May 17, 12

7. Continuous Deployment:
make failure cheap
Thursday, May 17, 12

8. Continuous Deployment:
small changes,
pushed frequently
Thursday, May 17, 12

9. Continuous Deployment:
small changes,
pushed frequently.
To production.
Thursday, May 17, 12

10. Continuous Deployment:
10,068 releases to
production in 2011
@ Etsy
Thursday, May 17, 12

11. Continuous Deployment:
How?
Thursday, May 17, 12

12. Continuous Deployment:
No branching.
“All existing revision control systems were
built by people who build installed
software”
- Paul Hammond,
Always Ship Trunk, Velocity 2010
Thursday, March 17, 2011
Thursday, May 17, 12

13. Continuous Deployment:
feature flags
if ($cfg[‘awesome_new_search’]) {
# new hotness
$rsp = do_solr();
} else {
# boring old stuff
$rsp = do_grep();
}
Thursday, May 17, 12

14. Continuous Deployment:
Ramp - ups
(on top of feature flags)
1. Launch to staff only
2. Launch to 1% of all users
3. Launch to members of a beta group
Thursday, May 17, 12

15. Continuous Deployment:
any engineer can launch a feature to
1% of users
Thursday, May 17, 12

16. Continuous Deployment:
~200 experiments
live right now
Thursday, May 17, 12

17. Metrics driven development:
optimize for learning
Thursday, May 17, 12

18. Metrics driven development:
measure everything
Thursday, May 17, 12

19. Metrics driven development:
getting started:
make it easy
Thursday, May 17, 12

20. Metrics driven development:
measure everything:
logs, monitoring, A/B, graphs,
StatsD
Thursday, May 17, 12

21. Metrics driven development:
StatsD @
Instagram, Pinterest, Github,
Mozilla, LAN.com, Zynga,
Kickstarter, LivingSocial and 70+
other companies
Thursday, May 17, 12

22. Metrics driven development:
StatsD
“We love statsd at Instagram. Written by Etsy, it’s a
network daemon that aggregates and rolls-up data into
Graphite. At its core, it has two types of statistics: counter
and timers. We use the counters to track everything from
number of signups per second to number of likes, and we
use timers to time generation of feeds, how long it takes to
follow users, and any other major action.”
- Mike Krieger, Founder and CTO, Instagram.
Thursday, May 17, 12

23. Metrics driven development:
StatsD
Open source,
a daemon for timing and counting things,
graphs rates, averages and 95th percentile,
20-30 minute install,
easy to use
Thursday, May 17, 12

24. Metrics driven development
StatsD::timing("page.render", $msec);
Thursday, May 17, 12

25. Metrics driven development
Thursday, May 17, 12

26. Metrics driven development
getting started:
your 5 core metrics
Thursday, May 17, 12

27. Metrics driven development
5 core metrics @ Etsy:
sign ups, logins, checkout, new
listings, posts in the bugs forums
Thursday, May 17, 12

28. Metrics driven development
378,000 metrics,
collected roughly once a
second
Thursday, May 17, 12

29. Metrics driven development
the confidence to
change
Thursday, May 17, 12

30. Engineering change:
Rapid experimentation
Adapt, find market fit.
Thursday, May 17, 12

31. Engineering change:
Rapid experimentation
Adapt, find market fit,
addictive to engineers.
Thursday, May 17, 12

32. Thank you!
Lean more: http://codeascraft.etsy.com
Thursday, May 17, 12