This document discusses managing failures and building resilience into systems at Etsy. Some key points: 1. Etsy has a complex architecture with many services and data stores that are functionally partitioned. This architecture is designed to limit the impact of failures. 2. Failures cannot be prevented, but they can be mitigated through techniques like redundant systems, small code changes, feature flags, extensive metrics collection, and resilient user interfaces. 3. Rather than focusing only on 100% uptime, product design also considers availability during failures through approaches like non-blocking interfaces that adapt to technical issues. 4. Building resilience is a shared responsibility of operations, engineering, product, and design teams through

1. On Failure and Resilience
Mike Brittain
DIRECTOR OF ENGINEERING, ETSY
@mikebrittain
Presented at 37signals on Aug 21, 2012

2. “Software Infrastructure”
“Framework” code, caching, ORM, file storage tier,
developer tools, CI/deployment, site performance,
front-end architecture.

3. Managing failures and building
resilience into systems, applications,
process, and people.

5. $61 M in goods sold in the marketplace
2.9 M items sold
1.2 B page views
Photo: http://www.etsy.com/shop/TheOldTimeJunkShop
http://www.etsy.com/blog/news/2012/etsy-statistics-june-weather-report/

6. Architecture
Linux, Apache, MySQL, PHP, Postgres,
Solr, Gearman, Memcache, Chef,
Hadoop, EC2/S3/EMR
30+ Logical data stores
(23 shards + more functionally partitioned)
Search and storage tiers as “services”

7. 150 Engineers + Designers + Product
(this was 20 in Feb 2010)
credit: martin_heigan (flickr)

8. Buyers, sellers, support,
developer api, i18n,
core infrastructure, storage,
payments, security, fraud
detection, big data and BI,
email delivery, corp IT,
operations, developer tools,
continuous integration and
testing, site performance,
search, advertising, seller
economics, mobile web,
iOS.

10. Zero Release Managers

11. There Will Be Fail
Credit: wilkee.deviantart.com

12. We cannot comprehend all of the ways in
which the individual parts of a complex
system will interact. We cannot know all
of the states and scenarios.
We cannot prevent failures.

13. Yet, we can mitigate them.
Redundant system architectures.
Small, well-understood changes to production.
Control application using config flags.
Gratuitous metrics collection.
Resilient user interfaces.
GameDay exercises.

14. “Uptime” is not binary.

15. Async
Convos Ads Auth
Tasks
Functionally Partitioned

16. Async
Convos Ads Auth
Tasks
Functionally Partitioned

17. 4 1
3 2 5
Async Async
Convos Convos Ads Ads Auth Auth
tasks tasks
Master-Master Replication

18. 4 1
3 2 5
Async Async
Convos Convos Ads Ads Auth Auth
tasks tasks
Master-Master Replication

19. 4 1
3 2 5
Async Async
Convos Convos Ads Ads Auth Auth
tasks tasks
Master-Master Replication

20. 1 5
3 2 4
shard1 shard1 shard2 shard2 shard3 shard3 shard4 shard4
~4% of listing data is
stored on shard3
Sharded Tables

21. 1 5
3 2 4
shard1 shard1 shard2 shard2 shard3 shard3 shard4 shard4
Sharded Tables

22. shard1 shard1 shard2 shard2 shard3 shard3 shard4 shard4
Outage is limited to
~4% of data set
Sharded Tables

23. “Uptime” is not binary.

24. Uptime of the application is the
responsibility of our Operations team.

25. Uptime of the application is the
responsibility of our Operations, Engineering,
Product, and Design teams.

26. Uptime of the application is the
responsibility of our Operations, Engineering,
Product, and Design teams.
If you are committing code, you are
operating the site.

27. Branching in Code

28. “All existing revision control systems
were built by people who build
installed software”
Always Ship Trunk
Paul Hammond
Velocity Conf 2010

29. Config Flags
Enable and disable features quickly.
Features for staff or for beta groups.
Percentage ramp-up of users or requests.
A/B “experiments.”

30. $cfg[‘new_search’] = array('enabled' => 'on');
$cfg[‘sign_in’] = array('enabled' => 'on');
$cfg[‘checkout’] = array('enabled' => 'on');
$cfg[‘homepage’] = array('enabled' => 'on');

31. $cfg[‘new_search’] = array('enabled' => 'on');
// Meanwhile...
if ($cfg[‘new_search’]) {
# New hotness
$results = do_solr();
} else {
# old and boring
$results = do_grep();
}

32. But...

33. “Doesn’t that mean you have conditionals
all over your code?”
Yes.

34. “Doesn’t that mean you have conditionals
all over your code?”
Yes.
“Does anyone ever clean those up?”
Sometimes.

35. “Doesn’t that mean you have conditionals
all over your code?”
Yes.
“Does anyone ever clean those up?”
Sometimes.
“That sounds like it sucks.”
Really?

36. “Doesn’t that mean you have conditionals
all over your code?”
Yes.
“Does anyone ever clean those up?”
Sometimes.
“That sounds like it sucks.”
Really?
“Wait a minute... all of the counter arguments are
in Comic Sans. WTF?!?
Oh, you noticed? ;)

37. +06:40
Site up, some seller tools disabled
00:00
Site down for maintenance
+01:47
Site up, disabled login and registration
+07:41
All features restored
DB Server Maintenance, June 16, 2012
http://etsystatus.com/2012/06/16/planned-outage-june-16th-7am-gmt/

38. “Uptime” is not binary.

39. Features are launched by flipping a
config flag, not by deploying
hundreds of lines of code.

40. “If Engineering at Etsy has a religion,
it’s the Church of Graphs.
Ian Malpass, Code as Craft
http://etsy.me/ePkoZB

43. THIS IS HOW
YOU RUN
A COMPLEX
SYSTEM
http://www.flickr.com/photos/flyforfun/2694158656/

44. Config flags
Operator Metrics
http://www.flickr.com/photos/flyforfun/2694158656/

45. Oh, you want to talk about how we collect
metrics and make graphs?
http://www.slideshare.net/mikebrittain/metricsdriven-engineering

46. Resilient User Interfaces

47. Interfaces and user experiences
that adapt to technical and
architectural failure.

54. /**
* Creates a database connection.
*/
public function __construct($host, $user, $pass, $db) {
parent::__construct($host, $user, $pass, $db);
if (mysqli_connect_error()) {
throw new DBConnection_Exception(
sprintf("Error: %s, %s",
mysqli_connect_errno(),
mysqli_connect_error()));
}
}

55. try {
$conn = new DBConnection('viewsdb.host', 'db_read_user',
'ssssshh!', 'views_db');
} catch (DBConnection_Exception $e) {
// TODO: Someone should figure out what to do if
// we can't connect to the views db.
throw $e;
}

58. Site navigation
Logo
Cute Picture
Generic, catch-all
error messaging....

60. Every back-end service is an
opportunity for failure.

64. 1 4
9
5
8 6
2 3
10
4 11
7
14
7
13
12

66. Critical Path

70. #srsly?

71. < 400 ms

72. Non-blocking Ajax

73. Google Calendar
Google Docs

74. GMail

75. “Oops, we aren’t able to
access click metrics right
now, do not worry — your
data is safe.”

76. Product design doesn’t stop
at 100% availability.

77. Dev Ops

78. Dev Ops
Product

79. 1 4
9
5
8 6
2 3
10
4 11
7
14
7
13
12

80. Operability Reviews

81. “What could possibly go wrong?”
What is changing about the architecture?
What kind of data access patterns are we using?
How much traffic, how many queries?
What metrics are we collecting?
Are there automated alerts?
How do we know the thresholds are right?
How do we turn it off?... and what happens when we do?

82. “What could possibly go wrong?”
What is changing about the architecture?
What kind of data access patterns are we using?
How much traffic, how many queries?
What metrics are we collecting?
Are there automated alerts?
How do we know the thresholds are right?
How do we turn it off? ...and what happens when we do?

83. “GameDay” Exercises

85. Pedro

86. Homepage (95th perc.)
Surprise!!!
Turning off multi-
language support
improves our page
generation times by
up to 25%.

87. (Blameless) Post-Mortems

88. How could this have gone better?
How quickly did we find out that something was wrong?
Did we communicate well to our visitors and each other?
Why did we have confidence that what we were doing was OK?
Did we have the right tools, did we use them properly?
Did we collect metrics, and could we find them?
Where did we make the wrong decisions?
What steps do we take to reduce the chance of this
happening again in the future?

89. “... an engineer who thinks they’re going to be
reprimanded are disincentivized to give the details
necessary to get an understanding of the mechanism,
pathology, and operation of the failure.
This lack of understanding of how the accident occurred
all but guarantees that it will repeat. If not with the
original engineer, another one in the future.”
John Allspaw
VP, Technical Operations, Etsy
http://codeascraft.etsy.com/2012/05/22/blameless-postmortems/

90. We should try to learn not only what went
wrong, but also what went right.

91. +06:40
Site up, some seller tools disabled
00:00
Site down for maintenance
+01:47
Site up, disabled login and registration
+07:41
All features restored
DB Server Maintenance, June 16, 2012
http://etsystatus.com/2012/06/16/planned-outage-june-16th-7am-gmt/

92. Operational Mindset
Dev Ops Product

93. Operational Mindset
Dev Ops Product
Business Priorities

94. Introspection

95. page views for error template

96. page views for error template
...or, how are we screwing our users?

97. Risk mitigation in a complex system
Redundant system architectures.
Small, well-understood changes to production.
Control application using config flags.
Gratuitous metrics collection.
Resilient user interfaces.
GameDay exercises.

98. Thank you.
Mike Brittain
mike@etsy.com
@mikebrittain

102. PHOTO
CREDITS
Flickr: roboppy
http://www.flickr.com/photos/51035735481@N01/163374138/
Flickr: jamesjyu
http://www.flickr.com/photos/32593095@N00/3465022/
Flickr: circulating
http://www.flickr.com/photos/26835318@N00/2318226026/