On March 12, 2019 Google experienced a 4 hour and 10 minute service disruption in their internal blob storage service. The outage was caused by a configuration change made by SREs on March 12th to reduce storage usage, which had the unintended side effect of overloading a key system. The document discusses principles for building highly available systems drawn from Google's experiences, including embracing constant change, hybrid architectures, observability, and value-based decision making.

1. New Rules
Expedia Group
@sallamar
https://www.subbu.org
Subbu Allamaraju

2. Faster, better, and cheaper
Safer, reliable

4. Change
Everything
else

6. Source: “Evolve of Die: High-Availability Design Principles Drawn from
Googles Network Infrastructure” by Ramesh Govindan et al, from the
Proceedings of the 2016 ACM SIGCOMM Conference
We find that failures are evenly distributed
across different network types and across
data, control, and management planes, but
that a large number of failures happen when
a network management operation is in
progress within the network.

7. ROOT CAUSE
On Monday 11 March 2019, Google SREs were alerted to a
significant increase in storage resources for metadata used
by the internal blob service. On Tuesday 12 March, to
reduce resource usage, SREs made a configuration change
which had a side effect of overloading a key part of the
system for looking up the location of blob data. The
increased load eventually lead to a cascading failure.
“On Tuesday 12 March 2019, Google's internal blob storage service experienced
a service disruption for a duration of 4 hours and 10 minutes.”

9. © 2019 Expedia Group 9
Invest for speed and safety
Rule 1

10. © 2019 Expedia Group 10Image ID 130985919 © Andrey Kryukov | Dreamstime.com
[changes, traffic, dependencies, side effects …]
[cloud services and building blocks…]

11. Releasing changes safely
Progressive deployments
Separating deploy step
from release
Feature flags
Canary releases
Compatible interfaces
Additive data model
changes
Shadow traffic
Predictable pipelines
Rollback
Change freeze

12. Sharded, replicated, or passive
non-fate-sharing architectures

13. Observability
Improve
steady-state
understanding

14. © 2019 Expedia Group 14
There is no substitute for
architecture and
operational hygiene
Rule 2

15. Big App
Big
Database
Big Ops

17. Retired
App
Big
App
Tech
Debt
Another
Big App

18. © 2019 Expedia Group 18
When change is constant,
hybrid is perpetual
Rule 3
tech, process, culture

19. What you
built
yesterday
What you
are building
now
What you
want to build
tomorrow

20. Tech
debt
Re-
platform
Grow the
business
Something
else
?

21. © 2019 Expedia Group 21
Stay calm and learn to
make value-based decisions
Rule 4

22. © 2019 Expedia Group 22
Embrace the messiness
Rule 5

23. Thank you
@sallamar
https://www.subbu.org
Expedia Group