High Availability at Yahoo!

High Availability at Yahoo! Michael Christian Infrastructure Resiliency July 29, 2010

Business Continuity Plan ning
Q: What is your plan if the datacenter burns down?
A: “Datacenters never actually go down.”
A: “Rebuild it from scratch somewhere else.”
A: “IBM will take care of it for us.”
A: “Sell my stock and move to Bali.”
(These are real quotes)

Datacenters DO go down!
Outages…
Network Instability
UPS Failures
HVAC Failures
Leaky Equipment
Squirrels
Flywheels
Explosions!
Maintenance & Migrations…
Core Router Upgrades
Massive Migrations
Power Systems Maintenance
(sometimes for the oddest of reasons)

Think BIG!
Levels upon levels of redundancy…
Two servers (duh)
Two switches?
Two uplinks per switch?
Two power distribution busses?
Two completely independent network planes touching all points?
Two Datacenters!
As complexity goes up to the extreme, reliability goes down.
If you solve for Datacenter-size disasters, the little stuff matters less
Plan for the macro: solve for the metro.

Mitigating Disaster
Focus on Impact Duration, not Incident Duration
Should be able to fail an entire datacenter out in minutes
Repairing datacenter damage could take weeks
Automation is the key!
DNS based GSLB systems can save you
Route users to the best possible service point
For a single lookup, return an optimal IP, based on distance/load/availability
Constant healthchecking to ensure availability
Proximity routing (various methods)
Akamai, Level3, Neustar, F5, Zeus, Brooklyn

Practice!
At Yahoo! we do full-scale live failover testing during peak traffic hours.
We shift millions of users across the country or across the world in minutes
Testing is non-impacting
Learn and build confidence
Start using this as part of daily life:
Staged Upgrades
Managing Risk
Migrations

Business Continuity Plan ning
“ When the servers crashed, effectively putting eBay out of business, Meg gathered her team, along with the best technology experts from around Silicon Valley, and they stayed there until it was fixed. Literally, sleeping at the office.”
Quote from Meg Whitman for Governor
Q: What is my plan if the datacenter burns down?
1) “Make sure my traffic failed over cleanly.”
2) “Roll over.”
3) “Go back to sleep.”

http://nosql.mypopescu.com	206
http://www.linkedin.com	9
http://static.slidesharecdn.com	3
http://www.hanrss.com	1
resource://brief-content	1
http://irr.posterous.com	1
http://passing.tk	1
http://atm09td.tumblr.com	1

High Availability at Yahoo!

by Chris Westin

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 223

Statistics

High Availability at Yahoo! Presentation Transcript