Getting to NO NINES
•
0 likes•1,311 views
Talk given by Mike Christian, VP, Site Reliability Engineering at Salesforce, at SRECon in April 2016 We pay an incredible amount of money to put our infrastructure into Tier 3/4 data centers, and still things fail. Worse yet, many of the failures are due to the complexity we've added to the system in a futile attempt to make it more "reliable." It's a false economy, as there is no such thing as a reliable data center. Switching to a resilience model not only solves the problem of keeping the service running when things fail; it also lets us build much lower tier data centers, for a fraction of the cost. No HVAC. No Generators. No UPS. Just an ephemeral building full of ephemeral servers. This talk will consist of real life failure examples, coupled with resiliency strategies that would have saved the day.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Getting to NO NINES
Similar to Getting to NO NINES (20)
More from Salesforce Engineering
More from Salesforce Engineering (17)
Recently uploaded
Recently uploaded (20)
Getting to NO NINES
- 1. © Copyright Peter Trimming and licensed for reuse under this Creative Commons Licence
- 2. Mike Christian GETTING TO NO NINES © Copyright Peter Trimming and licensed for reuse under this Creative Commons Licence
- 3. • Disaster Recovery • Airplane into building • Catastrophic fire • Earthquake There Once Was DR… • High Availability • Service Up, any cause - Varmits - Router Failures - Human Errors - Disasters • DR != HA © Copyright Rob Young and licensed for reuse under this Creative Commons Licence
- 4. Classic DR Planning: “So we don’t need to put that much effort into it.” “We will most likely never actually need to use this…” “We’re okay, we bought some insurance.” “Box… Checked!”
- 5. And everyone had a personal DR plan, right?
- 7. HA planning is something different (Assume failure and go from there…)
- 9. A partner once told me: “We have never lost a datacenter.”
- 10. Then, one week later they did. “No, it was just the edge routers.”
- 12. Even the best datacenters DO go down! For all sorts of unexpected reasons: • Network instability • HVAC failures • Power distribution and switching • UPS failures • Generator failures • Leaky roofs and equipment • Inadvertent fire suppression • Partial nuclear meltdown • Squirrels vs Internet! Maintenance & Migrations: • Core router upgrades • Infrastructure maintenance • Massive migrations
- 13. Bad things happen in the real world, and here’s why…
- 14. © Copyright Kevin Dooleyand licensed for reuse under this Creative Commons Licence
- 17. “When the servers crashed, effectively putting eBay out of business, Meg gathered her team, along with the best technology experts from around Silicon Valley, and they stayed there until it was fixed. Literally, sleeping at the office.” Quote from Meg Whitman for Governor
- 19. “Remember that Meg Whitman story? I was that guy! No kidding! I was cleaning up some space on the database, so I dropped an unused table, and it ended up being the main customer index. Doh! Meg came in and brought a ton of Oracle experts to sit with me all night, and rebuild the tablespace. DUDE, I LITERALLY SAVED EBAY!” -Ops Superhero -(Paraphrased) - (and Interview Candidate)
- 21. Fake Fires…
- 22. Chemical Suppression Dumps © Copyright Cohesion licensed for reuse under this Creative Commons Licence
- 23. Not Always Aborted… © Copyright Juan De Vojnikov and licensed for reuse under this Creative Commons Licence
- 24. HVAC Suppression (Not Always Wise) © Copyright PictoralEvidence and licensed for reuse under this Creative Commons Licence
- 26. Hurricane Wilma: Generator Fuel Shortage
- 27. Flywheels FTW!
- 29. Sometimes incidents can encompass entire metros…
- 30. 8/7/11, Impacted Companies • 1BOG 42 Spikes About.me Always Left America's Health Rankings Appbistro Appstores.com appgrinder.pl ArtCat ArtLog Assembla Ballpark bergholt.net bidwarrs Bigsoko Blitz.io Bloove Blue Print Log Blue Sombrero Bookingsync Boxoffice Mojo Big Prize on Facebook Buddy Events on Facebook Business Logic CanBIM Cash Crate Change.org Checkpoint Tracker CheckOrphan Chesscube Coderstack CollectorCarNation.com Comind CoTweet Cloud App chesscube Clique Code Spaces Comindwork Component Workshop Conga conne.cc ConsumerBell Crashpadder Creately Deallocated Objects Dealply DegreeOps Demeure DietPicture Dipity Discovr for iOS Doing Business in Canada dotcloud Dshini Easybib eCairn edocr eggie5 elog Engagor Expono EveryBlock Evite Geodata Check Good Joe Exceptional EveryTrail Explor Explore.to FathomDB FayerWayer Feedera FictFact Fitness Craze Flavorpill Flavors.me Flow FluxTracker followus fotopedia FormSpring Foursquare Frogpants GameChanger.io GazeHawk GenerationStation Get It Done GiantBomb.com Givmo Global Folders globalmediapro gmpx Grople GroupMe Guitarra Con Andres GumGum Henry Leutwyler Heroku hindawi Hollrback hootsuite HotPads Hubbr.net HyperAlerts iDent iDug Impure Indaba Music Index Tank Inventables Invox iRacing jaconda Jobs Score Jobbook joshdiehl.com Kekanto Kickstarter Krop Koofers Layers LetMeGo Lighthouse Listography Livefyre Localo LovingSports LunaTik LupaLuna Maansu Moby Picture MongoHQ MopApp My.SecondLife.com Mubi Muopia Neumodo Olapic OMAPpedia OnePlace Open Missouri NationalField.org Ninja Sports Noteleaf Nvidia's Developer Portal PandaBoard Pawesome Payday Loan Up Paymentez Peel PEG peopledock PowerCloud Pictorical piZap Postcrossing projectfedena projects.nytimes.com projects.propublica.org QuestionMoms Quora Qoop RaceBucket Racingjunk Radio MarketBeat Realmfpa.ru Receept RecordiaPro Reddit Repository Hosting Retickr Right Scale Rise Of Flight Rounds RoadFly Forum RSpec Documentation rstat.us Ruby Installer Saddleback Leather Scoopola ScriptFrenzy.org ScrumNinja ScrumPad SecretPoke Sencha Sendoid SendGig serifwebresources Shelfari ShortStack Shows for iPhone SindiQ SocialLamp Social Replay Sony Online Ent. Sohoos Someecards SonarApp Springpad SprocketList.com SpoofCard StarNews.in Start Wire STL Tweets streeteasy Street Fire SubMate Sunimi Survey Head Synclio Tanga Tapzilla TasteKid TeamLab TeamWork Live Tech Startup Center Tested textslide The MovieDB The MovieDB TrackStudio Trakt.tv TrapCall Typealoud Tzigla uFollow Usabilla Uweni VenueDriver.com Verkami Views.fm Vos Iz Neias Voxound vtravelled Vuze VYou Walkingspree Warener/Chapel Music Watchth.is WattPad Weddingbee Woorank Workout.io Worth1000 Wildfire XSplit Yabblr Yangutu YAYfriends! YLastic You Can Make This YouTrack Zaarly Zape ZKatter Zencoder
- 33. But Two Years Later…
- 35. Tainter’s Law
- 36. The Collapse of Complex Societies Joseph Tainter (Anthropologist/Historian)
- 37. So What? • As our platforms grow, scale, and evolve, they will develop innate complexity in an almost organic way • Redundancy adds additional complexity on top of that. • Too many layers of redundancy on top of an already complex system will ultimately lead to a decrease in reliability.
- 38. © Copyright Mshake3 and licensed for reuse under this Creative Commons Licence
- 39. So What Now? • Dr. Richard Cooke had it right. • At massive scale (and especially as we grow), the law of diminishing returns means our industry cannot build “reliable” systems. • But by embracing failure as a fundamental truth, we can design around it. • We need to stop focusing on reliable systems, and focus on resilient systems. • And we need to do that globally.
- 40. Let’s revisit our goals…
- 41. But if utility power is so reliable why do we need generators? We want to be as reliable as the power company. “UTILITY”
- 42. But if telcos are so reliable why do we need sat phones? We want to be as reliable as a telco. “TELCO”
- 43. We are not utilities. Utilities own the last mile. We own the first mile. It’s a different challenge to solve.
- 44. We shouldn’t be routing our “utilities” to the user. We need to route the users to the available “utilities.” It’s the internet, dummy!
- 45. In a properly designed global architecture, as long as ONE region is up, we have 100% availability.
- 46. What if we just solved cross-regional failover REALLY REALLY WELL ?
- 47. We can design resilient applications… That depend on ephemeral servers… That live in ephemeral datacenters (at half the cost)
- 48. NO HVAC
- 49. NO UPS
- 50. NO GENERATORS
- 51. NO NINES
- 52. FIVE ZEROS
- 53. + 1.00000
Editor's Notes
- http://commons.wikimedia.org/wiki/File:Squirrel_posing.jpg
- http://commons.wikimedia.org/wiki/File:Squirrel_posing.jpg
- http://commons.wikimedia.org/wiki/File:War_Of_The_Worlds_-_Plane_Crash_Set_(2562097108).jpg http://commons.wikimedia.org/wiki/File:ISS-11_Elektron_repair.jpg
- http://commons.wikimedia.org/wiki/File:Bali-IMG_4647.JPG I, the copyright holder of this work, release this work into the public domain. This applies worldwide. In some countries this may not be legally possible; if so: I grant anyone the right to use this work for any purpose, without any conditions, unless such conditions are required by law http://commons.wikimedia.org/wiki/File:Kitten-stare.jpg I, the copyright holder of this work, release this work into the public domain. This applies worldwide. In some countries this may not be legally possible; if so: I grant anyone the right to use this work for any purpose, without any conditions, unless such conditions are required by law
- This file is in the public domain because it was solely created by NASA. NASA copyright policy states that "NASA material is not protected by copyright unless noted".
- http://commons.wikimedia.org/wiki/File:Superhero_texting.jpg
- http://commons.wikimedia.org/wiki/File:Tacoma-narrows-bridge-collapse.jpg
- © Copyright Cohesion licensed for reuse under this Creative Commons Licence
- http://commons.wikimedia.org/wiki/File:Emergency_button,_And%C4%9Bl_metro_station,_line_B,_Prague,_Czech_Republic.jpg
- http://commons.wikimedia.org/wiki/File:HVAC_Ventilation_Exhaust.jpg
- http://commons.wikimedia.org/wiki/File:Hurricane_Wilma_200510212015.jpg This image is in the public domain because it contains materials that originally came from the U.S. National Oceanic and Atmospheric Administration, taken or made as part of an employee's official duties.
- http://commons.wikimedia.org/wiki/File:Photocopy_of_photograph._STEEL_PLANT,_1000-HORSEPOWER_CORLISS_STEAM_ENGINE_AND_FLYWHEEL_FOR_14-INCH_MILL,_1910._(From_the_Jefferson_County_Historical_society_Collection,_HAER_WASH,16-PORTO.V,1-36.tif This image or media file contains material based on a work of a National Park Service employee, created as part of that person's official duties. As a work of the U.S. federal government, such work is in the public domain. See the NPS website and NPS copyright policy for more information.
- http://commons.wikimedia.org/wiki/File:DAMAGE_CAUSED_BY_THE_EXPLOSION_OF_THE_CORLISS_ENGINE_FLYWHEEL._NO._2_ENGINE_HOUSE_(NO._7_MILL)._PHOTOCOPY_OF_1871_VIEW_LOOKING_WEST._From_the_collection_of_the_Manchester_Public_HABS_NH,6-MANCH,2-120.tif This image or media file contains material based on a work of a National Park Service employee, created as part of that person's official duties. As a work of the U.S. federal government, such work is in the public domain. See the NPS website and NPS copyright policy for more information.
- http://commons.wikimedia.org/wiki/File:SH-60B_helicopter_flies_over_Sendai.jpg This file is a work of a sailor or employee of the U.S. Navy, taken or made as part of that person's official duties. As a work of the U.S. federal government, the image is in the public domain.
- http://commons.wikimedia.org/wiki/File:Hokusai_Monster_Rat.jpg According to Japanese Copyright Law the copyright on this work has expired and is as such public domain. According to articles 51 and 57 of the copyright laws of Japan, under the jurisdiction of the Government of Japan all non-photographic works enter the public domain 50 years after the death of the creator (there being multiple creators, the creator who dies last) or 50 years after publication for anonymous or pseudonymous authors or for works whose copyright holder is an organization.
- http://commons.wikimedia.org/wiki/File:Cole_Thomas_The_Course_of_Empire_Destruction_1836.jpg This work is in the public domain in the United States, and those countries with a copyright term of life of the author plus 100 years or less.
- http://commons.wikimedia.org/wiki/File:Eugene_Dinsmore_Superhero_111007.jpg