6. Jesse Robbins CloudCamp 5minute Presentation
•
0 likes•180 views
The document discusses system failures and how failure is inevitable. It notes that as connectivity and dependence on technology increases, the potential for failures that impact many users also increases. Even with redundancies, all systems will fail eventually so providers and services must be prepared to quickly recover from inevitable outages and communicate effectively with users.
![Failure Happens ,[object Object],[email_address]](https://image.slidesharecdn.com/6-jesserobbins-cloudcamp-5minute-presentation-090522025336-phpapp01/85/6-Jesse-Robbins-CloudCamp-5minute-Presentation-1-320.jpg)
More Related Content
Viewers also liked
Viewers also liked (18)
More from Intel Corporation
More from Intel Corporation (20)
Recently uploaded
Recently uploaded (20)
6. Jesse Robbins CloudCamp 5minute Presentation
- 2. This will be on the test: FAILURE HAPPENS! FAILURE HAPPENS! FAILURE HAPPENS!
- 5. Good Book!
- 6. “ multiple and unexpected interactions of failures are inevitable” -Charles Perrow
- 9. Internet Routing... won’t .
- 14. #googlefail
- 15. YOU
- 16. Continuous Power... isn’t
- 17. 365 Main SF
- 18. 365 364.96 Main SF
- 22. Truck 1, Rackspace 0
- 23. Geography is a Single Point of Failure Single Point of Failure
- 30. Providers are baskets too.
- 33. Failure Happens. Anyone promising otherwise is either foolish or lying (or both). (or both). (or both). (or both).
- 34. Jesse Robbins [email_address] [email_address] June 22-24, 2009 Go Here!
Editor's Notes
- firefighters are usually considered to be about 75% paranoid and about 25% pyromaniac.
- Which means this sort of thing makes perfect sense to me at the time.
- The 365main site does not have a typical battery backup system. Instead they rely on Continuous Power Supplies (CPS) which use a flywheel driven alternator to generate electricity. The flywheel is connected to both a large diesel motor and an electric motor which runs on utility power. The flywheel is normally turned by the electric motor, and stores enough kinetic energy to power the alternator for up to 15 seconds. When utility power fails the diesel motor is supposed to start in under 5 seconds, well before the flywheel's kinetic energy is exhausted, providing uninterrupted electrical power.The advantage of a CPS over a battery-based system is that the power going to the datacenter is decoupled from the utility power. This eliminates the complex electrical switching required from most battery-based systems, making many CPS systems simpler and sometimes more reliable.
- In this incident, latent defects caused three generators to fail during start-up. No customers were affected until a fourth generator failed 30 seconds later, which overloaded the surviving backup system and caused power failures to 3 of 8 customer areas.What's most interesting is that the redundant design of the system is what caused it to fail so completely. The failure of the fourth generator should have only brought down one area instead of three. This kind of cascade failure is common in complex & tightly coupled systems. In my experience, these sorts of failure-modes are often identified and then promptly dismissed as being \"nearly impossible\". Unfortunately, the impossible often becomes reality.To put it another way... Failure Happens.
- Hurricane Katrina landed, and like many people I wanted to help.