Downloaded 29 times
Uploaded byBrian Troutwine
Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over
The document discusses fault-tolerance in real-time and networked systems, outlining three strategies: striving for perfection, hoping for the best, and embracing faults. Each option presents varying degrees of control and understanding of the problem domain, emphasizing the need for system design that anticipates and manages failures. Additionally, it highlights the significance of organization, process flexibility, and understanding network characteristics in building resilient systems.
More Related Content
Similar to Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over
Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over
- 1. Fault-ToleranceOnTheCheapMaking systems thatprobably won’t fall over.
- 2.
- 3.
- 4. I’m a real-time, networked systemsengineer.
- 5.
- 6. Real-Time Systems• Computationon a deadline
- 7. Real-Time Systems• Computationon a deadline • Fail-safe / Fail-operational
- 8. Real-Time Systems• Computationon a deadline • Fail-safe / Fail-operational • Guaranteed response / Best effort
- 9. Real-Time Systems• Computationon a deadline • Fail-safe / Fail-operational • Guaranteed response / Best effort • Resource adequate / inadequate
- 10.
- 11. Networked Systems• Out-of-ordermessages
- 12. Networked Systems• Out-of-ordermessages • No legitimate concept of “now”
- 13. Networked Systems• Out-of-ordermessages • No legitimate concept of “now” • High-latency transmission
- 14. Networked Systems• Out-of-ordermessages • No legitimate concept of “now” • High-latency transmission • Lossy transmission
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. • Total controlover the whole mechanism. Option 1: Perfection
- 28. • Total controlover the whole mechanism. • Total understanding of the problem domain. Option 1: Perfection
- 29. • Total controlover the whole mechanism. • Total understanding of the problem domain. • Specific, explicit system goals. Option 1: Perfection
- 30. • Total controlover the whole mechanism. • Total understanding of the problem domain. • Specific, explicit system goals. • Well-known service lifetime. Option 1: Perfection
- 31. Option 1: Perfection “TheyWrite the Right Stuff” Fast Company, 2005
- 32. • Extremely expensive. Option1: Perfection
- 33. • Extremely expensive. •Intentionally stifles creativity. Option 1: Perfection
- 34. • Extremely expensive. •Intentionally stifles creativity. • Design up front. Option 1: Perfection
- 35. • Extremely expensive. •Intentionally stifles creativity. • Design up front. • Complete control of the system is not complete. Option 1: Perfection
- 36. Option 2: Hope forthe Best
- 37. • Little up-frontknowledge of the problem domain. Option 2: Hope for the Best
- 38. • Little up-frontknowledge of the problem domain. • Implicit or short-term system goals. Option 2: Hope for the Best
- 39. • Little up-frontknowledge of the problem domain. • Implicit or short-term system goals. • No money down. Option 2: Hope for the Best
- 40. • Little up-frontknowledge of the problem domain. • Implicit or short-term system goals. • No money down. • Ingenuity under pressure. Option 2: Hope for the Best
- 41. Option 2: Hopefor the Best “Move fast and break things!”
- 42. • Ignorance ofproblem domain leads to long-term system issues. Option 2: Hope for the Best
- 43. • Ignorance ofproblem domain leads to long-term system issues. • Failures do propagate out toward users. Option 2: Hope for the Best
- 44. • Ignorance ofproblem domain leads to long-term system issues. • Failures do propagate out toward users. • No, money down! Option 2: Hope for the Best
- 45. • Ignorance ofproblem domain leads to long-term system issues. • Failures do propagate out toward users. • No, money down! • Hard to change cultural values. Option 2: Hope for the Best
- 46.
- 47. Option 3: EmbraceFaults • Partial control over the whole mechanism.
- 48. Option 3: EmbraceFaults • Partial control over the whole mechanism. • Partial understanding of the problem domain.
- 49. Option 3: EmbraceFaults • Partial control over the whole mechanism. • Partial understanding of the problem domain. • Sorta explicit system goals.
- 50. Option 3: EmbraceFaults • Partial control over the whole mechanism. • Partial understanding of the problem domain. • Sorta explicit system goals. • Able to spot a failure when you see one.
- 51. Option 3: EmbraceFaults “Fail fast. Either do the right thing or stop.” “Why Do Computers Stop and What Can Be Done About it?”, Jim Gray, 1985 (paraphrase)
- 52. Option 3: EmbraceFaults • Faults are isolated but must be resolved in production.
- 53. Option 3: EmbraceFaults • Faults are isolated but must be resolved in production. • Must carefully design for introspection.
- 54. Option 3: EmbraceFaults • Faults are isolated but must be resolved in production. • Must carefully design for introspection. • Moderate design up-front.
- 55. Option 3: EmbraceFaults • Faults are isolated but must be resolved in production. • Must carefully design for introspection. • Moderate design up-front. • Pay a little now, pay a little later.
- 56.
- 57. There are four conceptualstages to consider.
- 58.
- 59. The most atomic level ofthe system.
- 60. Progress here has anoutsized impact.
- 61.
- 62.
- 63.
- 64. Why test, when youcan prove?
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78. A finely builtmachine without a supporting organization is a disaster waiting to happen.
- 79. A finely builtmachine without organization is a disaster waiting to happen. Chernobyl STS-51-L Deepwater Horizon Magnitogorsk Damascus Incident Chevron Refinery BART ATC Asiana #214 Therac-25 New Orleans Levee
- 80. Correct the conditions thatallowed mistakes, as well as the mistake.
- 81.
- 82. Build flexible tools forexperts.
- 83.
- 84.
- 85.
- 86.
- 87. Every system carries the potentialfor its own destruction.
- 88. Some things aren’t worthbuilding.
- 89.
- 90. 0. The networkis unreliable. 1. Latency is non-zero. 2. Bandwidth is finite. 3. The network is insecure. 4. Topology changes. 5. There are many administrators. 6. Transport cost is non-zero. 7. The network is heterogenous.
- 91.
- 92. Recommended Reading“Normal Accidents:Living with High-Risk Technologies”, Charles Perrow “Digital Apollo: Human and Machine in Spaceflight”, David A. Mindel “Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety”, Eric Schlosser “Erlang Programming”, Simon Thompson and Francesco Cesarini “Steeltown, USSR”, Stephen Kotkin “Crash-Only Software”, George Candea and Armando Fox “The Truth About Chernobyl”, Grigorii Medvedev “Real-Time Systems: Design Principles for Distributed Embedded Applications”, Hermann Kopetz “ Th e Ap o l l o G u i d a n c e C o m p u t e r : Architecture and Operation”, Frank O’Brien “Why Do Computers Stop and What Can Be Done About It?”, Jim Gray “Thirteen: The Apollo Flight That Failed”, Henry S.F. Cooper Jr.