Download as PDF, PPTX
Uploaded byAtlassian
5 Essential Techniques for Building Fault-tolerant Systems
The document outlines five essential techniques for building fault-tolerant systems: containment, fail fast, escape, adjust, and learn. It emphasizes the importance of anticipating faults, implementing measures to minimize their impact, and fostering a culture of continuous improvement through monitoring and learning from incidents. These strategies aim to ensure system resilience and maintain availability even in the face of component failures.
More Related Content
Similar to 5 Essential Techniques for Building Fault-tolerant Systems
5 Essential Techniques for Building Fault-tolerant Systems
- 1. 5 Essential Techniquesfor Building Fault-tolerant Systems DIEGO BERRUETA | ENGINEERING PRINCIPAL | ATLASSIAN
- 2. Preston Rhea; Flickr(www.flickr.com/photos/prestonrhea/), CC-by
- 3.
- 5.
- 6.
- 7.
- 8.
- 9. Fault-tolerant system Continues tooperate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 10. Fault-tolerant system Continues tooperate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 11. Fault-tolerant system Continues tooperate in the event of faults in some of its components Fault Deviation from the normal state, either due to internal or external causes Failure Observable impact of a fault in a system, usually manifests as reduced availability
- 12. Bob Yeats (OSU);Flickr (www.flickr.com/photos/oregonstateuniversity/), CC-by Faults happen Preventing them may be technically or economically impractical
- 13. Systems can bedesigned and built to tolerate faults FAULT TOLERANCE
- 14. STABLE SYSTEM UNSTABLESYSTEM
- 15.
- 16. Contain the fault Usingphysical or logical barriers
- 17. Norlando Pobre; Flickr(www.flickr.com/photos/npobre/), CC-by
- 18. Shawn O’Neil; Flickr(www.flickr.com/photos/oneilsh/), CC-by
- 19. Pool separation Different poolsfor each task or dependency Brian Cantoni; Flickr (www.flickr.com/photos/cantoni/), CC-by
- 20. Asynchronous communication Sender andreceiver fail independently, receiver can catch up later Synchronous communication Propagates failures, context may be lost
- 21.
- 22. CLIENT ISOLATION NEW CLIENT CLIENTA CLIENT B
- 23. Avoid SPOF Find thecomponents which compromise the system How to contain faults Invest in redundancy Improve availability by having more than one of everything Build bulkheads Set up logic walls to reduce the blast radius
- 24.
- 25. A quick erroris better than a slow response FAIL FAST PRINCIPLE
- 26. Validate early Anticipate problemsand change course Carl Wycoff; Wikimedia Commons, CC-by
- 27. Reject politely Fast refusalis better than slow error Taber A.B.; Flickr (www.flickr.com/photos/andrewbain/), CC-by
- 28. Never wait long Protectlong tasks with timeouts Robert C-B.; Flickr (www.flickr.com/photos/29233640@N07/), CC-by
- 29. Watch out for slowness In-processlocks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 30. Watch out for slowness In-processlocks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 31. Watch out for slowness In-processlocks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 32. Watch out for slowness In-processlocks Database queries Sockets Remote APIs Albert Herring, Wikimedia Commons, CC-by
- 33. Fail fast: inpractice
- 34.
- 35. Decline service When overloaded,ask clients to come back later How to fail fast Never wait long Set a timeout for blocking calls and slow operations Validate early Avoid starting something that cannot be completed
- 36.
- 37. Circuit breakers Separate thefailing parts
- 38.
- 39. Fault-tolerance libraries Circuit breaking Avoidcascading failures during periods of turbulence Monitoring and alerting Observe the behaviour of all your dependencies Timeouts Time-bound any operation Fall-back Recover using an alternative path
- 40.
- 41. SERVICE EVOLUTION TENANT ATENANTA TENANT ATENANT B TENANT ATENANT C
- 42. Anticipate failure If itis not going to work, do not even try How to escape Degrade gracefully A cached result or a default value may be an alternative Detect problems Compare all interactions against error thresholds
- 43.
- 44. Communication Share initial expectations and status updates
- 45. Adjust Page size Growth Flow Retry Paginate queries Avoidunbounded database and remote queries
- 46. Adjust Page size Growth Flow Retry Beware ofthings that grow Clean up old data and set size limits
- 47. Adjust Page size Growth Flow Retry Negotiate speed Adjustdynamically using rate limits, request throttling, buffering and autoscaling
- 48. Adjust Page size Growth Flow Retry Insist smartly Retrytransient errors with timeouts and back-off
- 49.
- 50.
- 51. Insist smartly Transient errorscan be retried with back-off How to adjust Report availability Apply back pressure to prevent congestion Negotiate size Limit the cost of the job
- 52.
- 53. Monitor the behaviour Definemetrics, observe trends, set up alerts, capture logs Learn from experience Observe Test Reflect
- 54. Test with chaos Verifyyour hypothesis about fault-tolerance by continuously introducing chaos Learn from experience Observe Test Reflect
- 55. Learn from mistakes Eachincident is an opportunity to make the system more robust Learn from experience Observe Test Reflect
- 56.
- 57.
- 58.
- 59. Monitor and alert Understandthe behaviour of the system in production How to learn Reflect on incidents Analyse the root cause and prevent recurrences Test what if…? Deliberately introduce chaos to assess fault-tolerance
- 60.
- 61. Hope is nota strategy Test, observe, reflect Life starts after releasing “Code complete” is not “production ready” Be cynical Do not trust anybodyRobustness is an attitude
- 62. Some disasters can beprevented Build and test with failure in mind Faults are unavoidable Any possible fault will eventually happen
- 63. Leo Hidalgo; Flickr(www.flickr.com/photos/ileohidalgo/), CC-by
- 64. Release it! Michael Nygard SiteReliability Engineering Google References
- 65. Thank you! DIEGO BERRUETA| ENGINEERING PRINCIPAL | ATLASSIAN