Heroku's Ryan Smith took to the Waza 2013 stage to present "Predictable Failure" -- a look at how engineers can prepare for the inevitable. For more from Smith ping him at @ryandotsmith. For Waza videos stay tuned at http://blog.heroku.com or visit http://vimeo.com/herokuwaza

1. Predictable Failure
Building systems that fail in predictable ways.

2. About
● @ryandotsmith
● Engineer at Heroku for about 3 years

3. Things I have worked on
● Heroku API
● Messages Queues
● Usage & Billing
● DNS
● Log delivery & visibility

7. Your system will fail
It is important to comprehend the consequences

8. Critical Chain

9. What can you do?
● Make your apps less critical.
● Redundancy

10. No Redundancy - Simple
func Query(c Conn, query string) Result {
return c.DoQuery(query)
}

11. Redundancy - Complex
func Query(conns []Conn, query string) Result {
ch := make(chan Result, len(conns))
for _, conn := range conns {
go func(c Conn) {
ch <- c.DoQuery(query):
}(conn)
}
return <-ch
}

12. How many redundancies?

13. A component that just works

14. Independence
func add(x) {
sys.1 return x + y// Bug!
}
func add(x) {
sys.2 return x + y// Bug!
}

15. Most famous bug ever

16. Same function
Different implementation

17. Dormant Failures

18. Don't wait until disaster strikes to find out that
Your secondary RDMS has a full disk

19. Propagation

20. Conclusion
● Best if you don't have to build redundancy
● Consider the pitfalls if you do
○ Infinite regress
○ Indipendence
○ Dormant Failures
○ Propagation
● Don't be disillusioned 99.0 > 98.0