Heroku's Ryan Smith at Waza 2013: Predictable Failure

837 views

Published on

Heroku's Ryan Smith took to the Waza 2013 stage to present "Predictable Failure" -- a look at how engineers can prepare for the inevitable. For more from Smith ping him at @ryandotsmith.

For Waza videos stay tuned at http://blog.heroku.com or visit http://vimeo.com/herokuwaza

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Heroku's Ryan Smith at Waza 2013: Predictable Failure

  1. 1. Predictable FailureBuilding systems that fail in predictable ways.
  2. 2. About● @ryandotsmith● Engineer at Heroku for about 3 years
  3. 3. Things I have worked on ● Heroku API ● Messages Queues ● Usage & Billing ● DNS ● Log delivery & visibility
  4. 4. Your system will failIt is important to comprehend the consequences
  5. 5. Critical Chain
  6. 6. What can you do?● Make your apps less critical.● Redundancy
  7. 7. No Redundancy - Simplefunc Query(c Conn, query string) Result { return c.DoQuery(query)}
  8. 8. Redundancy - Complexfunc Query(conns []Conn, query string) Result { ch := make(chan Result, len(conns)) for _, conn := range conns { go func(c Conn) { ch <- c.DoQuery(query): }(conn) } return <-ch}
  9. 9. How many redundancies?
  10. 10. A component that just works
  11. 11. Independence func add(x) {sys.1 return x + y// Bug! } func add(x) {sys.2 return x + y// Bug! }
  12. 12. Most famous bug ever
  13. 13. Same functionDifferent implementation
  14. 14. Dormant Failures
  15. 15. Dont wait until disaster strikes to find out thatYour secondary RDMS has a full disk
  16. 16. Propagation
  17. 17. Conclusion● Best if you dont have to build redundancy● Consider the pitfalls if you do ○ Infinite regress ○ Indipendence ○ Dormant Failures ○ Propagation● Dont be disillusioned 99.0 > 98.0

×