Downloaded 19 times
Uploaded byMahesh Paolini-Subramanya
Erlang - Because S**t Happens
The document discusses the Erlang programming language and its focus on fault tolerance. It summarizes the "Big Six" principles of Erlang - concurrency, fault detection, fault identification, error encapsulation, code upgrade, and stable storage. It explains how Erlang implements processes that can crash and restart independently, with supervision to restart failed processes when needed. The principles of loose coupling and monitoring are also discussed in the context of building fault-tolerant systems and organizations.
More Related Content
Similar to Erlang - Because S**t Happens
Erlang - Because S**t Happens
- 1. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Erlang : Because S**t happensMahesh Paolini-Subramanya (@dieswaytoofast) V.P. R&D, Ubiquiti Networks
- 3.
- 4. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Horror! The Horror!
- 5. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Dude! WTF?!?!
- 6. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Surely you Tested?
- 8. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Fault Tolerance From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 9. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 10. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 11. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 12. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 13. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 14. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 15. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens erlang…
- 16. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 17. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 18. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Parallel From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 19. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Parallel Independent From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 20. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Parallel Independent Asynchronous message passing From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 21. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Parallel Independent Asynchronous message passing No shared state From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 22. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Parallel Independent Asynchronous message passing No shared state Immutable data structures From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 23. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 24. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 25. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens ** exception error: no match of right hand side value {error,unhappy}
- 26. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 27. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 28. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens ** exception error: no match of right hand side value {error,unhappy}
- 29. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 30. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 32. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Let it crash From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 33. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Let it crash (all the co-operating ones , that is) From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 34. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Let it crash (all the co-operating ones , that is) Supervisors can restart them From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 35. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Processes! Let it crash (all the co-operating ones , that is) Supervisors can restart them (When necessary!) From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 36. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 38. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 39. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 41. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens mnesia ‘n stuff
- 43. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Fault Tolerant System
- 44. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 45. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 46. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf LOOSECOUPLING
- 47. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Loose Coupling
- 48. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Loose Coupling Breeds Trust
- 49. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 50. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Loose Coupling Breeds Trust Devotes Brainpower appropriately
- 51. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 52. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Loose Coupling Breeds Trust Devotes Brainpower appropriately Minimizes bugs
- 54. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 55. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Fault Tolerance 60 - 90% of all SW projects fail 10 – 25% of all SW projects get abandoned
- 56. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 57. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Systems Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage
- 58. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Systems Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage M ONITORING
- 59. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Monitoring?
- 60. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Monitoring? Dashboards
- 61. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Monitoring? Dashboards Out of band systems
- 62. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Monitoring? Dashboards Out of band systems Polyglot safety
- 63. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Systems Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage
- 64. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Systems Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage POLYGLOT PERSISTENCE
- 68. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens (The day it all changed) Friday, June 29th 2012
- 69. • People • Vendors •Fraud Fault Tolerance
- 70. The Business Beware theBlack Swan
- 71. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Fault Tolerant Organization
- 72. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 73. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Concurrency The Big Six - Organizations
- 74. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 75. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 76. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens
- 77. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 78. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Concurrency Error encapsulation Fault detection The Big Six - Organizations
- 79. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six Concurrency Fault detection Fault Identification Error Encapsulation From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 81. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Organizations Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 82. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Concurrency Error encapsulation Fault detection Fault identification Code upgrade The Big Six - Organizations
- 83. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens The Big Six - Organizations Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 84. • Concurrency • Errorencapsulation • Fault detection • Fault identification • Code upgrade • Stable Storage The Big Six - Organizations
- 85.
- 86. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage EVERYW HERE!!!
- 87. V 1.0 ©Ubiquiti Networks, Inc. All Rights Reserved Erlang – because s**t happens Questions mahesh@dieswaytoofast.com @dieswaytoofast
Editor's Notes
- #2 An overall approach to Preparedness
- #3 The Buddha nature of erlang – Fault Tolerance
- #4 This is a story about unexpectedness.The only constant is change
- #5 We’ve all been confronted by this
- #6 Some of us have been confronted by this
- #7 And you always get asked this
- #8 There is only so much planning you can do. At some point, the 1000 year flood hits
- #9 The point being – shit happens, how are you going to deal with it?
- #10 Do more than one thing at a time
- #11 Know when something breaks
- #12 Know what broke
- #13 Don’t let it spread
- #14 Fix it
- #15 ‘save game’ so you can do-over
- #16 Butwe know this.
- #17 Lets take this one at a time
- #18 The Six Essential Characteristics of a Fault Tolerant System
- #19 Do more than one thing at a time
- #20 Do more than one thing at a time
- #21 Do more than one thing at a time
- #22 Do more than one thing at a time
- #23 Do more than one thing at a time
- #24 The Six Essential Characteristics of a Fault Tolerant System
- #25 Only one success case here
- #26 Fail fast and fail stop
- #27 Only one success case here
- #28 The Six Essential Characteristics of a Fault Tolerant System
- #29 The process tells everyone what happenedStack traces are your friend.Its Not Java!
- #30 Only one success case here
- #31 The Six Essential Characteristics of a Fault Tolerant System
- #32 Let it Crash
- #33 Do more than one thing at a time
- #34 Do more than one thing at a time
- #35 Do more than one thing at a time
- #36 Do more than one thing at a time
- #37 The Six Essential Characteristics of a Fault Tolerant System
- #38 Go get catfood
- #39 Let it Crash
- #40 The Six Essential Characteristics of a Fault Tolerant System
- #41 The back up bag
- #42 Processes are units of error encapsulation(and good for GC too!)
- #43 Fault Tolerance and SystemsThink of this FORMALLY
- #44 The bottom line – can your organization deal with the above issues?
- #45 The Six Essential Characteristics of a Fault Tolerant System
- #46 The Six Essential Characteristics of a Fault Tolerant System
- #47 The Six Essential Characteristics of a Fault Tolerant System
- #48 Loose Coupling, of course, gives us all these benefits
- #49 Loose Coupling, of course, gives us all these benefits
- #50 Builds trust Trust in the stupidity of people, trust that things will fail, trust that you will be affected
- #51 Loose Coupling, of course, gives us all these benefits
- #52 The amount of brainpower we have is limited.Reduce complexity by being able to focus on specific / limited areas
- #53 Loose Coupling, of course, gives us all these benefits
- #54 Andrew!
- #55 Isn’t Performance an issue w/ Loose Coupling?
- #56 remember the bit about failure? well, why optimize if you're going to fail anyhow? yeah yeah, you might fail because you don't perform, but that is rarely the problem
- #57 Work / Elegant / Fast;yes, that mine craft plugin you built might gt a million signupsit won’tseriously – it doesn't register statistically
- #58 Fault detection and identification?
- #59 Monitoring!
- #61 DashboardsOtherwise, how do you know whats going on?
- #62 Out of band access Don’t rely on the system to always tell you whats happening
- #63 Be PolyglotEverything fails – even erlang. (noooo)
- #64 Loose Coupling, of course, gives us all these benefits
- #65 Loose Coupling, of course, gives us all these benefits
- #66 Don’t cross the streams!
- #67 Fault Tolerance and OrganizationsThink of this FORMALLY
- #68 People fall ill
- #69 AWS will go down
- #70 CFOs run off to brazil
- #71 Tail Risk (Things that can never happen)This deserves its own section(financial crisis)
- #72 The bottom line – can your organization deal with the above issues?
- #73 The Six Essential Characteristics of a Fault Tolerant Organization
- #74 Easy right?Things can happen in parallel?
- #75 you'd be surprised at how poorly this gets done micromanagement (supervision gone bad) chaining (good for AR/AP, bad for decision making) e.g. – memos that get passed around for approval
- #77 “Bob is ill” Ahrens Fox!
- #78 "Bob was working on the financial projections. He has the flu, so won't be able to get to it for another week. And that means that we won't know what to buy for two weeks…”
- #79 (Ahrens Fox) "Bob was working on the financial projections. He has the flu, so won't be able to get to it for another week. And that means that we won't know what to buy for two weeks…”
- #80 The Six Essential Characteristics of a Fault Tolerant Organization
- #81 If bob is ill, will you survive
- #82 The Six Essential Characteristics of a Fault Tolerant Organization
- #83 Onboarding people is importantThis is not what it should feel like for Bob’s replacement
- #84 The Six Essential Characteristics of a Fault Tolerant Organization
- #85 Documentation, policies, corporate knowledge, corporate culture methods, access codes, Try to have work/knowledge distributed so that if/when you need to let someone go, things can continue till you get someone new!)
- #86 Ask yourself this. Over and over again…
- #87 Loose Coupling, of course, gives us all these benefits