Thinking Like An Erlanger: Processes, Protocols, and Failure Handling
•
3 likes•2,477 views
This document discusses Erlang concepts and how to think like an Erlanger. It provides examples of modeling systems like Game of Life and a stock exchange using Erlang's processes, message passing, and failure handling capabilities. Key points include: spawning many short-lived processes to model discrete entities, using protocols to define interactions, monitoring processes to restart them after failures, and transaction logs to ensure consistency after crashes. The document emphasizes focusing on interactions through messaging and designing systems to fail gracefully through Erlang's primitives.
Recommended
More Related Content
What's hot
What's hot (9)
Similar to Thinking Like An Erlanger: Processes, Protocols, and Failure Handling
Similar to Thinking Like An Erlanger: Processes, Protocols, and Failure Handling (20)
Recently uploaded
Recently uploaded (20)
Thinking Like An Erlanger: Processes, Protocols, and Failure Handling
- 1. Thinking Like An Erlanger Torben Hoffmann CTO @ Erlang Solutions torben.hoffmann@erlang-solutions.com @LeHoff
- 2. Do You Want To See The Erlang Code?
- 3. You Can’t Handle The Erlang Code!! Source: http://static7.businessinsider.com/image/4d9d0c7bcadcbbaf20120000/jack-a-few-good-men.jpg
- 6. General vs Domain Specific Telecom Erlang C++/Java Smaller gap = money!
- 7. Protocols
- 8. Paxos { Acceptors } Proposer Main Aux Learner | | | | | | -- Phase 2 -- X----------->|->|->| | | Accept!(N,I,V) | | | ! | | --- FAIL! --- |<-----------X--X--------------->| Accepted(N,I,V) | | | | | -- Failure detected (only 2 accepted) -- X----------->|->|------->| | Accept!(N,I,V) (re-transmit, include Aux) |<-----------X--X--------X------>| Accepted(N,I,V) | | | | | -- Reconfigure : Quorum = 2 -- X----------->|->| | | Accept!(N,I+1,W) (Aux not participating) |<-----------X--X--------------->| Accepted(N,I+1,W) | | | | | Source: https://en.wikipedia.org/wiki/Paxos_(computer_science)#Byzantine_Paxos
- 9. The Golden Trinity Of Erlang
- 10. Processes They are dirt cheap! Use lots of them! Lots! Focus on their interactions
- 11. Game of Life Simple cellular automaton Evolution of cells Discrete time
- 12. 2,3 neighbours: survive empty, 3 neighbours: new cell all others: empty
- 19. Traditional Approach 2-d array - one per time step
- 20. Issues with Traditional Approach Does not scale well Imperative data structures are ugly in Erlang
- 21. Basic Erlang Idea One process per cell Talk to neighbour cells
- 22. Collect from all neighbours Update own content and go to T+1
- 23. Is this Erlangy enough?
- 24. Spawn a process to collect from all neighbours
- 25. collector_loop([], NeighbourCount, Cell, Content) -> Cell ! {self(), {next_content, next_content(Content, NeighbourCount)}}; collector_loop(WaitingOn, NeighbourCount, Cell, Content) -> receive {cell_content, {{{_,_},_}=XYatT, NeighbourContent}} -> case lists:member(XYatT, WaitingOn) of true -> collector_loop(lists:delete(XYatT, WaitingOn), NeighbourCount + NeighbourContent, Cell, Content); false -> %% ignore messages we are not waiting for collector_loop(WaitingOn, NeighbourCount, Cell, Content) end end.
- 26. Will this work?
- 27. It works if… you only take one step at a time let the cells run freely… cells get out of sync requests for old time requests for future time
- 28. Fixes Old time requests: keep history Future time requests: queue the response
- 30. Supervising Cells The Erlang supervisor will restart with the original args… all state and history is lost Fix: monitor all cells step them forward to current time
- 33. Cell Mgr Code loop(#state{cells=Cells}=State) -> receive {'DOWN', Ref, process, _Pid, _Info} -> {value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells), NewPid = await_restart(XY), NewRef = erlang:monitor(process, NewPid), NextState = State#state{ cells=[{NewPid, NewRef, XY}|Rest]}, spawn ( fun () -> pacer(NewPid) end ), loop(NextState) end. Process down
- 34. Cell Mgr Code loop(#state{cells=Cells}=State) -> receive {'DOWN', Ref, process, _Pid, _Info} -> {value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells), NewPid = await_restart(XY), NewRef = erlang:monitor(process, NewPid), NextState = State#state{ cells=[{NewPid, NewRef, XY}|Rest]}, spawn ( fun () -> pacer(NewPid) end ), loop(NextState) end. Wait for supervisor to restart it
- 35. Cell Mgr Code loop(#state{cells=Cells}=State) -> receive {'DOWN', Ref, process, _Pid, _Info} -> {value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells), NewPid = await_restart(XY), NewRef = erlang:monitor(process, NewPid), NextState = State#state{ cells=[{NewPid, NewRef, XY}|Rest]}, spawn ( fun () -> pacer(NewPid) end ), loop(NextState) end. Get the new process up-to-date
- 36. Deadlock Remaining… N1 dies with Cell(i,j) queued
- 37. Re-cap A process per cell Short-lived processes for small tasks Focus on the protocols between processes Supervisor to restart Monitoring manager process to get restarted cells up-to-speed
- 38. Stock Exchange
- 39. The Trigger… Erlang-Questions on using ETS for sell and buy orders: http://erlang.org/pipermail/erlang- questions/2014-February/077969.html Painful…
- 40. An Exchange Connects buyers and sellers Buyers post buy intentions Sellers post sell intentions
- 41. Basic Erlang Idea One process per buy/sell intention Processes to negotiate deals by exchanging messages
- 42. Communication Use gproc as pub-sub mechanism to announce buy and sell intentions All buyers listen to sell intention All sellers listen to buy intentions
- 43. Can happen when Negotiation by 3-way handshake Deals priceseller pricebuyer
- 44. Buyer Arrives
- 46. Unique reference to identify the sell offer Seller’s Pid
- 49. 5 pt
- 50. Seller Arrives
- 53. What Can Go Wrong? 1. Buyer dies 3. Buyer dies 2. Seller dies 1 & 2 can be fixed by timing out Danger!! Seller has closed the deal on his side Simple re-start leaves the buyer at 3@5
- 54. Monitor each other Removes the need for timeouts Still not sure how far the other side got
- 55. Transaction Log Per Process Just replay back to the last state Issues: Messages cannot be replayed Must ask partner about their view on the status of the deal
- 56. Ledger Create Ledger process that tracks all completed deals Each buyer and seller get a unique OfferID when started
- 58. Thinking in Erlang Focus on protocols (MSCs) Ask “What could go wrong here?”
- 59. Tools Lots of processes!! Spawn short-lived processes for small things Supervisors Link and monitor gproc for registering and pub/sub Timeouts Transaction logs (ledgers)
- 60. Food for Thought What can I only do in Erlang? http://erlang.org/pipermail/erlang- questions/2014-November/ 081570.html You can avoid writing your own service framework. Craig Everett
- 61. Testing Async protocols are nasty Use EQC - Property Based Testing Focus on one process Mock the calls to others WIP in repo!
- 62. Code Game of life https://github.com/lehoff/egol Erlang Exchange https://github.com/lehoff/ erlang_exchange Deadlock left to fix Testing incomplete No ledger
- 63. Learning Erlang
- 65. Learning Erlang ESL training courses Learn You Some Erlang http://learnyousomeerlang.com/ Use the erlang-questions mailing list Do it hands-on Give it time to sink in!!!
- 66. Elixir Built on top of the Erlang VM More Ruby-like syntax Hygienic macros - easy to do DSLs Better support for data handling But… you still have to learn the Erlang programming model
- 67. Key building blocks Share nothing processes Message passing Fail fast approach Link/monitor concept You can deal with failures in a sensible manner because you have a language for them.