Massively Scalable, Massively Fault Tolerant

  • 940 views
Uploaded on

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
940
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
32
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. MASSIVELY SCALEABLEMASSIVELY FAULT-TOLERANT Trotter Cashion Philly Lambda, 2011
  • 2. ErlangBUT FIRST... AND OTP IN ACTION Buy this Book!http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • 3. ErlangNO SERIOUSLY... IN ACTION AND OTP http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • 4. ErlangAVAILABLE IN PDF... AND OTP IN ACTIONhttp://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • 5. AND NOW FOR ME...
  • 6. CO-FOUNDER
  • 7. @CASHION http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg
  • 8. github.com/trotter
  • 9. CHLOE• Realtime Web Server• Proxies WebSocket (and fallback) connections to your app• http://github.com/mashion/chloe• Written in Erlang/OTP (plus a lot of JavaScript)
  • 10. BAI PHILLY! Leaving in July... http://www.flickr.com/photos/dennis/156048151/
  • 11. HAI CALIFORNIA! See you in August... http://www.flickr.com/photos/salim/402618628/
  • 12. AN ERLANG PRIMER Yoinked from the Erlang Movie. Go Watch It!!!!
  • 13. THE BASICS• Functional• Single Assignment Variables• Pattern Matching is Big Here• Everything’s a Process• One Module Per File
  • 14. FUNCTIONALsay_hello() ->  io:fwrite("Ohai!n").
  • 15. ANONYMOUS FUNCTIONSmake_hello() ->  fun () -> io:fwrite("Ohai!n")end.Hello = make_hello().Hello().
  • 16. SINGLE ASSIGNMENT VARS1> Hello = 3.32> Hello = 2.** exception error: no match of right hand side value 23> f().ok4> Hello = 2.2
  • 17. OH NOES! WHAT WENT WRONG?destructuring_bind() ->  [{Name, Title}, _] = [{"Trotter", "Speaker"},                        {"Aaron", "Organizer"}],  io:fwrite("~pn", [Name]), %% => Trotter  io:fwrite("~pn", [Title]). %% => Speaker
  • 18. PATTERN MATCHING Erlang Rubymatch_me(1) -> def match_me(val)  loneliest;   case valmatch_me(2) ->   when 1  next_to_loneliest;     :loneliestmatch_me(friend) ->   when 2  all_ok;     :next_to_loneliestmatch_me(_) ->   when :friend  what_is_this.     :all_ok   else     :what_is_this   end end
  • 19. MAKING A PROCESS process.erlconcurrent_stuffs() ->  spawn(fun () -> timer:sleep(1000),                  io:fwrite("So slown") end),  spawn(fun () -> io:fwrite("Super fastn") end). erlang shell5> process:concurrent_stuffs().Super fast<0.39.0>So slow
  • 20. RECEIVE LOOPSfood_loop() ->  receive    pizza -> io:fwrite("lets eatn"),             food_loop();    death -> io:fwrite("all done heren");    _ -> io:fwrite("Nothing to don"),             food_loop()  end.
  • 21. SENDING MESSAGESmake_process() ->  spawn(?MODULE, food_loop, []).pizza_death() ->  Proc = make_process(),  Proc ! pizza,  Proc ! hi,  Proc ! death,  %% Goes into the void  Proc ! huh.
  • 22. HOW THAT WORKS process 1pizza_death
  • 23. HOW THAT WORKS process 1 spawn/3pizza_death
  • 24. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop
  • 25. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
  • 26. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
  • 27. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza
  • 28. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi
  • 29. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death
  • 30. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 31. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 32. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 33. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
  • 34. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
  • 35. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 36. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
  • 37. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
  • 38. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 39. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
  • 40. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
  • 41. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 42. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • 43. HOW THAT WORKS process 1 spawn/3pizza_death food_loop’s Mailbox pizza hi death huh
  • 44. HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death huh
  • 45. HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death
  • 46. BUT DOES IT SCALE? Hells yea! http://www.flickr.com/photos/24736216@N07/4808606178/
  • 47. TWO WAYS TO SCALE Name ImplementationHorizontally Buy a lot of machines Vertically Buy a really big machine
  • 48. ERLANG SCALES WELL ON BOTHWITH NEARLY ZERO CODE CHANGE
  • 49. BOOM!
  • 50. VERTICAL SCALING• Your processes will take advantage of all cores available• Yay! No code changes required
  • 51. HORIZONTAL SCALING• New nodes added to the cluster can be auto-detected• Code must know that a service could live on another node• You’ll need resource discovery
  • 52. CONNECTING NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: []3> io:fwrite("Ping? ~p~n", [net_adm:ping(AHost)]).Ping? pong4> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: [a@awesome-town.local5> q().
  • 53. SERVER REFERENCES Type Definition <Pid> globally unique process identifier RegisteredName locally registered process process locally registered on{RegisteredName, Node} another node {global, GlobalName} globally registered process
  • 54. LISTEN FOR INCOMING MESSAGElisten() ->  register(listener, self()),  listen_loop().listen_loop() ->  receive    {From, pizza} -> From ! "Nom noms", listen_loop();    {From, _} -> From ! "Boo!", listen_loop();    _ -> io:fwrite("Listen is dying!")  end.
  • 55. SENDING MESSAGE TO NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> {listener, AHost} ! {self(), pizza}.ok3> receive PizzaResp -> PizzaResp end.
  • 56. RESOURCE DISCOVERY Method Purpose add_target_resource(Type) specify a resource type you want to useadd_local_resource(Type, Pid) specify a resource type you have fetch_resources(Type) fetch all resources of specified type trade_resources() trade resource information See Chapter 8 of Erlang And OTP In Action
  • 57. HORIZONTAL CAVEATS• Passing large amounts of data between nodes is inefficient• Too many nodes will impact performance (Keep it < 100)• There are ways around both these problems
  • 58. OTP BASICSCause managing all the processes by hand is dumb Thanks, Homer
  • 59. OTPWTFBBQ?• Open Telecom Platform• Abstracts common systems behavior (servers, supervisors, ...)• Easier than pure Erlang!• Gives Erlang its reputation for fault tolerance
  • 60. BASIC CONSTRUCTSConstruct Purpose Encapsulates a large set of codeApplication meant to run as one unit.Supervisor Starts and restarts workers. Worker Actually does stuff
  • 61. HTTP://GITHUB.COM/MASHION/CHLOE Chloe (Application) chloe_sup (Supervisor) websocket_sup session_sup session_manager Other Stuffs (Supervisor) (Supervisor) (Worker)
  • 62. WORKERS• Use the `gen_server` behaviour (for the most part)• Contain your application logic• You define a module with callbacks• OTP runs a server calls back to module when messages arrive
  • 63. HOW WORKERS WORK client process (your code) gen_server functions gen_server code (part of OTP) gen_server callbacks worker code (your code) worker process
  • 64. SENDING MESSAGES TO WORKERSsay_hello_sync(Name) ->  gen_server:call(?SERVER, {say_hello, Name}).say_hello_async() ->  gen_server:cast(?SERVER, say_hello). gen_server_examples.erl
  • 65. EXAMPLE CALLBACKShandle_call({say_hello, Name}, _From, State) ->  Response = string:concat("Hello, ", Name),  {reply, {ok, Response}, State}.handle_cast(say_hello, State) ->  io:fwrite("Hello world!n"),  {noreply, State}. gen_server_examples.erl
  • 66. WORKER CALLBACKS Callback Purpose init(Args) Initialize the server processhandle_call(Request, From, State) Handle a synchronous call handle_cast(Request, State) Handle an asynchronous cast handle_info(Info, State) For timeouts and random messages terminate(Reason, State) Clean up the process when it diescode_change(OldVsn, State, Extra) Called when upgrading code
  • 67. WORKER FUNCTIONS Callback Purposestart_link(Module, Args, Options) Start a supervised worker start(Module, Args, Options) Start an unsupervised workercall(ServerRef, Request, Timeout) Synchronously call a worker cast(ServerRef, Request) Asynchronously cast to a worker man gen_server
  • 68. SUPERVISORS• Use the `supervisor` behaviour• Contain little to no application logic• Know how to start and restart failed workers• Keep your application from ever fully dying
  • 69. CHILD SPECS{Id, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
  • 70. CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
  • 71. CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
  • 72. CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
  • 73. CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
  • 74. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
  • 75. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines how to kill child processes.
  • 76. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Determines how to kill child processes.
  • 77. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Is this child a worker or a supervisor?
  • 78. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Is this child a worker or a supervisor?
  • 79. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Used by code reloading. Make it the same as the other module name.
  • 80. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, [session_manager]} Used by code reloading. Make it the same as the other module name.
  • 81. RESTART STRATEGIES Strategy When a child dies.... one_for_one restart it one_for_all restart all children restart it and all children after it rest_for_one in child list restart it (also allows you tosimple_one_for_one dynamically add children)
  • 82. ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 83. ONE_FOR_ONE Supervisor Child 2 Child 3
  • 84. ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 85. ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
  • 86. ONE_FOR_ALL Supervisor Child 2 Child 3
  • 87. ONE_FOR_ALL Supervisor Child 3
  • 88. ONE_FOR_ALL Supervisor
  • 89. ONE_FOR_ALL SupervisorChild 1
  • 90. ONE_FOR_ALL SupervisorChild 1 Child 2
  • 91. ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
  • 92. REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 93. REST_FOR_ONE SupervisorChild 1 Child 3
  • 94. REST_FOR_ONE SupervisorChild 1
  • 95. REST_FOR_ONE SupervisorChild 1 Child 2
  • 96. REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 97. SIMPLE_ONE_FOR_ONE Supervisor
  • 98. SIMPLE_ONE_FOR_ONE SupervisorChild 1
  • 99. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2
  • 100. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 101. SIMPLE_ONE_FOR_ONE Supervisor Child 2 Child 3
  • 102. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • 103. STARTING A SUPERVISORinit([]) ->  Worker = {gen_server_examples, {gen_server_examples, start_link, []},            permanent, 5000, worker, [gen_server_examples]},  RestartStrategy = {one_for_one, 5, 30},  {ok, {RestartStrategy, [Worker]}}.
  • 104. SUPERVISOR CALLBACKcallback purpose return data structure telling whatinit(Args) to supervise
  • 105. SUPERVISOR FUNCTIONS Callback Purpose start_link(Module, Args) Start a supervised supervisorstart_child(SupRef, ChildSpec) Start a child under the supervisor man supervisor
  • 106. FAULT TOLERANCE• When you hit an error, let the worker die• Supervisor will restart it• Either try what killed it again or move on• Life is grand
  • 107. LINKS• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book
  • 108. THANK YOU!• Questions?• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book• http://mashion.net