Your SlideShare is downloading. ×
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Massively Scalable, Massively Fault Tolerant
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Massively Scalable, Massively Fault Tolerant

989

Published on

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
989
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
34
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. MASSIVELY SCALEABLEMASSIVELY FAULT-TOLERANT Trotter Cashion Philly Lambda, 2011
    • 2. ErlangBUT FIRST... AND OTP IN ACTION Buy this Book!http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
    • 3. ErlangNO SERIOUSLY... IN ACTION AND OTP http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
    • 4. ErlangAVAILABLE IN PDF... AND OTP IN ACTIONhttp://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
    • 5. AND NOW FOR ME...
    • 6. CO-FOUNDER
    • 7. @CASHION http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg
    • 8. github.com/trotter
    • 9. CHLOE• Realtime Web Server• Proxies WebSocket (and fallback) connections to your app• http://github.com/mashion/chloe• Written in Erlang/OTP (plus a lot of JavaScript)
    • 10. BAI PHILLY! Leaving in July... http://www.flickr.com/photos/dennis/156048151/
    • 11. HAI CALIFORNIA! See you in August... http://www.flickr.com/photos/salim/402618628/
    • 12. AN ERLANG PRIMER Yoinked from the Erlang Movie. Go Watch It!!!!
    • 13. THE BASICS• Functional• Single Assignment Variables• Pattern Matching is Big Here• Everything’s a Process• One Module Per File
    • 14. FUNCTIONALsay_hello() ->  io:fwrite("Ohai!n").
    • 15. ANONYMOUS FUNCTIONSmake_hello() ->  fun () -> io:fwrite("Ohai!n")end.Hello = make_hello().Hello().
    • 16. SINGLE ASSIGNMENT VARS1> Hello = 3.32> Hello = 2.** exception error: no match of right hand side value 23> f().ok4> Hello = 2.2
    • 17. OH NOES! WHAT WENT WRONG?destructuring_bind() ->  [{Name, Title}, _] = [{"Trotter", "Speaker"},                        {"Aaron", "Organizer"}],  io:fwrite("~pn", [Name]), %% => Trotter  io:fwrite("~pn", [Title]). %% => Speaker
    • 18. PATTERN MATCHING Erlang Rubymatch_me(1) -> def match_me(val)  loneliest;   case valmatch_me(2) ->   when 1  next_to_loneliest;     :loneliestmatch_me(friend) ->   when 2  all_ok;     :next_to_loneliestmatch_me(_) ->   when :friend  what_is_this.     :all_ok   else     :what_is_this   end end
    • 19. MAKING A PROCESS process.erlconcurrent_stuffs() ->  spawn(fun () -> timer:sleep(1000),                  io:fwrite("So slown") end),  spawn(fun () -> io:fwrite("Super fastn") end). erlang shell5> process:concurrent_stuffs().Super fast<0.39.0>So slow
    • 20. RECEIVE LOOPSfood_loop() ->  receive    pizza -> io:fwrite("lets eatn"),             food_loop();    death -> io:fwrite("all done heren");    _ -> io:fwrite("Nothing to don"),             food_loop()  end.
    • 21. SENDING MESSAGESmake_process() ->  spawn(?MODULE, food_loop, []).pizza_death() ->  Proc = make_process(),  Proc ! pizza,  Proc ! hi,  Proc ! death,  %% Goes into the void  Proc ! huh.
    • 22. HOW THAT WORKS process 1pizza_death
    • 23. HOW THAT WORKS process 1 spawn/3pizza_death
    • 24. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop
    • 25. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
    • 26. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
    • 27. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza
    • 28. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi
    • 29. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death
    • 30. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 31. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 32. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 33. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
    • 34. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
    • 35. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 36. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
    • 37. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
    • 38. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 39. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
    • 40. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
    • 41. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 42. HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
    • 43. HOW THAT WORKS process 1 spawn/3pizza_death food_loop’s Mailbox pizza hi death huh
    • 44. HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death huh
    • 45. HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death
    • 46. BUT DOES IT SCALE? Hells yea! http://www.flickr.com/photos/24736216@N07/4808606178/
    • 47. TWO WAYS TO SCALE Name ImplementationHorizontally Buy a lot of machines Vertically Buy a really big machine
    • 48. ERLANG SCALES WELL ON BOTHWITH NEARLY ZERO CODE CHANGE
    • 49. BOOM!
    • 50. VERTICAL SCALING• Your processes will take advantage of all cores available• Yay! No code changes required
    • 51. HORIZONTAL SCALING• New nodes added to the cluster can be auto-detected• Code must know that a service could live on another node• You’ll need resource discovery
    • 52. CONNECTING NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: []3> io:fwrite("Ping? ~p~n", [net_adm:ping(AHost)]).Ping? pong4> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: [a@awesome-town.local5> q().
    • 53. SERVER REFERENCES Type Definition <Pid> globally unique process identifier RegisteredName locally registered process process locally registered on{RegisteredName, Node} another node {global, GlobalName} globally registered process
    • 54. LISTEN FOR INCOMING MESSAGElisten() ->  register(listener, self()),  listen_loop().listen_loop() ->  receive    {From, pizza} -> From ! "Nom noms", listen_loop();    {From, _} -> From ! "Boo!", listen_loop();    _ -> io:fwrite("Listen is dying!")  end.
    • 55. SENDING MESSAGE TO NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> {listener, AHost} ! {self(), pizza}.ok3> receive PizzaResp -> PizzaResp end.
    • 56. RESOURCE DISCOVERY Method Purpose add_target_resource(Type) specify a resource type you want to useadd_local_resource(Type, Pid) specify a resource type you have fetch_resources(Type) fetch all resources of specified type trade_resources() trade resource information See Chapter 8 of Erlang And OTP In Action
    • 57. HORIZONTAL CAVEATS• Passing large amounts of data between nodes is inefficient• Too many nodes will impact performance (Keep it < 100)• There are ways around both these problems
    • 58. OTP BASICSCause managing all the processes by hand is dumb Thanks, Homer
    • 59. OTPWTFBBQ?• Open Telecom Platform• Abstracts common systems behavior (servers, supervisors, ...)• Easier than pure Erlang!• Gives Erlang its reputation for fault tolerance
    • 60. BASIC CONSTRUCTSConstruct Purpose Encapsulates a large set of codeApplication meant to run as one unit.Supervisor Starts and restarts workers. Worker Actually does stuff
    • 61. HTTP://GITHUB.COM/MASHION/CHLOE Chloe (Application) chloe_sup (Supervisor) websocket_sup session_sup session_manager Other Stuffs (Supervisor) (Supervisor) (Worker)
    • 62. WORKERS• Use the `gen_server` behaviour (for the most part)• Contain your application logic• You define a module with callbacks• OTP runs a server calls back to module when messages arrive
    • 63. HOW WORKERS WORK client process (your code) gen_server functions gen_server code (part of OTP) gen_server callbacks worker code (your code) worker process
    • 64. SENDING MESSAGES TO WORKERSsay_hello_sync(Name) ->  gen_server:call(?SERVER, {say_hello, Name}).say_hello_async() ->  gen_server:cast(?SERVER, say_hello). gen_server_examples.erl
    • 65. EXAMPLE CALLBACKShandle_call({say_hello, Name}, _From, State) ->  Response = string:concat("Hello, ", Name),  {reply, {ok, Response}, State}.handle_cast(say_hello, State) ->  io:fwrite("Hello world!n"),  {noreply, State}. gen_server_examples.erl
    • 66. WORKER CALLBACKS Callback Purpose init(Args) Initialize the server processhandle_call(Request, From, State) Handle a synchronous call handle_cast(Request, State) Handle an asynchronous cast handle_info(Info, State) For timeouts and random messages terminate(Reason, State) Clean up the process when it diescode_change(OldVsn, State, Extra) Called when upgrading code
    • 67. WORKER FUNCTIONS Callback Purposestart_link(Module, Args, Options) Start a supervised worker start(Module, Args, Options) Start an unsupervised workercall(ServerRef, Request, Timeout) Synchronously call a worker cast(ServerRef, Request) Asynchronously cast to a worker man gen_server
    • 68. SUPERVISORS• Use the `supervisor` behaviour• Contain little to no application logic• Know how to start and restart failed workers• Keep your application from ever fully dying
    • 69. CHILD SPECS{Id, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
    • 70. CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
    • 71. CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
    • 72. CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
    • 73. CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
    • 74. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
    • 75. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines how to kill child processes.
    • 76. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Determines how to kill child processes.
    • 77. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Is this child a worker or a supervisor?
    • 78. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Is this child a worker or a supervisor?
    • 79. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Used by code reloading. Make it the same as the other module name.
    • 80. CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, [session_manager]} Used by code reloading. Make it the same as the other module name.
    • 81. RESTART STRATEGIES Strategy When a child dies.... one_for_one restart it one_for_all restart all children restart it and all children after it rest_for_one in child list restart it (also allows you tosimple_one_for_one dynamically add children)
    • 82. ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 83. ONE_FOR_ONE Supervisor Child 2 Child 3
    • 84. ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 85. ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
    • 86. ONE_FOR_ALL Supervisor Child 2 Child 3
    • 87. ONE_FOR_ALL Supervisor Child 3
    • 88. ONE_FOR_ALL Supervisor
    • 89. ONE_FOR_ALL SupervisorChild 1
    • 90. ONE_FOR_ALL SupervisorChild 1 Child 2
    • 91. ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
    • 92. REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 93. REST_FOR_ONE SupervisorChild 1 Child 3
    • 94. REST_FOR_ONE SupervisorChild 1
    • 95. REST_FOR_ONE SupervisorChild 1 Child 2
    • 96. REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 97. SIMPLE_ONE_FOR_ONE Supervisor
    • 98. SIMPLE_ONE_FOR_ONE SupervisorChild 1
    • 99. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2
    • 100. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 101. SIMPLE_ONE_FOR_ONE Supervisor Child 2 Child 3
    • 102. SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
    • 103. STARTING A SUPERVISORinit([]) ->  Worker = {gen_server_examples, {gen_server_examples, start_link, []},            permanent, 5000, worker, [gen_server_examples]},  RestartStrategy = {one_for_one, 5, 30},  {ok, {RestartStrategy, [Worker]}}.
    • 104. SUPERVISOR CALLBACKcallback purpose return data structure telling whatinit(Args) to supervise
    • 105. SUPERVISOR FUNCTIONS Callback Purpose start_link(Module, Args) Start a supervised supervisorstart_child(SupRef, ChildSpec) Start a child under the supervisor man supervisor
    • 106. FAULT TOLERANCE• When you hit an error, let the worker die• Supervisor will restart it• Either try what killed it again or move on• Life is grand
    • 107. LINKS• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book
    • 108. THANK YOU!• Questions?• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book• http://mashion.net

    ×