• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Massively Scalable, Massively Fault Tolerant
 

Massively Scalable, Massively Fault Tolerant

on

  • 1,055 views

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

This talk covers the basics of Erlang/OTP, a framework for highly scalable, highly fault tolerant applications.

Statistics

Views

Total Views
1,055
Views on SlideShare
1,052
Embed Views
3

Actions

Likes
1
Downloads
32
Comments
0

3 Embeds 3

https://twitter.com 1
http://twitter.com 1
https://blackboard.strayer.edu 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Massively Scalable, Massively Fault Tolerant Massively Scalable, Massively Fault Tolerant Presentation Transcript

  • MASSIVELY SCALEABLEMASSIVELY FAULT-TOLERANT Trotter Cashion Philly Lambda, 2011
  • ErlangBUT FIRST... AND OTP IN ACTION Buy this Book!http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • ErlangNO SERIOUSLY... IN ACTION AND OTP http://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • ErlangAVAILABLE IN PDF... AND OTP IN ACTIONhttp://bit.ly/otp-book Martin Logan Eric Merritt Richard Carlsson FOREWORD BY ULF WIGER MANNING
  • AND NOW FOR ME...
  • CO-FOUNDER
  • @CASHION http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg
  • github.com/trotter
  • CHLOE• Realtime Web Server• Proxies WebSocket (and fallback) connections to your app• http://github.com/mashion/chloe• Written in Erlang/OTP (plus a lot of JavaScript)
  • BAI PHILLY! Leaving in July... http://www.flickr.com/photos/dennis/156048151/
  • HAI CALIFORNIA! See you in August... http://www.flickr.com/photos/salim/402618628/
  • AN ERLANG PRIMER Yoinked from the Erlang Movie. Go Watch It!!!!
  • THE BASICS• Functional• Single Assignment Variables• Pattern Matching is Big Here• Everything’s a Process• One Module Per File
  • FUNCTIONALsay_hello() ->  io:fwrite("Ohai!n").
  • ANONYMOUS FUNCTIONSmake_hello() ->  fun () -> io:fwrite("Ohai!n")end.Hello = make_hello().Hello().
  • SINGLE ASSIGNMENT VARS1> Hello = 3.32> Hello = 2.** exception error: no match of right hand side value 23> f().ok4> Hello = 2.2
  • OH NOES! WHAT WENT WRONG?destructuring_bind() ->  [{Name, Title}, _] = [{"Trotter", "Speaker"},                        {"Aaron", "Organizer"}],  io:fwrite("~pn", [Name]), %% => Trotter  io:fwrite("~pn", [Title]). %% => Speaker
  • PATTERN MATCHING Erlang Rubymatch_me(1) -> def match_me(val)  loneliest;   case valmatch_me(2) ->   when 1  next_to_loneliest;     :loneliestmatch_me(friend) ->   when 2  all_ok;     :next_to_loneliestmatch_me(_) ->   when :friend  what_is_this.     :all_ok   else     :what_is_this   end end
  • MAKING A PROCESS process.erlconcurrent_stuffs() ->  spawn(fun () -> timer:sleep(1000),                  io:fwrite("So slown") end),  spawn(fun () -> io:fwrite("Super fastn") end). erlang shell5> process:concurrent_stuffs().Super fast<0.39.0>So slow
  • RECEIVE LOOPSfood_loop() ->  receive    pizza -> io:fwrite("lets eatn"),             food_loop();    death -> io:fwrite("all done heren");    _ -> io:fwrite("Nothing to don"),             food_loop()  end.
  • SENDING MESSAGESmake_process() ->  spawn(?MODULE, food_loop, []).pizza_death() ->  Proc = make_process(),  Proc ! pizza,  Proc ! hi,  Proc ! death,  %% Goes into the void  Proc ! huh.
  • HOW THAT WORKS process 1pizza_death
  • HOW THAT WORKS process 1 spawn/3pizza_death
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza “let’s eatn” hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi “Nothing to don” death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death “all done heren” huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 process 2 spawn/3pizza_death food_loop food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 spawn/3pizza_death food_loop’s Mailbox pizza hi death huh
  • HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death huh
  • HOW THAT WORKS process 1 spawn/3pizza_death pizza hi death
  • BUT DOES IT SCALE? Hells yea! http://www.flickr.com/photos/24736216@N07/4808606178/
  • TWO WAYS TO SCALE Name ImplementationHorizontally Buy a lot of machines Vertically Buy a really big machine
  • ERLANG SCALES WELL ON BOTHWITH NEARLY ZERO CODE CHANGE
  • BOOM!
  • VERTICAL SCALING• Your processes will take advantage of all cores available• Yay! No code changes required
  • HORIZONTAL SCALING• New nodes added to the cluster can be auto-detected• Code must know that a service could live on another node• You’ll need resource discovery
  • CONNECTING NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: []3> io:fwrite("Ping? ~p~n", [net_adm:ping(AHost)]).Ping? pong4> io:fwrite("Current nodes: ~p~n", [nodes()]).Current nodes: [a@awesome-town.local5> q().
  • SERVER REFERENCES Type Definition <Pid> globally unique process identifier RegisteredName locally registered process process locally registered on{RegisteredName, Node} another node {global, GlobalName} globally registered process
  • LISTEN FOR INCOMING MESSAGElisten() ->  register(listener, self()),  listen_loop().listen_loop() ->  receive    {From, pizza} -> From ! "Nom noms", listen_loop();    {From, _} -> From ! "Boo!", listen_loop();    _ -> io:fwrite("Listen is dying!")  end.
  • SENDING MESSAGE TO NODES$ erl -name b1> AHost = a@awesome-town.local.a@awesome-town.local2> {listener, AHost} ! {self(), pizza}.ok3> receive PizzaResp -> PizzaResp end.
  • RESOURCE DISCOVERY Method Purpose add_target_resource(Type) specify a resource type you want to useadd_local_resource(Type, Pid) specify a resource type you have fetch_resources(Type) fetch all resources of specified type trade_resources() trade resource information See Chapter 8 of Erlang And OTP In Action
  • HORIZONTAL CAVEATS• Passing large amounts of data between nodes is inefficient• Too many nodes will impact performance (Keep it < 100)• There are ways around both these problems
  • OTP BASICSCause managing all the processes by hand is dumb Thanks, Homer
  • OTPWTFBBQ?• Open Telecom Platform• Abstracts common systems behavior (servers, supervisors, ...)• Easier than pure Erlang!• Gives Erlang its reputation for fault tolerance
  • BASIC CONSTRUCTSConstruct Purpose Encapsulates a large set of codeApplication meant to run as one unit.Supervisor Starts and restarts workers. Worker Actually does stuff
  • HTTP://GITHUB.COM/MASHION/CHLOE Chloe (Application) chloe_sup (Supervisor) websocket_sup session_sup session_manager Other Stuffs (Supervisor) (Supervisor) (Worker)
  • WORKERS• Use the `gen_server` behaviour (for the most part)• Contain your application logic• You define a module with callbacks• OTP runs a server calls back to module when messages arrive
  • HOW WORKERS WORK client process (your code) gen_server functions gen_server code (part of OTP) gen_server callbacks worker code (your code) worker process
  • SENDING MESSAGES TO WORKERSsay_hello_sync(Name) ->  gen_server:call(?SERVER, {say_hello, Name}).say_hello_async() ->  gen_server:cast(?SERVER, say_hello). gen_server_examples.erl
  • EXAMPLE CALLBACKShandle_call({say_hello, Name}, _From, State) ->  Response = string:concat("Hello, ", Name),  {reply, {ok, Response}, State}.handle_cast(say_hello, State) ->  io:fwrite("Hello world!n"),  {noreply, State}. gen_server_examples.erl
  • WORKER CALLBACKS Callback Purpose init(Args) Initialize the server processhandle_call(Request, From, State) Handle a synchronous call handle_cast(Request, State) Handle an asynchronous cast handle_info(Info, State) For timeouts and random messages terminate(Reason, State) Clean up the process when it diescode_change(OldVsn, State, Extra) Called when upgrading code
  • WORKER FUNCTIONS Callback Purposestart_link(Module, Args, Options) Start a supervised worker start(Module, Args, Options) Start an unsupervised workercall(ServerRef, Request, Timeout) Synchronously call a worker cast(ServerRef, Request) Asynchronously cast to a worker man gen_server
  • SUPERVISORS• Use the `supervisor` behaviour• Contain little to no application logic• Know how to start and restart failed workers• Keep your application from ever fully dying
  • CHILD SPECS{Id, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
  • CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Unique name used to identify child process.
  • CHILD SPECS{session_manager, StartFunc, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Tuple {M, F, A} used to start the child.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, Restart, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines when a terminated process should be restarted.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, Shutdown, Type, Modules} Determines how to kill child processes.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Determines how to kill child processes.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, Type, Modules} Is this child a worker or a supervisor?
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Is this child a worker or a supervisor?
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, Modules} Used by code reloading. Make it the same as the other module name.
  • CHILD SPECS{session_manager, {session_manager, start_link, []}, permanent, 5000, worker, [session_manager]} Used by code reloading. Make it the same as the other module name.
  • RESTART STRATEGIES Strategy When a child dies.... one_for_one restart it one_for_all restart all children restart it and all children after it rest_for_one in child list restart it (also allows you tosimple_one_for_one dynamically add children)
  • ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • ONE_FOR_ONE Supervisor Child 2 Child 3
  • ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
  • ONE_FOR_ALL Supervisor Child 2 Child 3
  • ONE_FOR_ALL Supervisor Child 3
  • ONE_FOR_ALL Supervisor
  • ONE_FOR_ALL SupervisorChild 1
  • ONE_FOR_ALL SupervisorChild 1 Child 2
  • ONE_FOR_ALL SupervisorChild 1 Child 2 Child 3
  • REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • REST_FOR_ONE SupervisorChild 1 Child 3
  • REST_FOR_ONE SupervisorChild 1
  • REST_FOR_ONE SupervisorChild 1 Child 2
  • REST_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • SIMPLE_ONE_FOR_ONE Supervisor
  • SIMPLE_ONE_FOR_ONE SupervisorChild 1
  • SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2
  • SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • SIMPLE_ONE_FOR_ONE Supervisor Child 2 Child 3
  • SIMPLE_ONE_FOR_ONE SupervisorChild 1 Child 2 Child 3
  • STARTING A SUPERVISORinit([]) ->  Worker = {gen_server_examples, {gen_server_examples, start_link, []},            permanent, 5000, worker, [gen_server_examples]},  RestartStrategy = {one_for_one, 5, 30},  {ok, {RestartStrategy, [Worker]}}.
  • SUPERVISOR CALLBACKcallback purpose return data structure telling whatinit(Args) to supervise
  • SUPERVISOR FUNCTIONS Callback Purpose start_link(Module, Args) Start a supervised supervisorstart_child(SupRef, ChildSpec) Start a child under the supervisor man supervisor
  • FAULT TOLERANCE• When you hit an error, let the worker die• Supervisor will restart it• Either try what killed it again or move on• Life is grand
  • LINKS• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book
  • THANK YOU!• Questions?• http://bit.ly/erlang-otp-gist• http://bit.ly/otp-book• http://mashion.net