ERTS diagram

  • 100 views
Uploaded on

2012 08-24 - talk at WebCamp KL

2012 08-24 - talk at WebCamp KL

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
100
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. what is a stack in IT? Were talking about a silo of dependencies, like a tower, with the end-user sitting at the top.Material science and E&E engineering are the foundations near the bottom, which give us hardware. Hardware is hardly afirm foundation... it changes very often, like the currents in the sea, and so we have software operating systems likeWindows, Linux, and Android to deal with the frequent changes - giving us a longer term "ground floor", or "floatingplatform", upon which to base applications that provide value to end-users of IT. Most recently, distributed computing isbecoming very trendy - and you must be aware of salespeak like "cloud computing" and "data centres". Unfortunately,although some significant work has been done, at the research level, on operating systems that tie many separatecomputers together, there are no dominant solutions in this space - no solutions that have a level of acceptance that isas widespread as any of the aforementioned operating systems. The most popular solutions are currently a thin layer,called hypervisors, that fit between operating systems designed to work on single computers, and the many separatecomputers that need to be made to work together. The operating system remains a "single machine" model, and so it almostforces all application development to implement its own way of dealing with the coordination of work that needs to bedone among many computers. So lets say now, that youre a programmer who is quite flexible, and can learn to speak anylanguage you like, and you need to write an application that runs on many computers at the same time - an easy examplefor this crowd is an application that needs to serve 100,000 complex webpages per second. You might ask yourself, "whichis the hardware or software machine that already knows how to deal with most of the knitty-gritty of getting this jobdone?" Once you had an answer, you might then ask what language that software or hardware speaks, and then you wouldlearn that language, so that you could communicate with the hardware or software that already knows how to get the jobdone. It turns out that Ericsson has continuously developed and maintained for over 20 years, a piece of software that isvery clever at solving this problem problem, of getting many pieces of hardware to work together on performing a singlejob that helps some end-user. Ericsson calls this clever machine the "BEAM emulator", and it calls the framework forworking with the BEAM emulator, "the Open Telecoms Platform (OTP)". So BEAM is this smart machine, and OTP is a patternthat BEAM uses to think about distributed computing problems. Now how do we humans talk to BEAM? We would normally use alanguage called Erlang. This is why it is valuable to learn Erlang. (Zooming out for a second, lets remember thatlanguages dont do work - people, or machines, do work, and we only use languages to communicate with machines, as ifthey are little people.) The rest of todays talk is going to briefly cover three topics: 1) how BEAM gets distributedcomputing done, with the help of OTP, 2) how the Erlang language works, and 3) how Erlang or any other language is madecomprehensible to BEAM (technically: Im talking about BEAM as a compilation target)."stack"Im an end-user!Hardware: the shifting sandsOperating Systems: a plea for sanityApplications: I like to use it.Virtualisation: you mean we stillhave to use these mofos? (now: yes)designersfalala!fala..er..developersFU!!finance / opswe RFU-ed.investorsFFUU!!!!
  • 2. what is a stack in IT? Were talking about a silo of dependencies, like a tower, with the end-user sitting at the top.Material science and E&E engineering are the foundations near the bottom, which give us hardware. Hardware is hardly afirm foundation... it changes very often, like the currents in the sea, and so we have software operating systems likeWindows, Linux, and Android to deal with the frequent changes - giving us a longer term "ground floor", or "floatingplatform", upon which to base applications that provide value to end-users of IT. Most recently, distributed computing isbecoming very trendy - and you must be aware of salespeak like "cloud computing" and "data centres". Unfortunately,although some significant work has been done, at the research level, on operating systems that tie many separatecomputers together, there are no dominant solutions in this space - no solutions that have a level of acceptance that isas widespread as any of the aforementioned operating systems. The most popular solutions are currently a thin layer,called hypervisors, that fit between operating systems designed to work on single computers, and the many separatecomputers that need to be made to work together. The operating system remains a "single machine" model, and so it almostforces all application development to implement its own way of dealing with the coordination of work that needs to bedone among many computers. So lets say now, that youre a programmer who is quite flexible, and can learn to speak anylanguage you like, and you need to write an application that runs on many computers at the same time - an easy examplefor this crowd is an application that needs to serve 100,000 complex webpages per second. You might ask yourself, "whichis the hardware or software machine that already knows how to deal with most of the knitty-gritty of getting this jobdone?" Once you had an answer, you might then ask what language that software or hardware speaks, and then you wouldlearn that language, so that you could communicate with the hardware or software that already knows how to get the jobdone. It turns out that Ericsson has continuously developed and maintained for over 20 years, a piece of software that isvery clever at solving this problem problem, of getting many pieces of hardware to work together on performing a singlejob that helps some end-user. Ericsson calls this clever machine the "BEAM emulator", and it calls the framework forworking with the BEAM emulator, "the Open Telecoms Platform (OTP)". So BEAM is this smart machine, and OTP is a patternthat BEAM uses to think about distributed computing problems. Now how do we humans talk to BEAM? We would normally use alanguage called Erlang. This is why it is valuable to learn Erlang. (Zooming out for a second, lets remember thatlanguages dont do work - people, or machines, do work, and we only use languages to communicate with machines, as ifthey are little people.) The rest of todays talk is going to briefly cover three topics: 1) how BEAM gets distributedcomputing done, with the help of OTP, 2) how the Erlang language works, and 3) how Erlang or any other language is madecomprehensible to BEAM (technically: Im talking about BEAM as a compilation target).re-stack?Im an end-user!Applications: I like to use it.A magic box, please?(your fav. language / platform)
  • 3. what is a stack in IT? Were talking about a silo of dependencies, like a tower, with the end-user sitting at the top.Material science and E&E engineering are the foundations near the bottom, which give us hardware. Hardware is hardly afirm foundation... it changes very often, like the currents in the sea, and so we have software operating systems likeWindows, Linux, and Android to deal with the frequent changes - giving us a longer term "ground floor", or "floatingplatform", upon which to base applications that provide value to end-users of IT. Most recently, distributed computing isbecoming very trendy - and you must be aware of salespeak like "cloud computing" and "data centres". Unfortunately,although some significant work has been done, at the research level, on operating systems that tie many separatecomputers together, there are no dominant solutions in this space - no solutions that have a level of acceptance that isas widespread as any of the aforementioned operating systems. The most popular solutions are currently a thin layer,called hypervisors, that fit between operating systems designed to work on single computers, and the many separatecomputers that need to be made to work together. The operating system remains a "single machine" model, and so it almostforces all application development to implement its own way of dealing with the coordination of work that needs to bedone among many computers. So lets say now, that youre a programmer who is quite flexible, and can learn to speak anylanguage you like, and you need to write an application that runs on many computers at the same time - an easy examplefor this crowd is an application that needs to serve 100,000 complex webpages per second. You might ask yourself, "whichis the hardware or software machine that already knows how to deal with most of the knitty-gritty of getting this jobdone?" Once you had an answer, you might then ask what language that software or hardware speaks, and then you wouldlearn that language, so that you could communicate with the hardware or software that already knows how to get the jobdone. It turns out that Ericsson has continuously developed and maintained for over 20 years, a piece of software that isvery clever at solving this problem problem, of getting many pieces of hardware to work together on performing a singlejob that helps some end-user. Ericsson calls this clever machine the "BEAM emulator", and it calls the framework forworking with the BEAM emulator, "the Open Telecoms Platform (OTP)". So BEAM is this smart machine, and OTP is a patternthat BEAM uses to think about distributed computing problems. Now how do we humans talk to BEAM? We would normally use alanguage called Erlang. This is why it is valuable to learn Erlang. (Zooming out for a second, lets remember thatlanguages dont do work - people, or machines, do work, and we only use languages to communicate with machines, as ifthey are little people.) The rest of todays talk is going to briefly cover three topics: 1) how BEAM gets distributedcomputing done, with the help of OTP, 2) how the Erlang language works, and 3) how Erlang or any other language is madecomprehensible to BEAM (technically: Im talking about BEAM as a compilation target).Erlang/OTP?Im an end-user!Applications: I like to use it.A candidate for something thatdoesnt exist yet?
  • 4. OS: Hardware Abstraction(logicalcores)(RAM addressspace)a=>4b=>?c=>7etc.HEAP|v||s|<=|f||k||_|STACKAn OS Processassigned thisspace...Other OSProcesses...(hardwarecores)(hardwareRAM)1. An Erlang "Node a.k.a. Emulator a.k.a. VM" runs as an OS process.2. Internally it can manage 00s, 000s, or 000,000s of Erlangprocesses.3. Erlangprocessesare spawnedjust bypassingfunctionsintospawn().4. EachErlangprocess hasits ownheap andstack (implemented in the OS processs heap).5. Erlang processes can talk to each other viamessage passing (messages are copied between Erlangprocesses mailboxes).6. Erlang processes can be organisedinto supervision trees which providenodes with a framework for communalliving and dying (very biological).7. Architecturaldifference from e.g.Node.js:PreemptiveMultitasking ofErlang Processes(time sharing)One Scheduler perlogical coreOther Erlang nodes... Other OS processes...
  • 5. -module(dummy_module).-compile(native).-compile(export_all).dummy_function()->A = lala,B = "lala",C = [$l,97,$l,$a],D = <<"lala">>,E = <<$l,$a,$l,97>>,F = 437947750258039476268479386,G = (3.142 + 2), % 5.142H = "la" ++ [$l] ++ [97], % "lala"[1,2,3] == [ 1 | [2,3] ], % trueI = {a,b,{d,e}},J = lists:reverse([1,2,3]), % [3,2,1]K = fun()->{ok,[A,B,G]} end,K.% Then elsewhere, running% X = dummy_module:dummy_function(),% X().% would return% {ok, [lala,"lala",5.14199999... 5]}% declares the module name% targets MIs instead of byte codes% declaring the interface%function_name()->% VariableA = atom,% VariableB = "string",% VariableC = ditto, with less sugar,% VariableD = <<"binary">>,% VariableE = ditto, with less sugar,% VariableF = integer of any size,% Dynamic typing: floats + ints% Dynamic typing: strings% list [ head | tail ] syntax% VariableI = 3-tuple% VariableJ = module:function() call% VariableK = 1st-class fn, closure% dummy_function() can return theclosure% . ends dummy_functions declarationlanguage layer; basic data types, and syntax - dont try to remember it right nowPOPULAR: "patten matching" basically meansthat you can use wild-cards in switch-cases,even for calling different branches of afunction.
  • 6. compilation : script, to abstract format... which goes into the module "compile"Erlang has "leex" and "yecc"which do similar work, andoutput "Abstract FormattedErlang", which is fairlyregular/readable, andprobably a good compilationtarget for beginners (likeme).Theres also a lower-levelformat called Core Erlang.
  • 7. compilation : byte code and/or native code... what comes out of the "compile"moduleLLVM backendisexperimental, but FYI :)
  • 8. Further reading (top picks):- http://LearnYouSomeErlang.com(easy, prosaic, illustrated intro to everything)Search for it (or I can Dropbox the PDFs to you):- Vorreuters presentation on Hacking Erlang(on compiling custom/other languages to Erlangs AbstractFormat)- Zhangs 2011 thesis: Erlang on many cores(for a readable description of BEAM internals)- Erl-llvm thesis with the Greek cover page(more on lower-level & native compilation)- A study of ETS table implementation and performance(alternativs to Erlangs built-in in-memory key-value-storearchitecture/algorithms)- Erlang-embedded whitepaper(on getting a 3MB disk footprint, 16MB runtime)