Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

You shall not get excited

4,773 views

Published on

ffffuuuuconf 2010 presentation

Published in: Technology
  • Be the first to comment

You shall not get excited

  1. 1. You Shall NotGet Excited Ivan Ribeiro Rocha ivan.ribeiro@gmail.com @irr
  2. 2. Agenda ● Why not? ● ”History” about getting excited... ● What should I use...? ● My choice: Erlang ● Learning from experience...
  3. 3. Why not?
  4. 4. ”Once upon a time...” ”History” about getting excited...
  5. 5. (what is a ”technical” manager?)
  6. 6. …a ”technical” manager meets...
  7. 7. … a ”smart guy” (a.k.a salesman)
  8. 8. Deal! You deserveit!
  9. 9. ”no problem... we can fixit!”
  10. 10. ”relax... the guys will change it in the upcoming version...”
  11. 11. sometimes is not so easy to fix all problems
  12. 12. ”Ignorance more frequently begets confidence than does knowledge.” Charles Darwin
  13. 13. What should I use...?
  14. 14. Programming Languages and tools... ● Languages – Java, Ruby, Python, Perl, ... – C/C++, Lisp, Haskell, Ocaml, Erlang, ... ● Web Servers – Apache, nginx, ... – Jetty/Tomcat, ... – Yaws, inets, ...
  15. 15. Anyone using it?
  16. 16. Is it stable?
  17. 17. Does it scale?
  18. 18. (think about...) You're not:
  19. 19. what means scalable? - don’t design to scale infinitely - consider 5X - 50X growth - but > 100X requires redesign - break large systems into smaller services Jeff Dean http://bit.ly/clIJfL
  20. 20. no documentation...? ● Good APIs? ● Available books? ● % Comments...? ● Source code? ● Please! No ”magazines”...
  21. 21. still in doubt...?
  22. 22. Joe Armstrong Robert Virding Mike Williams Claes “Klacke” Wikström and more... My choice: Erlang
  23. 23. Learning from experience...
  24. 24. ● > 3 years and learning... ● back end development – REST API – memcached API – syslog API – MySQL and Mnesia (data storage)
  25. 25. ● functional + concurrent programming ● fault-tolerant and highly scalable ● very light-weight concurrency (processes) ● running massive systems for 20 years ● bit syntax and binaries (for protocol programming) ● links (encourages “let it crash” programming) ● ”concurrency belongs to the language and not the operating system”
  26. 26. Erlang (Why you should get excited...) Concurrent Programming –lots of processes ● the only way for processes to interact is through message passing – event based (epoll)
  27. 27. Erlang (Why you should get excited...) No shared memory – ”sharing is the property that prevents fault tolerance” – destructive shared data modifications do not occur!
  28. 28. Erlang (Why you should get excited...) It's easyto test and update your code
  29. 29. Erlang (Why you should get excited...) HTTP/JSON support – yaws (embedded) – mochiweb (json) – misultin –inets – httpc
  30. 30. Erlang (Why you should get excited...) many Libraries/tools for distributed programs... – net_[adm|kernel] net_kernel:connect_node(N). net_adm:ping(N). – rpc {R, E} = rpc:multicall(nodes(), Mod, Fun, [N], T), lists:foldl(fun(L, A) -> merge(A, [X || X <- L, not member(X, A)], N) end, C, R), – epmd...
  31. 31. Erlang (Why you should get excited...) ”Mnesia is a distributed DataBase Management System (DBMS), appropriate for telecommunications applications and other Erlang applications which require continuous operation and exhibit soft real-time properties” Mnesia is great but you should really know how to use it and you MUST architecture your application to get the best results...
  32. 32. Erlang (Why you should get excited...) OTP (Unix <-> C <==> OTP <-> Erlang) ”It’s an application operating system and a set of libraries and procedures used for building large-scale, fault-tolerant, distributed applications” – supervisor – applications – gen_server – gen_fsm – gen_event
  33. 33. ”We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.” Donald Knuth
  34. 34. Erlang (caveats: You must be careful...) Mnesia – can't handle very large data – 2 Gb ● ETS ● DETS ● MNESIA
  35. 35. Erlang (caveats: You must be careful...) Mnesia –2 Gb fragments (mod (2^X)) – avoid rehash: create more fragments... ● add ● move ● del*
  36. 36. Erlang (caveats: You must be careful...) Mnesia –CAP theorem –replicax partitioned networks
  37. 37. Erlang (caveats: You must be careful...) Mnesia – need to check replica consistency? ● vector clocks... (avoid dependencies) ● should try timestamps
  38. 38. Erlang (caveats: You must be careful...) Mnesia – event "running partitioned network" ● mnesia:set_master_nodes(L). ● must restart other nodes – if some node will not recover soon... ● mnesia:force_load_table(T).
  39. 39. Erlang (caveats: You must be careful...) Mnesia (QLC) ● avoid retrieve large data sets ● use cursors inside transactions mnesia:activity(sync_transaction, fun(X) -> QC = qlc:cursor(X), QR = qlc:next_answers(QC, N), qlc:delete_cursor(QC), QR end, [qlc:q([E || E <- mnesia:table(Tab)])], mnesia_frag).
  40. 40. Erlang (caveats: You must be careful...) Mnesia –load balance (read) – avoid ”overload” one instance mnesia_lib:set({T, where_to_read}, Node)
  41. 41. Erlang (caveats: You must be careful...) Logging – avoid overhead ● yaws ● error_logger – syslog ● udp ● tcp* ● gen_server + handle_cast
  42. 42. Erlang (caveats: You must be careful...) Messages – like a mailbox – per process (don't forget to read your messages)
  43. 43. Erlang (caveats: You must be careful...) OTP –only API – poor docs, books, articles...
  44. 44. Erlang (caveats: You must be careful...) OTP – avoid ”synchronize” long calls test(X) -> gen_server:call(?MODULE, {test, X}, 5000). handle_call({test, X}, From, State) → spawn(fun() → %% do some long task gen_server:reply(From, Reply) end), {noreply, State}.
  45. 45. Erlang (caveats: You must be careful...) Security – must be treated externally ● firewall ● private networks – cookie based -kernel inet_dist_use_interface {127,0,0,1} -kernel inet_dist_listen_min <min> -kernel inet_dist_listen_max <max> > erl -kernel inet_dist_listen_min 9001 inet_dist_listen_max 9005 (4369 TCP port, as it is used by epmd, ERL_EPMD_PORT environment variable)
  46. 46. Erlang (caveats: You must be careful...) Code swapping/replacement – Be careful with spawn inside old (replaced) module... it will die! – The code of a module can exist in two variants in a system: current and old (fully qualified function calls always refer to current code) – If a third instance of the module is loaded, the code server will remove (purge) the old code and any processes lingering in it will be terminated -module(m). -export([loop/0]). loop() -> receive code_switch -> m:loop(); Msg -> % ... loop() end.
  47. 47. Undocumented features... you must be careful!
  48. 48. Erlang (caveats: You must be careful...) Using async acceptors (prim_inet) – gen_tcp:accept(Listen) Usages: https://github.com/irr/erl-tutorials/blob/master/ts/ts1/src/ts.erl – prim_inet:async_accept(Socket, -1) Usages: https://github.com/irr/erl-tutorials/blob/master/ts/ts2/src/ts.erl http://svn.apache.org/viewvc/incubator/thrift/trunk/thrift_server.erl http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles ”It is undocumented because it is an internal module that is not ment to be called from applications. Its interface may change without warning in even the smallest patch.”
  49. 49. Erlang (caveats: You must be careful...) Using socket in http mode case gen_tcp:listen(Port, [binary, {packet, http}, {reuseaddr, true}, {active, false}, {backlog, 30}]) of ... case gen_tcp:recv(C#c.sock, 0, ?server_idle_timeout) of {ok, {http_header, _, 'Content-Length', _, Val}} -> ... {error, {http_error, "rn"}} -> ... {ok, http_eoh} -> ... Usage: http://www.trapexit.org/A_fast_web_server_demonstrating_some_undocumented_Erlang_features
  50. 50. Erlang (caveats: You must be careful...) 1> inet:ifget("eth0", [addr]). {ok,[{addr,{192,168,1,101}}]} 2> inet:getiflist(). {ok,["lo","eth0"]} 3> inet_parse:ntoa({192,168,1,101}). "192.168.1.101" 4> inet_parse:address("192.168.1.101"). {ok,{192,168,1,101}}
  51. 51. there is no silver bullet!
  52. 52. Any doubts?

×