Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Everybody Polyglot! - Cross-Language RPC with Erlang

7,392 views

Published on

Three different approaches to cross-language communication in Erlang: REST, Protocol Buffers, and Bert-RPC.

Published in: Technology, News & Politics
  • Be the first to comment

Everybody Polyglot! - Cross-Language RPC with Erlang

  1. 1. EVERYBODY POLYGLOT!(Cross-Language RPC with Erlang) ErlangDC · December 2011 Rusty Klophaus - @rustyio Basho Technologies
  2. 2. Languages have Strengths http://wordaligned.org/articles/distorted-software 2 @rustyio
  3. 3. Connections are Hard Serialization Versioning & Upgrades Data Type MismatchesSpeed, Bottlenecks & Back-Pressure Inadequate Tooling Context Switching 3 @rustyio
  4. 4. Connections are Hard http://wordaligned.org/articles/distorted-software 4 @rustyio
  5. 5. This TalkCreate An Example Service Any service will do, just need a framework for discussion.Expose Application via Interfaces • REST / JSON via Webmachine & Spooky • Protocol Buffers via erlang-protobuffs • BERT-RPC via Ernie ServerCode http://github.com/rustyio/ErlangRPCDemo 5 @rustyio
  6. 6. This TalkOur Application: A Sequence Server Erlang service that returns a sequence of numbers. 1 sequence(N) -> 2 List1 = lists:seq(1, N), 3 List2 = [list_to_binary(integer_to_list(X)) || X <- List1], 4 {ok, List2}; 5 6 %% sequence(2). 7 {ok, [<<"1">>, <<"2">>]}. 8 9 %% sequence(5). 10 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, <<"5">>]}. 11 12 %% sequence(50000). 13 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, ...]}. 6 @rustyio
  7. 7. Overview REST?Protocol Buffers? BERT-RPC? 7 @rustyio
  8. 8. REST / JSON - OverviewREST - Representational State Transfer Convention for talking to applications over HTTP Actions are Verbs are HTTP Methods (GET/PUT/POST/DELETE/...) Objects are nouns are URLs GET /users/5JSON - Javascript Object Notation Encode data as parseable Javascript Understood by everything Human Readable {"id":5,"first":"Rusty","last":"Klophaus"} 8 @rustyio
  9. 9. REST / JSON - Strengths & WeaknessesStrengths Simple, easy to poke around Good support in every language Composable - Caches, Reverse Proxies, Load BalancersWeaknesses General == More Handshaking/Metadata == More Overhead 9 @rustyio
  10. 10. Protocol Buffers - OverviewProtocol Buffers Developed by Google It’s not a protocol, it’s a format: • You provide the client / server logic • Useful for transmission AND storage Define data structures in .proto file, generate code http://code.google.com/apis/protocolbuffers/ https://github.com/ngerakines/erlang_protobuffs 10 @rustyio
  11. 11. Protocol Buffers - Strengths & WeaknessesStrengths Compact Add fields without breaking existing applications • Versioning is not strictWeaknesses Configuration file (.proto) with new syntax to learn Generated code Uneven language support 11 @rustyio
  12. 12. BERT-RPC - OverviewBERT-RPC Developed by GitHub (Tom Preston-Werner) BERT = Binary Erlang Term • Encoding mimics native Erlang serialization http://bert-rpc.org https://github.com/mojombo/ernie 1 % Request 2 {call, ernie_sequence, sequence, [3]} 3 4 % Response 5 {response, {ok, [<<"1">>, <<"2">>, <<"3">>]}} 12 @rustyio
  13. 13. BERT-RPC - Strengths & WeaknessesStrengths Easy to set up Agile CompactWeaknesses Uneven language support Less buzz than it deserves Not fully product-ized 13 @rustyio
  14. 14. Requests & ResponsesWhat does the chatter look like? 14 @rustyio
  15. 15. REST / JSON - Request & ResponseRequest GET /sequence/3 HTTP/1.1 User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 Host: localhost:8001 Accept: */*Response HTTP/1.1 200 OK Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) Date: Wed, 23 Nov 2011 22:09:13 GMT Content-Type: application/json Content-Length: 21 ["1","2","3"] 15 @rustyio
  16. 16. Protocol Buffers - Encodingrpc_demo.proto 1 message SequenceRequest { 2 required uint32 n = 1; 3 } 4 5 message SequenceResponse { 6 repeated bytes sequence = 1; 7 } 16 @rustyio
  17. 17. Protocol Buffers - RPCRequest 1 rpc_demo_pb:encode({sequencerequest, 3}). 2 <<8,3>>Response 1 rpc_demo_pb:encode({sequenceresponse, [<<"1">>, <<"2">>, <<"3">>]}). 2 <<10,1,49,10,1,50,10,1,51>> 17 @rustyio
  18. 18. BERT-RPC - Encoding1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]).2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>345 bert:encode([<<"1">>,<<"2">>,<<"3">>]).6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 18 @rustyio
  19. 19. BERT-RPC - Encoding 1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]). 2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 3 4 5 bert:encode([<<"1">>,<<"2">>,<<"3">>]). 6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 7 | | | | | | 8 | | | | | + Ascii value for 1 9 | | | | + Length of string (1)10 | | | + Next term is a string11 | | + Length of list (3)12 | + Next term is a list13 + Start of Erlang term 19 @rustyio
  20. 20. BERT-RPC - RPCRequest 1 bert:encode({call, ernie_sequence, sequence, [5]}). 3 <<131,104,4,100,0,4,99,97,108,108,100,0,14,101,114,110, 4 105,101,95,115,101,113,117,101,110,99,101,100,0,8,115, 5 101,113,117,101,110,99,101,107,0,1,5>>Response 1 bert:encode({reply,{ok,[<<"1">>,<<"2">>,<<"3">>,<<"4">>,<<"5">>]}}). 2 3 <<131,104,2,100,0,5,114,101,112,108,121,104,2,100,0,2,111, 4 107,108,0,0,0,5,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0, 5 1,51,109,0,0,0,1,52,109,0,0,0,1,53,106>> 20 @rustyio
  21. 21. Show me the Client Code! 21 @rustyio
  22. 22. REST / JSON - ClientsCURL curl http://localhost:8001/sequence/5Ruby Client 1 url = "http://localhost:8001/sequence/5" 2 resp = Net::HTTP.get_response(URI.parse(url)) 3 sequence = JSON.parse(resp.body) 22 @rustyio
  23. 23. Protocol Buffers - ClientRuby Client 1 require beefcake 2 3 class SequenceRequest 4 include Beefcake::Message 5 required :n, :int32, 1 6 end 7 8 class SequenceResponse 9 include Beefcake::Message 10 repeated :sequence, :string, 1 11 end 12 13 req = SequenceRequest.new(:n => 5) 14 Socket.tcp("localhost", 8003) do |socket| 15 socket.write(req.encode) 16 m = socket.read 17 return SequenceResponse.decode(m) 18 end 23 @rustyio
  24. 24. BERT-RPC - ClientRuby Client 1 require bert-rpc 2 3 svc = BERTRPC::Service.new(localhost, 9999) 4 sequence = svc.call.ernie_sequence.sequence(5) 24 @rustyio
  25. 25. Show me the Server Code! 25 @rustyio
  26. 26. REST / JSON - ServerSpooky Server 1 -module(spooky_sequence). 2 -behaviour(spooky). 3 -export([init/1, get/2]). 4 5 init([])-> 6 [{port, 8002}]. 7 8 get(_Req, ["sequence", Num])-> 9 case sequence:sequence(Num) of 10 {ok, List} -> 11 {200, mochijson2:encode(List)}; 12 {error, Error} -> 13 {500, io_lib:format("~w", [Error])} 14 end; 15 get(_Req, _)-> 16 {400, "Usage: /sequence/:Num:"}. 26 @rustyio
  27. 27. REST / JSON - ServerWebmachine Server 1 -module(webmachine_sequence). 2 -export([init/1, content_types_provided/2, to_json/2]). 3 -include_lib("webmachine/include/webmachine.hrl"). 4 5 -record(ctx, { list }). 6 7 init([]) -> 8 {ok, #ctx {}}. 9 10 content_types_provided(RD, Ctx) -> 11 Types = [{"application/json", to_json}], 12 {Types, RD, Ctx}. 13 14 to_json(RD, Ctx) -> 15 {ok, List} = sequence:sequence(N), 16 Body = mochijson2:encode(List), 17 {Body, RD, Ctx}. 27 @rustyio
  28. 28. Protocol Buffers - ServerErlang Server 1 %% Erlang RPC Demo 57 handle_info({tcp, _Sock, _Data}, State) -> 2 %% Copyright (c) 2011 Rusty Klophaus (@rustyio) 58 %% req =/= undefined: received a new request while another was in 3 %% See MIT-LICENSE for licensing information. 59 %% progress -> Error 4 60 lager:error("Received a new PB socket request" 5 -module(protobuff_server). 61 " while another was in progress"), 6 -behaviour(gen_server). 62 {stop, normal, State}; 7 63 8 -export([ 64 handle_info(_, State) -> % Ignore any late replies from gen_servers/messages from fsms 9 start_link/0, 65 {noreply, State}. 10 set_socket/2, 66 11 init/1, 67 terminate(_Reason, _State) -> 12 handle_call/3, 68 ok. 13 handle_cast/2, 69 14 handle_info/2, 70 code_change(_OldVsn, State, _Extra) -> {ok, State}. 15 terminate/2, 71 16 code_change/3, 72 %% =================================================================== 17 encode/1, 73 %% Handle PB Messages 18 decode/1]). 74 %% =================================================================== 19 75 20 -record(state, { sock }). 76 process_message(#sequencerequest { n = N }, State) -> 21 77 case sequence:sequence(N) of 22 -include("rpc_demo_pb.hrl"). 78 {ok, List} -> 23 79 Resp = #sequenceresponse { sequence = List }, 24 %% =================================================================== 80 send_msg(Resp, State); 25 %% Public API 81 {error, Reason} -> 26 %% =================================================================== 82 Msg = io_lib:format("~w", [Reason]), 27 83 Resp = #sequenceerror { message=Msg }, 28 start_link() -> 84 send_msg(Resp, State) 29 gen_server:start_link(?MODULE, [], []). 85 end. 30 86 31 set_socket(Pid, Socket) -> 87 %% Send a message to the client 32 gen_server:call(Pid, {set_socket, Socket}). 88 send_msg(Msg, State) -> 33 89 Pkt = encode(Msg), 34 init([]) -> 90 gen_tcp:send(State#state.sock, Pkt), 35 {ok, #state{}}. 91 State. 36 92 37 handle_call({set_socket, Socket}, _From, State) -> 93 encode(Msg) -> 38 inet:setopts(Socket, [{active, once}, {packet, 4}, {header, 1}]), 94 MsgType = element(1, Msg), 39 {reply, ok, State#state{sock = Socket}}. 95 [msg_code(Msg) | rpc_demo_pb:iolist(MsgType, Msg)]. 40 96 41 handle_cast(_Msg, State) -> 97 decode([MsgCode|MsgData]) -> 42 {noreply, State}. 98 MsgType = msg_type(MsgCode), 43 99 rpc_demo_pb:decode(MsgType, MsgData). 44 handle_info({tcp_closed, Socket}, State=#state{sock=Socket}) -> 100 45 {stop, normal, State}; 101 msg_code(#sequencerequest {}) -> 1; 46 handle_info({tcp_error, Socket, _Reason}, State=#state{sock=Socket}) -> 102 msg_code(#sequenceresponse {}) -> 2; 47 {stop, normal, State}; 103 msg_code(#sequenceerror {}) -> 3; 48 handle_info({tcp, _Sock, MsgData}, State=#state{sock=Socket}) -> 104 msg_code(Other) -> 49 Msg = decode(MsgData), 105 throw({unknown_pb_type, Other}). 50 case process_message(Msg, State) of 106 51 {pause, NewState} -> 107 msg_type(1) -> sequencerequest; 52 ok; 108 msg_type(2) -> sequenceresponse; 53 NewState -> 109 msg_type(3) -> sequenceerror; 54 inet:setopts(Socket, [{active, once}]) 110 msg_type(Other) -> 55 end, 111 throw({unknown_pb_code, Other}). 56 {noreply, NewState}; 28 @rustyio
  29. 29. BERT-RPC - ServerErnie Server - Erlang BERT-RPC Server 1 %% FILE: ernie.config 2 [ 3 {module, ernie_sequence}, 4 {type, native}, 5 {codepaths, []} 6 ]. 1 -module(ernie_sequence). 2 -export([sequence/1]). 3 4 sequence(N) -> 5 sequence:sequence(N). 29 @rustyio
  30. 30. BenchmarksHow fast is it? 30 @rustyio
  31. 31. BenchmarksDisclaimers Benchmarking is hard The shape of your data matters (size and complexity) Speed is not the only objective 31 @rustyio
  32. 32. Benchmarks:Encoding Speed 32 @rustyio
  33. 33. Encoding Speed - 5 items BERT JSON PB 33 @rustyio
  34. 34. Encoding Speed - 50 items BERT JSON PB 34 @rustyio
  35. 35. Encoding Speed - 500 items BERT JSON PB 35 @rustyio
  36. 36. Encoding Speed - 5,000 items BERT JSON PB 36 @rustyio
  37. 37. Encoding Speed - 50,000 items BERT JSON Sad Protocol Buffers :( 37 @rustyio
  38. 38. Encoding Speed - Ruby | 5 | 50 | 500 | 5,000 -----+---------+---------+---------+---------- JSON | 0.03 ms | 0.77 ms | 0.47 ms | 4.21 ms PB | 0.07 ms | 0.60 ms | 7.37 ms | 197.24 ms BERT | 0.08 ms | 0.52 ms | 4.97 ms | 52.42 ms 38 @rustyio
  39. 39. Encoding SpeedInterpretations BERT wins in Erlang, because it’s native. JSON wins in Ruby. Protocol Buffers is slow all around for complex data. • Note: This is different from large data. 39 @rustyio
  40. 40. Benchmarks: Single Hop$ ping dell.localPING dell.local (192.168.2.2): 56 data bytes64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.490 ms 40 @rustyio
  41. 41. BenchmarksMore Disclaimers First approach exhausted TCP connections “Fixed” by tunneling connections, which impacts results 41 @rustyio
  42. 42. Operation Performance - 5 items BERT PB PB SPOOKY WEBMACHINE 42 @rustyio
  43. 43. Operation Performance - 50 items BERT PB PB SPOOKY WEBMACHINE 43 @rustyio
  44. 44. Operation Performance - 500 items BERT PB PB SPOOKY WEBMACHINE 44 @rustyio
  45. 45. Operation Performance - 5,000 items BERT PB PB SPOOKY WEBMACHINE 45 @rustyio
  46. 46. Encoding SpeedInterpretations Performance is fairly even for simple data. PB is clear loser for data with lots of items. Webmachine pays a tax for considering entire HTTP decision tree. Take these with a grain of salt, everything is tunneled over SSH. 46 @rustyio
  47. 47. Webmachine - HTTP Decision Tree 47 @rustyio
  48. 48. Benchmarks: Multiple Hops$ ping rusty.ioPING rusty.io (173.203.217.46): 56 data bytes64 bytes from 173.203.217.46: icmp_seq=0 ttl=53 time=49.550 ms 48 @rustyio
  49. 49. Operation Performance - 5 items BERT PB PB SPOOKY WEBMACHINE 49 @rustyio
  50. 50. Operation Performance - 50 items BERT PB PB SPOOKY WEBMACHINE 50 @rustyio
  51. 51. Operation Performance - 500 items BERT PB PB SPOOKY WEBMACHINE 51 @rustyio
  52. 52. Operation Performance - 5,000 items BERT PB PB SPOOKY WEBMACHINE 52 @rustyio
  53. 53. Operation Performance - Multiple HopsInterpretations No clear winner. Network speed / variability is the bottleneck. Take these with a grain of salt, everything is tunneled over SSH. 53 @rustyio
  54. 54. The ResultsRecommendations Start with REST / JSON Optimize for performance with Protocol Buffers Get adventurous with BERT-RPC • Easy to set up == Easy to back outGet Involved! Protocol Buffers NIF? JSON NIF? General Ernie (BERT-RPC) improvements: • Re-use connections, better packaging & documentation 54 @rustyio
  55. 55. Honorable MentionsOther RPC Frameworks: Thrift - http://thrift.apache.org/ (Originally Facebook) Avro - http://avro.apache.org/docs/current/ XML-RPC CORBA - http://www.erlang.org/doc/man/corba.html UBF - http://www.sics.se/~joe/ubf/site/home.html MsgPack - http://msgpack.org/ Etch - http://incubator.apache.org/projects/etch.html ASN.1 - http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_OneErlang <-> Other Code: Erlang "ports" - integration via stdin/stdout. Bifs - Extend Erlang with new functions."Fake" Erlang Nodes: JInterface (it is... not great) C Nodes - http://www.erlang.org/doc/tutorial/cnode.html 55 @rustyio
  56. 56. Related Talks• Anton Lavrik - Piqi-RPC: exposing Erlang services via JSON, XML and Google Protocol Buffers over HTTP http://www.erlang-factory.com/conference/SFBay2011/speakers/AntonLavrik• Tom Preston-Werner - BERT is to Erlang as JSON is to JavaScript http://www.erlang-factory.com/conference/ErlangUserConference2009/speakers/ TomPrestonWerner• Todd Lipcon - Thrift Avro/Erlang Bindings http://www.erlang-factory.com/conference/SFBay2010/speakers/toddlipcon• Cliff Moon - Building Polyglot Distributed Systems with Jinterface http://www.erlang-factory.com/conference/SFBay2011/speakers/CliffMoon• Kresten Krab Thorup - Erjang - A JVM-based Erlang VM http://www.erlang-factory.com/conference/SFBay2010/speakers/ KrestenKrabThorup• Yurii Rashkovskii - Beam.JS: Erlang meets JavaScript http://www.erlang-factory.com/conference/SFBay2011/speakers/YuriiRashkovskii 56 @rustyio
  57. 57. Thanks!Questions? 57 @rustyio

×