Everybody Polyglot! - Cross-Language RPC with Erlang

  • 5,723 views
Uploaded on

Three different approaches to cross-language communication in Erlang: REST, Protocol Buffers, and Bert-RPC.

Three different approaches to cross-language communication in Erlang: REST, Protocol Buffers, and Bert-RPC.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,723
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
38
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. EVERYBODY POLYGLOT!(Cross-Language RPC with Erlang) ErlangDC · December 2011 Rusty Klophaus - @rustyio Basho Technologies
  • 2. Languages have Strengths http://wordaligned.org/articles/distorted-software 2 @rustyio
  • 3. Connections are Hard Serialization Versioning & Upgrades Data Type MismatchesSpeed, Bottlenecks & Back-Pressure Inadequate Tooling Context Switching 3 @rustyio
  • 4. Connections are Hard http://wordaligned.org/articles/distorted-software 4 @rustyio
  • 5. This TalkCreate An Example Service Any service will do, just need a framework for discussion.Expose Application via Interfaces • REST / JSON via Webmachine & Spooky • Protocol Buffers via erlang-protobuffs • BERT-RPC via Ernie ServerCode http://github.com/rustyio/ErlangRPCDemo 5 @rustyio
  • 6. This TalkOur Application: A Sequence Server Erlang service that returns a sequence of numbers. 1 sequence(N) -> 2 List1 = lists:seq(1, N), 3 List2 = [list_to_binary(integer_to_list(X)) || X <- List1], 4 {ok, List2}; 5 6 %% sequence(2). 7 {ok, [<<"1">>, <<"2">>]}. 8 9 %% sequence(5). 10 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, <<"5">>]}. 11 12 %% sequence(50000). 13 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, ...]}. 6 @rustyio
  • 7. Overview REST?Protocol Buffers? BERT-RPC? 7 @rustyio
  • 8. REST / JSON - OverviewREST - Representational State Transfer Convention for talking to applications over HTTP Actions are Verbs are HTTP Methods (GET/PUT/POST/DELETE/...) Objects are nouns are URLs GET /users/5JSON - Javascript Object Notation Encode data as parseable Javascript Understood by everything Human Readable {"id":5,"first":"Rusty","last":"Klophaus"} 8 @rustyio
  • 9. REST / JSON - Strengths & WeaknessesStrengths Simple, easy to poke around Good support in every language Composable - Caches, Reverse Proxies, Load BalancersWeaknesses General == More Handshaking/Metadata == More Overhead 9 @rustyio
  • 10. Protocol Buffers - OverviewProtocol Buffers Developed by Google It’s not a protocol, it’s a format: • You provide the client / server logic • Useful for transmission AND storage Define data structures in .proto file, generate code http://code.google.com/apis/protocolbuffers/ https://github.com/ngerakines/erlang_protobuffs 10 @rustyio
  • 11. Protocol Buffers - Strengths & WeaknessesStrengths Compact Add fields without breaking existing applications • Versioning is not strictWeaknesses Configuration file (.proto) with new syntax to learn Generated code Uneven language support 11 @rustyio
  • 12. BERT-RPC - OverviewBERT-RPC Developed by GitHub (Tom Preston-Werner) BERT = Binary Erlang Term • Encoding mimics native Erlang serialization http://bert-rpc.org https://github.com/mojombo/ernie 1 % Request 2 {call, ernie_sequence, sequence, [3]} 3 4 % Response 5 {response, {ok, [<<"1">>, <<"2">>, <<"3">>]}} 12 @rustyio
  • 13. BERT-RPC - Strengths & WeaknessesStrengths Easy to set up Agile CompactWeaknesses Uneven language support Less buzz than it deserves Not fully product-ized 13 @rustyio
  • 14. Requests & ResponsesWhat does the chatter look like? 14 @rustyio
  • 15. REST / JSON - Request & ResponseRequest GET /sequence/3 HTTP/1.1 User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 Host: localhost:8001 Accept: */*Response HTTP/1.1 200 OK Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) Date: Wed, 23 Nov 2011 22:09:13 GMT Content-Type: application/json Content-Length: 21 ["1","2","3"] 15 @rustyio
  • 16. Protocol Buffers - Encodingrpc_demo.proto 1 message SequenceRequest { 2 required uint32 n = 1; 3 } 4 5 message SequenceResponse { 6 repeated bytes sequence = 1; 7 } 16 @rustyio
  • 17. Protocol Buffers - RPCRequest 1 rpc_demo_pb:encode({sequencerequest, 3}). 2 <<8,3>>Response 1 rpc_demo_pb:encode({sequenceresponse, [<<"1">>, <<"2">>, <<"3">>]}). 2 <<10,1,49,10,1,50,10,1,51>> 17 @rustyio
  • 18. BERT-RPC - Encoding1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]).2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>345 bert:encode([<<"1">>,<<"2">>,<<"3">>]).6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 18 @rustyio
  • 19. BERT-RPC - Encoding 1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]). 2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 3 4 5 bert:encode([<<"1">>,<<"2">>,<<"3">>]). 6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>> 7 | | | | | | 8 | | | | | + Ascii value for 1 9 | | | | + Length of string (1)10 | | | + Next term is a string11 | | + Length of list (3)12 | + Next term is a list13 + Start of Erlang term 19 @rustyio
  • 20. BERT-RPC - RPCRequest 1 bert:encode({call, ernie_sequence, sequence, [5]}). 3 <<131,104,4,100,0,4,99,97,108,108,100,0,14,101,114,110, 4 105,101,95,115,101,113,117,101,110,99,101,100,0,8,115, 5 101,113,117,101,110,99,101,107,0,1,5>>Response 1 bert:encode({reply,{ok,[<<"1">>,<<"2">>,<<"3">>,<<"4">>,<<"5">>]}}). 2 3 <<131,104,2,100,0,5,114,101,112,108,121,104,2,100,0,2,111, 4 107,108,0,0,0,5,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0, 5 1,51,109,0,0,0,1,52,109,0,0,0,1,53,106>> 20 @rustyio
  • 21. Show me the Client Code! 21 @rustyio
  • 22. REST / JSON - ClientsCURL curl http://localhost:8001/sequence/5Ruby Client 1 url = "http://localhost:8001/sequence/5" 2 resp = Net::HTTP.get_response(URI.parse(url)) 3 sequence = JSON.parse(resp.body) 22 @rustyio
  • 23. Protocol Buffers - ClientRuby Client 1 require beefcake 2 3 class SequenceRequest 4 include Beefcake::Message 5 required :n, :int32, 1 6 end 7 8 class SequenceResponse 9 include Beefcake::Message 10 repeated :sequence, :string, 1 11 end 12 13 req = SequenceRequest.new(:n => 5) 14 Socket.tcp("localhost", 8003) do |socket| 15 socket.write(req.encode) 16 m = socket.read 17 return SequenceResponse.decode(m) 18 end 23 @rustyio
  • 24. BERT-RPC - ClientRuby Client 1 require bert-rpc 2 3 svc = BERTRPC::Service.new(localhost, 9999) 4 sequence = svc.call.ernie_sequence.sequence(5) 24 @rustyio
  • 25. Show me the Server Code! 25 @rustyio
  • 26. REST / JSON - ServerSpooky Server 1 -module(spooky_sequence). 2 -behaviour(spooky). 3 -export([init/1, get/2]). 4 5 init([])-> 6 [{port, 8002}]. 7 8 get(_Req, ["sequence", Num])-> 9 case sequence:sequence(Num) of 10 {ok, List} -> 11 {200, mochijson2:encode(List)}; 12 {error, Error} -> 13 {500, io_lib:format("~w", [Error])} 14 end; 15 get(_Req, _)-> 16 {400, "Usage: /sequence/:Num:"}. 26 @rustyio
  • 27. REST / JSON - ServerWebmachine Server 1 -module(webmachine_sequence). 2 -export([init/1, content_types_provided/2, to_json/2]). 3 -include_lib("webmachine/include/webmachine.hrl"). 4 5 -record(ctx, { list }). 6 7 init([]) -> 8 {ok, #ctx {}}. 9 10 content_types_provided(RD, Ctx) -> 11 Types = [{"application/json", to_json}], 12 {Types, RD, Ctx}. 13 14 to_json(RD, Ctx) -> 15 {ok, List} = sequence:sequence(N), 16 Body = mochijson2:encode(List), 17 {Body, RD, Ctx}. 27 @rustyio
  • 28. Protocol Buffers - ServerErlang Server 1 %% Erlang RPC Demo 57 handle_info({tcp, _Sock, _Data}, State) -> 2 %% Copyright (c) 2011 Rusty Klophaus (@rustyio) 58 %% req =/= undefined: received a new request while another was in 3 %% See MIT-LICENSE for licensing information. 59 %% progress -> Error 4 60 lager:error("Received a new PB socket request" 5 -module(protobuff_server). 61 " while another was in progress"), 6 -behaviour(gen_server). 62 {stop, normal, State}; 7 63 8 -export([ 64 handle_info(_, State) -> % Ignore any late replies from gen_servers/messages from fsms 9 start_link/0, 65 {noreply, State}. 10 set_socket/2, 66 11 init/1, 67 terminate(_Reason, _State) -> 12 handle_call/3, 68 ok. 13 handle_cast/2, 69 14 handle_info/2, 70 code_change(_OldVsn, State, _Extra) -> {ok, State}. 15 terminate/2, 71 16 code_change/3, 72 %% =================================================================== 17 encode/1, 73 %% Handle PB Messages 18 decode/1]). 74 %% =================================================================== 19 75 20 -record(state, { sock }). 76 process_message(#sequencerequest { n = N }, State) -> 21 77 case sequence:sequence(N) of 22 -include("rpc_demo_pb.hrl"). 78 {ok, List} -> 23 79 Resp = #sequenceresponse { sequence = List }, 24 %% =================================================================== 80 send_msg(Resp, State); 25 %% Public API 81 {error, Reason} -> 26 %% =================================================================== 82 Msg = io_lib:format("~w", [Reason]), 27 83 Resp = #sequenceerror { message=Msg }, 28 start_link() -> 84 send_msg(Resp, State) 29 gen_server:start_link(?MODULE, [], []). 85 end. 30 86 31 set_socket(Pid, Socket) -> 87 %% Send a message to the client 32 gen_server:call(Pid, {set_socket, Socket}). 88 send_msg(Msg, State) -> 33 89 Pkt = encode(Msg), 34 init([]) -> 90 gen_tcp:send(State#state.sock, Pkt), 35 {ok, #state{}}. 91 State. 36 92 37 handle_call({set_socket, Socket}, _From, State) -> 93 encode(Msg) -> 38 inet:setopts(Socket, [{active, once}, {packet, 4}, {header, 1}]), 94 MsgType = element(1, Msg), 39 {reply, ok, State#state{sock = Socket}}. 95 [msg_code(Msg) | rpc_demo_pb:iolist(MsgType, Msg)]. 40 96 41 handle_cast(_Msg, State) -> 97 decode([MsgCode|MsgData]) -> 42 {noreply, State}. 98 MsgType = msg_type(MsgCode), 43 99 rpc_demo_pb:decode(MsgType, MsgData). 44 handle_info({tcp_closed, Socket}, State=#state{sock=Socket}) -> 100 45 {stop, normal, State}; 101 msg_code(#sequencerequest {}) -> 1; 46 handle_info({tcp_error, Socket, _Reason}, State=#state{sock=Socket}) -> 102 msg_code(#sequenceresponse {}) -> 2; 47 {stop, normal, State}; 103 msg_code(#sequenceerror {}) -> 3; 48 handle_info({tcp, _Sock, MsgData}, State=#state{sock=Socket}) -> 104 msg_code(Other) -> 49 Msg = decode(MsgData), 105 throw({unknown_pb_type, Other}). 50 case process_message(Msg, State) of 106 51 {pause, NewState} -> 107 msg_type(1) -> sequencerequest; 52 ok; 108 msg_type(2) -> sequenceresponse; 53 NewState -> 109 msg_type(3) -> sequenceerror; 54 inet:setopts(Socket, [{active, once}]) 110 msg_type(Other) -> 55 end, 111 throw({unknown_pb_code, Other}). 56 {noreply, NewState}; 28 @rustyio
  • 29. BERT-RPC - ServerErnie Server - Erlang BERT-RPC Server 1 %% FILE: ernie.config 2 [ 3 {module, ernie_sequence}, 4 {type, native}, 5 {codepaths, []} 6 ]. 1 -module(ernie_sequence). 2 -export([sequence/1]). 3 4 sequence(N) -> 5 sequence:sequence(N). 29 @rustyio
  • 30. BenchmarksHow fast is it? 30 @rustyio
  • 31. BenchmarksDisclaimers Benchmarking is hard The shape of your data matters (size and complexity) Speed is not the only objective 31 @rustyio
  • 32. Benchmarks:Encoding Speed 32 @rustyio
  • 33. Encoding Speed - 5 items BERT JSON PB 33 @rustyio
  • 34. Encoding Speed - 50 items BERT JSON PB 34 @rustyio
  • 35. Encoding Speed - 500 items BERT JSON PB 35 @rustyio
  • 36. Encoding Speed - 5,000 items BERT JSON PB 36 @rustyio
  • 37. Encoding Speed - 50,000 items BERT JSON Sad Protocol Buffers :( 37 @rustyio
  • 38. Encoding Speed - Ruby | 5 | 50 | 500 | 5,000 -----+---------+---------+---------+---------- JSON | 0.03 ms | 0.77 ms | 0.47 ms | 4.21 ms PB | 0.07 ms | 0.60 ms | 7.37 ms | 197.24 ms BERT | 0.08 ms | 0.52 ms | 4.97 ms | 52.42 ms 38 @rustyio
  • 39. Encoding SpeedInterpretations BERT wins in Erlang, because it’s native. JSON wins in Ruby. Protocol Buffers is slow all around for complex data. • Note: This is different from large data. 39 @rustyio
  • 40. Benchmarks: Single Hop$ ping dell.localPING dell.local (192.168.2.2): 56 data bytes64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.490 ms 40 @rustyio
  • 41. BenchmarksMore Disclaimers First approach exhausted TCP connections “Fixed” by tunneling connections, which impacts results 41 @rustyio
  • 42. Operation Performance - 5 items BERT PB PB SPOOKY WEBMACHINE 42 @rustyio
  • 43. Operation Performance - 50 items BERT PB PB SPOOKY WEBMACHINE 43 @rustyio
  • 44. Operation Performance - 500 items BERT PB PB SPOOKY WEBMACHINE 44 @rustyio
  • 45. Operation Performance - 5,000 items BERT PB PB SPOOKY WEBMACHINE 45 @rustyio
  • 46. Encoding SpeedInterpretations Performance is fairly even for simple data. PB is clear loser for data with lots of items. Webmachine pays a tax for considering entire HTTP decision tree. Take these with a grain of salt, everything is tunneled over SSH. 46 @rustyio
  • 47. Webmachine - HTTP Decision Tree 47 @rustyio
  • 48. Benchmarks: Multiple Hops$ ping rusty.ioPING rusty.io (173.203.217.46): 56 data bytes64 bytes from 173.203.217.46: icmp_seq=0 ttl=53 time=49.550 ms 48 @rustyio
  • 49. Operation Performance - 5 items BERT PB PB SPOOKY WEBMACHINE 49 @rustyio
  • 50. Operation Performance - 50 items BERT PB PB SPOOKY WEBMACHINE 50 @rustyio
  • 51. Operation Performance - 500 items BERT PB PB SPOOKY WEBMACHINE 51 @rustyio
  • 52. Operation Performance - 5,000 items BERT PB PB SPOOKY WEBMACHINE 52 @rustyio
  • 53. Operation Performance - Multiple HopsInterpretations No clear winner. Network speed / variability is the bottleneck. Take these with a grain of salt, everything is tunneled over SSH. 53 @rustyio
  • 54. The ResultsRecommendations Start with REST / JSON Optimize for performance with Protocol Buffers Get adventurous with BERT-RPC • Easy to set up == Easy to back outGet Involved! Protocol Buffers NIF? JSON NIF? General Ernie (BERT-RPC) improvements: • Re-use connections, better packaging & documentation 54 @rustyio
  • 55. Honorable MentionsOther RPC Frameworks: Thrift - http://thrift.apache.org/ (Originally Facebook) Avro - http://avro.apache.org/docs/current/ XML-RPC CORBA - http://www.erlang.org/doc/man/corba.html UBF - http://www.sics.se/~joe/ubf/site/home.html MsgPack - http://msgpack.org/ Etch - http://incubator.apache.org/projects/etch.html ASN.1 - http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_OneErlang <-> Other Code: Erlang "ports" - integration via stdin/stdout. Bifs - Extend Erlang with new functions."Fake" Erlang Nodes: JInterface (it is... not great) C Nodes - http://www.erlang.org/doc/tutorial/cnode.html 55 @rustyio
  • 56. Related Talks• Anton Lavrik - Piqi-RPC: exposing Erlang services via JSON, XML and Google Protocol Buffers over HTTP http://www.erlang-factory.com/conference/SFBay2011/speakers/AntonLavrik• Tom Preston-Werner - BERT is to Erlang as JSON is to JavaScript http://www.erlang-factory.com/conference/ErlangUserConference2009/speakers/ TomPrestonWerner• Todd Lipcon - Thrift Avro/Erlang Bindings http://www.erlang-factory.com/conference/SFBay2010/speakers/toddlipcon• Cliff Moon - Building Polyglot Distributed Systems with Jinterface http://www.erlang-factory.com/conference/SFBay2011/speakers/CliffMoon• Kresten Krab Thorup - Erjang - A JVM-based Erlang VM http://www.erlang-factory.com/conference/SFBay2010/speakers/ KrestenKrabThorup• Yurii Rashkovskii - Beam.JS: Erlang meets JavaScript http://www.erlang-factory.com/conference/SFBay2011/speakers/YuriiRashkovskii 56 @rustyio
  • 57. Thanks!Questions? 57 @rustyio