Three different approaches to cross-language communication in Erlang: REST, Protocol Buffers, and Bert-RPC.

1. EVERYBODY POLYGLOT!
(Cross-Language RPC with Erlang)
ErlangDC · December 2011
Rusty Klophaus - @rustyio
Basho Technologies

2. Languages have Strengths
http://wordaligned.org/articles/distorted-software
2 @rustyio

3. Connections are Hard
Serialization
Versioning & Upgrades
Data Type Mismatches
Speed, Bottlenecks & Back-Pressure
Inadequate Tooling
Context Switching
3 @rustyio

4. Connections are Hard
http://wordaligned.org/articles/distorted-software
4 @rustyio

5. This Talk
Create An Example Service
Any service will do, just need a framework for discussion.
Expose Application via Interfaces
• REST / JSON via Webmachine & Spooky
• Protocol Buffers via erlang-protobuffs
• BERT-RPC via Ernie Server
Code
http://github.com/rustyio/ErlangRPCDemo
5 @rustyio

6. This Talk
Our Application: A Sequence Server
Erlang service that returns a sequence of numbers.
1 sequence(N) ->
2 List1 = lists:seq(1, N),
3 List2 = [list_to_binary(integer_to_list(X)) || X <- List1],
4 {ok, List2};
5
6 %% sequence(2).
7 {ok, [<<"1">>, <<"2">>]}.
8
9 %% sequence(5).
10 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, <<"5">>]}.
11
12 %% sequence(50000).
13 {ok, [<<"1">>, <<"2">>, <<"3">>, <<"4">>, ...]}.
6 @rustyio

7. Overview
REST?
Protocol Buffers?
BERT-RPC?
7 @rustyio

8. REST / JSON - Overview
REST - Representational State Transfer
Convention for talking to applications over HTTP
Actions are Verbs are HTTP Methods (GET/PUT/POST/DELETE/...)
Objects are nouns are URLs
GET /users/5
JSON - Javascript Object Notation
Encode data as parseable Javascript
Understood by everything
Human Readable
{"id":5,"first":"Rusty","last":"Klophaus"}
8 @rustyio

9. REST / JSON - Strengths & Weaknesses
Strengths
Simple, easy to poke around
Good support in every language
Composable - Caches, Reverse Proxies, Load Balancers
Weaknesses
General == More Handshaking/Metadata == More Overhead
9 @rustyio

10. Protocol Buffers - Overview
Protocol Buffers
Developed by Google
It’s not a protocol, it’s a format:
• You provide the client / server logic
• Useful for transmission AND storage
Define data structures in .proto file, generate code
http://code.google.com/apis/protocolbuffers/
https://github.com/ngerakines/erlang_protobuffs
10 @rustyio

11. Protocol Buffers - Strengths & Weaknesses
Strengths
Compact
Add fields without breaking existing applications
• Versioning is not strict
Weaknesses
Configuration file (.proto) with new syntax to learn
Generated code
Uneven language support
11 @rustyio

12. BERT-RPC - Overview
BERT-RPC
Developed by GitHub (Tom Preston-Werner)
BERT = Binary Erlang Term
• Encoding mimics native Erlang serialization
http://bert-rpc.org
https://github.com/mojombo/ernie
1 % Request
2 {call, ernie_sequence, sequence, [3]}
3
4 % Response
5 {response, {ok, [<<"1">>, <<"2">>, <<"3">>]}}
12 @rustyio

13. BERT-RPC - Strengths & Weaknesses
Strengths
Easy to set up
Agile
Compact
Weaknesses
Uneven language support
Less buzz than it deserves
Not fully product-ized
13 @rustyio

14. Requests & Responses
What does the chatter look like?
14 @rustyio

15. REST / JSON - Request & Response
Request
GET /sequence/3 HTTP/1.1
User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4
OpenSSL/0.9.8r zlib/1.2.5
Host: localhost:8001
Accept: */*
Response
HTTP/1.1 200 OK
Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
Date: Wed, 23 Nov 2011 22:09:13 GMT
Content-Type: application/json
Content-Length: 21
["1","2","3"]
15 @rustyio

16. Protocol Buffers - Encoding
rpc_demo.proto
1 message SequenceRequest {
2 required uint32 n = 1;
3 }
4
5 message SequenceResponse {
6 repeated bytes sequence = 1;
7 }
16 @rustyio

17. Protocol Buffers - RPC
Request
1 rpc_demo_pb:encode({sequencerequest, 3}).
2 <<8,3>>
Response
1 rpc_demo_pb:encode({sequenceresponse, [<<"1">>, <<"2">>, <<"3">>]}).
2 <<10,1,49,10,1,50,10,1,51>>
17 @rustyio

18. BERT-RPC - Encoding
1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]).
2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>
3
4
5 bert:encode([<<"1">>,<<"2">>,<<"3">>]).
6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>
18 @rustyio

19. BERT-RPC - Encoding
1 term_to_binary([<<"1">>,<<"2">>,<<"3">>]).
2 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>
3
4
5 bert:encode([<<"1">>,<<"2">>,<<"3">>]).
6 <<131,108,0,0,0,3,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,1,51,106>>
7 | | | | | |
8 | | | | | + Ascii value for '1'
9 | | | | + Length of string (1)
10 | | | + Next term is a string
11 | | + Length of list (3)
12 | + Next term is a list
13 + Start of Erlang term
19 @rustyio

20. BERT-RPC - RPC
Request
1 bert:encode({call, ernie_sequence, sequence, [5]}).
3 <<131,104,4,100,0,4,99,97,108,108,100,0,14,101,114,110,
4 105,101,95,115,101,113,117,101,110,99,101,100,0,8,115,
5 101,113,117,101,110,99,101,107,0,1,5>>
Response
1 bert:encode({reply,{ok,[<<"1">>,<<"2">>,<<"3">>,<<"4">>,<<"5">>]}}).
2
3 <<131,104,2,100,0,5,114,101,112,108,121,104,2,100,0,2,111,
4 107,108,0,0,0,5,109,0,0,0,1,49,109,0,0,0,1,50,109,0,0,0,
5 1,51,109,0,0,0,1,52,109,0,0,0,1,53,106>>
20 @rustyio

21. Show me the Client Code!
21 @rustyio

22. REST / JSON - Clients
CURL
curl http://localhost:8001/sequence/5
Ruby Client
1 url = "http://localhost:8001/sequence/5"
2 resp = Net::HTTP.get_response(URI.parse(url))
3 sequence = JSON.parse(resp.body)
22 @rustyio

23. Protocol Buffers - Client
Ruby Client
1 require 'beefcake'
2
3 class SequenceRequest
4 include Beefcake::Message
5 required :n, :int32, 1
6 end
7
8 class SequenceResponse
9 include Beefcake::Message
10 repeated :sequence, :string, 1
11 end
12
13 req = SequenceRequest.new(:n => 5)
14 Socket.tcp("localhost", 8003) do |socket|
15 socket.write(req.encode)
16 m = socket.read
17 return SequenceResponse.decode(m)
18 end
23 @rustyio

24. BERT-RPC - Client
Ruby Client
1 require 'bert-rpc'
2
3 svc = BERTRPC::Service.new('localhost', 9999)
4 sequence = svc.call.ernie_sequence.sequence(5)
24 @rustyio

25. Show me the Server Code!
25 @rustyio

26. REST / JSON - Server
Spooky Server
1 -module(spooky_sequence).
2 -behaviour(spooky).
3 -export([init/1, get/2]).
4
5 init([])->
6 [{port, 8002}].
7
8 get(_Req, ["sequence", Num])->
9 case sequence:sequence(Num) of
10 {ok, List} ->
11 {200, mochijson2:encode(List)};
12 {error, Error} ->
13 {500, io_lib:format("~w", [Error])}
14 end;
15 get(_Req, _)->
16 {400, "Usage: /sequence/:Num:"}.
26 @rustyio

27. REST / JSON - Server
Webmachine Server
1 -module(webmachine_sequence).
2 -export([init/1, content_types_provided/2, to_json/2]).
3 -include_lib("webmachine/include/webmachine.hrl").
4
5 -record(ctx, { list }).
6
7 init([]) ->
8 {ok, #ctx {}}.
9
10 content_types_provided(RD, Ctx) ->
11 Types = [{"application/json", to_json}],
12 {Types, RD, Ctx}.
13
14 to_json(RD, Ctx) ->
15 {ok, List} = sequence:sequence(N),
16 Body = mochijson2:encode(List),
17 {Body, RD, Ctx}.
27 @rustyio

28. Protocol Buffers - Server
Erlang Server
1 %% Erlang RPC Demo
57 handle_info({tcp, _Sock, _Data}, State) ->
2 %% Copyright (c) 2011 Rusty Klophaus (@rustyio)
58 %% req =/= undefined: received a new request while another was in
3 %% See MIT-LICENSE for licensing information.
59 %% progress -> Error
4
60 lager:error("Received a new PB socket request"
5 -module(protobuff_server).
61 " while another was in progress"),
6 -behaviour(gen_server).
62 {stop, normal, State};
7
63
8 -export([
64 handle_info(_, State) -> % Ignore any late replies from gen_servers/messages from fsms
9 start_link/0,
65 {noreply, State}.
10 set_socket/2,
66
11 init/1,
67 terminate(_Reason, _State) ->
12 handle_call/3,
68 ok.
13 handle_cast/2,
69
14 handle_info/2,
70 code_change(_OldVsn, State, _Extra) -> {ok, State}.
15 terminate/2,
71
16 code_change/3,
72 %% ===================================================================
17 encode/1,
73 %% Handle PB Messages
18 decode/1]).
74 %% ===================================================================
19
75
20 -record(state, { sock }).
76 process_message(#sequencerequest { n = N }, State) ->
21
77 case sequence:sequence(N) of
22 -include("rpc_demo_pb.hrl").
78 {ok, List} ->
23
79 Resp = #sequenceresponse { sequence = List },
24 %% ===================================================================
80 send_msg(Resp, State);
25 %% Public API
81 {error, Reason} ->
26 %% ===================================================================
82 Msg = io_lib:format("~w", [Reason]),
27
83 Resp = #sequenceerror { message=Msg },
28 start_link() ->
84 send_msg(Resp, State)
29 gen_server:start_link(?MODULE, [], []).
85 end.
30
86
31 set_socket(Pid, Socket) ->
87 %% Send a message to the client
32 gen_server:call(Pid, {set_socket, Socket}).
88 send_msg(Msg, State) ->
33
89 Pkt = encode(Msg),
34 init([]) ->
90 gen_tcp:send(State#state.sock, Pkt),
35 {ok, #state{}}.
91 State.
36
92
37 handle_call({set_socket, Socket}, _From, State) ->
93 encode(Msg) ->
38 inet:setopts(Socket, [{active, once}, {packet, 4}, {header, 1}]),
94 MsgType = element(1, Msg),
39 {reply, ok, State#state{sock = Socket}}.
95 [msg_code(Msg) | rpc_demo_pb:iolist(MsgType, Msg)].
40
96
41 handle_cast(_Msg, State) ->
97 decode([MsgCode|MsgData]) ->
42 {noreply, State}.
98 MsgType = msg_type(MsgCode),
43
99 rpc_demo_pb:decode(MsgType, MsgData).
44 handle_info({tcp_closed, Socket}, State=#state{sock=Socket}) ->
100
45 {stop, normal, State};
101 msg_code(#sequencerequest {}) -> 1;
46 handle_info({tcp_error, Socket, _Reason}, State=#state{sock=Socket}) ->
102 msg_code(#sequenceresponse {}) -> 2;
47 {stop, normal, State};
103 msg_code(#sequenceerror {}) -> 3;
48 handle_info({tcp, _Sock, MsgData}, State=#state{sock=Socket}) ->
104 msg_code(Other) ->
49 Msg = decode(MsgData),
105 throw({unknown_pb_type, Other}).
50 case process_message(Msg, State) of
106
51 {pause, NewState} ->
107 msg_type(1) -> sequencerequest;
52 ok;
108 msg_type(2) -> sequenceresponse;
53 NewState ->
109 msg_type(3) -> sequenceerror;
54 inet:setopts(Socket, [{active, once}])
110 msg_type(Other) ->
55 end,
111 throw({unknown_pb_code, Other}).
56 {noreply, NewState};
28 @rustyio

29. BERT-RPC - Server
Ernie Server - Erlang BERT-RPC Server
1 %% FILE: ernie.config
2 [
3 {module, ernie_sequence},
4 {type, native},
5 {codepaths, []}
6 ].
1 -module(ernie_sequence).
2 -export([sequence/1]).
3
4 sequence(N) ->
5 sequence:sequence(N).
29 @rustyio

30. Benchmarks
How fast is it?
30 @rustyio

31. Benchmarks
Disclaimers
Benchmarking is hard
The shape of your data matters (size and complexity)
Speed is not the only objective
31 @rustyio

32. Benchmarks:
Encoding Speed
32 @rustyio

33. Encoding Speed - 5 items
BERT JSON PB
33 @rustyio

34. Encoding Speed - 50 items
BERT JSON PB
34 @rustyio

35. Encoding Speed - 500 items
BERT JSON PB
35 @rustyio

36. Encoding Speed - 5,000 items
BERT JSON PB
36 @rustyio

37. Encoding Speed - 50,000 items
BERT JSON
Sad Protocol Buffers :(
37 @rustyio

38. Encoding Speed - Ruby
| 5 | 50 | 500 | 5,000
-----+---------+---------+---------+----------
JSON | 0.03 ms | 0.77 ms | 0.47 ms | 4.21 ms
PB | 0.07 ms | 0.60 ms | 7.37 ms | 197.24 ms
BERT | 0.08 ms | 0.52 ms | 4.97 ms | 52.42 ms
38 @rustyio

39. Encoding Speed
Interpretations
BERT wins in Erlang, because it’s native.
JSON wins in Ruby.
Protocol Buffers is slow all around for complex data.
• Note: This is different from large data.
39 @rustyio

40. Benchmarks:
Single Hop
$ ping dell.local
PING dell.local (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.490 ms
40 @rustyio

41. Benchmarks
More Disclaimers
First approach exhausted TCP connections
“Fixed” by tunneling connections, which impacts results
41 @rustyio

42. Operation Performance - 5 items
BERT PB PB SPOOKY WEBMACHINE
42 @rustyio

43. Operation Performance - 50 items
BERT PB PB SPOOKY WEBMACHINE
43 @rustyio

44. Operation Performance - 500 items
BERT PB PB SPOOKY WEBMACHINE
44 @rustyio

45. Operation Performance - 5,000 items
BERT PB PB SPOOKY WEBMACHINE
45 @rustyio

46. Encoding Speed
Interpretations
Performance is fairly even for simple data.
PB is clear loser for data with lots of items.
Webmachine pays a tax for considering entire HTTP decision tree.
Take these with a grain of salt, everything is tunneled over SSH.
46 @rustyio

47. Webmachine - HTTP Decision Tree
47 @rustyio

48. Benchmarks:
Multiple Hops
$ ping rusty.io
PING rusty.io (173.203.217.46): 56 data bytes
64 bytes from 173.203.217.46: icmp_seq=0 ttl=53 time=49.550 ms
48 @rustyio

49. Operation Performance - 5 items
BERT PB PB SPOOKY WEBMACHINE
49 @rustyio

50. Operation Performance - 50 items
BERT PB PB SPOOKY WEBMACHINE
50 @rustyio

51. Operation Performance - 500 items
BERT PB PB SPOOKY WEBMACHINE
51 @rustyio

52. Operation Performance - 5,000 items
BERT PB PB SPOOKY WEBMACHINE
52 @rustyio

53. Operation Performance - Multiple Hops
Interpretations
No clear winner.
Network speed / variability is the bottleneck.
Take these with a grain of salt, everything is tunneled over SSH.
53 @rustyio

54. The Results
Recommendations
Start with REST / JSON
Optimize for performance with Protocol Buffers
Get adventurous with BERT-RPC
• Easy to set up == Easy to back out
Get Involved!
Protocol Buffers NIF?
JSON NIF?
General Ernie (BERT-RPC) improvements:
• Re-use connections, better packaging & documentation
54 @rustyio

55. Honorable Mentions
Other RPC Frameworks:
Thrift - http://thrift.apache.org/ (Originally Facebook)
Avro - http://avro.apache.org/docs/current/
XML-RPC
CORBA - http://www.erlang.org/doc/man/corba.html
UBF - http://www.sics.se/~joe/ubf/site/home.html
MsgPack - http://msgpack.org/
Etch - http://incubator.apache.org/projects/etch.html
ASN.1 - http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One
Erlang <-> Other Code:
Erlang "ports" - integration via stdin/stdout.
Bifs - Extend Erlang with new functions.
"Fake" Erlang Nodes:
JInterface (it is... not great)
C Nodes - http://www.erlang.org/doc/tutorial/cnode.html
55 @rustyio

56. Related Talks
• Anton Lavrik - Piqi-RPC: exposing Erlang services via JSON, XML and Google
Protocol Buffers over HTTP
http://www.erlang-factory.com/conference/SFBay2011/speakers/AntonLavrik
• Tom Preston-Werner - BERT is to Erlang as JSON is to JavaScript
http://www.erlang-factory.com/conference/ErlangUserConference2009/speakers/
TomPrestonWerner
• Todd Lipcon - Thrift Avro/Erlang Bindings
http://www.erlang-factory.com/conference/SFBay2010/speakers/toddlipcon
• Cliff Moon - Building Polyglot Distributed Systems with Jinterface
http://www.erlang-factory.com/conference/SFBay2011/speakers/CliffMoon
• Kresten Krab Thorup - Erjang - A JVM-based Erlang VM
http://www.erlang-factory.com/conference/SFBay2010/speakers/
KrestenKrabThorup
• Yurii Rashkovskii - Beam.JS: Erlang meets JavaScript
http://www.erlang-factory.com/conference/SFBay2011/speakers/YuriiRashkovskii
56 @rustyio

57. Thanks!
Questions?
57 @rustyio