Amazon's Dynamo in POE and Erlang @ YAPC::Asia 2008 LT

Developing Amazon’s Dynamo in POE and Erlang – Prologue of “From POE to Erlang” takemaru

Dynamo ?  Amazon CTO Werner Vogels

Dynamo ?  Key-value  my %hash = ( key1 => “value1”, key2 => “value2”, );  memcached

Dynamo ?  Key-value 

Dynamo ?  Key-value  Latency < 300ms

Dynamo ?  Key-value  Lock

Dynamo ?  Key-value  …………… …………… …………… …………… …………… …………… Dynamo Bigtable/GFS

Kai – Yet Anothoer Amazon’s Dynamo  Dynamo  POE  Erlang memcache

 Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

 3  Consistent Hashing Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

 1 Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

 2   Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

  Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

POE (Event Driven)  Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State my $State; sub recv_from_client; sub recv_from_node; Dynamo Dynamo sub timeout; Node Node sub another_error;

POE (Event Driven)  Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_client { $cli_sock->recv(my $req, 1024); Dynamo Dynamo Node Node

POE (Event Driven)  3 Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node state_init($cli_sock); my @nodes = choose_nodes($req); State for my $node (@nodes) { $sock = IO::Socket::Inet->new(…); $sock->write($req); $poe->kernel->select_read( Dynamo Dynamo $sock, Node Node ‘recv_from_node’, $cli_sock, ); }

POE (Event Driven)  1 Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_node { my $cli_sock = $poe->args->[2]; $node_sock->read(my $res, 1024); Dynamo Dynamo stat_add_res($cli_sock, $res); Node Node my $count = stat_count_res($cli_sock); if ($count == 2) {

POE (Event Driven)  2  Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_node { my $cli_sock = $poe->args->[2]; $node_sock->read(my $res, 1024); Dynamo Dynamo stat_add_res($cli_sock, $res); Node Node my $count = stat_count_res($cli_sock); if ($count == 2) { my $res = stat_uniq_res($cli_sock); $cli_sock->write($res); }

POE (Event Driven)  Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State Client Receive from Client Receive from Node Node

Erlang  Dynamo Dynamo Node Node Process for Client Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node

Erlang  Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node recv(Sock) -> {ok, Req} = gen_tcp:recv(Sock, 0), lists:foreach( Dynamo Dynamo fun(Node) -> spawn(Mod, map, [Node, Req, self()]) end, choose_nodes(Req) Node Node ), Res = gather(2, []),

Erlang  3 Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), Dynamo Dynamo Node Node

Erlang  1 Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), receive Dynamo Dynamo {tcp, Sock, Res} -> Node Node send(Parent, Res);

Erlang  2  Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), receive Dynamo Dynamo {tcp, Sock, Res} -> Node Node send(Parent, Res);

Erlang  2  Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node Res = gather(2, []), gen_tcp:send(Sock, Res). gather(0, Acc) -> Acc; Dynamo Dynamo gather(N, Acc) -> Node Node receive Res -> gather(N-1, uniq([Res|Acc])), after 100 -> % timeout Acc end.

Erlang  Process for Node Process for Client Dynamo Dynamo Client Node Node Client Process for Client Process for Node Node

 Erlang       

Amazon's Dynamo in POE and Erlang @ YAPC::Asia 2008 LT

0 comments

Notes on slide 1

2 Favorites & 1 Event