Developing Amazon’s Dynamo in POE and Erlang
             – Prologue of “From POE to Erlang”


                           ...
Dynamo            ?
  Amazon   CTO   Werner Vogels
Dynamo               ?
  Key-value

   


                   my %hash = (
                       key1 => “value1”,
     ...
Dynamo         ?
  Key-value

   
Dynamo         ?
  Key-value

   
Dynamo            ?
  Key-value

                     Latency




               < 300ms
Dynamo         ?
  Key-value

                  Lock
Dynamo                   ?
  Key-value

   


                             ……………
                             ……………

   ...
Kai – Yet Anothoer Amazon’s Dynamo
         Dynamo
         POE




         Erlang   




                       memcache
              

                        Dynamo   Dynamo
                         Node
    Node




                   Dyn...
                              3
         Consistent Hashing       
                                        Dynamo   Dyna...
  1             

                          Dynamo   Dynamo
                           Node
    Node




                ...
  2
   
                     Dynamo   Dynamo
                       Node
    Node




                 Dynamo          ...
 
                   
                         Dynamo   Dynamo
                          Node
    Node




             ...
POE (Event Driven)
                                         

                                                Dynamo     ...
POE (Event Driven)
                                 

                                            Dynamo          Dynamo
...
POE (Event Driven)
                             3                   

                                            Dynamo ...
POE (Event Driven)
   1                          

                                             Dynamo          Dynamo
  ...
POE (Event Driven)
   2
     
                                              Dynamo          Dynamo
                     ...
POE (Event Driven)
                                               

               Receive from Client
 Timeout
   Receiv...
Erlang
                             

                                          Dynamo   Dynamo
                         ...
Erlang
                                      

                                                Dynamo          Dynamo
   ...
Erlang
                                3

                                             Dynamo          Dynamo
           ...
Erlang
   1                            

                                             Dynamo          Dynamo
            ...
Erlang
   2
    
                                             Dynamo          Dynamo
                                   ...
Erlang
   2
    
                                             Dynamo          Dynamo
                                   ...
Erlang
                             
                                          Process for Node
               Process fo...
  Erlang

   

   

        


   

        


   

        
Upcoming SlideShare
Loading in...5
×

Amazon's Dynamo in POE and Erlang @ YAPC::Asia 2008 LT

2,333

Published on

Published in: Technology, News & Politics
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,333
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
64
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Amazon's Dynamo in POE and Erlang @ YAPC::Asia 2008 LT

  1. 1. Developing Amazon’s Dynamo in POE and Erlang – Prologue of “From POE to Erlang” takemaru
  2. 2. Dynamo ?   Amazon CTO Werner Vogels
  3. 3. Dynamo ?   Key-value   my %hash = ( key1 => “value1”, key2 => “value2”, );   memcached
  4. 4. Dynamo ?   Key-value  
  5. 5. Dynamo ?   Key-value  
  6. 6. Dynamo ?   Key-value   Latency < 300ms
  7. 7. Dynamo ?   Key-value   Lock
  8. 8. Dynamo ?   Key-value   …………… …………… …………… …………… …………… …………… Dynamo Bigtable/GFS
  9. 9. Kai – Yet Anothoer Amazon’s Dynamo   Dynamo   POE   Erlang memcache
  10. 10.   Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  11. 11.   3   Consistent Hashing Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  12. 12.   1 Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  13. 13.   2     Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  14. 14.     Dynamo Dynamo Node Node Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  15. 15. POE (Event Driven)   Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State my $State; sub recv_from_client; sub recv_from_node; Dynamo Dynamo sub timeout; Node Node sub another_error;
  16. 16. POE (Event Driven)   Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_client { $cli_sock->recv(my $req, 1024); Dynamo Dynamo Node Node
  17. 17. POE (Event Driven)   3 Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node state_init($cli_sock); my @nodes = choose_nodes($req); State for my $node (@nodes) { $sock = IO::Socket::Inet->new(…); $sock->write($req); $poe->kernel->select_read( Dynamo Dynamo $sock, Node Node ‘recv_from_node’, $cli_sock, ); }
  18. 18. POE (Event Driven)   1 Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_node { my $cli_sock = $poe->args->[2]; $node_sock->read(my $res, 1024); Dynamo Dynamo stat_add_res($cli_sock, $res); Node Node my $count = stat_count_res($cli_sock); if ($count == 2) {
  19. 19. POE (Event Driven)   2   Dynamo Dynamo Node Node Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State sub recv_from_node { my $cli_sock = $poe->args->[2]; $node_sock->read(my $res, 1024); Dynamo Dynamo stat_add_res($cli_sock, $res); Node Node my $count = stat_count_res($cli_sock); if ($count == 2) { my $res = stat_uniq_res($cli_sock); $cli_sock->write($res); }
  20. 20. POE (Event Driven)   Receive from Client Timeout Receive from Node Dynamo Dynamo Client Node Node State Client Receive from Client Receive from Node Node
  21. 21. Erlang   Dynamo Dynamo Node Node Process for Client Dynamo Dynamo Client Node Node Dynamo Dynamo Node Node
  22. 22. Erlang   Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node recv(Sock) -> {ok, Req} = gen_tcp:recv(Sock, 0), lists:foreach( Dynamo Dynamo fun(Node) -> spawn(Mod, map, [Node, Req, self()]) end, choose_nodes(Req) Node Node ), Res = gather(2, []),
  23. 23. Erlang   3 Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), Dynamo Dynamo Node Node
  24. 24. Erlang   1 Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), receive Dynamo Dynamo {tcp, Sock, Res} -> Node Node send(Parent, Res);
  25. 25. Erlang   2   Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node map(Node, Req, Parent) -> {ok, Sock} = gen_tcp:connect(Node, …), gen_tcp:send(Sock, Req), receive Dynamo Dynamo {tcp, Sock, Res} -> Node Node send(Parent, Res);
  26. 26. Erlang   2   Dynamo Dynamo Node Node Processes for Nodes Process for Client Dynamo Dynamo Client Node Node Res = gather(2, []), gen_tcp:send(Sock, Res). gather(0, Acc) -> Acc; Dynamo Dynamo gather(N, Acc) -> Node Node receive Res -> gather(N-1, uniq([Res|Acc])), after 100 -> % timeout Acc end.
  27. 27. Erlang   Process for Node Process for Client Dynamo Dynamo Client Node Node Client Process for Client Process for Node Node
  28. 28.   Erlang              
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×