Riak Core:
                             Building Distributed
                             Applications Without
           ...
You suddenly feel
             an uncontrollable desire to
                    learn Erlang.




  http://www.flickr.com/ph...
What is Riak Core?
                             How does it work?
                             How can you use it?




   ...
Distributed, scalable, failure-tolerant.




                                 4
Wednesday, October 6, 2010
Distributed, scalable, failure-tolerant.

                             No central coordinator.
                           ...
Distributed, scalable, failure-tolerant.

                               Horizontally scalable;
                          ...
Distributed, scalable, failure-tolerant.

                                 Always available.
                             ...
Basho Technologies

               Riak KV
                    Distributed, scalable, failure-tolerant key/value datastore...
Wednesday, October 6, 2010
Riak   Riak    Riak
                             KV     Core   Search




Wednesday, October 6, 2010
Riak Core is an Erlang library
                        that helps you build
                distributed, scalable, failure...
Amazon Dynamo




Wednesday, October 6, 2010
“We Generalized the
                             Dynamo Architecture and
                             Open-Sourced the Bit...
Wait, doesn’t *Erlang* let you build
               distributed, scalable, failure-tolerant
                           app...
Erlang makes it easy to connect the
               components of your application.
                             Client
   ...
Riak Core helps you build a service that
               harnesses the power of many nodes.

                             N...
“People use languages
                                        other than Erlang?!?
                                       ...
How does Riak Core work?




                                        18
Wednesday, October 6, 2010
Command   ObjectName, Payload




Wednesday, October 6, 2010
Predictable Routing




                                     20
Wednesday, October 6, 2010
Hash the Object Name
                   Command    ObjectName, Payload



                              SHA1(ObjName), Pay...
A Naive Approach
                   Command                ObjectName, Payload



                                        ...
A Naive Approach
                   Command                ObjectName, Payload



                                        ...
"All problems in computer
                               science can be solved by
                             another lev...
Add VNodes

               What
                    Virtual Node. Logical subdivision of the cluster.
                    ...
Routing with Consistent Hash
                   Command                            ObjectName, Payload



                ...
Adding a Node
                   Command                            ObjectName, Payload



                               ...
Removing a Node
                   Command                            ObjectName, Payload



                             ...
The Ring




                             Hash Location




Wednesday, October 6, 2010
The Ring




                             Preflist




Wednesday, October 6, 2010
Writing Replicas (n_val)




                                   Preflist when N=3



Wednesday, October 6, 2010
Routing Around Failures




                              X
                                   Preflist when N=3
          ...
Location of the Routing Layer




                                          33
Wednesday, October 6, 2010
Router in the Middle

                                      Client                    Client                  Client




 ...
Riak Core - Router on Each Node

                                      Client                    Client                   ...
Eventually - Router on the Client

                                      Client                    Client                 ...
No Shared State


                              Router          Router          Router          Router    Router
         ...
Gossip


                               Local      Incoming
                             Ring State   Ring State




     ...
Handoff

               When
                    Nodes is added to the system.
                    Node is removed from th...
Not Mentioned
                                Vector Clocks
                                Merkle Trees
                 ...
Distinguished
                                               gentlemen
                                                   ...
How do you use Riak Core?




                                        42
Wednesday, October 6, 2010
Two Things to Think About

               Command
                    Command = ObjectName, Payload
                    Th...
VNode Module
               Startup/Shutdown
                  init([Partition]) ->
                   {ok, State}

      ...
VNode Module
               Handoff Coordination
                  handoff_starting(Node, State) ->
                   {Bo...
Start the riak_core application

                                                      riak_core
                         ...
Start the riak_core application

                                                      riak_core
                         ...
Start the riak_core application




                             Start, coordinate, and supervise handoff.




           ...
Start the riak_core application

                                                      riak_core
                         ...
Start the riak_core application

                                                      riak_core
                         ...
Start the riak_core application

                                                      riak_core
                         ...
In your application...

                                                      riak_core
                             riak_...
In your application...

                                                      riak_core
                             riak_...
In your application...
                                                      riak_core
                             riak_c...
Start Sending Commands

            # Figure out the preflist...
            {_Verb, ObjName, _Payload} = Command,
       ...
Review

               Riak Core
                    Open source Erlang library for building distributed, scalable,
      ...
At the center of the universe,
                                     beneath the bottom-most turtle,
                      ...
At the center of the universe,
                             beneath the bottom-most turtle,
                              ...
Thanks! Questions?

               Learn More
                    More Information: http://wiki.basho.com
                ...
END




Wednesday, October 6, 2010
Upcoming SlideShare
Loading in...5
×

Riak Core: Building Distributed Applications Without Shared State

7,161

Published on

Riak Core--an open-source Erlang library created by Basho Technologies that powers Riak KV and Riak Search--allows developers to build distributed, scalable, failure-tolerant applications based on a generalized version of Amazon's Dynamo architecture. In this talk, Rusty explains why Riak Core was built, discuss what problems it solves and how it works, and walk through the steps to using Riak Core in an Erlang application.

Published in: Technology
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,161
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
212
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide

Riak Core: Building Distributed Applications Without Shared State

  1. 1. Riak Core: Building Distributed Applications Without Shared State Commercial Users of Functional Programming Baltimore, MD · October 2010 Rusty Klophaus (@rklophaus) Basho Technologies Wednesday, October 6, 2010
  2. 2. You suddenly feel an uncontrollable desire to learn Erlang. http://www.flickr.com/photos/procsilas/18014203 2 Wednesday, October 6, 2010
  3. 3. What is Riak Core? How does it work? How can you use it? 3 Wednesday, October 6, 2010
  4. 4. Distributed, scalable, failure-tolerant. 4 Wednesday, October 6, 2010
  5. 5. Distributed, scalable, failure-tolerant. No central coordinator. Easy to setup/operate. 5 Wednesday, October 6, 2010
  6. 6. Distributed, scalable, failure-tolerant. Horizontally scalable; add commodity hardware to get more X. 6 Wednesday, October 6, 2010
  7. 7. Distributed, scalable, failure-tolerant. Always available. No single point of failure. Self-healing. 7 Wednesday, October 6, 2010
  8. 8. Basho Technologies Riak KV Distributed, scalable, failure-tolerant key/value datastore. Started as a “Dynamo clone”. Map/Reduce, Lightweight Data Relations, Client APIs Riak Search Distributed, scalable, failure-tolerant full-text search engine. Near Realtime, Riak KV Integration, Solr Support 8 Wednesday, October 6, 2010
  9. 9. Wednesday, October 6, 2010
  10. 10. Riak Riak Riak KV Core Search Wednesday, October 6, 2010
  11. 11. Riak Core is an Erlang library that helps you build distributed, scalable, failure-tolerant applications. 11 Wednesday, October 6, 2010
  12. 12. Amazon Dynamo Wednesday, October 6, 2010
  13. 13. “We Generalized the Dynamo Architecture and Open-Sourced the Bits.” 13 Wednesday, October 6, 2010
  14. 14. Wait, doesn’t *Erlang* let you build distributed, scalable, failure-tolerant applications? 14 Wednesday, October 6, 2010
  15. 15. Erlang makes it easy to connect the components of your application. Client Service A Service B Service C Queue E Resource D Wednesday, October 6, 2010
  16. 16. Riak Core helps you build a service that harnesses the power of many nodes. Node A Node B Node C Node D Node E Node F Node G Node H Service Node I Node J Node K Node L Node M Node N Node O ... Wednesday, October 6, 2010
  17. 17. “People use languages other than Erlang?!? I find that hilarious.” -Al Gore actually said this to me. http://www.flickr.com/photos/scobleizer/2216445692 Wednesday, October 6, 2010
  18. 18. How does Riak Core work? 18 Wednesday, October 6, 2010
  19. 19. Command ObjectName, Payload Wednesday, October 6, 2010
  20. 20. Predictable Routing 20 Wednesday, October 6, 2010
  21. 21. Hash the Object Name Command ObjectName, Payload SHA1(ObjName), Payload 0 to 2^160 Wednesday, October 6, 2010
  22. 22. A Naive Approach Command ObjectName, Payload SHA1(ObjName), Payload Node A Node B Node C Node D Wednesday, October 6, 2010
  23. 23. A Naive Approach Command ObjectName, Payload SHA1(ObjName), Payload Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  24. 24. "All problems in computer science can be solved by another level of indirection." - David Wheeler 24 Wednesday, October 6, 2010
  25. 25. Add VNodes What Virtual Node. Logical subdivision of the cluster. Handles incoming commands, does work, replies. For Parallelism # of VNodes = maximum concurrent requests For Rebalancing the Cluster Smallest block that can be shifted to a new node. For Resiliance The system restarts failed VNodes. 25 Wednesday, October 6, 2010
  26. 26. Routing with Consistent Hash Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Wednesday, October 6, 2010
  27. 27. Adding a Node Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  28. 28. Removing a Node Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  29. 29. The Ring Hash Location Wednesday, October 6, 2010
  30. 30. The Ring Preflist Wednesday, October 6, 2010
  31. 31. Writing Replicas (n_val) Preflist when N=3 Wednesday, October 6, 2010
  32. 32. Routing Around Failures X Preflist when N=3 and node 0 is down. Wednesday, October 6, 2010
  33. 33. Location of the Routing Layer 33 Wednesday, October 6, 2010
  34. 34. Router in the Middle Client Client Client Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  35. 35. Riak Core - Router on Each Node Client Client Client Router Router Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  36. 36. Eventually - Router on the Client Client Client Client Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  37. 37. No Shared State Router Router Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
  38. 38. Gossip Local Incoming Ring State Ring State Are rings equivalent? Strictly descendent? Or different? Wednesday, October 6, 2010
  39. 39. Handoff When Nodes is added to the system. Node is removed from the system. Node has temporarily failed. What Ship the data backing a VNode from one node to another. 39 Wednesday, October 6, 2010
  40. 40. Not Mentioned Vector Clocks Merkle Trees Bloom Filters 40 Wednesday, October 6, 2010
  41. 41. Distinguished gentlemen prefer Erlang. http://www.flickr.com/photos/rebcal/3987226359 Wednesday, October 6, 2010
  42. 42. How do you use Riak Core? 42 Wednesday, October 6, 2010
  43. 43. Two Things to Think About Command Command = ObjectName, Payload The commands/requests/operations that you will send through the system. VNode Module The callback module that will receive the commands. 43 Wednesday, October 6, 2010
  44. 44. VNode Module Startup/Shutdown init([Partition]) -> {ok, State} terminate(State) -> ok Commands handle_command(Cmd, Sender, State) -> {noreply, State1} | {reply, Reply, State1} handle_handoff_command(Cmd, Sender, State) -> {noreply, State1} | {reply, ok, State1} 44 Wednesday, October 6, 2010
  45. 45. VNode Module Handoff Coordination handoff_starting(Node, State) -> {Bool, State1} encode_handoff_data(Data, State) -> <<Binary>>. handle_handoff_data(Data, Sender, State) -> {reply, ok, State1} handoff_finished(Node, State) -> {ok, State1} 45 Wednesday, October 6, 2010
  46. 46. Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... application:start(riak_core). 46 Wednesday, October 6, 2010
  47. 47. Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Supervise vnode processes. 47 Wednesday, October 6, 2010
  48. 48. Start the riak_core application Start, coordinate, and supervise handoff. 48 Wednesday, October 6, 2010
  49. 49. Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Maintain cluster membership information. 49 Wednesday, October 6, 2010
  50. 50. Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Monitor node liveness, broadcast to registered modules. 50 Wednesday, October 6, 2010
  51. 51. Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Send ring information to other nodes. Reconcile different views of the cluster. Rebalance cluster when nodes join or leave. 51 Wednesday, October 6, 2010
  52. 52. In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Start the vnodes for your application. Master = { riak_X_vnode_master, { riak_core_vnode_master, start_link, [riak_X_vnode] }, permanent, 5000, worker, [riak_core_vnode_master] }, {ok, { {one_for_one, 5, 10}, [Master]} }. 52 Wednesday, October 6, 2010
  53. 53. In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Tell riak_core that your application is ready to receive requests. riak_core:register_vnode_module(riak_X_vnode), riak_core_node_watcher:service_up(riak_X, self()) 53 Wednesday, October 6, 2010
  54. 54. In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Join to an existing node in the cluster. riak_core_gossip:send_ring(ClusterNode, node()) 54 Wednesday, October 6, 2010
  55. 55. Start Sending Commands # Figure out the preflist... {_Verb, ObjName, _Payload} = Command, PrefList = riak_core_apl:get_apl(ObjName, NVal, riak_X), # Send the command... riak_core_vnode_master:command(PrefList, Command, riak_X_vnode_master) 55 Wednesday, October 6, 2010
  56. 56. Review Riak Core Open source Erlang library for building distributed, scalable, failure tolerant applications. Continual improvement in the coming months. Riak KV Key/Value datastore with map/reduce based on Riak Core. Riak Search Full-text, near real-time search engine based on Riak Core. 56 Wednesday, October 6, 2010
  57. 57. At the center of the universe, beneath the bottom-most turtle, is a cluster of Erlang nodes. http://www.flickr.com/photos/jurvetson/469492885 Wednesday, October 6, 2010
  58. 58. At the center of the universe, beneath the bottom-most turtle, is a cluster of Erlang nodes. With 100% uptime. http://www.flickr.com/photos/jurvetson/469492885 Wednesday, October 6, 2010
  59. 59. Thanks! Questions? Learn More More Information: http://wiki.basho.com Amazon’s Dynamo Paper Get the Code http://hg.basho.com/riak_core Get in Touch Twitter: @rklophaus, @basho/team Email: rusty@basho.com 59 Wednesday, October 6, 2010
  60. 60. END Wednesday, October 6, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×