• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Riak Core: Building Distributed Applications Without Shared State
 

Riak Core: Building Distributed Applications Without Shared State

on

  • 7,298 views

Riak Core--an open-source Erlang library created by Basho Technologies that powers Riak KV and Riak Search--allows developers to build distributed, scalable, failure-tolerant applications based on a ...

Riak Core--an open-source Erlang library created by Basho Technologies that powers Riak KV and Riak Search--allows developers to build distributed, scalable, failure-tolerant applications based on a generalized version of Amazon's Dynamo architecture. In this talk, Rusty explains why Riak Core was built, discuss what problems it solves and how it works, and walk through the steps to using Riak Core in an Erlang application.

Statistics

Views

Total Views
7,298
Views on SlideShare
6,849
Embed Views
449

Actions

Likes
20
Downloads
197
Comments
0

10 Embeds 449

http://nosql.mypopescu.com 361
http://www.nosqldatabases.com 58
http://irr.posterous.com 16
http://static.slidesharecdn.com 4
http://www.linkedin.com 2
http://www.docshut.com 2
http://www.slashdocs.com 2
https://twitter.com 2
applewebdata://62A4CE7A-4DD3-4693-A9A0-E094EF82A752 1
resource://brief-content 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Riak Core: Building Distributed Applications Without Shared State Riak Core: Building Distributed Applications Without Shared State Presentation Transcript

    • Riak Core: Building Distributed Applications Without Shared State Commercial Users of Functional Programming Baltimore, MD · October 2010 Rusty Klophaus (@rklophaus) Basho Technologies Wednesday, October 6, 2010
    • You suddenly feel an uncontrollable desire to learn Erlang. http://www.flickr.com/photos/procsilas/18014203 2 Wednesday, October 6, 2010
    • What is Riak Core? How does it work? How can you use it? 3 Wednesday, October 6, 2010
    • Distributed, scalable, failure-tolerant. 4 Wednesday, October 6, 2010
    • Distributed, scalable, failure-tolerant. No central coordinator. Easy to setup/operate. 5 Wednesday, October 6, 2010
    • Distributed, scalable, failure-tolerant. Horizontally scalable; add commodity hardware to get more X. 6 Wednesday, October 6, 2010
    • Distributed, scalable, failure-tolerant. Always available. No single point of failure. Self-healing. 7 Wednesday, October 6, 2010
    • Basho Technologies Riak KV Distributed, scalable, failure-tolerant key/value datastore. Started as a “Dynamo clone”. Map/Reduce, Lightweight Data Relations, Client APIs Riak Search Distributed, scalable, failure-tolerant full-text search engine. Near Realtime, Riak KV Integration, Solr Support 8 Wednesday, October 6, 2010
    • Wednesday, October 6, 2010
    • Riak Riak Riak KV Core Search Wednesday, October 6, 2010
    • Riak Core is an Erlang library that helps you build distributed, scalable, failure-tolerant applications. 11 Wednesday, October 6, 2010
    • Amazon Dynamo Wednesday, October 6, 2010
    • “We Generalized the Dynamo Architecture and Open-Sourced the Bits.” 13 Wednesday, October 6, 2010
    • Wait, doesn’t *Erlang* let you build distributed, scalable, failure-tolerant applications? 14 Wednesday, October 6, 2010
    • Erlang makes it easy to connect the components of your application. Client Service A Service B Service C Queue E Resource D Wednesday, October 6, 2010
    • Riak Core helps you build a service that harnesses the power of many nodes. Node A Node B Node C Node D Node E Node F Node G Node H Service Node I Node J Node K Node L Node M Node N Node O ... Wednesday, October 6, 2010
    • “People use languages other than Erlang?!? I find that hilarious.” -Al Gore actually said this to me. http://www.flickr.com/photos/scobleizer/2216445692 Wednesday, October 6, 2010
    • How does Riak Core work? 18 Wednesday, October 6, 2010
    • Command ObjectName, Payload Wednesday, October 6, 2010
    • Predictable Routing 20 Wednesday, October 6, 2010
    • Hash the Object Name Command ObjectName, Payload SHA1(ObjName), Payload 0 to 2^160 Wednesday, October 6, 2010
    • A Naive Approach Command ObjectName, Payload SHA1(ObjName), Payload Node A Node B Node C Node D Wednesday, October 6, 2010
    • A Naive Approach Command ObjectName, Payload SHA1(ObjName), Payload Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • "All problems in computer science can be solved by another level of indirection." - David Wheeler 24 Wednesday, October 6, 2010
    • Add VNodes What Virtual Node. Logical subdivision of the cluster. Handles incoming commands, does work, replies. For Parallelism # of VNodes = maximum concurrent requests For Rebalancing the Cluster Smallest block that can be shifted to a new node. For Resiliance The system restarts failed VNodes. 25 Wednesday, October 6, 2010
    • Routing with Consistent Hash Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Wednesday, October 6, 2010
    • Adding a Node Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • Removing a Node Command ObjectName, Payload SHA1(ObjName), Payload VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • The Ring Hash Location Wednesday, October 6, 2010
    • The Ring Preflist Wednesday, October 6, 2010
    • Writing Replicas (n_val) Preflist when N=3 Wednesday, October 6, 2010
    • Routing Around Failures X Preflist when N=3 and node 0 is down. Wednesday, October 6, 2010
    • Location of the Routing Layer 33 Wednesday, October 6, 2010
    • Router in the Middle Client Client Client Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • Riak Core - Router on Each Node Client Client Client Router Router Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • Eventually - Router on the Client Client Client Client Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • No Shared State Router Router Router Router Router VNode VNode VNode VNode VNode VNode VNode VNode 0 1 3 4 2 5 6 7 Node A Node B Node C Node D Node E Wednesday, October 6, 2010
    • Gossip Local Incoming Ring State Ring State Are rings equivalent? Strictly descendent? Or different? Wednesday, October 6, 2010
    • Handoff When Nodes is added to the system. Node is removed from the system. Node has temporarily failed. What Ship the data backing a VNode from one node to another. 39 Wednesday, October 6, 2010
    • Not Mentioned Vector Clocks Merkle Trees Bloom Filters 40 Wednesday, October 6, 2010
    • Distinguished gentlemen prefer Erlang. http://www.flickr.com/photos/rebcal/3987226359 Wednesday, October 6, 2010
    • How do you use Riak Core? 42 Wednesday, October 6, 2010
    • Two Things to Think About Command Command = ObjectName, Payload The commands/requests/operations that you will send through the system. VNode Module The callback module that will receive the commands. 43 Wednesday, October 6, 2010
    • VNode Module Startup/Shutdown init([Partition]) -> {ok, State} terminate(State) -> ok Commands handle_command(Cmd, Sender, State) -> {noreply, State1} | {reply, Reply, State1} handle_handoff_command(Cmd, Sender, State) -> {noreply, State1} | {reply, ok, State1} 44 Wednesday, October 6, 2010
    • VNode Module Handoff Coordination handoff_starting(Node, State) -> {Bool, State1} encode_handoff_data(Data, State) -> <<Binary>>. handle_handoff_data(Data, Sender, State) -> {reply, ok, State1} handoff_finished(Node, State) -> {ok, State1} 45 Wednesday, October 6, 2010
    • Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... application:start(riak_core). 46 Wednesday, October 6, 2010
    • Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Supervise vnode processes. 47 Wednesday, October 6, 2010
    • Start the riak_core application Start, coordinate, and supervise handoff. 48 Wednesday, October 6, 2010
    • Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Maintain cluster membership information. 49 Wednesday, October 6, 2010
    • Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Monitor node liveness, broadcast to registered modules. 50 Wednesday, October 6, 2010
    • Start the riak_core application riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Send ring information to other nodes. Reconcile different views of the cluster. Rebalance cluster when nodes join or leave. 51 Wednesday, October 6, 2010
    • In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Start the vnodes for your application. Master = { riak_X_vnode_master, { riak_core_vnode_master, start_link, [riak_X_vnode] }, permanent, 5000, worker, [riak_core_vnode_master] }, {ok, { {one_for_one, 5, 10}, [Master]} }. 52 Wednesday, October 6, 2010
    • In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Tell riak_core that your application is ready to receive requests. riak_core:register_vnode_module(riak_X_vnode), riak_core_node_watcher:service_up(riak_X, self()) 53 Wednesday, October 6, 2010
    • In your application... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... riak_core riak_core_vnode_sup riak_core_handoff_* riak_core_node_* X_vnode X_vnode riak_core_ring_* riak_core_gossip_* X_vnode ... Join to an existing node in the cluster. riak_core_gossip:send_ring(ClusterNode, node()) 54 Wednesday, October 6, 2010
    • Start Sending Commands # Figure out the preflist... {_Verb, ObjName, _Payload} = Command, PrefList = riak_core_apl:get_apl(ObjName, NVal, riak_X), # Send the command... riak_core_vnode_master:command(PrefList, Command, riak_X_vnode_master) 55 Wednesday, October 6, 2010
    • Review Riak Core Open source Erlang library for building distributed, scalable, failure tolerant applications. Continual improvement in the coming months. Riak KV Key/Value datastore with map/reduce based on Riak Core. Riak Search Full-text, near real-time search engine based on Riak Core. 56 Wednesday, October 6, 2010
    • At the center of the universe, beneath the bottom-most turtle, is a cluster of Erlang nodes. http://www.flickr.com/photos/jurvetson/469492885 Wednesday, October 6, 2010
    • At the center of the universe, beneath the bottom-most turtle, is a cluster of Erlang nodes. With 100% uptime. http://www.flickr.com/photos/jurvetson/469492885 Wednesday, October 6, 2010
    • Thanks! Questions? Learn More More Information: http://wiki.basho.com Amazon’s Dynamo Paper Get the Code http://hg.basho.com/riak_core Get in Touch Twitter: @rklophaus, @basho/team Email: rusty@basho.com 59 Wednesday, October 6, 2010
    • END Wednesday, October 6, 2010