0
Building Distributed
Systems With Riak Core
      Andy Gross (@argv0)
         VP Engineering
             Basho
       De...
Riak K/V
• Distributed Key-Value Store
• Based on Amazon’s Dynamo
• HTTP and Binary (Protocol Buffers) APIs
• Data access ...
High-Level Dynamo
• Decentralized (no “master” nodes)
• Homogeneous (all nodes can do anything)
• Vector clocks (no relian...
N, R, W Values
• N = number of replicas to store (on
  distinct nodes)
• R = number of replica responses needed
  for a su...
Harvesting A
         Framework
• We noticed that Riak code fell into one of
  two categories
  • Code specific to K/V stor...
Distributed
        Coordination
• Making many machines act like one
• Division of labor
• Load balancing
• State storage
...
Riak Core Applications

    Your App       Riak K/V



           Riak Core
Riak Core Applications
             Your App
   Your
   App       Riak K/V


          Riak Core
Riak Core Abstractions

• Virtual Nodes
• Preference Lists
• Ring Event Watchers
• Node Event Watchers
Virtual Nodes

• Primary actor in a Dynamo-based system
• Handles load for (1/num_partitions)
• Implements commands dispat...
Preference Lists
• Lists of virtual nodes obtained by hashing a
  request (document, sessionid, etc).
• Allows any node to...
Ring Event Watchers

• Notified when ring state changes due to
  node addition/removal
• API: ring_update(NewRing)
• Can mo...
Node Event Watchers

• Nodes run and advertise “services”
• API: service_update(Services)
• Active service list used to ge...
Use cases
• If distributed systems isn’t your core
  business, outsource it!
• Providing a distribution layer on top of
  ...
Current Status and
       Roadmap
• Erlang-only now, but not for long (HTTP
  and PB APIs coming)
• Some harvesting left t...
Thanks!

• http://wiki.basho.com
• http://github.com/basho
• http://twitter.com/basho/team
• irc://freenode.net/#riak
• Ri...
Building Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak Core
Building Distributed Systems With Riak and Riak Core
Upcoming SlideShare
Loading in...5
×

Building Distributed Systems With Riak and Riak Core

12,926

Published on

My talk from DevNationSF 2010

Published in: Technology
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
12,926
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
211
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide

Transcript of "Building Distributed Systems With Riak and Riak Core"

  1. 1. Building Distributed Systems With Riak Core Andy Gross (@argv0) VP Engineering Basho DevNation SF 2010
  2. 2. Riak K/V • Distributed Key-Value Store • Based on Amazon’s Dynamo • HTTP and Binary (Protocol Buffers) APIs • Data access by {Bucket, Key} • Javascript Map/Reduce • Link Walking • Pluggable Storage (Bitcask, InnoDB, ...)
  3. 3. High-Level Dynamo • Decentralized (no “master” nodes) • Homogeneous (all nodes can do anything) • Vector clocks (no reliance on physical time) • Gossip Protocol (no global state) • Consistent Hashing for replica placement (a local calculation for each node)
  4. 4. N, R, W Values • N = number of replicas to store (on distinct nodes) • R = number of replica responses needed for a successful read (specified per-request) • W = number of replica responses needed for a successful write (specified per- request)
  5. 5. Harvesting A Framework • We noticed that Riak code fell into one of two categories • Code specific to K/V storage • “generic” distributed systems code • So we split Riak into K/V and Core
  6. 6. Distributed Coordination • Making many machines act like one • Division of labor • Load balancing • State storage • Mutual exclusion/locking
  7. 7. Riak Core Applications Your App Riak K/V Riak Core
  8. 8. Riak Core Applications Your App Your App Riak K/V Riak Core
  9. 9. Riak Core Abstractions • Virtual Nodes • Preference Lists • Ring Event Watchers • Node Event Watchers
  10. 10. Virtual Nodes • Primary actor in a Dynamo-based system • Handles load for (1/num_partitions) • Implements commands dispatched from clients • Handles handoff when nodes join/leave
  11. 11. Preference Lists • Lists of virtual nodes obtained by hashing a request (document, sessionid, etc). • Allows any node to compute document locations • Central to replication in Riak • Down nodes are filtered out, replaced with next-best nodes in the ring.
  12. 12. Ring Event Watchers • Notified when ring state changes due to node addition/removal • API: ring_update(NewRing) • Can modify ring state in an app-specific fashion
  13. 13. Node Event Watchers • Nodes run and advertise “services” • API: service_update(Services) • Active service list used to generate per-app preference lists.
  14. 14. Use cases • If distributed systems isn’t your core business, outsource it! • Providing a distribution layer on top of non-distributed systems like: • Couch, Redis, Memcached • Implementing your own systems.
  15. 15. Current Status and Roadmap • Erlang-only now, but not for long (HTTP and PB APIs coming) • Some harvesting left to do (versioned objects, ring/node handler utilities) • Project templates - skeleton code for writing Riak Core-based systems. • Stronger consistency models (with a Paxos/ ZAB-like protocol)
  16. 16. Thanks! • http://wiki.basho.com • http://github.com/basho • http://twitter.com/basho/team • irc://freenode.net/#riak • Riak SF Meetup (on meetup.com) • Visit us! 795 Folsom @ 4th (Twitter Bldg.)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×