Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Distributed Systems With Riak and Riak Core

16,725 views

Published on

My talk from DevNationSF 2010

Published in: Technology
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT, ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Distributed Systems With Riak and Riak Core

  1. 1. Building Distributed Systems With Riak Core Andy Gross (@argv0) VP Engineering Basho DevNation SF 2010
  2. 2. Riak K/V • Distributed Key-Value Store • Based on Amazon’s Dynamo • HTTP and Binary (Protocol Buffers) APIs • Data access by {Bucket, Key} • Javascript Map/Reduce • Link Walking • Pluggable Storage (Bitcask, InnoDB, ...)
  3. 3. High-Level Dynamo • Decentralized (no “master” nodes) • Homogeneous (all nodes can do anything) • Vector clocks (no reliance on physical time) • Gossip Protocol (no global state) • Consistent Hashing for replica placement (a local calculation for each node)
  4. 4. N, R, W Values • N = number of replicas to store (on distinct nodes) • R = number of replica responses needed for a successful read (specified per-request) • W = number of replica responses needed for a successful write (specified per- request)
  5. 5. Harvesting A Framework • We noticed that Riak code fell into one of two categories • Code specific to K/V storage • “generic” distributed systems code • So we split Riak into K/V and Core
  6. 6. Distributed Coordination • Making many machines act like one • Division of labor • Load balancing • State storage • Mutual exclusion/locking
  7. 7. Riak Core Applications Your App Riak K/V Riak Core
  8. 8. Riak Core Applications Your App Your App Riak K/V Riak Core
  9. 9. Riak Core Abstractions • Virtual Nodes • Preference Lists • Ring Event Watchers • Node Event Watchers
  10. 10. Virtual Nodes • Primary actor in a Dynamo-based system • Handles load for (1/num_partitions) • Implements commands dispatched from clients • Handles handoff when nodes join/leave
  11. 11. Preference Lists • Lists of virtual nodes obtained by hashing a request (document, sessionid, etc). • Allows any node to compute document locations • Central to replication in Riak • Down nodes are filtered out, replaced with next-best nodes in the ring.
  12. 12. Ring Event Watchers • Notified when ring state changes due to node addition/removal • API: ring_update(NewRing) • Can modify ring state in an app-specific fashion
  13. 13. Node Event Watchers • Nodes run and advertise “services” • API: service_update(Services) • Active service list used to generate per-app preference lists.
  14. 14. Use cases • If distributed systems isn’t your core business, outsource it! • Providing a distribution layer on top of non-distributed systems like: • Couch, Redis, Memcached • Implementing your own systems.
  15. 15. Current Status and Roadmap • Erlang-only now, but not for long (HTTP and PB APIs coming) • Some harvesting left to do (versioned objects, ring/node handler utilities) • Project templates - skeleton code for writing Riak Core-based systems. • Stronger consistency models (with a Paxos/ ZAB-like protocol)
  16. 16. Thanks! • http://wiki.basho.com • http://github.com/basho • http://twitter.com/basho/team • irc://freenode.net/#riak • Riak SF Meetup (on meetup.com) • Visit us! 795 Folsom @ 4th (Twitter Bldg.)

×