Link Walking with Riak
Upcoming SlideShare
Loading in...5
×
 

Link Walking with Riak

on

  • 2,641 views

Describe how to create, manage and traverse links in Riak KV.

Describe how to create, manage and traverse links in Riak KV.

Statistics

Views

Total Views
2,641
Views on SlideShare
2,636
Embed Views
5

Actions

Likes
3
Downloads
18
Comments
1

4 Embeds 5

http://www.linkedin.com 2
http://www.slideshare.net 1
http://paper.li 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Link Walking with Riak Link Walking with Riak Presentation Transcript

  • Link Walking with Riak (KV) Susan Potter / Finsignia June 25, 2011
  • Riak Ecosystem Riak Search Riak KV Riak Core
  • Riak Ecosystem Riak Search Riak KV Riak Core
  • Riak Ecosystem Riak Search Riak KV Riak Core
  • Riak Ecosystem Riak Search Riak KV Riak Core
  • Riak Ecosystem Riak Search Riak KV Riak Core
  • % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • Problem Domain
  • Problem Domain
  • Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1 => object linked to album with tag/type “producer”
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1 => all albums that “producers” on album A have released
  • Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] _,_,0 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] _,_,0/album,release,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • Traversing Links[3/4] • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • Possibilities Object metadata extensions?
  • Limitations Not a graph database!
  • References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • Credits • Basho & Riak logos Courtesy of Basho • Icy chained fence photo http://www.flickr.com/photos/existentist/342177746/ • Frozen chain photo http://www.flickr.com/photos/skedonk/4197921511/ • Hook and chain photo http://www.flickr.com/photos/37491467@N07/3871760093/
  • Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter
  • Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter