Link Walking with Riak

  • 2,289 views
Uploaded on

Describe how to create, manage and traverse links in Riak KV.

Describe how to create, manage and traverse links in Riak KV.

More in: Technology , Design
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
2,289
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
18
Comments
1
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Link Walking with Riak (KV) Susan Potter / Finsignia June 25, 2011
  • 2. Riak Ecosystem Riak Search Riak KV Riak Core
  • 3. Riak Ecosystem Riak Search Riak KV Riak Core
  • 4. Riak Ecosystem Riak Search Riak KV Riak Core
  • 5. Riak Ecosystem Riak Search Riak KV Riak Core
  • 6. Riak Ecosystem Riak Search Riak KV Riak Core
  • 7. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 8. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 9. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 10. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 11. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 12. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 13. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 14. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 15. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 16. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 17. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 18. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 19. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 20. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 21. Problem Domain
  • 22. Problem Domain
  • 23. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 24. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 25. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 26. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 27. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 28. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 29. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 30. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 31. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 32. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 33. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 34. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 35. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 36. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 37. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 38. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 39. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 40. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • 41. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • 42. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1 => object linked to album with tag/type “producer”
  • 43. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 44. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 45. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 46. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 47. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 48. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 49. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 50. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 51. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1 => all albums that “producers” on album A have released
  • 52. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 53. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 54. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 55. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 56. Traversing Links[3/4] _,_,0 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 57. Traversing Links[3/4] _,_,0/album,release,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 58. Traversing Links[3/4] • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 59. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 60. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 61. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 62. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 63. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 64. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 65. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 66. Possibilities Object metadata extensions?
  • 67. Limitations Not a graph database!
  • 68. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 69. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 70. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 71. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 72. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 73. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 74. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 75. Credits • Basho & Riak logos Courtesy of Basho • Icy chained fence photo http://www.flickr.com/photos/existentist/342177746/ • Frozen chain photo http://www.flickr.com/photos/skedonk/4197921511/ • Hook and chain photo http://www.flickr.com/photos/37491467@N07/3871760093/
  • 76. Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter
  • 77. Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter