Your SlideShare is downloading. ×
Link Walking with Riak
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Link Walking with Riak

2,396
views

Published on

Describe how to create, manage and traverse links in Riak KV.

Describe how to create, manage and traverse links in Riak KV.

Published in: Technology, Design

1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,396
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
1
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Link Walking with Riak (KV) Susan Potter / Finsignia June 25, 2011
  • 2. Riak Ecosystem Riak Search Riak KV Riak Core
  • 3. Riak Ecosystem Riak Search Riak KV Riak Core
  • 4. Riak Ecosystem Riak Search Riak KV Riak Core
  • 5. Riak Ecosystem Riak Search Riak KV Riak Core
  • 6. Riak Ecosystem Riak Search Riak KV Riak Core
  • 7. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 8. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 9. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 10. % whoami# curl http://susanpotter.net/whoami/windycitydb2011.json • 15 years RDBMS • 3-4 years KV databases (CouchDB and now Riak) • 1 year Redis • Using graph database in next part of platform
  • 11. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 12. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 13. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 14. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 15. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 16. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 17. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 18. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 19. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 20. Scope of Talk • Problem Domain • not Consistent Hashing or DHTs • Linking Objects • not Vector Clocks • Traversing Links • not Gossip Protocol • Link Maintainence • not Hinted Handoff • not Read Repair • not Eventual Consistency
  • 21. Problem Domain
  • 22. Problem Domain
  • 23. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 24. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 25. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 26. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 27. Linking[1/2]curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’ http://host:8098/riak/album/Born-This-Way data ...
  • 28. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 29. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 30. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 31. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 32. Linking[2/2] • Two Dimensional both direction and type/tag • 0..* links per object of any type/tag • Metadata traversing not load, parse, then lookup cycle • Model sufficiency otherwise graph database might be more useful • Link to other buckets (in same Riak cluster)
  • 33. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 34. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 35. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 36. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 37. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 38. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 39. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1• GET /riak/album/Born-This-Way/_,producer,1
  • 40. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • 41. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1
  • 42. Traversing Links[1/4]• GET /riak/album/Born-This-Way/artist,_,1 => all artists linked to album regardless of link tag/type• GET /riak/album/Born-This-Way/artist,performer,1 => artists linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/artist,producer,1 => artists linked to album with tag/type “producer”• GET /riak/album/Born-This-Way/_,performer,1 => object linked to album with tag/type “performer”• GET /riak/album/Born-This-Way/_,producer,1 => object linked to album with tag/type “producer”
  • 43. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 44. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 45. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 46. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 47. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1• GET /riak/album/A/artist,producer,0/album,release,1
  • 48. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 49. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 50. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1
  • 51. Traversing Links[2/4]• GET /riak/album/A/artist,_,1/artist,collaborator,1 => all artists associated with album A and all artists that have collaborated with these artists• GET /riak/album/A/artist,performer,0/artist,collaborator,1 => all “collaborators” of “performers” on album A excluding “performers”• GET /riak/album/A/artist,producer,0/album,release,1 => all albums that “producers” on album A have released
  • 52. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 53. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 54. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 55. Traversing Links[3/4] artist,performer,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 56. Traversing Links[3/4] _,_,0 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 57. Traversing Links[3/4] _,_,0/album,release,1 • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 58. Traversing Links[3/4] • Bucket name e.g. artist, album • Tag (or type) name e.g. performer, producer • Return (or “keep”) useful in nested traversals • Wildcarding ’_’ for bucket and tag means “any” • Nesting traverse on multiple levels • Multiple Sources? how to walk links from multiple objects?
  • 59. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 60. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 61. curl -X POST -H “Content-type: application/json” http://host:8098/mapred data @- {"inputs":[ ["artist","Ke$ha"], ["artist", "Lady+Gaga"] ], "query":[ {"link": { "tag": "release", "bucket": "album" }}, {"map":{ "language":"javascript", "source":"function(v) return [v]; "} } ]}^D
  • 62. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 63. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 64. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 65. Link Maintenance • Commit Hooks can use either pre- or post-commit hooks • Automatic Backlinks can use commit hooks to generate backlinks automatically • Trigger Messages links could only be used as triggers • Delete Cleanup post-commit hook with backlinks can clean up links to new orphans automatically
  • 66. Possibilities Object metadata extensions?
  • 67. Limitations Not a graph database!
  • 68. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 69. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 70. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 71. References[1/2] • Riak Wiki http://wiki.basho.com/ • Riak Mailing List riak-users@lists.basho.com • Riak Contrib http://contrib.basho.com/ • Riak Code Repos http://github.com/basho
  • 72. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 73. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 74. References[2/2] • Secondary index support in Riak? Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806 • riak_core applications e.g. Riak Search, riak_zab, riak_pipe • Comet KV database supports Active Storage Objects (ASO), handlers http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
  • 75. Credits • Basho & Riak logos Courtesy of Basho • Icy chained fence photo http://www.flickr.com/photos/existentist/342177746/ • Frozen chain photo http://www.flickr.com/photos/skedonk/4197921511/ • Hook and chain photo http://www.flickr.com/photos/37491467@N07/3871760093/
  • 76. Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter
  • 77. Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter