Schema Design for Riak

  • 12,029 views
Uploaded on

Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design. …

Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design.

* How do you structure data when you don't have tables and foreign keys?
* When should you denormalize, add links, or create map-reduce queries?
* Where will Riak be a natural fit and where will it be challenging?

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
12,029
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
241
Comments
0
Likes
18

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide















































































































Transcript

  • 1. Schema Design for Riak Sean Cribbs Developer Advocate basho
  • 2. Thinking Non-Relationally basho
  • 3. “There is no spoon schema.” basho
  • 4. There is an application. basho
  • 5. What you lose basho
  • 6. What you lose •Tables basho
  • 7. What you lose •Tables •Foreign keys and constraints basho
  • 8. What you lose •Tables •Foreign keys and constraints •ACID basho
  • 9. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners basho
  • 10. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners •Declarative query language (SQL) basho
  • 11. What you gain basho
  • 12. What you gain •More flexible, fluid designs basho
  • 13. What you gain •More flexible, fluid designs •More natural data representations basho
  • 14. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain basho
  • 15. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain •Reduced operational complexity basho
  • 16. Lose the crutches basho
  • 17. Lose the crutches •Sparse data (optional/multi-value fields) basho
  • 18. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures basho
  • 19. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers basho
  • 20. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers •Innovative access patterns basho
  • 21. Eventual Consistency basho
  • 22. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) basho
  • 23. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging basho
  • 24. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes basho
  • 25. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes •Weigh risk of conflicts and emergent issues basho
  • 26. Know your data basho
  • 27. Make your Top 10 list basho
  • 28. Make your Top 10 list •Frequently requested pages/screens basho
  • 29. Make your Top 10 list •Frequently requested pages/screens •Slow queries basho
  • 30. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes basho
  • 31. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes •Complicated joins or aggregations basho
  • 32. Analyze basho
  • 33. Analyze •Interdependencies, coupling basho
  • 34. Analyze •Interdependencies, coupling •Cardinalities of relationships basho
  • 35. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern basho
  • 36. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern •Shoehorned structures basho
  • 37. Example basho
  • 38. basho
  • 39. basho
  • 40. Top queries basho
  • 41. List Features basho
  • 42. Reorder Features basho
  • 43. Regroup Features basho
  • 44. Edit Feature basho
  • 45. List Folders basho
  • 46. Project Statistics basho
  • 47. projects users folders features milestones project_items Current Schema basho
  • 48. projects users folders artificial ordering features milestones project_items Current Schema basho
  • 49. projects users additional key for stats folders artificial ordering features milestones project_items Current Schema basho
  • 50. projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
  • 51. collaborator projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
  • 52. collaborator projects users additional key for stats owner BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
  • 53. Observations basho
  • 54. Observations •Features and milestones are disjunct, except in list participation basho
  • 55. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data basho
  • 56. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data •Duplication of relationships - collaborator-owner, project-folder-item basho
  • 57. List/Reorder Features basho
  • 58. List/Reorder Features • AJAX postback was already “total” ordering basho
  • 59. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” basho
  • 60. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” • Membership, order are coupled concepts basho
  • 61. Links in Riak basho
  • 62. Links in Riak •Lightweight pointers, like HTML <link> tags. basho
  • 63. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag basho
  • 64. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag •Traversable through “link-walking” basho
  • 65. Riak: List/Reorder Features basho
  • 66. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag basho
  • 67. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort basho
  • 68. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk basho
  • 69. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk • Concurrent writes may conflict basho
  • 70. Regroup Features basho
  • 71. Regroup Features • Changing foreign key to folder basho
  • 72. Regroup Features • Changing foreign key to folder • Recalculate order for both lists (multiple UPDATE statements) basho
  • 73. Riak: Regroup Features basho
  • 74. Riak: Regroup Features •Already solved for us by folder owning list as links basho
  • 75. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated basho
  • 76. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) basho
  • 77. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) • No transactions, need extra info for conflict resolution basho
  • 78. Edit Feature basho
  • 79. Edit Feature • Required two representations (text, tree), both stored as BLOBs basho
  • 80. Edit Feature • Required two representations (text, tree), both stored as BLOBs • Application deserializes tree to present web form, recreates text on save basho
  • 81. Composition basho
  • 82. Composition •Sometimes called “denormalization” basho
  • 83. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent basho
  • 84. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent •JSON, XML, Erlang terms basho
  • 85. Riak: Edit Feature basho
  • 86. Riak: Edit Feature •Already denormalized! basho
  • 87. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing basho
  • 88. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing •Individual objects larger basho
  • 89. List Folders basho
  • 90. List Folders • Number of folders is small per project basho
  • 91. List Folders • Number of folders is small per project • Essentially unordered within project basho
  • 92. Riak: List Folders basho
  • 93. Riak: List Folders • Store links to folder objects (or entire list) in project basho
  • 94. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders basho
  • 95. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
  • 96. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
  • 97. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
  • 98. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get • List keys is expensive folders • Two writes required, or move feature links/ order to project basho
  • 99. Project Statistics basho
  • 100. Project Statistics • Relational design had per-feature counters (maintained by app) basho
  • 101. Project Statistics • Relational design had per-feature counters (maintained by app) • Extra foreign key sped up aggregation by reducing joins basho
  • 102. Riak: Project Statistics basho
  • 103. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! basho
  • 104. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects basho
  • 105. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects •Query: Follow links to features, map per- feature counters, reduce summarize the counters basho
  • 106. Awkward Fits basho
  • 107. Awkward Fits •User uniqueness constraint basho
  • 108. Awkward Fits •User uniqueness constraint •“double-click” problem basho
  • 109. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions basho
  • 110. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership basho
  • 111. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership •M-to-N is hard basho
  • 112. Review basho
  • 113. Review •Analyze your relational model basho
  • 114. Review •Analyze your relational model •Identify pain points, take stats basho
  • 115. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives basho
  • 116. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives •Test, Measure, Repeat! basho
  • 117. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com basho
  • 118. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com sean@basho.com @seancribbs basho