Schema Design for Riak

13,242
-1

Published on

Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design.

* How do you structure data when you don't have tables and foreign keys?
* When should you denormalize, add links, or create map-reduce queries?
* Where will Riak be a natural fit and where will it be challenging?

Published in: Technology
0 Comments
22 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
13,242
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
275
Comments
0
Likes
22
Embeds 0
No embeds

No notes for slide















































































































  • Schema Design for Riak

    1. 1. Schema Design for Riak Sean Cribbs Developer Advocate basho
    2. 2. Thinking Non-Relationally basho
    3. 3. “There is no spoon schema.” basho
    4. 4. There is an application. basho
    5. 5. What you lose basho
    6. 6. What you lose •Tables basho
    7. 7. What you lose •Tables •Foreign keys and constraints basho
    8. 8. What you lose •Tables •Foreign keys and constraints •ACID basho
    9. 9. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners basho
    10. 10. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners •Declarative query language (SQL) basho
    11. 11. What you gain basho
    12. 12. What you gain •More flexible, fluid designs basho
    13. 13. What you gain •More flexible, fluid designs •More natural data representations basho
    14. 14. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain basho
    15. 15. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain •Reduced operational complexity basho
    16. 16. Lose the crutches basho
    17. 17. Lose the crutches •Sparse data (optional/multi-value fields) basho
    18. 18. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures basho
    19. 19. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers basho
    20. 20. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers •Innovative access patterns basho
    21. 21. Eventual Consistency basho
    22. 22. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) basho
    23. 23. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging basho
    24. 24. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes basho
    25. 25. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes •Weigh risk of conflicts and emergent issues basho
    26. 26. Know your data basho
    27. 27. Make your Top 10 list basho
    28. 28. Make your Top 10 list •Frequently requested pages/screens basho
    29. 29. Make your Top 10 list •Frequently requested pages/screens •Slow queries basho
    30. 30. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes basho
    31. 31. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes •Complicated joins or aggregations basho
    32. 32. Analyze basho
    33. 33. Analyze •Interdependencies, coupling basho
    34. 34. Analyze •Interdependencies, coupling •Cardinalities of relationships basho
    35. 35. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern basho
    36. 36. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern •Shoehorned structures basho
    37. 37. Example basho
    38. 38. basho
    39. 39. basho
    40. 40. Top queries basho
    41. 41. List Features basho
    42. 42. Reorder Features basho
    43. 43. Regroup Features basho
    44. 44. Edit Feature basho
    45. 45. List Folders basho
    46. 46. Project Statistics basho
    47. 47. projects users folders features milestones project_items Current Schema basho
    48. 48. projects users folders artificial ordering features milestones project_items Current Schema basho
    49. 49. projects users additional key for stats folders artificial ordering features milestones project_items Current Schema basho
    50. 50. projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    51. 51. collaborator projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    52. 52. collaborator projects users additional key for stats owner BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    53. 53. Observations basho
    54. 54. Observations •Features and milestones are disjunct, except in list participation basho
    55. 55. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data basho
    56. 56. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data •Duplication of relationships - collaborator-owner, project-folder-item basho
    57. 57. List/Reorder Features basho
    58. 58. List/Reorder Features • AJAX postback was already “total” ordering basho
    59. 59. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” basho
    60. 60. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” • Membership, order are coupled concepts basho
    61. 61. Links in Riak basho
    62. 62. Links in Riak •Lightweight pointers, like HTML <link> tags. basho
    63. 63. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag basho
    64. 64. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag •Traversable through “link-walking” basho
    65. 65. Riak: List/Reorder Features basho
    66. 66. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag basho
    67. 67. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort basho
    68. 68. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk basho
    69. 69. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk • Concurrent writes may conflict basho
    70. 70. Regroup Features basho
    71. 71. Regroup Features • Changing foreign key to folder basho
    72. 72. Regroup Features • Changing foreign key to folder • Recalculate order for both lists (multiple UPDATE statements) basho
    73. 73. Riak: Regroup Features basho
    74. 74. Riak: Regroup Features •Already solved for us by folder owning list as links basho
    75. 75. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated basho
    76. 76. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) basho
    77. 77. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) • No transactions, need extra info for conflict resolution basho
    78. 78. Edit Feature basho
    79. 79. Edit Feature • Required two representations (text, tree), both stored as BLOBs basho
    80. 80. Edit Feature • Required two representations (text, tree), both stored as BLOBs • Application deserializes tree to present web form, recreates text on save basho
    81. 81. Composition basho
    82. 82. Composition •Sometimes called “denormalization” basho
    83. 83. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent basho
    84. 84. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent •JSON, XML, Erlang terms basho
    85. 85. Riak: Edit Feature basho
    86. 86. Riak: Edit Feature •Already denormalized! basho
    87. 87. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing basho
    88. 88. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing •Individual objects larger basho
    89. 89. List Folders basho
    90. 90. List Folders • Number of folders is small per project basho
    91. 91. List Folders • Number of folders is small per project • Essentially unordered within project basho
    92. 92. Riak: List Folders basho
    93. 93. Riak: List Folders • Store links to folder objects (or entire list) in project basho
    94. 94. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders basho
    95. 95. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    96. 96. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    97. 97. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    98. 98. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get • List keys is expensive folders • Two writes required, or move feature links/ order to project basho
    99. 99. Project Statistics basho
    100. 100. Project Statistics • Relational design had per-feature counters (maintained by app) basho
    101. 101. Project Statistics • Relational design had per-feature counters (maintained by app) • Extra foreign key sped up aggregation by reducing joins basho
    102. 102. Riak: Project Statistics basho
    103. 103. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! basho
    104. 104. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects basho
    105. 105. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects •Query: Follow links to features, map per- feature counters, reduce summarize the counters basho
    106. 106. Awkward Fits basho
    107. 107. Awkward Fits •User uniqueness constraint basho
    108. 108. Awkward Fits •User uniqueness constraint •“double-click” problem basho
    109. 109. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions basho
    110. 110. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership basho
    111. 111. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership •M-to-N is hard basho
    112. 112. Review basho
    113. 113. Review •Analyze your relational model basho
    114. 114. Review •Analyze your relational model •Identify pain points, take stats basho
    115. 115. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives basho
    116. 116. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives •Test, Measure, Repeat! basho
    117. 117. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com basho
    118. 118. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com sean@basho.com @seancribbs basho

    ×