0
Schema Design
           for Riak
             Sean Cribbs
          Developer Advocate




basho
Thinking
        Non-Relationally


basho
“There is no spoon
            schema.”


basho
There is an
        application.


basho
What you lose




basho
What you lose

        •Tables




basho
What you lose

        •Tables
        •Foreign keys and constraints




basho
What you lose

        •Tables
        •Foreign keys and constraints
        •ACID



basho
What you lose

        •Tables
        •Foreign keys and constraints
        •ACID
        •Sophisticated query planners

...
What you lose

        •Tables
        •Foreign keys and constraints
        •ACID
        •Sophisticated query planners
 ...
What you gain




basho
What you gain

        •More flexible, fluid designs




basho
What you gain

        •More flexible, fluid designs
        •More natural data representations



basho
What you gain

        •More flexible, fluid designs
        •More natural data representations
        •Scaling without pai...
What you gain

        •More flexible, fluid designs
        •More natural data representations
        •Scaling without pai...
Lose the crutches




basho
Lose the crutches

        •Sparse data (optional/multi-value fields)




basho
Lose the crutches

        •Sparse data (optional/multi-value fields)
        •Richer data structures



basho
Lose the crutches

        •Sparse data (optional/multi-value fields)
        •Richer data structures
        •Meaningful i...
Lose the crutches

        •Sparse data (optional/multi-value fields)
        •Richer data structures
        •Meaningful i...
Eventual Consistency




basho
Eventual Consistency
        •You are already dealing with
          inconsistency (e.g. caching, read slave)




basho
Eventual Consistency
        •You are already dealing with
          inconsistency (e.g. caching, read slave)

        •Pl...
Eventual Consistency
        •You are already dealing with
          inconsistency (e.g. caching, read slave)

        •Pl...
Eventual Consistency
        •You are already dealing with
          inconsistency (e.g. caching, read slave)

        •Pl...
Know your data



basho
Make your Top 10 list




basho
Make your Top 10 list

        •Frequently requested pages/screens




basho
Make your Top 10 list

        •Frequently requested pages/screens
        •Slow queries



basho
Make your Top 10 list

        •Frequently requested pages/screens
        •Slow queries
        •Secondary indexes


basho
Make your Top 10 list

        •Frequently requested pages/screens
        •Slow queries
        •Secondary indexes
      ...
Analyze




basho
Analyze

        •Interdependencies, coupling




basho
Analyze

        •Interdependencies, coupling
        •Cardinalities of relationships



basho
Analyze

        •Interdependencies, coupling
        •Cardinalities of relationships
        •Access pattern


basho
Analyze

        •Interdependencies, coupling
        •Cardinalities of relationships
        •Access pattern
        •Sho...
Example



basho
basho
basho
Top queries
basho
List Features
basho
Reorder Features
basho
Regroup Features
basho
Edit Feature
basho
List Folders
basho
Project Statistics
basho
projects          users


               folders


        features   milestones

           project_items



        Curr...
projects          users


                      folders

   artificial
   ordering    features   milestones

              ...
projects          users
           additional
          key for stats


                             folders

   artificial...
projects          users
           additional
          key for stats
                                                    ...
collaborator


                            projects               users
           additional
          key for stats
    ...
collaborator


                            projects                users
           additional
          key for stats    ...
Observations




basho
Observations

        •Features and milestones are disjunct,
          except in list participation




basho
Observations

        •Features and milestones are disjunct,
          except in list participation

        •Opacity of s...
Observations

        •Features and milestones are disjunct,
          except in list participation

        •Opacity of s...
List/Reorder Features




basho
List/Reorder Features

        • AJAX postback was
          already “total” ordering




basho
List/Reorder Features

        • AJAX postback was
           already “total” ordering

        • List is really a propert...
List/Reorder Features

        • AJAX postback was
           already “total” ordering

        • List is really a propert...
Links in Riak




basho
Links in Riak

        •Lightweight pointers, like HTML <link>
          tags.




basho
Links in Riak

        •Lightweight pointers, like HTML <link>
          tags.

        •Composed of bucket, key, tag


ba...
Links in Riak

        •Lightweight pointers, like HTML <link>
          tags.

        •Composed of bucket, key, tag
    ...
Riak: List/Reorder
             Features




basho
Riak: List/Reorder
                Features
        •Store list of features and milestones as
          links on folder, w...
Riak: List/Reorder
                Features
        •Store list of features and milestones as
          links on folder, w...
Riak: List/Reorder
                Features
        •Store list of features and milestones as
          links on folder, w...
Riak: List/Reorder
                Features
        •Store list of features and milestones as
          links on folder, w...
Regroup Features




basho
Regroup Features

        • Changing foreign key
          to folder




basho
Regroup Features

        • Changing foreign key
          to folder

        • Recalculate order for
          both lists...
Riak: Regroup
           Features




basho
Riak: Regroup
                  Features
        •Already solved for us by folder owning
          list as links




basho
Riak: Regroup
                  Features
        •Already solved for us by folder owning
          list as links

        ...
Riak: Regroup
                  Features
        •Already solved for us by folder owning
          list as links

        ...
Riak: Regroup
                  Features
        •Already solved for us by folder owning
          list as links

        ...
Edit Feature




basho
Edit Feature
        • Required two
          representations (text,
          tree), both stored as
          BLOBs




b...
Edit Feature
        • Required two
          representations (text,
          tree), both stored as
          BLOBs

    ...
Composition




basho
Composition

        •Sometimes called “denormalization”




basho
Composition

        •Sometimes called “denormalization”
        •Store dependent data in the same object
          as the...
Composition

        •Sometimes called “denormalization”
        •Store dependent data in the same object
          as the...
Riak: Edit Feature




basho
Riak: Edit Feature

        •Already denormalized!




basho
Riak: Edit Feature

        •Already denormalized!
        •Now parse tree can have the same
          format (JSON, XML),...
Riak: Edit Feature

        •Already denormalized!
        •Now parse tree can have the same
          format (JSON, XML),...
List Folders




basho
List Folders

        • Number of folders is
          small per project




basho
List Folders

        • Number of folders is
          small per project

        • Essentially unordered
          within...
Riak: List Folders




basho
Riak: List Folders
        • Store links to folder
          objects (or entire list) in
          project




basho
Riak: List Folders
        • Store links to folder
          objects (or entire list) in
          project

        • Simp...
Riak: List Folders
        • Store links to folder
          objects (or entire list) in
          project

        • Simp...
Riak: List Folders
        • Store links to folder         • Parameterized bucket
          objects (or entire list) in   ...
Riak: List Folders
        • Store links to folder         • Parameterized bucket
          objects (or entire list) in   ...
Riak: List Folders
        • Store links to folder         • Parameterized bucket
          objects (or entire list) in   ...
Project Statistics




basho
Project Statistics

        • Relational design had
          per-feature counters
          (maintained by app)




basho
Project Statistics

        • Relational design had
          per-feature counters
          (maintained by app)

        ...
Riak: Project Statistics




basho
Riak: Project Statistics
        •Use scatter-gather: Map-reduce was
          made for this type of problem!




basho
Riak: Project Statistics
        •Use scatter-gather: Map-reduce was
          made for this type of problem!

        •Ta...
Riak: Project Statistics
        •Use scatter-gather: Map-reduce was
          made for this type of problem!

        •Ta...
Awkward Fits




basho
Awkward Fits

        •User uniqueness constraint




basho
Awkward Fits

        •User uniqueness constraint
         •“double-click” problem




basho
Awkward Fits

        •User uniqueness constraint
         •“double-click” problem
        •Existing authentication soluti...
Awkward Fits

        •User uniqueness constraint
         •“double-click” problem
        •Existing authentication soluti...
Awkward Fits

        •User uniqueness constraint
         •“double-click” problem
        •Existing authentication soluti...
Review




basho
Review

        •Analyze your relational model




basho
Review

        •Analyze your relational model
        •Identify pain points, take stats



basho
Review

        •Analyze your relational model
        •Identify pain points, take stats
        •Design some alternatives...
Review

        •Analyze your relational model
        •Identify pain points, take stats
        •Design some alternatives...
Plug
        Interested in learning about support,
        consulting, or Enterprise features?
          
        Email in...
Plug
        Interested in learning about support,
        consulting, or Enterprise features?
          
        Email in...
Upcoming SlideShare
Loading in...5
×

Schema Design for Riak

12,612

Published on

Moving to Riak involves a number of changes from the status quo of RDBMS systems, one of which is taking greater control over your schema design.

* How do you structure data when you don't have tables and foreign keys?
* When should you denormalize, add links, or create map-reduce queries?
* Where will Riak be a natural fit and where will it be challenging?

Published in: Technology
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
12,612
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
262
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide















































































































  • Transcript of "Schema Design for Riak"

    1. 1. Schema Design for Riak Sean Cribbs Developer Advocate basho
    2. 2. Thinking Non-Relationally basho
    3. 3. “There is no spoon schema.” basho
    4. 4. There is an application. basho
    5. 5. What you lose basho
    6. 6. What you lose •Tables basho
    7. 7. What you lose •Tables •Foreign keys and constraints basho
    8. 8. What you lose •Tables •Foreign keys and constraints •ACID basho
    9. 9. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners basho
    10. 10. What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners •Declarative query language (SQL) basho
    11. 11. What you gain basho
    12. 12. What you gain •More flexible, fluid designs basho
    13. 13. What you gain •More flexible, fluid designs •More natural data representations basho
    14. 14. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain basho
    15. 15. What you gain •More flexible, fluid designs •More natural data representations •Scaling without pain •Reduced operational complexity basho
    16. 16. Lose the crutches basho
    17. 17. Lose the crutches •Sparse data (optional/multi-value fields) basho
    18. 18. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures basho
    19. 19. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers basho
    20. 20. Lose the crutches •Sparse data (optional/multi-value fields) •Richer data structures •Meaningful identifiers •Innovative access patterns basho
    21. 21. Eventual Consistency basho
    22. 22. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) basho
    23. 23. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging basho
    24. 24. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes basho
    25. 25. Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes •Weigh risk of conflicts and emergent issues basho
    26. 26. Know your data basho
    27. 27. Make your Top 10 list basho
    28. 28. Make your Top 10 list •Frequently requested pages/screens basho
    29. 29. Make your Top 10 list •Frequently requested pages/screens •Slow queries basho
    30. 30. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes basho
    31. 31. Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes •Complicated joins or aggregations basho
    32. 32. Analyze basho
    33. 33. Analyze •Interdependencies, coupling basho
    34. 34. Analyze •Interdependencies, coupling •Cardinalities of relationships basho
    35. 35. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern basho
    36. 36. Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern •Shoehorned structures basho
    37. 37. Example basho
    38. 38. basho
    39. 39. basho
    40. 40. Top queries basho
    41. 41. List Features basho
    42. 42. Reorder Features basho
    43. 43. Regroup Features basho
    44. 44. Edit Feature basho
    45. 45. List Folders basho
    46. 46. Project Statistics basho
    47. 47. projects users folders features milestones project_items Current Schema basho
    48. 48. projects users folders artificial ordering features milestones project_items Current Schema basho
    49. 49. projects users additional key for stats folders artificial ordering features milestones project_items Current Schema basho
    50. 50. projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    51. 51. collaborator projects users additional key for stats BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    52. 52. collaborator projects users additional key for stats owner BLOB folders columns artificial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho
    53. 53. Observations basho
    54. 54. Observations •Features and milestones are disjunct, except in list participation basho
    55. 55. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data basho
    56. 56. Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data •Duplication of relationships - collaborator-owner, project-folder-item basho
    57. 57. List/Reorder Features basho
    58. 58. List/Reorder Features • AJAX postback was already “total” ordering basho
    59. 59. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” basho
    60. 60. List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” • Membership, order are coupled concepts basho
    61. 61. Links in Riak basho
    62. 62. Links in Riak •Lightweight pointers, like HTML <link> tags. basho
    63. 63. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag basho
    64. 64. Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag •Traversable through “link-walking” basho
    65. 65. Riak: List/Reorder Features basho
    66. 66. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag basho
    67. 67. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort basho
    68. 68. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk basho
    69. 69. Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk • Concurrent writes may conflict basho
    70. 70. Regroup Features basho
    71. 71. Regroup Features • Changing foreign key to folder basho
    72. 72. Regroup Features • Changing foreign key to folder • Recalculate order for both lists (multiple UPDATE statements) basho
    73. 73. Riak: Regroup Features basho
    74. 74. Riak: Regroup Features •Already solved for us by folder owning list as links basho
    75. 75. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated basho
    76. 76. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) basho
    77. 77. Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) • No transactions, need extra info for conflict resolution basho
    78. 78. Edit Feature basho
    79. 79. Edit Feature • Required two representations (text, tree), both stored as BLOBs basho
    80. 80. Edit Feature • Required two representations (text, tree), both stored as BLOBs • Application deserializes tree to present web form, recreates text on save basho
    81. 81. Composition basho
    82. 82. Composition •Sometimes called “denormalization” basho
    83. 83. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent basho
    84. 84. Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent •JSON, XML, Erlang terms basho
    85. 85. Riak: Edit Feature basho
    86. 86. Riak: Edit Feature •Already denormalized! basho
    87. 87. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing basho
    88. 88. Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing •Individual objects larger basho
    89. 89. List Folders basho
    90. 90. List Folders • Number of folders is small per project basho
    91. 91. List Folders • Number of folders is small per project • Essentially unordered within project basho
    92. 92. Riak: List Folders basho
    93. 93. Riak: List Folders • Store links to folder objects (or entire list) in project basho
    94. 94. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders basho
    95. 95. Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    96. 96. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    97. 97. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho
    98. 98. Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get • List keys is expensive folders • Two writes required, or move feature links/ order to project basho
    99. 99. Project Statistics basho
    100. 100. Project Statistics • Relational design had per-feature counters (maintained by app) basho
    101. 101. Project Statistics • Relational design had per-feature counters (maintained by app) • Extra foreign key sped up aggregation by reducing joins basho
    102. 102. Riak: Project Statistics basho
    103. 103. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! basho
    104. 104. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects basho
    105. 105. Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects •Query: Follow links to features, map per- feature counters, reduce summarize the counters basho
    106. 106. Awkward Fits basho
    107. 107. Awkward Fits •User uniqueness constraint basho
    108. 108. Awkward Fits •User uniqueness constraint •“double-click” problem basho
    109. 109. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions basho
    110. 110. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership basho
    111. 111. Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership •M-to-N is hard basho
    112. 112. Review basho
    113. 113. Review •Analyze your relational model basho
    114. 114. Review •Analyze your relational model •Identify pain points, take stats basho
    115. 115. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives basho
    116. 116. Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives •Test, Measure, Repeat! basho
    117. 117. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com basho
    118. 118. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com sean@basho.com @seancribbs basho
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×