Schema Design for Riak

Schema Design for Riak Sean Cribbs Developer Advocate basho

Thinking Non-Relationally basho

“There is no spoon schema.” basho

There is an application. basho

What you lose basho

What you lose •Tables basho

What you lose •Tables •Foreign keys and constraints basho

What you lose •Tables •Foreign keys and constraints •ACID basho

What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners basho

What you lose •Tables •Foreign keys and constraints •ACID •Sophisticated query planners •Declarative query language (SQL) basho

What you gain basho

What you gain •More ﬂexible, ﬂuid designs basho

What you gain •More ﬂexible, ﬂuid designs •More natural data representations basho

What you gain •More ﬂexible, ﬂuid designs •More natural data representations •Scaling without pain basho

What you gain •More ﬂexible, ﬂuid designs •More natural data representations •Scaling without pain •Reduced operational complexity basho

Lose the crutches basho

Lose the crutches •Sparse data (optional/multi-value ﬁelds) basho

Lose the crutches •Sparse data (optional/multi-value ﬁelds) •Richer data structures basho

Lose the crutches •Sparse data (optional/multi-value ﬁelds) •Richer data structures •Meaningful identiﬁers basho

Lose the crutches •Sparse data (optional/multi-value ﬁelds) •Richer data structures •Meaningful identiﬁers •Innovative access patterns basho

Eventual Consistency basho

Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) basho

Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging basho

Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes basho

Eventual Consistency •You are already dealing with inconsistency (e.g. caching, read slave) •Plan for EC instead of triaging •Determine resolution processes •Weigh risk of conﬂicts and emergent issues basho

Know your data basho

Make your Top 10 list basho

Make your Top 10 list •Frequently requested pages/screens basho

Make your Top 10 list •Frequently requested pages/screens •Slow queries basho

Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes basho

Make your Top 10 list •Frequently requested pages/screens •Slow queries •Secondary indexes •Complicated joins or aggregations basho

Analyze basho

Analyze •Interdependencies, coupling basho

Analyze •Interdependencies, coupling •Cardinalities of relationships basho

Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern basho

Analyze •Interdependencies, coupling •Cardinalities of relationships •Access pattern •Shoehorned structures basho

Example basho

Top queries basho

List Features basho

Reorder Features basho

Regroup Features basho

Edit Feature basho

List Folders basho

Project Statistics basho

projects users folders features milestones project_items Current Schema basho

projects users folders artiﬁcial ordering features milestones project_items Current Schema basho

projects users additional key for stats folders artiﬁcial ordering features milestones project_items Current Schema basho

projects users additional key for stats BLOB folders columns artiﬁcial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho

collaborator projects users additional key for stats BLOB folders columns artiﬁcial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho

collaborator projects users additional key for stats owner BLOB folders columns artiﬁcial serialized ordering features milestones parse tree formatted project_items plain text Current Schema basho

Observations basho

Observations •Features and milestones are disjunct, except in list participation basho

Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data basho

Observations •Features and milestones are disjunct, except in list participation •Opacity of serialized BLOB data •Duplication of relationships - collaborator-owner, project-folder-item basho

List/Reorder Features basho

List/Reorder Features • AJAX postback was already “total” ordering basho

List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” basho

List/Reorder Features • AJAX postback was already “total” ordering • List is really a property of the parent “folder” • Membership, order are coupled concepts basho

Links in Riak basho

Links in Riak •Lightweight pointers, like HTML <link> tags. basho

Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag basho

Links in Riak •Lightweight pointers, like HTML <link> tags. •Composed of bucket, key, tag •Traversable through “link-walking” basho

Riak: List/Reorder Features basho

Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag basho

Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort basho

Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk basho

Riak: List/Reorder Features •Store list of features and milestones as links on folder, with index as tag •Query: Follow links to items, map values (with index), reduce sort •One write to order list, easy to link-walk • Concurrent writes may conﬂict basho

Regroup Features basho

Regroup Features • Changing foreign key to folder basho

Regroup Features • Changing foreign key to folder • Recalculate order for both lists (multiple UPDATE statements) basho

Riak: Regroup Features basho

Riak: Regroup Features •Already solved for us by folder owning list as links basho

Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated basho

Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) basho

Riak: Regroup Features •Already solved for us by folder owning list as links •List order still recalculated •Two writes (vs. N for relational) • No transactions, need extra info for conﬂict resolution basho

Edit Feature basho

Edit Feature • Required two representations (text, tree), both stored as BLOBs basho

Edit Feature • Required two representations (text, tree), both stored as BLOBs • Application deserializes tree to present web form, recreates text on save basho

Composition basho

Composition •Sometimes called “denormalization” basho

Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent basho

Composition •Sometimes called “denormalization” •Store dependent data in the same object as the parent •JSON, XML, Erlang terms basho

Riak: Edit Feature basho

Riak: Edit Feature •Already denormalized! basho

Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing basho

Riak: Edit Feature •Already denormalized! •Now parse tree can have the same format (JSON, XML), exposing it to other types of processing •Individual objects larger basho

List Folders basho

List Folders • Number of folders is small per project basho

List Folders • Number of folders is small per project • Essentially unordered within project basho

Riak: List Folders basho

Riak: List Folders • Store links to folder objects (or entire list) in project basho

Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders basho

Riak: List Folders • Store links to folder objects (or entire list) in project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho

Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho

Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get folders • Two writes required, or move feature links/ order to project basho

Riak: List Folders • Store links to folder • Parameterized bucket objects (or entire list) in based on project key project • One write to add folder • Simple link-walking or map-reduce to get • List keys is expensive folders • Two writes required, or move feature links/ order to project basho

Project Statistics basho

Project Statistics • Relational design had per-feature counters (maintained by app) basho

Project Statistics • Relational design had per-feature counters (maintained by app) • Extra foreign key sped up aggregation by reducing joins basho

Riak: Project Statistics basho

Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! basho

Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects basho

Riak: Project Statistics •Use scatter-gather: Map-reduce was made for this type of problem! •Take advantage of now-transparent internal format of feature objects •Query: Follow links to features, map per- feature counters, reduce summarize the counters basho

Awkward Fits basho

Awkward Fits •User uniqueness constraint basho

Awkward Fits •User uniqueness constraint •“double-click” problem basho

Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions basho

Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership basho

Awkward Fits •User uniqueness constraint •“double-click” problem •Existing authentication solutions •Project membership •M-to-N is hard basho

Review basho

Review •Analyze your relational model basho

Review •Analyze your relational model •Identify pain points, take stats basho

Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives basho

Review •Analyze your relational model •Identify pain points, take stats •Design some alternatives •Test, Measure, Repeat! basho

Plug Interested in learning about support, consulting, or Enterprise features? Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com basho

Plug Interested in learning about support, consulting, or Enterprise features? Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com sean@basho.com @seancribbs basho

http://basho.com	1407
http://nosql.mypopescu.com	163
http://coderwall.com	36
http://basho.co.jp	32
http://crowdczar.com	15
http://localhost	5
http://localhost:3000	4
http://localhost:3000	4
http://translate.googleusercontent.com	2
http://www.slideshare.net	2
http://twitter.com	1
http://www-new.basho.com	1

Schema Design for Riak

by Sean Cribbs

Accessibility

Categories

Upload Details

Usage Rights

12 Embeds 1,672

Statistics

Schema Design for Riak Presentation Transcript