Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Slides: Moving from a Relational Model to NoSQL

1,117 views

Published on

Businesses are quickly moving to NoSQL databases to power their modern applications. However, a technology migration involves risk, especially if you have to change your data model. What if you could host a relatively unmodified RDBMS schema on your NoSQL database, then optimize it over time?

We’ll show you how Couchbase makes it easy to:

• Use SQL for JSON to query your data and create joins
• Optimize indexes and perform HashMap queries
• Build applications and analysis with NoSQL

Published in: Data & Analytics
  • Login to see the comments

Slides: Moving from a Relational Model to NoSQL

  1. 1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Moving from a Relational Model to NoSQL Leveraging a Flexible JSON Schema Matthew Groves – Product Marketing Manager
  2. 2. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 2 Objectives • Give an overview of migrating an existing relational database to NoSQL (Couchbase) • Give you a sense for some of the technical issues / differences • We will not get into deep details (i.e. dialect details, different ways to write a query, how indexes are handled, etc)
  3. 3. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 1 2 3 4 5 Quick Intro to NoSQL Quick Intro to Couchbase Migration Options Optimizing Demo AGENDA
  4. 4. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Quick Intro to NoSQL
  5. 5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 5 Consider + Purchase Operate + Service Build + Fulfill Massively Interactive Enterprises e-Commerce Supply Chain Internet of Things CUSTOMER EMPLOYEE MACHINE Drivers of NoSQL
  6. 6. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 6 Application Requirements Are Dramatically Different NoSQL Transactions Interactions Transactional Analytical SCALABLE SECURE FLEXIBILE DISTRIBUTED LOW TCO
  7. 7. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 7 Application Requirements Are Dramatically Different Oracle 1K:1 Consumer 1:1 1M:1 IoT Legacy Databases Insufficient Relational divide Couchbase
  8. 8. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 8 What is JSON? JavaScript Object Notation • Lightweight data interchange format • Based on a subset of JavaScript • Uses a hierarchical key/value syntax Data Types • Object: Unordered collection of key/value pairs • Array: ordered list of 0 or more values • Number: Integer or Float • String: Unicode in double quotes • Boolean: true or false • Null: Empty Value <""> { "JSON": "JavaScript Object Notation", "Desc": [ "Lightweight data interchange format", "Based on a subset of JavaScript", "Uses a hierarchical key/value syntax" ], "DataTypes": { "Object": { "Desc": "Unordered collection of key/value pairs", "Ex01": { "key": "value" } }, "Array": { "Desc": "Ordered list of 0 or more values", "Ex01": [ "apples", 12, true ] }, "Number": { "Desc": "Integer or Float", "Ex01": 12, "Ex02": 1234.56789 }, "String": { "Desc": "Unicode in double quotes", "Ex01": "Hello World!" }, "Boolean": { "Desc": "true or false", "Ex01": true, "Ex02": false }, "Null": { "Desc": "Empty Value", "Ex01": "" } }
  9. 9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 9 Why JSON for Data? JSON is easy to read and write JSON is ubiquitous • ~ 170 JSON parsing libs • ~ 60 different languages/frameworks • json.org – resource lists • Standardizing in RFC 8259 JSON objects make great "documents" • Object => "document" / "subdocument" • Internal key/value pairs => "attributes"
  10. 10. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 10 { '_type': 'user', 'age': 23, 'name': 'Mint, Mikey', 'admin': false } Tables vs document sets (simple) USER user_id age name admin blue123-4 42 Blue, Betty true orange456-7 31 Orange, Oscar false { 'age': 42, 'name': 'Blue, Betty', 'admin': true } id: blue123-4 Relational DB Tables Rows { '_type': 'product', 'qty;: 1966, 'name': 'Blarg Nozzle', 'stock': true } { 'qty;: 1966, 'name': 'Blarg Nozzle', 'stock': true } id: abc123 PRODUCT prod_id qty name stock abc123 1966 Blarg Nozzle true xyz456 5280 Hepto Shaft false NoSQL Bucket Collections Documents
  11. 11. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 11 Schema “enforcement” vs schema agility USER user_id age name admin blue123-4 42 Blue, Betty true orange456-7 31 Orange, Oscar false Relational { '_type': 'product', 'qty;: 1966, 'name': 'Blarg Nozzle', 'stock': true } { 'type': 'product', 'qty;: 1966, 'name': 'Blarg Nozzle', 'stock': true } id: product:: abc123 PRODUCT prod_id qty name stock abc123 1966 Blarg Nozzle true xyz456 5280 Hepto Shaft false def321 ??? Foo Modulator false { 'qty': 0, 'name': 'Foo Modulator', 'stock': false, 'restock': true } id: def321 restock = true ? { '_type': 'user', 'age': 23, 'name': 'Mint, Mikey', 'admin': false } { 'age': 42, 'name': 'Blue, Betty', 'admin': true } id: blue123-4 NoSQL
  12. 12. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 12 Timestamp format choices • ISO 8601 • Time component array • Epoch/Unix
  13. 13. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Quick Intro to Couchbase
  14. 14. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 14 Couchbase | NoEQUAL Agility/versatility Performance Easy to manage
  15. 15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 15 What is Couchbase? • NoSQL Database that stores JSON documents • Includes a Key-Value Store for fast lookups • Supports SQL and ACID transactions • Includes Indexing • Includes Full Text Search capability • Memory-First, shared-nothing architecture • Asynchronous architecture • Masterless
  16. 16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 16 PRIVATE CLOUD Cloud-Native: Deploy On Premise or In Any Cloud OPTIMIZED FOR FLEXIBILITY, CUSTOMIZABILITY AND PERFORMANCE OPTIMIZED FOR SIMPLICITY, PERFORMANCE AND EASE OF USE PUBLIC CLOUD FULLY MANAGED DBaaS
  17. 17. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 17 COUCHBASE LITE SYNC GATEWAY COUCHBASE SERVER Lightweight embedded NoSQL database with full CRUD and query functionality. Secure web gateway with synchronization, data access, and data integration APIs for accessing, integrating, and synchronizing data over the web. Highly scalable, highly available, high performance NoSQL database server. Securely Manage & Sync Data from any Cloud to the Edge
  18. 18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 18 Cross Data Center Replication (XDCR)
  19. 19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Migration Options
  20. 20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 20 Scenario Existing application architected on an RDBMS, it works well, but there are a lot of changes coming and new expected demands in terms of scalability.
  21. 21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 21 Issues to consider • Risk tolerance • Skills / expertise • Migration cost • Performance • Scale
  22. 22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 22 How do you migrate? • 1 Rewrite: No migration, write the whole thing over • 2 Redesign Schema: Keep your business logic, rewrite your data layer and schema, totally redesign your schema on NoSQL best practices • 3 Refactor First: Keep everything but refactor your data logic and RDBMS schema into a best practices NoSQL schema • 4 Optimize Later: Host your schema with as few changes as possible, get the application running on the new technology, refactor/optimize the schema as necessary for performance • 5 Just Host It: Host your schema with as few changes as possible. Risk Effort
  23. 23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Migrating Without Denormalizing
  24. 24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 24 Conventional relational data Normalize – model data as minimally redundant tables, and rely on in-memory JOINs ✔Storage minimized ✔Schema enforced ✔Ad hoc queries well supported by SQL ✔Record-centric, 2D data paradigm CustomerID Name Address Email XYZ234 Acme 9 W. 2nd al@acme ProductID Name Description 774477 Widget Fit for all uses. 115588 Gadget Just what I need. PayTypeID Name 001 PayPal OrderID Customer PayType 123 XYZ234 001 OrderID ProductID Qty 123 774477 2 123 115588 3
  25. 25. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 25 Collection Relational Database vs Bucket (simple) USER user_id age name admin blue123-4 42 Blue, Betty true orange456-7 31 Orange, Oscar false Relational table { 'type': 'user', 'age': 31, 'name': 'Orange, Oscar', 'admin’: false } id: user::orang e456-7 { 'age': 42, 'name': 'Blue, Betty', 'admin': true } id: blue123-4
  26. 26. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 26 { 'type': 'user', 'age': 31, 'name': 'Orange, Oscar', 'admin’: false } id: user::orang e456-7 Collection Rows/records vs documents (simple) Copyright © 2019 Couchbase, Inc. USER user_id age name admin blue123-4 42 Blue, Betty true orange456-7 31 Orange, Oscar false Relational table { 'age': 42, 'name': 'Blue, Betty', 'admin': true } id: blue123-4
  27. 27. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 27 Collection Columns vs attributes/parameters/fields USER user_id age name admin blue123-4 42 Blue, Betty true orange456-7 31 Orange, Oscar false Relational table { 'age': 31, 'name': 'Orange, Oscar', 'admin’: false } id: orange456-7 { 'age': 42, 'name': 'Blue, Betty', 'admin': true } id: blue123-4
  28. 28. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 28 Document data { "date" : "2021-01-06", "orderNum" : "ABC123", "customer": { "name": "Acme", "address": "9 W. 2nd", "email": "al@acme" }, "items": [{ "qty": 2, "name": "Widget", "description": "Fit for all uses." },{ "qty": 3, "name": "Gadget", "description": "Just what I need." }] } order::123 ORDER id date orderNum 123 2021-01-06 ABC123 ITEM order_id qty name description 123 2 Widget Fit for all… 123 3 Gadget Just what I ... CUSTOMER order_id name address email 123 Acme 9 W. 2nd al@acme
  29. 29. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 29 Document data De-normalize – optimize for planned application usage ✔Data can directly align with object model ✔Reduced JOIN and Tx pressure ✔Simpler data-layer code ✔Low latency CRUD and streaming APIs ✔Indexing for alternate access patterns ✔Queries well supported by SQL++ (N1QL)
  30. 30. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 30 The New Query SELECT l.city FROM Travel.dbo.LANDMARKS l WHERE l.city IN (SELECT city FROM Travel.dbo.AIRPORTS) RDBMS Queries SELECT l.city FROM Travel._ default.landmark l WHEREl.city IN (SELECT VALUE city FROM Travel._ default.airport ) N1QL Queries
  31. 31. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 31 Query Differences • Table = collection • ANSI JOINs and other SQL conventions are supported. • There are other “N1QLisms” that are mainly to do with JSON like “unnest” or “nest” dealing with embedded documents. • Indexes are very important in Couchbase SELECT player.name, player.level, stats.loadtime, SUM(CASE WHEN hist.result = "won" THEN 1 ELSE 0 END) AS wins FROM jungleville_stats AS stats UNNEST stats.`pvp - hist` AS hist JOIN jungleville AS player ON KEYS stats.uuid GROUP BY player, stats ORDER BY wins DESC, player.level DESC
  32. 32. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 32 Indexes • meta(bucket) allows you to refer to the field names • GSI = Global Secondary Index CREATE INDEX person_role on profiles( person_role_name ) WHERE meta(profiles).type="person" USING GSI
  33. 33. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 33 Keys to success • NoSQL Database feature support (SQL, ACID, Joins) • Understanding • Buckets • Scopes • Collections • SQL Dialect transitions • Optimize Later
  34. 34. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Optimizing
  35. 35. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 35 Key considerations • Joining and transacting is costly (especially in a distributed architecture) and doesn’t scale as well horizontally. • Your well-designed application code is a guide. “Person” contains “Addresses” but looks up “Orders” • Think in terms of Entities. In the scope of your application does this have a “lifecycle” outside of the thing that always references it. • Can you reduce the number of explicit transactions required? • Can you reduce the number of joins? • Have you met your performance/scale requirement
  36. 36. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 36 Key considerations JOINs / ACID App code as a guide Entities
  37. 37. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 37 Reduce Joins/Transactions { 'acct_no': 'ABC1', 'balances': { 'savings': 100, 'checking': 200 } } UPDATE Accounts SET balances.savings = 100 AND balances.checking = 200 WHERE acct_no = "ABC1" AND type = "account"; acct_no balance ABC1 100 UPDATE savings SET balance = 100 WHERE acct_no = "ABC1"; savings acct_no balance ABC1 200 UPDATE checking SET balance = 200 WHERE acct_no = "ABC1"; checking N1QL: Atomic document update SQL: ACID transaction START TRANSACTION COMMIT
  38. 38. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 38 Analyze your logical model ✔Order embeds Items, external Product (1:n) and Paytype (1:1) docs Customer Order 1:n Paytype 1:1 Items 1:n Product 1:1 Name Name Quantity Name Description Price Is this your object?
  39. 39. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 39 ✔Order embeds Items, external Product (1:n) and Paytype (1:1) docs Customer Order 1:n Paytype 1:1 Items 1:n Product 1:1 Name Name Quantity Name Description Price Is this your object? Embedding
  40. 40. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 40 Analyze your logical model ✔Order embeds Paytype and refs Items which embed Product Customer Order 1:n Paytype 1:1 Items 1:n Product 1:1 Name Name Quantity Name Description Price Are these your objects?
  41. 41. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 41 Embedding + Referring ✔Order embeds Paytype and refs Items which embed Product Customer Order 1:n Paytype 1:1 Items 1:n Product 1:1 Name Name Quantity Name Description Price Are these your objects? Customer Paytype
  42. 42. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 42 Objects vs. object arrays
  43. 43. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 43 Conclusions • Migrate first, optimize later (reduces risk, increases performance/scale) • Joins / Multi-document transactions are more costly in a distributed architecture • Expect a 1-1 migration to be slower, require more hardware • Embed documents as an optimization • SQL Queries can be converted • ACID transactions are available (but not needed as often when optimized)
  44. 44. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. 44 Data Access: N1QL = SQL for JSON Create User Get User Update User Delete User SELECT * FROM users WHERE users.lastName = “Johnson” UPDATE users SET status = “Platinum” WHERE users.lastName = “Johnson” DELETE FROM users WHERE users.firstName = “Shane” 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 INSERT INTO VALUES Users (KEY, VALUE) ( “user::100”, { “firstName”: ”Shane”, “lastName”: ”Johnson” } );
  45. 45. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. Couchbase Demo
  46. 46. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2020. All rights reserved. https://www.couchbase.com/products/cloud

×