Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Agile document models & data structures – Couchbase Connect 2016

751 views

Published on

Developing on a flexible schema system like Couchbase makes you more agile in responding to changing needs. However, to really make development easy you still need to map the underlying documents to data structures in higher level languages and have a data model supporting your access patterns.

Couchbase Server 4.5 added an innovative JSON streaming-parser in the core of the server enabling Sub-Document operations over the super-efficient binary protocol. Using this, Data Structures such as Lists, Maps, Sets and Queues have been added to Couchbase SDKs. Since the underlying representation is still JSON, it retains the flexible data modeling needed. Of course, services such as N1QL and Full Text Search work exactly as expected on these same JSON documents.

In this session, you will learn what to consider when planning your data model and learn how the new Data Structures and Sub-Document operations can make your Couchbase apps more efficient. The session will show an example data model and how Couchbase data structures map up to high-level APIs in platforms such as Java’s Collections Framework or functions in other platforms like Go.

Published in: Software

Agile document models & data structures – Couchbase Connect 2016

  1. 1. ©2016 Couchbase Inc. Agile Document Models & Data Structures 1
  2. 2. ©2016 Couchbase Inc.©2016 Couchbase Inc. SpeakingYour Language • Topics for today: • Data structures - tie into native language collection interfaces • Sub-document - lower level access with focused power • Data modeling with Couchbase • Session: “Picking the right API for the right job” • SDK Goal: complex data access made easy • More than just a document storage/retrieval system • Tight SDK integration is key • Consistent, transparent developer experience across languages 2
  3. 3. ©2016 Couchbase Inc. 3 Data Structures API
  4. 4. ©2016 Couchbase Inc.©2016 Couchbase Inc. Couchbase SDK Data Structures API • Target SDK release along with 4.6 • Builds on awesomeness of sub-documentAPI • Simplified access without touching whole document • Make JSON data types transparent • Native integration of Map, List, Set, Queues… • Java Collections Framework • .NET System.Collections • Python, Node.js, Go 4
  5. 5. ©2016 Couchbase Inc.©2016 Couchbase Inc. Typical Document Data Access JSON DocCB JSON ObjectSDK Collections FrameworkApp 5
  6. 6. ©2016 Couchbase Inc.©2016 Couchbase Inc. Simplified Data Structure Access JSON DocCB Collections Framework SDK DS 6 App “user1”: {“name”:... , “address”:.. , “favs”: [...]}, “user2”:{“name” , “address” ..., ”favs”: [...]}, for (String f : favs) {} “user1”: {“name”:... , “address”:.. , “favs”: [...]}, “user2”:{“name” , “address” ..., ”favs”: [...]},
  7. 7. ©2016 Couchbase Inc.©2016 Couchbase Inc. Targeted Collection Updates Item From Collection App Sub-doc Update CB 7 MapAdd(“user1”,”favs”, “newfav”) “user1”: {“name”:... , “address”:.. , “favs”: [...]}, “user2”:{“name” , “address” ..., ”favs”: [...]},
  8. 8. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Four Data Structures… Structure JSON Type JSON Example Lists - Append, prepend, insert - Size/count JSON Array: [… , ... ] [ 1, 2, “abc” ] Maps - Add/remove by key - Size/count JSON Object: { “key”: “value”} { “name”: “value” } Sets - Specialized add/remove - Unique values - Size/count JSON Array: [ … , ... ] [ 1, 3, 6, 8 ] Queue - First in – first out - Pop – retrieve/remove - Size/count JSON Array: [… , ... ] [ “task1”, “task2”, “task3” ] remove 1... [ “task2”, “task3”, “task4” ] 8
  9. 9. ©2016 Couchbase Inc.©2016 Couchbase Inc. Consistent Access Across Languages Functions Lists ListGet ListPush ListShift ListDelete ListSet ListSize >namesList = bucket.ListGet(“key”) >print namesList [‘name1’,’name2’,’name3’] Maps MapGet MapRemove MapSize MapSet Sets SetAdd SetExists SetSize SetRemove Queue QueuePush QueuePop QueueSize QueueRemove 9 • Idiomatic -vs- functional • Java Collections Framework • .NET System.Collections • As well as functional approach * Experimental features alert: may add/remove to this list – feedback welcome!
  10. 10. ©2016 Couchbase Inc.©2016 Couchbase Inc. Consistent Access Across Languages 10 Collections Approach Lists List<String> namesList = new CouchbaseArrayList<String>("key", bucket); for (String name : namesList) { … } Maps var namesDict = new CouchbaseDictionary<string, Poco>(_bucket, “key”); namesDict.Add(“newkey1”, new Poco { Name = “poco1” }); Sets var namesSet = new CouchbaseSet<Poco>(_bucket, "pocos"); namesSet.Add(new Poco { Key = "poco1", Name = "Poco-pica" }); namesSet.Remove(new Poco {Key = "poco1", Name = "Poco-pica"}); foreach(var poco in namesSet){ … } Queue var namesQueue = new CouchbaseQueue<Poco>(_bucket, key); namesQueue.Enqueue(new Poco { Name = "pcoco1" }); var item = namesQueue.Peek(); • Support for advanced capabilities of collection frameworks
  11. 11. ©2016 Couchbase Inc. 11 Sub-Document API
  12. 12. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sub-Document API “The sub-document API enables you to access parts of JSON documents (sub-documents) efficiently without requiring the transfer of the entire document over the network. This improves performance and brings better efficiency to the network IO path, especially when working with largeJSON documents.” • First released in 4.5, support cross SDK • Efficient document lookup, insert & update • Powerful lower level control, focusing on particular elements • Keep work on server • Two methods available – lookup and mutate/change 12
  13. 13. ©2016 Couchbase Inc.©2016 Couchbase Inc. Digging Below Data Structures Data Structures API Sub-Document API MapGet(key, mapkey) LookupIn(key).get(mapkey) MapRemove(key, mapkey) MutateIn(key).remove(mapkey) MapSet(key, mapkey, value, createMap) MutateIn(key).(mapkey, value, create_doc=createMap) 13
  14. 14. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sub-Document API 14 Operations LookupIn LookupIn(key, operation(path)) Get Exists Execute MutateIn MutateIn(key, operation(path, value)) Counter Insert Remove Replace Upsert Execute arrayAddunique arrayAppend arrayInsert arrayPrepend Chaining Operations MutateIn(key, operation(path, value), operation(path, value), operation(path, value)) Returns SubdocResult<rc=0x0, key='map1', cas=0x14b6458980042, specs=(Spec<GET, 'subkey1'>, Spec<EXISTS, 'subkey1'>), results=[(0, u'subvalue1'), (0, None)]>
  15. 15. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sample Sub-Document Lookup 15 LookupIn(key, operation(path)) LookupIn(‘copilotmark’) .get(‘phones.number') .execute(); LookupIn(‘copilotmark’) .exists(‘phones’) .get(‘phones.number') .get(‘gender’) .execute(); SubdocResult<rc=0x0, key=’copilotmark', cas=0x14b6458980042, specs=(Spec<EXISTS, ‘phones’>, <GET, ’phones.number'>, <GET, ‘gender’), results=[(0, None,), (0, ’212-771-1834’), (0, u’male')]>
  16. 16. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sample Sub-Document Change 16 MutateIn(key, path, value) MutateIn(‘copilotmark’) .replace(‘phones.number’, ‘212-787-2212’) .upsert(‘nickname’, ‘Freddie’) .execute()
  17. 17. ©2016 Couchbase Inc. 17 Data Modeling for Couchbase Server
  18. 18. ©2016 Couchbase Inc.©2016 Couchbase Inc. What is Data Modeling? 18 • A data model is a conceptual representation of the data structures that are required by a database • The data structures include the data objects, the associations between data objects, and the rules which govern operations on the objects.
  19. 19. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Modeling Approaches 19 NoSQL Relaxed Normalization schema implied by structure fields may be empty, duplicate, or missing Relational Required Normalization schema enforced by db same fields in all records • Minimize data inconsistencies (one item = one location) • Reduced update cost (no duplicated data) • Preserve storage resources • Optimized to planned/actual access patterns • Flexibly with software architecture • Supports clustered architecture • Reduced server overhead
  20. 20. ©2016 Couchbase Inc.©2016 Couchbase Inc. Modeling Couchbase Documents 20 • Couchbase Server is a document database • Data is stored in JSON documents, not in tables • Relational databases rely on an explicit pre-defined schema to describe the structure of data • JSON documents are self-describing
  21. 21. ©2016 Couchbase Inc.©2016 Couchbase Inc. What and Why JSON? 21 • What is JSON? – Schema flexibility – Lightweight data interchange format – Based on JavaScript – Programming language independent – Field names must be unique • Why JSON? – Less verbose – Can represent Objects and Arrays (including nested documents) There is NO IMPEDENCE MISMATCH between a JSON Document and a Java Object
  22. 22. ©2016 Couchbase Inc.©2016 Couchbase Inc. JSON Design Choices 22 • Couchbase Server neither enforces nor validates for any particular document structure • Choices that impact JSON document design: – Single Root Attributes – Objects vs. Arrays – Array ElementTypes – Timestamp Formats – Property Names – Empty and Null PropertyValues – JSON Schema
  23. 23. ©2016 Couchbase Inc.©2016 Couchbase Inc. Root Attributes vs. Embedded Attributes 23 • The choice of having a single root attribute or the “type” attribute embedded.
  24. 24. ©2016 Couchbase Inc.©2016 Couchbase Inc. Root Attributes vs. Embedded Attributes 24 • Accessing the document with a root attribute SELECT track.* FROM couchmusic
  25. 25. ©2016 Couchbase Inc.©2016 Couchbase Inc. Root Attributes vs. Embedded Attributes 25 • Accessing the document with the “type” attribute SELECT * FROM couchmusic WHERE type=‘track’
  26. 26. ©2016 Couchbase Inc.©2016 Couchbase Inc. Objects vs. Arrays 26 • The choice of having an object type, or an array type
  27. 27. ©2016 Couchbase Inc.©2016 Couchbase Inc. Objects vs. Arrays 27 • How would the object look like? class UserProfile{ Phone phones; } class Phone{ String cell; String landline; }
  28. 28. ©2016 Couchbase Inc.©2016 Couchbase Inc. Objects vs. Arrays 28 • How would the object look like? class UserProfile{ List<Phone> phones; } class Phone{ String number; String type; }
  29. 29. ©2016 Couchbase Inc.©2016 Couchbase Inc. Array ElementTypes Array of strings Array of objects 29 • Array elements can be simple types, objects or arrays:
  30. 30. ©2016 Couchbase Inc.©2016 Couchbase Inc. Array ElementTypes Array of strings 30 • Array elements can be simple types, objects or arrays: class Playlist{ List<String> tracks; } ... String trackId = tracks.get(1); JsonDocument trackDocument = bucket.get(trackId) Multiple get() calls to retrieve the document. Worth it?
  31. 31. ©2016 Couchbase Inc.©2016 Couchbase Inc. Array ElementTypes 31 • Array elements can be simple types, objects or arrays: class Playlist{ List<Track> tracks; } ... myPlaylist.getTracks() .get(1).getArtistName(); Limited Denormalization: commonly needed data (e.g., title) in local object, detail available in referenced foreign document
  32. 32. ©2016 Couchbase Inc.©2016 Couchbase Inc. Timestamp Formats Array of time components String (ISO 8601) Number (Unix style) (Epoch) • Working and dealing with timestamps has been challenging ever since • When storing timestamps, you have at least 3 options: 16
  33. 33. ©2016 Couchbase Inc.©2016 Couchbase Inc. Observed Practices withTimestamp Formats • Storing as Epoch will help you to easily sort the documents • If you wanted the documents to be sorted in the order of their “last update” time • SELECT * FROM couchmusic WHERE type = ‘track’ ORDER BY updates DESC • Storing date as array format helps • To grouping 16
  34. 34. ©2016 Couchbase Inc.©2016 Couchbase Inc. Taking Advantage of Storing Date as an Array • Group options can be specified to control the execution of the view • The group and group_level options are only useful when a Reduce function has been defined in the correspondingView • The group_level option, used when the key is anArray, determines how many elements of the key are used when aggregating the results. 16
  35. 35. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of View group_level = 1 Key Value [2014] 36 [2015] 20 Execute Reduce Key Value [2014,11,29,18,49,36] 3 [2014,12,03,20,11,26] 5 [2014,12,03,23,37,21] 2 [2014,12,06,10,12,19] 8 [2014,12,09,05,01,26] 3 [2014,12,18,01,04,30] 11 [2014,12,26,18,34,44] 4 [2015,01,03,16,48,32] 7 [2015,01,03,20,20,06] 5 [2015,01,15,08,17,28] 8 Copyright © 2015 Couchbase, Inc. 35 • For the data below with Reduce function defined as _sum and group_level = 1
  36. 36. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of View group_level = 2 Key Value [2014,11] 3 [2014,12] 33 [2015,01] 20 Key Value [2014,11,29,18,49,36] 3 [2014,12,03,20,11,26] 5 [2014,12,03,23,37,21] 2 [2014,12,06,10,12,19] 8 [2014,12,09,05,01,26] 3 [2014,12,18,01,04,30] 11 [2014,12,26,18,34,44] 4 [2015,01,03,16,48,32] 7 [2015,01,03,20,20,06] 5 [2015,01,15,08,17,28] 8 Copyright © 2015 Couchbase, Inc. 36 • For the data below with Reduce function defined as _sum and group_level = 2 Execute Reduce
  37. 37. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of View group_level = 3 Key Value [2014,11,29,18,49,36] 3 [2014,12,03,20,11,26] 5 [2014,12,03,23,37,21] 2 [2014,12,06,10,12,19] 8 [2014,12,09,05,01,26] 3 [2014,12,18,01,04,30] 11 [2014,12,26,18,34,44] 4 [2015,01,03,16,48,32] 7 [2015,01,03,20,20,06] 5 [2015,01,15,08,17,28] 8 Key Value [2014,11,29] 3 [2014,12,03] 7 [2015,12,06] 8 [2015,12,09] 3 [2015,12,18] 11 [2015,12,26] 4 [2014,01,03] 12 [2014,01,15] 8 Copyright © 2015 Couchbase, Inc. 37 • For the data below with Reduce function defined as _sum and group_level = 3 Execute Reduce
  38. 38. ©2016 Couchbase Inc.©2016 Couchbase Inc. Empty and Null PropertyValues 38 • Keep in mind that JSON supports optional properties • If a property has a null value, consider dropping it from the JSON, unless there's a good reason not to • N1QL makes it easy to test for missing or null property values • Be sure your application code handles the case where a property value is missing SELECT * FROM couchmusic1 WHERE userprofile.address IS NULL; SELECT * FROM couchmusic1 WHERE userprofile.gender IS MISSING;
  39. 39. ©2016 Couchbase Inc.©2016 Couchbase Inc. Empty, Null and Missing PropertyValues 39 { countryCode: “UK”, currencyCode: “GBP”, region: “Europe” } { countryCode: “UK”, currencyCode: “GBP”, region: “” } WHERE region IS NOT MISSING, IS NOT NULL, IS VALUED WHERE region IS NOT MISSING, IS NOT NULL, IS NOT VALUED { countryCode: “UK”, currencyCode: “GBP” } { countryCode: “UK”, currencyCode: “GBP”, region: null } WHERE region IS MISSING WHERE region IS NULL
  40. 40. ©2016 Couchbase Inc.©2016 Couchbase Inc. JSON Schema 40 • Couchbase Server pays absolutely no attention to the shape of your JSON documents so long as they are well-formed • There are times when it is useful to validate that a JSON document conforms to some expected shape • JSON Schema is a JSON-based format for defining the structure of JSON data • There are implementations for most popular programming languages • Learn more here: http://json-schema.org
  41. 41. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of JSON Schema 41
  42. 42. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of JSON Schema –Type Specification Available type specifications include: • array • boolean • integer • number • object • string • enum 42
  43. 43. ©2016 Couchbase Inc.©2016 Couchbase Inc. Type specific validations include: • minimum • maximum • minLength • maxLength • format • pattern 43 Example of JSON Schema –Type SpecificValidation
  44. 44. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of JSON Schema – Required Properties Required properties can be specified for each object 44
  45. 45. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example of JSON Schema – Additional Properties Additional properties can be disabled 45
  46. 46. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Nesting (aka Denormalization) 46 • As you know, relational database design promotes separating data using normalization, which doesn’t scale • For NoSQL systems, we often avoid normalization so that we can scale • Nesting allows related objects to be organized into a hierarchical tree structure where you can have multiple levels of grouping • Rule of thumb is to nest no more than 3 levels deep unless there is a very good reason to do so • You will often want to include a timestamp in the nested data
  47. 47. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example #1 of Data Nesting • Playlist with owner attribute containing username of corresponding userprofile 47 Document Key: copilotmarks61569
  48. 48. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example #1 of Data Nesting • Playlist with owner attribute containing a subset of the corresponding userprofile 48 * Note the inclusion of the updated attribute
  49. 49. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example #2 of Data Nesting • Playlist with tracks attribute containing an array of track IDs 49
  50. 50. ©2016 Couchbase Inc.©2016 Couchbase Inc. Example #2 of Data Nesting • Playlist with tracks attribute containing an array of track objects 50 * Note the inclusion of the updated attribute
  51. 51. ©2016 Couchbase Inc. 51 Key Design
  52. 52. ©2016 Couchbase Inc.©2016 Couchbase Inc. Choices with JSON Key Design 52 • A key formed of attributes that exist in the real world: – Phone numbers – Usernames – Social security numbers – Account numbers – SKU, UPC or QR codes – Device IDs • Often the first choice for document keys • Be careful when working with any personally identifiable information (PII), sensitive personal information (SPI) or protected health information (PHI)
  53. 53. ©2016 Couchbase Inc.©2016 Couchbase Inc. Surrogate Keys 53 • We often use surrogate keys when no obvious natural key exist • They are not derived from application data • They can be generated values – 3305311F4A0FAAFEABD001D324906748B18FB24A (SHA-1) – 003C6F65-641A-4CGA-8E5E-41C947086CAE (UUID) • They can be sequential numbers (often implemented using the Counter feature of Couchbase Server) – 456789, 456790, 456791, …
  54. 54. ©2016 Couchbase Inc.©2016 Couchbase Inc. KeyValue Patterns • Common practice for users of Couchbase Server to follow patterns for formatting key values by using symbols such as single or double colons • DocType::ID – userprofile::fredsmith79 – playlist::003c6f65-641a-4c9a-8e5e-41c947086cae • AppName::DocType::ID – couchmusic::userprofile::fredsmith79 54 Enables Multi-Tenency – pizza::user::101 – Pizza::user::102 – burger::user::101 – burger::user::102
  55. 55. ©2016 Couchbase Inc.©2016 Couchbase Inc. Lookup Key Pattern 55 • The purpose of the Lookup Key Pattern is to allow multiple ways to reach the same data, essentially a secondary index • For example, we want to lookup a Userprofile by their email address instead of their ID • To accomplish this, we create another small document that refers to the Userprofile document we are interested in • Implementing this pattern is straightforward, just create an additional document containing a single property that stores the key to the primary document • With the introduction of N1QL, this pattern will be less commonly used
  56. 56. ©2016 Couchbase Inc.©2016 Couchbase Inc. Lookup Key Pattern 56 userprofile::copilotmarks61569 andy.bowman@games.com JSON andy.bowman@games.com • Lookup document can be JsonDocument or StringDocument
  57. 57. ©2016 Couchbase Inc. 57 Trade-offs in Data Modeling
  58. 58. ©2016 Couchbase Inc.©2016 Couchbase Inc. MakingTough Choices 58 • We must also make trade-offs in data modeling: – Document size – Atomicity – Complexity – Speed
  59. 59. ©2016 Couchbase Inc.©2016 Couchbase Inc. Document Size 59 • Couchbase Server supports documents up to 20 Mb • Larger documents take more disk space, more time to transfer across the network and more time to serialize/deserialize • If you are dealing with documents that are potentially large (greater than 1 Mb), you must test thoroughly to find out if speed of access is adequate as you scale. If not, you will need to break up the document into smaller ones. • You may need to limit the number of dependent child objects you embed
  60. 60. ©2016 Couchbase Inc.©2016 Couchbase Inc. Atomicity 60 • Atomicity in Couchbase Server is at the document level • Couchbase Server does not support transactions • They can be simulated if you are willing to write and maintain additional code to implement them (generally not recommended) • If you absolutely need changes to be atomic, they will have to be part of the same document • The maximum document size for Couchbase Server may limit how much data you can store in a single document
  61. 61. ©2016 Couchbase Inc.©2016 Couchbase Inc. Complexity 61 • Complexity affects every area of software systems including data modeling • The complexity of queries (N1QL) • The complexity of code for updating multiple copies of the same data
  62. 62. ©2016 Couchbase Inc.©2016 Couchbase Inc. Speed 62 • As it relates to data modeling, speed of access is critical • When using N1QL to access data, keep in mind that query by document key is fastest and query by secondary index is usually much slower • If implementing an interactive use case, you will want to avoid using JOINs • You can use data duplication to improve the speed of accessing related data and thus trade improved speed for greater complexity and larger document size • Keep in mind that CouchbaseViews can be used when up to the second accuracy is not required
  63. 63. ©2016 Couchbase Inc.©2016 Couchbase Inc. Remember 63 SDK get() is faster than (get by key) N1QL with MOI is faster than N1QL with GSI is faster than Model you document key, such that you document can be retrieved with the key, if possible, than a N1QL query
  64. 64. ©2016 Couchbase Inc.©2016 Couchbase Inc. Embed vs. Refer 64 • All of the previous trade-offs are usually rolled into a single decision – whether to embed or refer • When to embed: – Reads greatly outnumber writes – You're comfortable with the slim risk of inconsistent data across the multiple copies – You're optimizing for speed of access • When to refer: – Consistency of the data is a priority – You want to ensure your cache is used efficiently – The embedded version would be too large or complex
  65. 65. ©2016 Couchbase Inc.©2016 Couchbase Inc. Next Steps • Flexible data access is key to solutions using document stores • Join us for discussion on Forums or discuss with our experts here • https://forums.couchbase.com • https://developer.couchbase.com/server 65
  66. 66. ©2016 Couchbase Inc. 66 GetTrained on Couchbase http://training.couchbase.com CS300: Couchbase NoSQL Server Administration CD220: Developing Couchbase NoSQL Applications CD210: Couchbase NoSQL Data Modeling, Querying, andTuning Using N1QL CD257: Developing Couchbase Mobile NoSQL Applications
  67. 67. ©2016 Couchbase Inc. 67 Tyler Mitchell Senior Product Manager, SDK tyler@couchbase.com @1tylermitchell Clarence J MTauro, Ph.D. Senior Instructor clarence@couchbase.com @javapsyche
  68. 68. ©2016 Couchbase Inc. ThankYou! 68

×