Couchbase App Development: Documents, schemas and their relationships

2,662 views

Published on

You will see examples of JSON document schemas and learn how to model object relationships to best take advantage of Couchbase's low latency document access and map reduce query functionality.

In this webinar you'll learn:

How to model object relationships in Couchbase
Examples of different JSON document schemas
How to take advantage of the powerful map reduce query functionality

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,662
On SlideShare
0
From Embeds
0
Number of Embeds
224
Actions
Shares
0
Downloads
41
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Couchbase App Development: Documents, schemas and their relationships

  1. 1. Developing  With  Documents:   Schemas  and  Rela7onships Chris  Anderson Architect 1
  2. 2. www.couchbase.com/webinars 2
  3. 3. What  we’ll  talk  about• Working  with  JSON  documents• RunBme-­‐driven  paFerns  for • linking  between  documents • embedding  data • fetching  mulBple  documents• Map  reduce  indexing  paFerns 3
  4. 4. JSON  DOCUMENTS 4
  5. 5. Couchbase  Server  is  a  Document  Database hFp://marBnfowler.com/bliki/AggregateOrientedDatabase.html This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster. 5
  6. 6. Document  Database o::1001 { uid:  ji22jd, customer:  Ann, line_items:  [   {  sku:  0321293533,  quan:  3,    unit_price:  48.0  }, {  sku:  0321601912,  quan:  1,  unit_price:  39.0  }, {  sku:  0131495054,  quan:  1,  unit_price:  51.0  }   ], payment:  {  type:  Amex,  expiry:  04/2001,   last5:  12345  } } • Easy  to  distribute  data • Makes  sense  to  applica5on  programmers 6
  7. 7. JSON  Documents• Maps  more  closely  to  external  API• CRUD  Opera5ons,  lightweight  schema myDocument = { “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language.”, “status”: { “apis” : true, “databases” : “document” } }• Stored  under  a  unique  idenfier  key client.set(“mydocumentid”, myDocument); mySavedDocument = client.get(“mydocumentid”); 7
  8. 8. Object  to  JSON  back  to  ObjectUser  Object u::jasdeep@couchbase.com string uid { set() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } User  Object u::jasdeep@couchbase.com string uid { get() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } 8
  9. 9. Object  to  JSON  back  to  ObjectUser  Object u::jasdeep@couchbase.com string uid { set() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } User  Object u::jasdeep@couchbase.com string uid { get() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } 8
  10. 10. Object  to  JSON  back  to  ObjectUser  Object u::jasdeep@couchbase.com string uid { set() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } User  Object u::jasdeep@couchbase.com string uid { get() “uid”:  123456, string firstname “firstname”:  “jasdeep”, string lastname “lastname”:  “Jaitla”, int age “age”:  22, “favorite_colors”:  [“blue”,  “black”], array favorite_colors “email”:  “jasdeep@couchbase.com” string email } 8
  11. 11. Meta  +  Document  Body{ Document      "brewery":  "New  Belgium  Brewing", user data,      "name":  "1554  Enlightened  Black  Ale", can be anything      "abv":  5.5,      "descripon":  "Born  of  a  flood...",      "category":  "Belgian  and  French  Ale",      "style":  "Other  Belgian-­‐Style  Ales",      "updated":  "2010-­‐07-­‐22  20:00:20"} unique ID “vintage” date format from an SQL dump{ >_<      "id"  :  "beer_Enlightened_Black_Ale”, Metadata          ... identifier,{ expiration, etc 9
  12. 12. No  More  ALTER  table• No  More  alter  table• More  producBve  developers!• Emergent  schema—the  next  session  is   about  views• This  session  is  about  working  directly  with   documents  from  interacBve  applicaBon   code  • Schema  driven  by  code 10
  13. 13. LIVE  DATA 11
  14. 14. RealBme  InteracBve  CRUD Couchbase#Server#2.0#Architecture# 8092$ 11211$ 11210$ Query#API# Memcapable##1.0# Memcapable##2.0# Moxi$ Query$Engine$ REST$management$API/Web$UI$ vBucket#state#and#replica?on#manager# Memcached$ Global#singleton#supervisor# Rebalance#orchestrator# Configura?on#manager# Node#health#monitor# Process#monitor# Heartbeat# Couchbase$EP$Engine$ storage#interface# $$$$$CouchStore$ h"p$ on#each#node# one#per#cluster# Persistence$Layer$ Erlang/OTP$ HTTP# Erlang#port#mapper# Distributed#Erlang# 8091$ 4369$ 21100$>$21199$ 14# • Requirement:  high-­‐performance  with  high-­‐ concurrency  and  dynamic  scale • Key/value  API  is  the  interacve  low-­‐latency  path 12
  15. 15. Couchbase  Basic  OperaBons• get  (key) –  Retrieve  a  document• set  (key,  value) –  Store  a  document,  overwrites  if  exists• add  (key,  value) –  Store  a  document,  error/excepBon  if  exists• replace  (key,  value) –  Store  a  document,  error/excepBon  if  doesn’t  exist• cas  (key,  value,  cas) –  Compare  and  swap,  mutate  document  only  if  it  hasn’t  changed   while  execuBng  this  operaBon 13
  16. 16. Couchbase  Basic  OperaBons  (cont’d) Atomic  Counters  are  a  special  structure  in  Couchbase,  they   are  executed  in  order  and  are  PosiBve  Integer  Values • set  (key,  value) –  Use  set  to  iniBalize  the  counter • cb.set(“my_counter”,  1) • incr  (key) –  Increase  an  atomic  counter  value,  default  by  1 • cb.incr(“my_counter”)  #  now  it’s  2 • decr  (key) –  Decrease  an  atomic  counter  value,  default  by  1 • cb.decr(“my_counter”)  #  now  it’s  1 14
  17. 17. Simple  Example  in  Ruby #  example.rb #  user.rb require  “./user.rb” require  “rubygems” require  “couchbase” u1  =  User.new({   :email  =>  “jasdeep@scalabl3.com”, class  User :name  =>  “Jasdeep  Jaitla”, aFr_accessor    :name,  :email,  :Btle,  :twiFer :Btle  =>  “Scalability  Sherpa”, :twiFer  =>  “@scalabl3 def  ini7alize(aFr  =  {}) })              aFr.each  do  |name,  value|              seFer  =  "#{name}=" u1.save              next  unless  respond_to?(seFer)              send(seFer,  value)              end end def  save C  =  Couchbase.bucket C.  set(@email.downcase,  self.to_json) end end 15
  18. 18. RunBme  Driven  Schema • What’s  in  the  database  looks  more  like  your  code • Thinking  about  throughput,  latency,  update  and  read  paFerns  is  the   new  data  modeling • Data  flows  get  more  aFenBon  than  data  at  rest • Performance  Oriented  Architecture  hFp://jchrisa.net/drl/_design/ sofa/_list/post/post-­‐page?startkey=%5B%22Performance-­‐Oriented-­‐ Architecture%22%5D • When  should  I  split  a  data-­‐structure  into  mulBple  documents? • Generally  the  more  useful  your  document  is  as  a  standalone  enBty,   the  beFer. • Documents  that  grow  without  bound  are  bad 16
  19. 19. LINK  BETWEEN  DOCUMENTS 17
  20. 20. Let’s  Add  Comments  and  RaBngs  to  the  Beer • Challenge  linking  items  together • Whether  to  grow  an  exis5ng  item  or  store   independent  documents • No  transac5onality  between  documents! good w/ burgers{      "brewery":  "New  Belgium  Brewing", I give that a 5!      "name":  "1554  Enlightened  Black  Ale",      "abv":  5.5,      "descripon":  "Born  of  a  flood...", tastes like      "category":  "Belgian  and  French  Ale", college!      "style":  "Other  Belgian-­‐Style  Ales",      "updated":  "2010-­‐07-­‐22  20:00:20"}
  21. 21. Let’s  Add  Comments  and  RaBngs  to  the  Beer • We’ll  put  comments  in  their  own  document • And  add  the  ra5ngs  to  the  beer  document  itself. {{      "brewery":  "New  Belg      "type":  "comment",      "name":  "1554  Enligh      "about_id":  "beer_Enlightened_Black_Ale",      "abv":  5.5,      "user_id":  525,      "descripon":  "Born  o      "text":  "tastes  like  college!",      "category":  "Belgian  a      "updated":  "2010-­‐07-­‐22  20:00:20"      "style":  "Other  Belgia}      "updated":  "2010-­‐07-­‐    “rangs”  :  {        “525”  :  5, I give that a 5!        “30”  :  4,        “1044”  :  2  },
  22. 22. { {      "brewery":  "New  Belgium  Brewing",      "type":  "user",      "name":  "1554  Enlightened  Black  Ale",      "user_id":  525,      "abv":  5.5,      "name":  "Chris",      "descrip7on":  "Born  of  a  flood...",      "email":  "jchris@couchbase.com"      "category":  "Belgian  and  French  Ale",      "style":  "Other  Belgian-­‐Style  Ales",}      "updated":  "2010-­‐07-­‐22  20:00:20",    “ra7ngs”  :  { I give tha        “525”  :  5, t a 5!        “30”  :  4,{        “1044”  :  2      "type":  "comment",  },      "about_id":  "beer_Enlightened_Black_Ale",    “comments”  :  [      "user_id":  525,          “f1e62”,      "text":  "tastes  like  college!",          “6ad8c”      "updated":  "2010-­‐07-­‐22  20:00:20"      ]} }
  23. 23. Do  it:  save  the  comment  document • Set  at  the  id  “f1e62” client.set(“f1e62”,{      "type":  "comment", create a new document      "about_id":  "beer_Enlightened_Black_Ale",      "user_id":  525,      "text":  "tastes  like  college!",      "updated":  "2010-­‐07-­‐22  20:00:20" { });      "id":  "f1e62" } 21
  24. 24. { Link  between  comments  and  beers      "brewery":  "New  Belgium        "name":  "1554  Enlightene      "abv":  5.5, {      "descripon":  "Born  of  a  fl      "type":  "comment",      "category":  "Belgian  and  F      "about_id":  "beer_Enlightened_Black_Ale",link to      "style":  "Other  Belgian-­‐Sty beer      "user_id":  525,      "updated":  "2010-­‐07-­‐22  2      "text":  "tastes  like  college!",    “rangs”  :  {      "updated":  "2010-­‐07-­‐22  20:00:20"        “525”  :  5, }        “30”  :  4, {        “1044”  :  2      "id":  "f1e62"  }, }    “comment_ids”  :  [          “f1e62”, link to          “6ad8c” comments      ] }
  25. 25. How  to:  look  up  comments  from  a  beer • SERIALIZED  LOOP   beer = client.get(“beer:A_cold_one”); beer.comment_ids.each { |id| comments.push(client.get(id)); }• FAST  MULTI-­‐KEY  LOOKUP beer = client.get(“beer:A_cold_one”); comments = client.multiGet(beer.comment_ids) • ASYNC  VIEW  QUERY comments = client.query(“myapp”,“by_comment_on”, {:key => “beer:A_cold_one”}); figure  hFp://www.ibm.com/developerworks/webservices/library/ws-­‐sdoarch/ 23
  26. 26. How  to:  add  a  raBng  to  a  beer • Other  users  are  raBngs  beers  also,  so  we  use  a  CAS  update – we  don’t  want  to  accidentally  overwrite  another  users  raBng  that  is  being  saved  at  the   same  Bme  as  ours • Best  pracBce  is  to  use  a  lambda  so  the  client  can  retry cb.cas("mykey") do |doc| doc["ratings"][current_user.id] = my_rating doc end Actor  1 Actor  2 Success CAS  mismatch   &  retry Couchbase  Server 24
  27. 27. Object  Graph  With  Shared  InteracBve  Updates • Challenge:  higher  level  data  structures • Objects  shared  across  mul5ple  users • Mixed  object  sets  (upda5ng  some  private  and  some   shared  objects) figure  hFp://www.ibm.com/developerworks/webservices/library/ws-­‐sdoarch/ 25
  28. 28. Get  With  Lock  (GETL) • Ozen  referred  to  as  “GETL” • PessimisBc  concurrency  control • Locks  have  a  short  TTL • Locks  released  with  CAS  operaBons • Useful  when  working  with  object  graphs
  29. 29. Developing  with  Views:See  Inside  the  Data J  Chris  Anderson Architect 27
  30. 30. What  we’ll  talk  about• Lifecycle  of  a  view • Index  definiBon,  build,  and  query  phase • Consistency  opBons  (async  by  default)• Emergent  Schema  -­‐  Views  and  Documents• PaFerns: • Secondary  index • Basic  aggregaBons  (avg  raBngs  by  brewery) • Time-­‐based  analyBcs  with  group_level • Leaderboard 28
  31. 31. VIEW  LIFECYCLE:DEFINE  -­‐  BUILD  -­‐  QUERY 29
  32. 32. View  DefiniBon  (in  JavaScript) like: CREATE INDEX city ON brewery city; 30
  33. 33. Distributed  Index  Build  Phase • OpBmized  for  lookups,  in-­‐order  access  and  aggregaBons • All  view  reads  from  disk  (different  performance  profile) • View  builds  against  every  document  on  every  node – This  is  why  you  should  group  them  in  a  design  document • AutomaBcally  kept  up  to  date SERVER  1 SERVER  2 SERVER  3 AcBve  Docs AcBve  Docs AcBve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC 31
  34. 34. Dynamic  Range  Queries  with  OpBonal  AggregaBon• Efficiently  fetch  an  row  or  group  of  related  rows.• Queries  use  cached  values  from  B-­‐tree  inner  nodes  when  possible• Take  advantage  of  in-­‐order  tree  traversal  with  group_level  queries ?startkey=“J”&endkey=“K” {“rows”:[{“key”:“Juneau”,“value”:null}]} SERVER  1 SERVER  2 SERVER  3 AcBve  Docs AcBve  Docs AcBve  Docs Doc  5 DOC Doc  4 DOC Doc  1 DOC Doc  2 DOC Doc  7 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  6 DOC Replica  Docs Replica  Docs Replica  Docs Doc  4 DOC Doc  6 DOC Doc  7 DOC Doc  1 DOC Doc  3 DOC Doc  9 DOC Doc  8 DOC Doc  2 DOC Doc  5 DOC 3
  35. 35. EMERGENT  SCHEMA 33
  36. 36. Emergent  Schema • Falls  out  of  your  key-­‐value  usage • Helps  to  know  whats  efficient • Mostly  you  can  relax"Capture  the  users  intent" JSON.org Github  API Twioer  API 34
  37. 37. QUERY  PATTERN:FIND  BY  ATTRIBUTE 3
  38. 38. Find  documents  by  a  specific  aFribute • Lets  find  beers  by  brewery_id! 36
  39. 39. The  index  definiBon 37
  40. 40. The  result  set:  beers  keyed  by  brewery_id 38
  41. 41. QUERY  PATTERN:BASIC  AGGREGATIONS 3
  42. 42. Use  a  built-­‐in  reduce  funcBon  with  a  group  query • Lets  find  average  abv  for  each  brewery! 40
  43. 43. We  are  reducing  doc.abv  with  _stats 41
  44. 44. Group  reduce  (reduce  by  unique  key) 42
  45. 45. QUERY  PATTERN:TIME-­‐BASED  ROLLUPS 4
  46. 46. Find  paFerns  in  beer  comments  by  Bme {      "type":  "comment",      "about_id":  "beer_Enlightened_Black_Ale",      "user_id":  525,      "text":  "tastes  like  college!", 7mestamp      "updated":  "2010-­‐07-­‐22  20:00:20" } {      "id":  "f1e62" }
  47. 47. Query  with  group_level=2  to  get  monthly  rollups 45
  48. 48. dateToArray()  is  your  friend () rr ay T oA te da• String  or  Integer  based  Bmestamps• Output  opBmized  for  group_level  queries• array  of  JSON  numbers:   [2012,9,21,11,30,44] 46
  49. 49. group_level=2  results• Monthly  rollup• Sorted  by  5me—sort  the  query  results  in  your  applica4on   if  you  want  to  rank  by  value—no  chained  map-­‐reduce 47
  50. 50. group_level=3  -­‐  daily  results  -­‐  great  for  graphing• Daily,  hourly,  minute  or  second  rollup  all  possible  with   the  same  index.• hp://crate.im/posts/couchbase-­‐views-­‐reddit-­‐data/ 48
  51. 51. QUERY  PATTERN: LEADERBOARD 4
  52. 52. Aggregate  value  stored  in  a  document • Lets  find  the  top-­‐rated  beers! {      "brewery":  "New  Belgium  Brewing",      "name":  "1554  Enlightened  Black  Ale",      "abv":  5.5,      "descripon":  "Born  of  a  flood...",      "category":  "Belgian  and  French  Ale",      "style":  "Other  Belgian-­‐Style  Ales",      "updated":  "2010-­‐07-­‐22  20:00:20",    “rangs”  :  {        “jchris”  :  5, ra7ngs        “scalabl3”  :  4,        “damienkatz”  :  1  }, 50    “comments”  :  [
  53. 53. Sort  each  beer  by  its  average  raBng • Lets  find  the  top-­‐rated  beers! average 51
  54. 54. WHAT  NOT  TO  WRITE 5
  55. 55. Most  common  mistakes• Reduces  that  don’t  reduce• Trying  to  do  too  many  things  with  one  view• Emi€ng  too  much  data  into  a  view  value• ExpecBng  view  query  performance  to  be  as  fast  as  get/set• Recursive  queries  require  applicaBon  code. 53
  56. 56. GEOGRAPHIC  INDEX 5
  57. 57. Experimental  Status• Not  yet  using  Superstar  trees   • (only  fast  on  large  clusters)• OpBmized  for  bulk  loading 5
  58. 58. FULL  TEXT  INDEX 5
  59. 59. ElasBc  Search  Adapter • ElasBc  Search  is  good  for  ad-­‐hoc  queries  and  faceted  browsing • Our  adapter  is  aware  of  changing  Couchbase  topology • Indexed  by  ElasBc  Search  azer  stored  to  disk  in  Couchbase Elas7cSearch 57
  60. 60. THANK  YOU!HTTP://WWW.COUCHBASE.COM @JCHRIS 5

×