Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CouchConf-Berlin-Querying-with-Couchbase

447 views

Published on

Published in: Technology
  • Be the first to comment

CouchConf-Berlin-Querying-with-Couchbase

  1. 1. 1
  2. 2. Introduction to MapReduce 2
  3. 3. A View Is: 3
  4. 4. A View Is:A named pair of functions: a map function 3
  5. 5. A View Is:A named pair of functions: a map function a reduce function (optional) 3
  6. 6. A View Is:A named pair of functions: a map function a reduce function (optional)An entry in a design document 3
  7. 7. A View Is:A named pair of functions: a map function a reduce function (optional)An entry in a design documentA disk file of indexed results 3
  8. 8. A Map Function Is: 4
  9. 9. A Map Function Is:Called with every database document 4
  10. 10. A Map Function Is:Called with every database documentAn emitter of a key and a value 4
  11. 11. A Reduce Function Is: 5
  12. 12. A Reduce Function Is:Called once with the map results 5
  13. 13. A Reduce Function Is:Called once with the map resultsA simplifier (it reduces map output) 5
  14. 14. Example "President" Document (1 of 44) 6
  15. 15. Example "President" Document (1 of 44){ "_id":"5d5f25254ef8fd62d6b9f2db642a8fc2", "_rev": "1-157b2928bec2def71485cc751af7de37", "type":"president", "presidency":1, "name":"George Washington", "wikipedia_entry":"http://en.wikipedia.org/wiki/George_Washington", "took_office":1789, "left_office":1797, "party":"Independent", "home_state":"Virginia"} 6
  16. 16. Example "Event" Document (1 of 883) 7
  17. 17. Example "Event" Document (1 of 883){ "_id":"5d5f25254ef8fd62d6b9f2db642a9f7d", "_rev": "1-9630b35932dedbd4d31138aaf3385847", "type":"event", "year":1791, "event":"The independent Vermont Republic becomes the 14th state"} 7
  18. 18. Design Document{... "_id":"_design/design_document", "_rev": "1-9630b35932dedbd4d31138aaf3385847", "views": { "party_state_name": { "map":"function ... ", "reduce": " ... "}, "president_events": { "map":"function ... "}, "president_names": { "map":"function ... "}, "presidents": { "map":"function ... ", "reduce": " ... "}, "time_in_office": { "map":"function .... ", "reduce": " ... "}, "total_time_in_office": { "map":"function .... ", "reduce": " ... "} }, ...} 8
  19. 19. Design Document special{... "_id":"_design/design_document", "_rev": "1-9630b35932dedbd4d31138aaf3385847", "views": { "party_state_name": { "map":"function ... ", "reduce": " ... "}, "president_events": { "map":"function ... "}, "president_names": { "map":"function ... "}, "presidents": { "map":"function ... ", "reduce": " ... "}, "time_in_office": { "map":"function .... ", "reduce": " ... "}, "total_time_in_office": { "map":"function .... ", "reduce": " ... "} }, ...} 8
  20. 20. Design Document special you choose{... "_id":"_design/design_document", "_rev": "1-9630b35932dedbd4d31138aaf3385847", "views": { "party_state_name": { "map":"function ... ", "reduce": " ... "}, "president_events": { "map":"function ... "}, "president_names": { "map":"function ... "}, "presidents": { "map":"function ... ", "reduce": " ... "}, "time_in_office": { "map":"function .... ", "reduce": " ... "}, "total_time_in_office": { "map":"function .... ", "reduce": " ... "} }, ...} 8
  21. 21. president_names 9
  22. 22. Invoke a View 10
  23. 23. Invoke a Viewcurl -X GEThttp://localhost:5984/presidents/_design/design_doc/_view/president_names 11
  24. 24. Invoke a Viewcurl -X GEThttp://localhost:5984/presidents/_design/design_doc/_view/president_names{"total_rows":44,"offset":0,"rows":[ {"id":"...","key":1789,"value":"George Washington"}, {"id":"...","key":1797,"value":"John Adams"}, {"id":"...","key":1801,"value":"Thomas Jefferson"}, {"id":"...","key":1809,"value":"James Madison"}, ...]} emitting document ids are always included 11
  25. 25. Invoke a View 12
  26. 26. Invoke a View 12
  27. 27. Under the Hood: Views CouchDBSpidermonkey Disk ICU 13
  28. 28. Under the Hood: Viewshttp://localhost:5984/presidents/_design/design_doc/_view/president_names CouchDB Spidermonkey Disk ICU 13
  29. 29. Under the Hood: Views Erlang HTTPhttp://localhost:5984/presidents/_design/design_doc/_view/president_names CouchDB Spidermonkey Disk ICU 13
  30. 30. Under the Hood: Views Erlang HTTP mod_couchhttp://localhost:5984/presidents/_design/design_doc/_view/president_names CouchDB Spidermonkey Disk ICU 13
  31. 31. Under the Hood: Views Erlang HTTP mod_couch query server CouchDB storage enginehost:5984/presidents/_design/design_doc/_view/president_names Spidermonkey Disk ICU 13
  32. 32. Under the Hood: Views{"total_rows":44, "offset":0, "rows":[...]} Erlang HTTP mod_couch query server CouchDB storage engine view Spidermonkey Disk ICU 13
  33. 33. Under the Hood: Views{"total_rows":44, "offset":0, "rows":[...]} CouchDB Spidermonkey Disk ICU 14
  34. 34. Under the Hood: Views{"total_rows":44, "offset":0, "rows":[...]} {"id":"...","key":1789,"value":"George Washington"}, {"id":"...","key":1797,"value":"John Adams"}, ... CouchDB Spidermonkey Disk ICU 14
  35. 35. Fetch Documents Matching a KeyGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?key=1993 any valid JSON 15
  36. 36. Fetch Documents Matching a KeyGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?key=1993 total view rows offset into rows{"total_rows":44,"offset":41,"rows":[! ! {"id":"...","key":1993,"value":"Bill Clinton"}]} matching key 15
  37. 37. Invoke a View 16
  38. 38. Invoke a View 16
  39. 39. Get a Key Range of DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?startkey=1790&endkey=1810 17
  40. 40. Get a Key Range of DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?startkey=1790&endkey=1810{"total_rows":44,"offset":1,"rows":[! {"id":"...","key":1797,"value":"John Adams"},! {"id":"...","key":1801,"value":"Thomas Jefferson"},! {"id":"...","key":1809,"value":"James Madison"}]} 17
  41. 41. Invoke a View 18
  42. 42. Invoke a View 18
  43. 43. Limit the Number of DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=2 19
  44. 44. Limit the Number of DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=2{"total_rows":44,"offset":0,"rows":[! {"id":"...","key":1789,"value":"George Washington"},! {"id":"...","key":1797,"value":"John Adams"},]} 19
  45. 45. Invoke a View 20
  46. 46. Invoke a View 20
  47. 47. presidents (_count) 21
  48. 48. Invoke a View 22
  49. 49. Reduce: _count{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { "presidents": { "map":"function(doc) { ! if(doc.type == president) { ! ! emit(doc.took_office, doc) }}", "reduce":"_count" } }, ...} 23
  50. 50. _count FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/presidents 24
  51. 51. _count FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/presidents{"rows":[! {"key":null,"value":44}]} 24
  52. 52. Invoke a View 25
  53. 53. Invoke a View 25
  54. 54. total_time_in_office (_sum) 26
  55. 55. Invoke a View 27
  56. 56. A Map Function for _sum{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { total_time_in_office": { "map":"function(doc) { ! ! if(doc.type == president) { ! ! ! emit(doc.name, doc.left_office - doc.took_office) ! }}", "reduce":"_sum" } }, ...} 28
  57. 57. A Map Function for _sum{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { total_time_in_office": { value is number of years in office "map":"function(doc) { ! ! if(doc.type == president) { ! ! ! emit(doc.name, doc.left_office - doc.took_office) ! }}", "reduce":"_sum" } }, will be sorted by name ...} 28
  58. 58. Reduce: _sum{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { total_time_in_office": { "map":"function(doc) { ! ! if(doc.type == president) { ! ! ! emit(doc.name, doc.left_office - doc.took_office) ! }}", "reduce":"_sum" } }, ...} 29
  59. 59. Reduce: _sum{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { total_time_in_office": { _sum requires number values "map":"function(doc) { ! ! if(doc.type == president) { ! ! ! emit(doc.name, doc.left_office - doc.took_office) ! }}", "reduce":"_sum" } }, ...} 29
  60. 60. _sum FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/total_time_in_office 30
  61. 61. _sum FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/total_time_in_office{"rows":[! {"key":null,"value":232}]} 30
  62. 62. Invoke a View 31
  63. 63. Invoke a View 31
  64. 64. time_in_office (_stats) 32
  65. 65. Invoke a View 33
  66. 66. Reduce: _stats{ "_id":"_design/design_doc", "_rev": "1-157b2928bec2def71485cc751af7de37", "views": { "time_in_office": { "map":"function(doc) { ! ! if(doc.type == president) { ! ! ! emit(doc.name, doc.left_office - doc.took_office) ! }}", "reduce":"_stats" } }, ...} 34
  67. 67. _stats FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/time_in_office 35
  68. 68. _stats FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/time_in_office{"rows":[! {"key":null,! "value":{ ! "sum":232,"count":43,"min":0,"max":12,"sumsqr":1546 }]} 35
  69. 69. Invoke a View 36
  70. 70. Invoke a View 36
  71. 71. View Trees 37
  72. 72. Disk-Based View Tree k=size of interior node A-R n=number of keys A-H I-Rdepth= log k(n) A-C D-F G-H I-L N-R A B C D F G H I K L N O Q R leaves 38
  73. 73. Disk-Based View Tree root k=size of interior node A-R n=number of keys A-H interior nodes I-Rdepth= log k(n) A-C D-F G-H I-L N-R A B C D F G H I K L N O Q R leaves 38
  74. 74. _count Nodes root 14 7 reductions 7 3 2 2 3 4A B C D F G H I K L N O Q R keys 39
  75. 75. _count Nodes root A-R 14 A-H 7 reductions I-R 7 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L N O Q R keys 40
  76. 76. Inserting a New Document A-R 14 A-H I-R 7 7 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R 41
  77. 77. Inserting a New Document A-R 14 A-H I-R 7 7 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R new key 41
  78. 78. Inserting a New Document A-R 14 A-H I-R 7 7 M-R 5 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R new key 41
  79. 79. Inserting a New Document A-R 14 I-R 8 A-H I-R 7 7 M-R 5 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R new key 41
  80. 80. Inserting a New Document A-R 15 A-R 14 I-R 8 A-H I-R 7 7 M-R 5 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R new key 41
  81. 81. Inserting a New Document new root A-R 15 A-R 14 new reductions I-R 8 A-H I-R 7 7 M-R 5 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R new key 41
  82. 82. Committing the Change A-R 15 A-R 14 I-R 8 A-H I-R 7 7 M-R 5 A-C D-F G-H I-L N-R 3 2 2 3 4A B C D F G H I K L M N O Q R 42
  83. 83. Committing the Change A-R 15 14 A-H I-R 7 87 A-C D-F G-H I-L M-R N-R 3 2 2 3 5 4A B C D F G H I K L M N O Q R 42
  84. 84. Getting a Key Range A-R 14 A-H I-R 7 7 A-C D-F G-H I-L M-R 3 2 2 3 5A B C D F G H I K L M N O Q R startkey endkey 43
  85. 85. Key Range Reduction 15 7 8 3 2 2 3 5A B C D F G H I K L M N O Q R startkey endkey 44
  86. 86. Key Range Reduction 15 (8) (3) 7 8 (5) 3 (1) 2 2 3 5 (2)A B C D F G H I K L M N O Q R startkey endkey 44
  87. 87. More Ways to Use Views 45
  88. 88. Skip the Reduce FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/time_in_office?reduce=false 46
  89. 89. Skip the Reduce FunctionGET http://localhost:5984/presidents/_design/design_doc/_view/time_in_office?reduce=false{"total_rows":44,"offset":0,"rows":[ {"id":"...","key":"Abraham Lincoln","value":5}, {"id":"...","key":"Andrew Jackson","value":8}, {"id":"...","key":"Andrew Johnson","value":4}, ...]} 46
  90. 90. Invoke a View 47
  91. 91. Invoke a View 47
  92. 92. Reversing the Order of ResultsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true 48
  93. 93. Reversing the Order of ResultsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true{"total_rows":44,"offset":0,"rows":[ {"id":"...","key":2009,"value":"Barack Obama"}, {"id":"...","key":2001,"value":"George W. Bush"}, {"id":"...","key":1993,"value":"Bill Clinton"}, {"id":"...","key":1989,"value":"George H. W. Bush"}, ...]} 48
  94. 94. Invoke a View 49
  95. 95. Invoke a View 49
  96. 96. Reversing the Order of a RangeGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&startkey=1850&endkey=1790 50
  97. 97. Reversing the Order of a Range startkey and endkey are reversed, tooGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&startkey=1850&endkey=1790 50
  98. 98. Reversing the Order of a Range startkey and endkey are reversed, tooGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&startkey=1850&endkey=1790{"total_rows":44,"offset":0,"rows":[ {"id":"...","key":2009,"value":"Barack Obama"}, {"id":"...","key":2001,"value":"George W. Bush"}, {"id":"...","key":1993,"value":"Bill Clinton"}, {"id":"...","key":1989,"value":"George H. W. Bush"}, ...]} 50
  99. 99. Invoke a View 51
  100. 100. Invoke a View 51
  101. 101. Ignore a Given Number of RowsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=10&skip=1 52
  102. 102. Ignore a Given Number of RowsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=10&skip=1 avoid large values 52
  103. 103. Invoke a View 53
  104. 104. Invoke a View 53
  105. 105. Paginating (Initial Page)// first page of documentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=2 54
  106. 106. Paginating (Initial Page)// first page of documentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?limit=2{"total_rows":44,"offset":0,"rows":[! {"id":"...","key":1789,"value":"George Washington"},! {"id":"...","key":1797,"value":"John Adams"},]} last key of result 54
  107. 107. Paginating (Successive Pages) 55
  108. 108. Paginating (Successive Pages) last key of previous result// successive pagesGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?startkey=1797&skip=1&limit=2 dont include the first document 55
  109. 109. Paginating (Successive Pages)// successive pagesGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?startkey=1797&skip=1&limit=2{"total_rows":44,"offset":2,"rows":[! {"id":"...","key":1801,"value":"Thomas Jefferson"},! {"id":"...","key":1809,"value":"James Madison"},]} 55
  110. 110. Invoke a View 56
  111. 111. Invoke a View 56
  112. 112. Paginating in Reverse Order// first page of documentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&limit=2 57
  113. 113. Paginating in Reverse Order// first page of documentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&limit=2// successive pages last key of previous resultGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?descending=true&startkey=1797&skip=1&limit=2 dont include the first document 57
  114. 114. Invoke a View 58
  115. 115. Invoke a View 58
  116. 116. Using a Stale ViewGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?stale=ok 59
  117. 117. Invoke a View 60
  118. 118. Invoke a View 60
  119. 119. Invoke a View 61
  120. 120. Invoke a View 61
  121. 121. Invoke a View 62
  122. 122. Invoke a View 62
  123. 123. Invoke a View 63
  124. 124. Invoke a View 63
  125. 125. Updating the View Immediately AfterGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?stale=update_after 64
  126. 126. party_state_name(group & group_level) 65
  127. 127. Invoke a View 66
  128. 128. Group LevelMap Keys Group Level 1 Group Level 2["a",1,1] ["a"] ["a",1]["a",3,4] ["a",3]["a",3,8]["b",2,6] ["b"] ["b",2]["b",2,6]["c",1,5] ["c"] ["c",1]["c",4,2] ["c",4] 67
  129. 129. Group Level Map Keys Group Level 1 Group Level 2 ["a",1,1] ["a"] ["a",1] ["a",3,4] ["a",3] ["a",3,8] ["b",2,6] ["b"] ["b",2] ["b",2,6] ["c",1,5] ["c"] ["c",1] ["c",4,2] ["c",4] GET http://localhost:5984/my_db/_design/ddoc/_view/v1? group_level=2only applies to reduce views 67
  130. 130. Invoke a View 68
  131. 131. Group Level 69
  132. 132. Group Level 69
  133. 133. Group Level 70
  134. 134. Group Level 70
  135. 135. Group 71
  136. 136. Group GET http://localhost:5984/my_db/_design/greeting/_view/v1? group=true one output row for each unique keyequivalent to group_level=infinity 71
  137. 137. Group Level 72
  138. 138. Group Level 72
  139. 139. Including Full DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?include_docs=true 73
  140. 140. Including Full DocumentsGET http://localhost:5984/presidents/_design/design_doc/_view/president_names?include_docs=true{"total_rows":44,"offset":0,"rows":[{"id":"...","key":1789,"value":"George Washington","doc":{"_id":"...","_rev":"1-...","presidency":"1","wikipedia_entry":"http://en.wikipedia.org/...","took_office":1789,"left_office":1797,"party":"Independent","home_state":"Virginia","name":"George Washington","type":"president"}},! ,...]} 73
  141. 141. Group Level 74
  142. 142. Group Level 74
  143. 143. Emitting with include_docs=true 75
  144. 144. Emitting with include_docs=true includes the latest rev of the emitterfunction(doc) { emit("key", aValue) } 75
  145. 145. Emitting with include_docs=true includes the latest rev of the emitterfunction(doc) { emit("key", aValue) } includes this rev of the emitterfunction(doc) { emit("key",{"_rev":doc._rev; value:doc}) } 75
  146. 146. Emitting with include_docs=true includes the latest rev of the emitterfunction(doc) { emit("key", aValue) } includes this rev of the emitterfunction(doc) { emit("key",{"_rev":doc._rev; value:doc}) } includes document with id foofunction(doc) { emit("key",{"_id":"foo", "value":doc}) } 75
  147. 147. Requesting Specific Keys 76
  148. 148. Requesting Specific KeysPOST -H "Content-Type:application/json"http://localhost:5984/presidents/_design/design_doc/_view/president_names-d {"keys":[1789, 1929, 1993, ... ]} 76
  149. 149. Requesting Specific KeysPOST -H "Content-Type:application/json"http://localhost:5984/presidents/_design/design_doc/_view/president_names-d {"keys":[1789, 1929, 1993, ... ]}POST -H "Content-Type:application/json"http://localhost:5984/presidents/_design/design_doc/_view/president_names?include_docs=true-d {"keys":[1789, 1929, 1993, ... ]} 76
  150. 150. Group Level 77
  151. 151. Group Level 77
  152. 152. president_events (join) 78
  153. 153. Invoke a View 79
  154. 154. Collating Joins"views": { "president_events": "function(doc) { if (doc.type == president) { emit([doc.took_office], doc.name); } else if (doc.type == event) { emit([doc.year, 0], doc.event); } }"} 80
  155. 155. Collating Joins year took office"views": { "president_events": "function(doc) { if (doc.type == president) { emit([doc.took_office], doc.name); } else if (doc.type == event) { emit([doc.year, 0], doc.event); } }"} 80
  156. 156. Collating Joins year took office"views": { one-element array "president_events": "function(doc) { if (doc.type == president) { emit([doc.took_office], doc.name); } else if (doc.type == event) { emit([doc.year, 0], doc.event); } }"} 80
  157. 157. Collating Joins year took office"views": { one-element array "president_events": "function(doc) { if (doc.type == president) { emit([doc.took_office], doc.name); } else if (doc.type == event) { emit([doc.year, 0], doc.event); } }" year of event} 80
  158. 158. Collating Joins year took office"views": { one-element array "president_events": "function(doc) { if (doc.type == president) { emit([doc.took_office], doc.name); } else if (doc.type == event) { emit([doc.year, 0], doc.event); } }" year of event} second array element 80
  159. 159. Join Presidents and EventsGET http://localhost:5984/presidents/_design/design_doc/_view/president_events 81
  160. 160. Join Presidents and EventsGET http://localhost:5984/presidents/_design/design_doc/_view/president_events{"total_rows":883,"offset":0,"rows":[ {"id":"...","key":[1789],"value":"George Washington"}, {"id":"...","key":[1790,0],"value":"Rhode Island ratifies the Constitution and becomes 13th state"}, {"id":"...","key":[1791,0],"value":"Bill of Rights ratified"}, {"id":"...","key":[1791,0],"value":"First Bank of the United States chartered"}, ...]} 81
  161. 161. Group Level 82
  162. 162. Group Level 82
  163. 163. End 83

×