Webinar   Couchbase 104 - Views and Indexing
 

Webinar Couchbase 104 - Views and Indexing

on

  • 1,639 views

Learn the architecture and use of Views, the structure of Map-Reduce functions, design documents, querying views and view query parameters, primary aggregate reduces and grouping, eventual consistency ...

Learn the architecture and use of Views, the structure of Map-Reduce functions, design documents, querying views and view query parameters, primary aggregate reduces and grouping, eventual consistency of indexes and strategies of use.

What will be covered during this training:

What are Indexes
What is a Map-Reduce
Understanding Design Documents
Admin Console Overview
Anatomy of Map Functions
Batch Processing
Range Querying, Index-Key Querying, Set Querying
RDBMS Queries vs. Map-Reduce Queries
Grouping and Group Level
Eventual Consistency and Stale Parameter
Tips for Creating Views and Sandboxing Tests

Statistics

Views

Total Views
1,639
Views on SlideShare
749
Embed Views
890

Actions

Likes
2
Downloads
81
Comments
1

2 Embeds 890

http://www.couchbase.com 878
http://ofisconf.saftas.com.tr 12

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • I'm working creating index using CouchDB for multidimensional objects (including Documents in space). Comments and suggestions will be appreciated. The idea is to generate and store a document ID for retrieval and range queries.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Webinar   Couchbase 104 - Views and Indexing Webinar Couchbase 104 - Views and Indexing Presentation Transcript

  • Couchbase  104:  Views  and  Indexing Jasdeep  Jaitla Technical  Evangelist email:  jasdeep@couchbase.com twi0er:  @scalabl3
  • WHAT  IS  A  VIEW?
  • Views are Indexes • Indexes are methodologies to speed up access to information • Examples: - Dewey Decimal System - Card Catalogs - Hierarchal File Folders • In databases, Indexes are specialized structures for searching for data, typically one or two key fields
  • Indexing Subsystem • Storing data and Indexing data are separate systems in all databases • In explicit schema scenarios (RDBMS), Indexes are optimized based on the data type(s) • In flexible schema scenarios Map-Reduce is used to create indexes
  • What is Map-Reduce? • Map-Reduce is a technique designed for dealing with Big Data and processing in parallel in distributed systems • Map-Reduce is also specifically designed for dealing with unstructured or semi-structured data • Map functions identify data with collections, process them, and output transformed values • Reduce functions take the output of Map functions and perform numeric aggregate calculations on them
  • Views: Map-Reduce Indexes • In Couchbase, Map-Reduce is specifically used to create Indexes • Map functions are applied to JSON documents and they output or "emit" data that is organized in an Index form emit() CRUD Operations MAP() (processed)
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } }
  • Sample View function (doc, meta) { if (doc.type == “beer” && doc.brewery_id && doc.name) { emit(doc.name, doc.abv); } } • Creates an Index of Beer Names (doc.name) and the Alcohol By Volume values (doc.abv) - Filters Documents • Only JSON Documents with json key doc.type == "beer" • and doc.brewery_id is non-null • and doc.name is non-null - Outputs • Beer Name (doc.name) [searchable] • Beer Alcohol By Volume (doc.abv) [row value]
  • ARCHITECTURE
  • Storage to Index Couchbase Server RAM Cache EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Cluster Machine View Engine Indexers
  • Storage to Index Couchbase Server RAM Cache storage ops EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Cluster Machine View Engine Indexers
  • Views: Eventual Consistency Couchbase Server RAM Cache EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Cluster Machine View Engine Indexers
  • Views: Eventual Consistency Time 1 RAM Cache storage ops Couchbase Server EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Cluster Machine View Engine Indexers
  • Views: Eventual Consistency Time 1 RAM Cache get Couchbase Server EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Cluster Machine View Engine Indexers
  • Views: Eventual Consistency Time 1 RAM Cache get Couchbase Server EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Replica Couchbase Cluster Machine Time 2
  • Why  Use  Map-­‐Reduce  Indexes? • Index  (Find)  Documents  by  different  JSON  Values     • Query  Documents  by  JSON  Values     • Create  StaXsXcs  and  Aggregates   When  are  Indexes  Necessary? • Documents  are  Keyed  by  Random  ProperXes  (UUID,  GUID,  etc.)   • IteraXng  through  Lists  of  Documents  with  Random  Keys   • IteraXng  through  Lists  of  Documents  on  different  JSON  ProperXes   (i.e.  all  User  docs,  all  Product  docs,  by  Timestamp,  etc.)  
  • ANATOMY  OF  A  VIEW
  • Buckets  >>  Design  Documents  >>  Views Couchbase Bucket
  • Buckets  >>  Design  Documents  >>  Views Couchbase Bucket Design Document 1 View View View Design Document 2 View View
  • Buckets  >>  Design  Documents  >>  Views Couchbase Bucket Indexers Are Allocated Per Design Doc Design Document 1 View View View Design Document 2 Can Only Access Data in the Bucket Namespace All Updated at Same Time View View
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map function(doc,  meta)  {   emit(doc.username,  doc.email)   }
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map json doc function(doc,  meta)  {   emit(doc.username,  doc.email)   }
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map json doc doc metadata function(doc,  meta)  {   emit(doc.username,  doc.email)   }
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map json doc doc metadata function(doc,  meta)  {   emit(doc.username,  doc.email)   } create row
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map json doc doc metadata function(doc,  meta)  {   emit(doc.username,  doc.email)   } create row indexed key
  • Map()  FuncXon  =>  Index Every Document passes through View Map() functions Map json doc doc metadata function(doc,  meta)  {   emit(doc.username,  doc.email)   } create row indexed key output value(s)
  • Single  Element  Keys  (Text  Key) Map function(doc,  meta)  {   emit(doc.email,  doc.points)   }
  • Single  Element  Keys  (Text  Key) Map function(doc,  meta)  {   emit(doc.email,  doc.points)   } text key
  • Single  Element  Keys  (Text  Key) Map function(doc,  meta)  {   emit(doc.email,  doc.points)   } text key meta.id doc.email doc.points u::1 abba@couchbase.com 1000 u::35 jasdeep@couchbase.com 1200 u::20 zorro@couchbase.com 900
  • Compound  Keys  (Array) Array Based Index Keys get sorted as Strings, but can be grouped by array elements Map function(doc,  meta)  {   emit(dateToArray(doc.timestamp),  1)   }
  • Compound  Keys  (Array) Array Based Index Keys get sorted as Strings, but can be grouped by array elements Map function(doc,  meta)  {   emit(dateToArray(doc.timestamp),  1)   } array key
  • Compound  Keys  (Array) Array Based Index Keys get sorted as Strings, but can be grouped by array elements Map function(doc,  meta)  {   emit(dateToArray(doc.timestamp),  1)   } meta.id dateToArray(doc.3mestamp) value array key u::20 [2012,10,9,18,45] 1 u::1 [2012,9,26,11,15] 1 u::35 [2012,8,13,2,12] 1
  • 32 3
  • QUERYING  VIEWS 32 3
  • View Query Parameters •  key$=$“”$ ­  used%for%exact%match%of%index1key% •  keys$=$[]$ ­  used%for%matching%set%of%index1keys% •  startkey/endkey$=$“”$ ­  used%for%range%queries%on%index1keys% •  startkey_docID/endkey_docID$=$“”$ ­  used%for%range%queries%on%meta.id% •  stale=[false,$update_a;er,$true]$ ­  used%to%decide%indexer%behavior%from%client% •  group/group_by$ ­  used%with%reduces%to%aggregate%with%grouping%
  • Most  Common  Query’s  Are  Ranges doc.email meta.id abba@couchbase.com u::1 beta@couchbase.com u::7 jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Most  Common  Query’s  Are  Ranges doc.email meta.id abba@couchbase.com u::1 ?startkey=”b1”  &  endkey=”zZ” beta@couchbase.com u::7 Pulls  the  Index-­‐Keys   between  UTF-­‐8  Range   specified  by  the   startkey  and  endkey. jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Most  Common  Query’s  Are  Ranges doc.email meta.id abba@couchbase.com u::1 ?startkey=”bz”  &  endkey=”zn” beta@couchbase.com u::7 Pulls  the  Index-­‐Keys   between  UTF-­‐8  Range   specified  by  the   startkey  and  endkey. jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Most  Common  Query’s  Are  Ranges doc.email meta.id abba@couchbase.com u::1 beta@couchbase.com u::7 jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Index-­‐Key  Matching doc.email meta.id abba@couchbase.com u::1 beta@couchbase.com u::7 jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Index-­‐Key  Matching doc.email abba@couchbase.com Match  a  Single  Index-­‐Key u::1 beta@couchbase.com u::7 jasdeep@couchbase.com ?key=”math@couchbase.com”   meta.id u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Index-­‐Key  Set  Matches doc.email abba@couchbase.com ?keys=[“math@couchbase.com”,   “ye3@couchbase.com”] Query  MulXple  in  the   Set  (Array  NotaXon) meta.id u::1 beta@couchbase.com u::7 jasdeep@couchbase.com u::2 math@couchbase.com u::5 ma0@couchbase.com u::6 yeX@couchbase.com u::4 zorro@couchbase.com u::3
  • Understanding  CollaXon  Order Byte  Order 1234567890  <  a-­‐z  <  A-­‐Z Unicode  Colla3on 1234567890  <  aAbBcCdDeEfFgGhHiIjJkKlLmM... a < á < A < Á < b If  it  were  Byte  Order  2  Queries  Merged: startkey="y"&endkey="z"  merged  with  startkey="Y"&endkey="Z" With  Unicode  Colla3on  gets  both  y  and  Y: startkey="y"&endkey="z"
  • Understanding Stale stale  =  UPDATE_AFTER  (default  if  nothing  is  specified)   always  get  fastest  response   can  take  two  queries  to  read  your  own  writes   stale  =  OK   auto  update  will  trigger  eventually   might  not  see  your  own  writes  for  a  few  minutes   least  frequent  updates  -­‐>  least  resource  impact   stale  =  FALSE   Use  with  Persistence  observe  if  data  needs  to  be  included  in  view  results   BUT  aware  of  delay  it  adds,  only  use  when  really  required
  • Built-In Reduces • Are faster than creating your own reduces for the same information - _count • gives count for number of items in Index - _sum • sums value parameters (for numeric values only) - _stats • gives sum, count, min, max and sum of squares for statistics ! !
  • Custom Reduces • Are a bit tricky at first, it's a skill! • Learn about it through our docs, practice first, most common problem in custom reduces is that they don't "reduce" the data • Can be creatively used! • Always do it in a separate Design Document to sandbox it from your existing Views, if you have a logic problem or error it won't interrupt existing Views
  • 32 3
  • BEER  SAMPLE  VIEW 32 3
  • Beer  Sample  Database  Example meta {! } doc {! "name": "Aventinus Weizenstarkbier / Doppel "id": "110f37fa30",! "rev": "1-000000000",! Weizen Bock",! "abv": 8.2,! "expiration": 0,! "ibu": 0,! "flags": 0,! "srm": 0,! "type": "json"! "upc": 0,! "type": "beer",! "brewery_id": "110f1f2012",! "updated": "2010-07-22 20:00:20",! "description": "Dark-ruby, almost black-colored and streaked with fine top-fermenting yeast, this beer has a compact and persistent head. This is a very intense wheat doppelbock with a complex spicy chocolate-like arome with a hint of banana and raisins. On the palate, you experience a soft touch and on the tongue it is very rich and complex, though fresh with a hint of caramel. It finishes in a rich soft and lightly bitter impression.",! "style": "South German-Style Weizenbock",! "category": "German Ale"! }
  • Beer  Sample  Database  Example meta {! } doc {! alcohol by volume (abv) "name": "Aventinus Weizenstarkbier / Doppel "id": "110f37fa30",! "rev": "1-000000000",! Weizen Bock",! "abv": 8.2,! "expiration": 0,! document key "ibu": 0,! "flags": 0,! "srm": 0,! brewery_id (key) "type": "json"! "upc": 0,! "type": "beer",! "brewery_id": "110f1f2012",! "updated": "2010-07-22 20:00:20",! "description": "Dark-ruby, almost black-colored and streaked with fine top-fermenting yeast, this beer has a compact and persistent head. This is a very intense wheat doppelbock with a complex spicy chocolate-like arome with a hint of banana and raisins. On the palate, you experience a soft touch and on the tongue it is very rich and complex, though fresh with a hint of caramel. It finishes in a rich soft and lightly bitter impression.",! "style": "South German-Style Weizenbock",! "category": "German Ale"! }
  • Map  FuncXon  -­‐  Index  DefiniXon 30
  • Map  FuncXon  -­‐  Index  DefiniXon +row 30
  • Map  FuncXon  -­‐  Index  DefiniXon +row indexed key 30
  • Map  FuncXon  -­‐  Index  DefiniXon +row indexed key value(s) 30
  • Result  Set  -­‐  Brewery  ID’s  by  Beer 31
  • Result  Set  -­‐  Brewery  ID’s  by  Beer brewery_id alcohol by volume (abv) document key (of the beer) 31
  • Reduce  Values  (doc.abv)  with  _stats 34 34
  • Reduce  Values  (doc.abv)  with  _stats add _stats built-in reduction 34 34
  • Query  with  Group  and  Reduce Find average alcohol by volume per brewery. 33
  • Query  with  Group  and  Reduce Find average alcohol by volume per brewery. add _stats built-in reduction set group=true & reduce=true 33
  • Groups  Brewery_ID’s,  Reduces  for  Stats Brewery ID’s are Grouped, and _stats collected (Reduced) 35 35
  • Groups  Brewery_ID’s,  Reduces  for  Stats group=true & reduce=true number of beers by this brewery min abv max abv Brewery ID’s are Grouped, and _stats collected (Reduced) 35 35
  • INTERFACE  DEMO
  • Q  &  A