Couchbase 104:
Views and Indexing
Todd Greenstein
Solution Architect
[Agenda]
[1] What are Views
[2] Architecture in Couchbase
[3] Uses
[4] View Anatomy (Map/Reduce)
[5] Examples
Indexing Subsystem
• Storing data and Indexing data are separate
systems in all databases
• In explicit schema scenarios (...
Map Reduce
• Map-Reduce is a technique designed for dealing
with Big Data and processing in parallel in
distributed system...
Couchbase Map Reduce
• In Couchbase, Map-Reduce is specifically used
to create Indexes
• Map functions are applied to JSON...
[Architecture]
Storage to Index
Views: Eventual Consistency
Time 1
Time 2
Buckets >> Design Documents >> Views
Couchbase Bucket
Design Document 1 Design Document 2
View ViewViewViewView
Indexers A...
33 2
Eventually indexed Views – Data flow2
Managed Cache
DiskQueue
Disk
Replication
Queue
App Server
Couchbase Server Node...
Index building details
•All the views within a design document are incrementally updated
when the view is accessed or auto...
Automatic Index Updates
• Views can be updated automatically according to:
– Update Interval (time period); default 5000 m...
Manual Index Updates
• All queries support ‘stale’ parameter to view
• Allows application to control updates
– Has perform...
Understanding Stale
stale = UPDATE_AFTER (default if nothing is specified)
always get fastest response
can take two querie...
Replica Indexes
• In addition to data replicas, optionally create replica for indexes
– Build an index using the data in r...
[Uses]
Use Cases
• Iteration
• Retrieval (UUID/GUID)
• Ranges (Parameters)
• Aggregating (Built in Reduces)
• Statistical Analysis
[View Anatomy]
Map() Function => Index
function(doc, meta) {
emit(doc.username, doc.email)
} indexed key output value(s)create row
json d...
Single Element Keys (Text Key)
function(doc, meta) {
emit(doc.email, doc.points)
} text key
Map
met
a.id
doc.email
doc.po
...
Compound Keys (Array)
function(doc, meta) {
emit(dateToArray(doc.timestamp), 1)
} array key
Array Based Index Keys get sor...
Reduce Functions
• Optional
• Operate on View Value emitted by map function
• Built-in reduce
– _count – provides a count ...
View Query Parameters
[Examples]
Sample Inventory Bucket
{
"item":"KS-00114629 Dark Grey
Glasses",
"vendor":"kate spade",
"category":"glasses"
}
{
"item":"...
Couchbase 104 Webinar
Upcoming SlideShare
Loading in …5
×

Couchbase 104 Webinar

1,166 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,166
On SlideShare
0
From Embeds
0
Number of Embeds
409
Actions
Shares
0
Downloads
26
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • 1.  A set request comes in from the application .
    2.  Couchbase Server responses back that they key is written
    3. Couchbase Server then Replicates the data out to memory in the other nodes
    4. At the same time it is put the data into a write que to be persisted to disk
  • Couchbase 104 Webinar

    1. 1. Couchbase 104: Views and Indexing Todd Greenstein Solution Architect
    2. 2. [Agenda] [1] What are Views [2] Architecture in Couchbase [3] Uses [4] View Anatomy (Map/Reduce) [5] Examples
    3. 3. Indexing Subsystem • Storing data and Indexing data are separate systems in all databases • In explicit schema scenarios (RDBMS), Indexes are optimized based on the data type(s) • In flexible schema scenarios Map-Reduce is used to create indexes
    4. 4. Map Reduce • Map-Reduce is a technique designed for dealing with Big Data and processing in parallel in distributed systems • Map-Reduce is also specifically designed for dealing with unstructured or semi-structured data • Map functions identify data with collections, process them, and output transformed values • Reduce functions take the output of Map functions and perform numeric aggregate calculations on them
    5. 5. Couchbase Map Reduce • In Couchbase, Map-Reduce is specifically used to create Indexes • Map functions are applied to JSON documents and they output or "emit" data that is organized in an Index form
    6. 6. [Architecture]
    7. 7. Storage to Index
    8. 8. Views: Eventual Consistency Time 1 Time 2
    9. 9. Buckets >> Design Documents >> Views Couchbase Bucket Design Document 1 Design Document 2 View ViewViewViewView Indexers Are Allocated Per Design Doc All Updated at Same TimeAll Updated at Same Time All Updated at Same Time Can Only Access Data in the Bucket Namespace Can Only Access Data in the Bucket Namespace
    10. 10. 33 2 Eventually indexed Views – Data flow2 Managed Cache DiskQueue Disk Replication Queue App Server Couchbase Server Node Doc 1Doc 1 Doc 1 To other node View engine Doc 1
    11. 11. Index building details •All the views within a design document are incrementally updated when the view is accessed or auto-indexing kicks in •The entire view is recreated if the view definition has changed •The index information stored on disk consists of the combination of both the key and value information defined within your view.
    12. 12. Automatic Index Updates • Views can be updated automatically according to: – Update Interval (time period); default 5000 milliseconds – Update Documents (number of changed documents); default 5000 changes – Replica Update Interval for index replicas • Can be set globally • Can be set on individual design documents – Configured using REST API • Operate independently of manual updates • Only updates with eligible documents
    13. 13. Manual Index Updates • All queries support ‘stale’ parameter to view • Allows application to control updates – Has performance considerations • Update request for one view updates all views within corresponding design document • Can be triggered through UI or REST API as well • Only updates with eligible documents
    14. 14. Understanding Stale stale = UPDATE_AFTER (default if nothing is specified) always get fastest response can take two queries to read your own writes stale = OK auto update will trigger eventually might not see your own writes for a few minutes least frequent updates -> least resource impact stale = FALSE Use with Persistence observe if data needs to be included in view results BUT aware of delay it adds, only use when really required
    15. 15. Replica Indexes • In addition to data replicas, optionally create replica for indexes – Build an index using the data in replica vBuckets • Enabled per bucket (see bucket config) or per design document (REST API) • Implies additional CPU and I/O overhead – Each node must update index for active data and replica data • In the event of failover: – Without replica indexes, queries may be out of date for longer while rebuilt – Replica indexes enabled if present and queries remain consistent – Replicas indexes are rebuilt on replica nodes
    16. 16. [Uses]
    17. 17. Use Cases • Iteration • Retrieval (UUID/GUID) • Ranges (Parameters) • Aggregating (Built in Reduces) • Statistical Analysis
    18. 18. [View Anatomy]
    19. 19. Map() Function => Index function(doc, meta) { emit(doc.username, doc.email) } indexed key output value(s)create row json doc doc metadata Every Document passes through View Map() functions Map
    20. 20. Single Element Keys (Text Key) function(doc, meta) { emit(doc.email, doc.points) } text key Map met a.id doc.email doc.po ints u::1 abba@couchbase.co m 1000 u::3 5 jasdeep@couchbase. com 1200 u::2 0 zorro@couchbase.co m 900
    21. 21. Compound Keys (Array) function(doc, meta) { emit(dateToArray(doc.timestamp), 1) } array key Array Based Index Keys get sorted as Strings, but can be grouped by array elements Map meta. id dateToArray(doc.timest amp) valu e u::20 [2012,10,9,18,45] 1 u::1 [2012,9,26,11,15] 1 u::35 [2012,8,13,2,12] 1
    22. 22. Reduce Functions • Optional • Operate on View Value emitted by map function • Built-in reduce – _count – provides a count of unique keys – _sum – provides a sum total of values – _stats – provides statistics (max, min, avg, etc.) of values • Automatically re-reduces up B-tree • Results stored pre-computed for fast access • Custom reductions possible if needed but less performant
    23. 23. View Query Parameters
    24. 24. [Examples]
    25. 25. Sample Inventory Bucket { "item":"KS-00114629 Dark Grey Glasses", "vendor":"kate spade", "category":"glasses" } { "item":"IH-00884501 Triple ring Shoulder bag", "vendor":"iron heart", "category":"handbag", ”discount":"15" }

    ×