Your SlideShare is downloading. ×
0

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Couchbase TLV Dev track 04 - power techniques with indexing

736

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
736
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
32
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Built on the Javascript V8 engine. Our query language is simple Javascript, so very easy to write our map functions.
  • How we can write our map functions: We could use a Single Element key, or “Primary key.”
  • Group_level queriesWe can use the built in dateToArray Javascript helper function to find a rollup of i.e. documents edited by date (by year, month, day, hour etc.)Group_level=2 we'd segment by year,month   and    Group_level=3  we'd segment by year,month,day etc.
  • Per data bucket, we have multiple Design Docs which contain the view definitions for a number of views.  This means our views are all batched together to be incrementally updated.  Best practise is splitting our views up into relevant ownerships / writers.  So i.e. 1 Design Document holds all the views for the Frontend UI of the website, and another Design Document holds the views for the Backend Admin interface (used to list and edit users, or posts etc etc.)In a worst case Design Doc scenario, there would be a 1 view in a dozen design documents, meaning we have 12 view functions to run, whereas we should structure it as multiple views per design document.  But, getting the balance right is important, as we also wouldn't want to have a design document with 100 views in it!When we change 1 view definition, it will update the index for the ENTIRE design doc, this is why it's logical to split views into relevant Design Doc categories etc.
  • First, walk through the optionsThen mention Observe
  • Transcript

    • 1. Developing with Couchbase: Power Techniques with Indexing Michael Nitschinger Engineer, Developer Solutions
    • 2. Agenda • Introduction to Indexing and Querying in Couchbase • Understand Map/Reduce Basics • Architectural Overview • Simple Indexes • Simple Queries
    • 3. Indexing and Querying
    • 4. Views are Indexes Indexes help to speed up access to data Doc2 Doc3 Doc1 Index Doc1 Doc3 Doc4 Doc2 Doc5
    • 5. Couchbase Server 2.0: Views • Storing and Indexing Data are separate processes • In RDBMS, Indexes are optimized based on fixed data types. • Map-Reduce is a flexible approach helping to Index unstructured data.
    • 6. Map-Reduce in General • The map function locates data items and outputs optimized data structures • The reduce function aggregates the output from a map function. • Together: very good for semi-structured and distributed data. Map Output Map Output Reduce Map Output Map Output
    • 7. Couchbase Server Map-Reduce In Couchbase, Map-Reduce is specifically used to create an Index. Map functions are applied to JSON Documents and they output or “emit” a data structure designed to be rapidly queried and traversed. emit() CRUD Operations MAP() (processed)
    • 8. Couchbase Server Views • Create a View of beer names • Filter only Documents with a JSON key type == beer and also has JSON keys brewery_id and name • Output the beer name, and a Alcohol By Volume (ABV) value
    • 9. Couchbase Server Views • Views can cover a few different use cases - Simple secondary indexes (the most common) - Complex secondary, tertiary and composite indexes - Aggregation functions (reduction) • Example: count the number of North American Ales - Organizing related data
    • 10. Map() Function => Index Every changed document goes through all map functions Map Content Metadata function(doc, meta) { emit(doc.username, doc.email) } create row indexed key output value(s)
    • 11. Single Element Keys (Text Key) Map function(doc, meta) { emit(doc.email, null) } text key doc.email meta.id abba@couchbase.com u::1 jasdeep@couchbase.com u::2 zorro@couchbase.com u::3
    • 12. Compound Keys (Array) Array Based Index Keys get sorted as Strings, but can be grouped by array elements Map function(doc, meta) { emit(dateToArray(doc.timestamp), 1) } dateToArray(doc.timestam array key value p) [2012,7,9,18,45] 1 [2012,8,26,11,15] 1 [2012,9,13,2,12] 1
    • 13. Indexing Architecture App Server Doc 1 Couchbase Server Node To other node Replication Queue Doc 1 Doc 1 3 Doc Updated in RAM Cache First Disk Queue 3 2 Managed Cache Disk Doc 1 All Documents & Updates Pass Through View Engine View Engine Indexer Updates Indexes After On Disk, in Batches
    • 14. Buckets >> Design Documents >> Views Beer-Sample Indexers Are Allocated Per Design Doc Beers by_name by_abv Breweries all All Updated at Same Time All Updated at Same Time location beers
    • 15. Querying Views: Parameters
    • 16. Parameters used in View Querying • key = “” - used for exact match of index-key • keys = [] - used for matching set of index-keys • startkey/endkey = “” - used for range queries on index-keys • startkey_docID/endkey_docID = “” - used for range queries on meta.id • stale=[false, update_after, true] - used to decide indexer behavior from client • group/group_by - used with reduces to aggregate with grouping
    • 17. Query Pattern: Range
    • 18. Index-Key Matching doc.email abba@couchbase.com Match a Single Index-Key u::1 beta@couchbase.com u::7 jasdeep@couchbase.com ?key=”math@couchbase.com” meta.id u::2 math@couchbase.com u::5 matt@couchbase.com u::6 yeti@couchbase.com u::4 zorro@couchbase.com u::3
    • 19. Range Query doc.email meta.id abba@couchbase.com u::1 ?startkey=”math@couchbase.com” ?startkey=”bz” endkey=”zz” ?startkey=”b1” & endkey=”zn” &endkey=”math@couchbase.com” beta@couchbase.com u::7 Range of a single item Pulls the Index-Keys (can also UTF-8 Range between be done with key= parameter). specified by the startkey and endkey. jasdeep@couchbase.com u::2 math@couchbase.com u::5 matt@couchbase.com u::6 yeti@couchbase.com u::4 zorro@couchbase.com u::3
    • 20. Index-Key Set Matches doc.email abba@couchbase.com Query Multiple in the Set (Array Notation) u::1 beta@couchbase.com u::7 jasdeep@couchbase.com ?keys=[“math@couchbase.com”, “yeti@couchbase.com”] meta.id u::2 math@couchbase.com u::5 matt@couchbase.com u::6 yeti@couchbase.com u::4 zorro@couchbase.com u::3
    • 21. Query Pattern: Basic Aggregations
    • 22. Simple secondary Index • Find the ABV for each brewery
    • 23. Aggregation: Reducing doc.abv with _stats
    • 24. Group reduce (reduce by unique key)
    • 25. Querying from Views Querying from Ruby Client
    • 26. Query Pattern: Time Based Rollups
    • 27. Find Comment Counts By Time { timestam p "type": "comment", "about_id": "beer_Enlightened_Black_Ale", "user_id": 525, "text": "tastes like college!", "updated": "2010-07-22 20:00:20" } { "id": "u525_c1" }
    • 28. dateToArray() converts DateTime strings to Array of values • String or Integer based timestamps • Output optimized for group_level queries • Generates an array of JSON numbers: [2012,9,21,11,30,44]
    • 29. Query with group_level=2 to get monthly rollups
    • 30. group_level=3 - daily results - great for graphing • Daily, hourly, minute or second rollup all possible with the same index. • http://crate.im/posts/couchbase-views-redditdata/
    • 31. Query Pattern: Leaderboard
    • 32. Aggregate value stored in a document • Lets find the top-rated beers! { ratings "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "style": "Other Belgian-Style Ales", "updated": "2010-07-22 20:00:20", “ratings” : { “ingenthr” : 5, “jchris” : 4, “scalabl3” : 5, “damienkatz” : 1 }, “comments” : [ “f1e62”, “6ad8c“ ] }
    • 33. Sort each beer by its average rating • Lets find the top-rated beers! 34
    • 34. Q&A
    • 35. Thanks!

    ×