Querying Riak Just Got Easier - Introducing Secondary Indices
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Querying Riak Just Got Easier - Introducing Secondary Indices

on

  • 19,966 views

This presentation introduces new Riak KV functionality called Secondary Indexes. Secondary Indices allows a developer to retrieve data by attribute value, rather than by primary key....

This presentation introduces new Riak KV functionality called Secondary Indexes. Secondary Indices allows a developer to retrieve data by attribute value, rather than by primary key.

Currently, a developer coding outside of Riak’s key/value based access must maintain their own indexes into the data using links, other Riak objects, or external systems. This is straightforward for simple use cases, but can add substantial coding and data modeling for complex applications. By formalizing an approach and building index support directly into Riak KV, we remove this burden from the application developer while preserving Riak’s core benefits, including scalability and tolerance against hardware failure and network partitions.

The presentation covers usage, capabilities, limitations, and lessons learned.

Statistics

Views

Total Views
19,966
Views on SlideShare
19,737
Embed Views
229

Actions

Likes
22
Downloads
156
Comments
3

24 Embeds 229

http://www.nosql.io 38
http://nosql.io 32
http://staging.conferize.com 29
http://irr.posterous.com 25
http://twitter.com 20
http://ruben.i.conferize.com 16
http://lanyrd.com 16
https://twitter.com 12
https://coolchevy.org.ua 6
http://speakerrate.com 6
http://mundo-powerpoints.blogspot.com 5
http://nuevospowerpoints.blogspot.com 5
http://www.onlydoo.com 3
http://tweetedtimes.com 3
http://trunk.ly 2
http://us-w1.rockmelt.com 2
http://www.coolchevy.org.ua 2
http://webcache.googleusercontent.com 1
http://www.twylah.com 1
http://10.0.1.2 1
http://www.linkedin.com 1
http://posterous.com 1
http://www.i.conferize.com 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • One question about this approach. If indexes are based only on headers included with the KV write, does that preclude adding an index for existing values in a bucket w/o rewriting every record?

    In the RDBMS world it's very common to add an index to solve performance problems or facilitate a newly added query. Being unable to do this would be a serious drawback.
    Are you sure you want to
    Your message goes here
    Processing…
  • I post it also on NoSQL.io .
    Are you sure you want to
    Your message goes here
    Processing…
  • Database tradeoffs as RPG classes, I like it!
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Querying Riak Just Got Easier - Introducing Secondary Indices Presentation Transcript

  • 1. Querying Riak Just Got Easier Secondary Indices in Riak OSCON Data Portland, Oregon · July 2011 Basho Technologies rusty@basho.com Rusty Klophaus twitter: rustyio
  • 2. tl;dr:Secondary Indices fundamentally change data modeling in Riak. Model one-to-one, one-to-many, or many-to-many relationships simply and efficiently. 2
  • 3. But first, a little bitabout tradeoffs and NoSQL. 3
  • 4. Which one would you choose?It depends on their abilities.It also depends on the quest. Fools! 5
  • 5. Rule #1 Your character will fare better in certain environmentsdepending on his (or her) abilities. 6
  • 6. Rule #2There are always tradeoffs 7
  • 7. Databases are like RPG character classes.They focus on certain abilities. 8
  • 8. Database Abilities (1/2)Schema Flexible Schema ⟩ Pre-Defined Schema ⟩ Typed Fields ⟩ Untyped Fields ⟩ BlobOperation Skew Mostly Writes ⟩ 50/50 ⟩ Mostly ReadsDisk Persistence Every Operation ⟩ Delayed Batch ⟩ Flush ⟩ NeverTransactions Global ⟩ Table ⟩ Object ⟩ NoneData Relations Ad-Hoc ⟩ Pre-Defined ⟩ NoneOperation Order Random ⟩ Sequential 9
  • 9. Database Abilities (2/2)Secondary Queries Ad-Hoc ⟩ Pre-Determined ⟩ NoneNative Data Types Tables ⟩ XML ⟩ JSON ⟩ Text ⟩ BlobScalable Data Center ⟩ Cluster ⟩ Single MachineFailure-Tolerance Data Center ⟩ Network ⟩ Machine ⟩ Disk ⟩ Sector ⟩ NoneStability Predictable Latency ⟩ Variable LatencyPerformance Ops Per SecondAnd so on... 10
  • 10. For a long time, industry focused on just one manifestation of database abilities: SELECT * FROM Quests «Relational Database» 11
  • 11. The World Has ChangedGlobal Internet “Everyone is connected...” ile Soc ob ial MMobile Computing Global “...all the time...”Social Networking “...producing more data (in more varieties) than ever.” 12
  • 12. NoSQL is about alternatives. Focus on different abilities, environments, and tradeoffs. 13
  • 13. As a database consumer, you need to understand thebasic tradeoffs of each solution. In turn, database producers should strive to make those tradeoffs clear. 14
  • 14. There are always tradeoffs.Databases that claim to doeverything well are lying. 15
  • 15. Where Does Riak Fit? 16
  • 16. 17
  • 17. Have you ever been burned by: hardware failure? overloaded servers? emergencies at 2am? 17
  • 18. Does your quest involve:SLAs that mention uptime or latency? data that you can’t afford to lose? 18
  • 19. If so, Riak’s tradeoffs will make sense. If not, they won’t. 19
  • 20. Riak KV - Some Tradeoffs (1/3)Amazon’s Dynamo Architecture ✔Distributed, scalable, no single point of failure. ✘ No transactions; trade strong consistency for eventual consistency. 20
  • 21. Riak KV - Some Tradeoffs (2/3)Extremely Focused on Operations ✔Simple to install, manage, connect a cluster. ✔Has been called “plumbing”, ie: it just works. ✘ Historically, developer-facing features lagged behind. This is rapidly changing. # Scale out... riak-admin join nodename@hostname # Scale back in... riak-admin leave 21
  • 22. Riak KV - Some Tradeoffs (3/3)Key/Value Model ✔Simple, straightforward, content-type agnostic. ✘ More difficult to discover your data. (Queryability.) Let’s dive deeper into “queryability.” 22
  • 23. Current Options for Querying RiakMapReduce Provide set of starting keys, filter via map. MapReduce is meant for calculations/aggregations, not queries.Riak Search Full-text search in Riak. Opinionated, assumes your document is prose.Roll Your Own Indices Difficult to get right. More code to maintain. Often introduces SPOFs. 23
  • 24. New Feature: Secondary Indices 24
  • 25. What are Secondary Indices? 25
  • 26. What Are Secondary Indices?Goals Provide *simple* indexing on Riak objects. Maintain Riak’s operational advantages. Make a developer’s life easier.How Does It Work? At write time, tag your data with key/value metadata. Query the metadata, get matching objects. 26
  • 27. For example...BUCKET KEY VALUE INDEX METADATAloot gauntlet24 category: armor price: 400 ... “Gauntlets of Shininess” 27
  • 28. Index an Object# Store an object with:# Bucket: loot# Key: gauntlet24# Fields:# - category: armor# - price: 400curl -X PUT -d "OPAQUE_VALUE" -H "x-riak-index-category_bin: armor" -H "x-riak-index-price_int: 400" http://127.0.0.1:8098/riak/loot/gauntlet24 28
  • 29. Query the Index# Query for category_bin = "armor"curl http://127.0.0.1:8098/buckets/loot/index/category_bin/armor{"keys":["gauntlet24"]}# Query for price_int between 300 and 500curl http://127.0.0.1:8098/buckets/loot/index/price_int/300/500{"keys":["gauntlet24"]} 29
  • 30. Query Syntax /buckets/$BUCKET/index/$FIELDNAME/$VALUE /buckets/$BUCKET/index/$FIELDNAME/$START/$END$BUCKET Bucket to query.$FIELDNAME Must end with “_bin” for binaries, “_int” for integers. Special field $key for key range lookups.$VALUE / $START / $END Equality or range queries. 30
  • 31. Data Modeling withSecondary Indices 31
  • 32. Key/Value Lookups Retrieve a user’s session. Retrieve an object by key. sessions/ { 8b6cfaa foo: "bar", ... } 32
  • 33. Key/Value Lookups Use Case Retrieve a user’s session. Retrieve an object by key. Generic Case Object sessions/ { 8b6cfaa foo: "bar", ... } Key Value 33
  • 34. Alternate Keys / One-to-One Relationships Retrieve a user by username or by email address. An object has multiple names. users/ { rusty username: "rusty", email: "rusty@basho.com", twitter: "rustyio", ... } emails/ "rusty" rusty@basho.com 34
  • 35. Alternate Keys Wit Retieve a user by username or by email address. Seco h nda Ind ry An object has multiple names. ices ! users/ { rusty username: "rusty", email: "rusty@basho.com", twitter: "rustyio", ... } Indexes: email_bin: rusty@basho.com twitter_bin: rustyio 35
  • 36. Ownership / One-to-Many Relationships A person has many cars. Parent has a *small* number of children. people/ { frank cars: [ { plate: "ET7-B928", color: "red", type: "corvette", } ... ] } 36
  • 37. Ownership / One-to-Many Relationships Wit A person has many cars. Seco h nda Ind ry Parent has a *small* number of children. ices ! people/ { frank cars: [ { plate: "ET7-B928", color: "red", type: "corvette", }, ... ] } Indexes: cars_plate_bin: ET7-B928 cars_plate_bin: BUB-7911 ... 37
  • 38. Ownership / One-to-Many Relationships A user has many status updates. Parent has a *large* number of children. users/ { rustyio status_updates: [ "18258713", "87187597", "71117389", ... ] } statuses/ { 18258713 author: "rustyio" reply_to: "barackobama" text: "Sorry, cant hang out now, Im speaking at OSCON Data." }38
  • 39. Ownership / One-to-Many Relationships Wit A user has many status updates. Seco h nda Ind ry Parent has a *large* number of children. ices ! users/ { rustyio ... } statuses/ { 18258713 author: "rustyio" reply_to: "barackobama" text: "Sorry, cant hang out now, Im speaking at OSCON Data." } Indexes: author_bin: rustyio reply_to_bin: barackobama 39
  • 40. Membership / Many-to-Many Relationships A user joins one or more clubs, a club has users. A group has many members. users/ { rusty clubs: [ "dc_larping_club", ... ] } clubs/ { dc_larpers users: [ "rusty", "frank" ] } 40
  • 41. Membership / Many-to-Many Relationships Wit A user joins one or more clubs, a club has users. Seco h nda Ind ry A group has many members. ices ! users/ { rusty clubs: [ "dc_larping_club", "nascar_fans", ... ] } Indexes: club_bin: dc_larping_club club_bin: nascar_fans clubs/ ... dc_larpers 41
  • 42. What Were The Challenges? 42
  • 43. Challenge: Ambitious Prototyping (1/5)Why Difficult? Early prototypes contained support for: • A SQL-like query language (RQL), with compound queries • Sorting and pagination, with intelligent caching • Inline map/reduce • Extensible data types Arguably too clever. Allowed developers to shoot selves in foot.How Solved? Ruthlessly cut features & simplify. 43
  • 44. Challenge: Data Types (2/5)Why Difficult? What type is a given field? Naming convention, or global dictionary? What if the user wants to change the type? What if the user provides a value of the wrong type?How Solved? Field type determined by suffix. (field1_bin, field2_int) Different type == different field name. Pre-commit hook to validate data types. 44
  • 45. Challenge: Disk Based Storage (3/5)Why Difficult? Disk is slow. (http://highscalability.com/numbers-everyone-should-know) Need data structures that are both read and write efficient, for data of unpredictable sizes and shapes.How Solved? Leverage merge_index (data engine from Riak Search). Investigate LevelDB (library from Google) 45
  • 46. Challenge: Atomicity (4/5)Why Difficult? Need to keep the index synchronized with the object value. Account for eventual consistency, siblings, handoff, replication, etc. What happens during node failure / partial cluster situations?How Solved? Make the KV object the authoritative data. Indexed data is discarded if the object moves. 46
  • 47. Challenge: System is Distributed (5/5)Why Difficult? Index is split over many different partitions. *Don’t* need to query every partition. *Do* need to be smart about which partitions to query.How Solved? Extend riak_core (distribution layer) with ability to broadcast command to covering set of partitions. (h/t to Kelly McLaughlin, @_klm) 47
  • 48. Indexing Client API User tags the object with metadata.Cluster Write Coordinator Validate that metadata parses. riak_kv_put_fsm PUT Request Node VNode Coordinator riak_core_vnode_master VNode (Virtual Node) Core VNode riak_core_vnode KV VNode riak_kv_vnode Backend Stores object in bitcask, riak_kv_index_backend index metadata in merge_index.
  • 49. Querying Client User issues a query API against metadata.Cluster Query Coordinator riak_index_query_fsm Coverage Logic riak_core_coverage_fsm Query Request Node VNode Coordinator riak_core_vnode_master VNode (Virtual Node) Core VNode riak_core_vnode KV VNode riak_kv_vnode Backend Runs query against index, riak_kv_index_backend replies with results.
  • 50. Next StepsPublish We are open source. Code is available now (for the foolhardy.) Beta version soon (for the adventurous.) Included in Riak version 1.0 (for the masses.) 50
  • 51. API design is like sex: Make one mistake andsupport it for the rest of your life. - @joshbloch(Everything is subject to change.) 51
  • 52. About Basho Technologies<plug type=“shameless”> • Distributed company: ~30 people • Cambridge, MA / San Francisco, CA / Reston, VA • “We hate downtime, we hate overtime.” • Riak KV (Open Source) • Riak Support, Services, and Enterprise Features ($)</plug> 52
  • 53. Thanks! / Questions? Also at OSCON... Rusty Klophaus Mark Phillupsrusty@basho.com mark@basho.com @rustyio @pharkmillups Ryan Zezeski Tony Falcorzezeski@basho.com tony@basho.com @rzezeski @antonyfalco
  • 54. END