Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Advanced querying

1,516 views

Published on

Crea

Published in: Technology, Business
  • Be the first to comment

Advanced querying

  1. 1. Advanced Querying Brian Mitchell (strmpnk)
  2. 2. Query
  3. 3. Queryfinding the right information
  4. 4. Query finding the right informationscanning and processing data
  5. 5. Query finding the right informationscanning and processing data traversing data structures
  6. 6. Everything ends up in some sort of data structure.
  7. 7. B-tree B-tree B-tree B-tree B-tree B-tree B-tree B-tree B-tree B-treeshallow, append only, compressed, awesome
  8. 8. I/O
  9. 9. I/Oall of your data structures are limited by the medium Throughput (MB/s) Latency (microseconds) 3000 2250 1500 750 0 HDD SSD RAM
  10. 10. Obviously RAM is good. Cheap too. Not unlimited.
  11. 11. Not unlimited.
  12. 12. all your data working set
  13. 13. all your data working setKeep it in RAM
  14. 14. "Working Set"• recently accessed documents• replicating documents• compaction files• index files
  15. 15. Controlling Working Set Size• smaller documents • short object keys, less repetition• smaller databases • increases locality and minimizes compaction overhead• fewer or smaller views • multi-purpose • avoid repeating document data
  16. 16. Primary IndexYour first line of defense against bloat
  17. 17. Function of an Index Key Value
  18. 18. Function of a Primary Index In Couchbase Key Doc
  19. 19. UniquenessA B C
  20. 20. UniquenessA B C B
  21. 21. Uniqueness Semantic KeyingA B C B
  22. 22. One FileAlways Fresh, No Extra Cleaning
  23. 23. Secondary Index aka. View• Projects a new sequence• Custom mapped values• M-N• Links back to source document
  24. 24. View Techniques• Join by collation• Page by key• Foreign includes• Cheap aggregates• Flexible grouping
  25. 25. Join By CollationContact A Contact B Note for A Note for B Note for A
  26. 26. Join By CollationContact A Contact B Note for A Note for B Note for A Emit A B A-note B-note A-note
  27. 27. Join By CollationContact A Contact B Note for A Note for B Note for A Emit A A-note A-note B B-note
  28. 28. Page By KeyA B C D E
  29. 29. Page By Key limit=2A B C D E
  30. 30. Page By Key limit=2A B C D E limit=2&start_key=Bufff0
  31. 31. Foreign Includes A B Emit a a
  32. 32. Foreign Includes A B Reference_id=A _id=B
  33. 33. Cheap Aggregates• It pays to know your data well• Reduce values are stored inline with the view b-tree• Small values take very little space• Nice built-in reduce functions• Not just for user visible data
  34. 34. Flexible Grouping2008-10-02 2008-08-17 2009-02-12 Emit[2008,10] [2008, 8] [2009, 2]
  35. 35. Flexible Grouping2008-10-02 2008-08-17 2009-02-12 Emit[2008,10] [2008, 8] [2009, 2]
  36. 36. Flexible Grouping2008-10-02 2008-08-17 2009-02-12 Emit[2008,10] [2008, 8] [2009, 2]
  37. 37. Traditional CouchDB
  38. 38. 20%10% 70%
  39. 39. 20% 20% 20% 20%10% 10% 10% 10% 70% 70% 70% 70% 20% 20% 20% 20%10% 10% 10% 10% 70% 70% 70% 70% 20% 20% 20% 20%10% 10% 10% 10% 70% 70% 70% 70%
  40. 40. Clustering
  41. 41. Single Key
  42. 42. Single Key
  43. 43. Single Key
  44. 44. Single Key
  45. 45. Query
  46. 46. Query
  47. 47. Query
  48. 48. Query
  49. 49. Alternatives
  50. 50. Manual Indexing• Store an index as a document• Good properties for mostly static indexing• Cluster friendly• Create custom constrains (uniqueness)• Snapshot of a slow query for speed
  51. 51. GeoCouch• R-tree based• First-class Erlang • improved with view engine refactor• Can be abused for multi-dimensional queries • more than just geo-data
  52. 52. CouchDB Lucene• Based on CouchDB Externals• Limited to Couchbase Single Server• Faceted queries• Full-text indexing
  53. 53. Hybrid• Application managed• Allow stand alone service to work with Couchbase cluster • i.e. Solr, Redis, PostgreSQL• Complex concurrency• More moving parts
  54. 54. Fintwitter: @strmpnk email: b@p2p.io

×