Scaling Box-Search: Gearing up for Petabyte Scale - Shubhro Roy & Anthony Urbanowicz, Box
•Download as PPTX, PDF•
0 likes•541 views
The document discusses how Box is scaling its search capabilities to handle a petabyte-scale index. It outlines Box's existing search infrastructure and scaling issues, and proposes solutions like key range partitioning, shard growth handling, and optimized shard allocation to improve query performance and cluster utilization as the index grows towards 1 petabyte in size over the next 5 years. The talk also covers future work like integrating key range partitioning into SolrCloud and building a search federation service to classify and rank results across collections.
Recommended
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Scaling Box-Search: Gearing up for Petabyte Scale - Shubhro Roy & Anthony Urbanowicz, Box
Similar to Scaling Box-Search: Gearing up for Petabyte Scale - Shubhro Roy & Anthony Urbanowicz, Box (20)
More from Lucidworks
More from Lucidworks (20)
Recently uploaded
Recently uploaded (20)
Scaling Box-Search: Gearing up for Petabyte Scale - Shubhro Roy & Anthony Urbanowicz, Box
- 1. Scaling Box-Search: Gearing up for petabyte scale
- 2. 2Scaling Box-Search: Gearing up for petabyte scale Human perception of Realtime 13 milliseconds
- 3. 3Scaling Box-Search: Gearing up for petabyte scale Anthony UrbanowiczShubhro Roy Staff Software Engineer Box Senior Software Engineer Box
- 4. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 5. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 6. Cloud Content Management platform that enables you to securely manage, share and access your content from anywhere
- 7. 7Scaling Box-Search: Gearing up for petabyte scale Search use cases • Quick-search • Full-text content search • Metadata search • Search API
- 8. 8Scaling Box-Search: Gearing up for petabyte scale Search index stats / Indexed files : Hundreds of billions / Replicated index size: 600+ TB / Files indexed per day: Hundreds of millions / Rate of index growth: ~2 TB / month / Projected index size in 5 years : ~1 petabyte
- 9. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 10. 10Scaling Box-Search: Gearing up for petabyte scale Index pipeline WebApp/ API Shard 0 Shard 1 Shard n Pub-Sub Queue Search Workers Events Offline Store (HBase) Search WorkersSearch Worker Cluster Shard 0 Shard 1 Shard n Cluster Offline ReIndex fileId % numShards = shardId Representation Service
- 11. 11Scaling Box-Search: Gearing up for petabyte scale Query Pipeline WebApp/ API Shard 0 Shard 1 Shard n Cluster Query Cerebro Rate-Limiter Circuit-Breaker Re-Ranker Federator Shard 0 Shard 1 Shard n Cluster App-DB (MySQL) fq = <permissions filter>
- 12. 12Scaling Box-Search: Gearing up for petabyte scale Cluster Management Shard 0 Shard 1 Shard n Cluster Search Worker Cerebro Zookeeper Sops Shard Health Monitor Commands
- 13. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 14. 14Scaling Box-Search: Gearing up for petabyte scale Major pain points / Queries bottlenecked by slowest shard / Single user/enterprise can overwhelm the whole cluster / Query latency increases as we add more machines
- 15. 15Scaling Box-Search: Gearing up for petabyte scale Solution: enterprise based sharding / Natural grouping of documents / Deterministic sharding approach / Reduces query fan-out to just one shard in most cases / Better index compression / Better utilization of filter-cache
- 16. 16Scaling Box-Search: Gearing up for petabyte scale Solr composite-id routing / ShardingKey / k ! documentId / k: number of bits to use from the sharding key / ShardingKey = hash(enterpriseId) / Solr Query: q=solr&_route_=<enterpriseId>/k!
- 17. 17Scaling Box-Search: Gearing up for petabyte scale • Non-uniform shard sizes • Need to generate and maintain mapping of enterpriseId K • Can’t handle new enterprises without offline rebuild Issues with composite-id routing 200,000,000 400,000,000 600,000,000 800,000,000 1,000,000,000 1,200,000,000 1,400,000,000 1,600,000,000 1,800,000,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 Shard Document Counts
- 18. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 19. 19Scaling Box-Search: Gearing up for petabyte scale Key ranges n/k keys Dynamic key range partitioning • Unique sharding key per document • Sort sharding keys in byte order • Partition rows into k buckets • Each shard has n/k documents Logical shard 0 Logical shard 1 Logical shard 2 Logical shard 3 Logical shard 4 Logical shard 5
- 20. 20Scaling Box-Search: Gearing up for petabyte scale • h(enterpriseId): All documents of an enterprise are grouped together • h(folderId): d-gap compression • h(fileId): uniqueness Sharding key 4 bytes h(enterprise_id) h(folder_id) h(file_id) 4 bytes 8 bytes
- 21. 21Scaling Box-Search: Gearing up for petabyte scale Shard routing Index routing Query routingShard boundaries Shardingkey Binary search Binary search + Linear probing H(enterpriseid)
- 22. 22Scaling Box-Search: Gearing up for petabyte scale Lumos architecture
- 23. 23Scaling Box-Search: Gearing up for petabyte scale • Write ShardingKey to HBase at Index time • Stores keys in sorted byte-order • MR job scans HBase table to generate shard boundaries for a cluster • Maintain shard boundaries per cluster in HBase HBase metastore
- 24. 24Scaling Box-Search: Gearing up for petabyte scale Shard routing Search Worker / Cerebro Lumos Controller Shard 0 Shard 1 Shard n HBase Zookeeper Index / Query Request Cluster
- 25. 25Scaling Box-Search: Gearing up for petabyte scale No free lunch
- 26. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 27. 27Scaling Box-Search: Gearing up for petabyte scale • Enterprises grow at different rates • Machines have max capacity of 3TB • Some shards exceed capacity while some had lot to spare Shard growth with live indexing FILECOUNT SHARD Shard growth in 2 weeks
- 28. 28Scaling Box-Search: Gearing up for petabyte scale • Spill new index requests to shard(s) with most capacity • Optimal use of cluster capacity • Automated load-balancing without any downtime Solution: Shard spilling
- 29. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 30. 30Scaling Box-Search: Gearing up for petabyte scale Shard allocation Key ranges Node 1 Node 2 Logical shard 0 Logical shard 1 Logical shard 2 Logical shard 3 Logical shard 4 Logical shard 5 Physical shard 0 Physical shard 1 Physical shard 2 Physical shard 3 Physical shard 4 Physical shard 5
- 31. 31Scaling Box-Search: Gearing up for petabyte scale Shard allocation : Binpacking B I N P A C K I N G Key ranges Logical shard 0 Logical shard 1 Logical shard 2 Logical shard 3 Logical shard 4 Logical shard 5 Node 1 Node 2 Physical shard 0 Physical shard 1 Physical shard 2 Physical shard 3 Physical shard 4 Physical shard 5
- 32. 32Scaling Box-Search: Gearing up for petabyte scale • Compute QPS per logical shard • Minimize overall QPS at each machine • NP- hard problem • Greedy approximation: descending first fit Shard allocation: Binpacking Minimize 𝑗=0 𝑛 𝑤𝑖 𝑥𝑖𝑗 for each bin 𝑥𝑖𝑗 𝜖 {0,1} 𝑤𝑖 weight of items n = number of bins
- 33. 33Scaling Box-Search: Gearing up for petabyte scale Shard allocation : machine QPS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 QPS Hours Machine QPS (Random Shard -> Machine) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 QPS Hours Machine QPS Binpacking
- 34. Agenda / Search @ Box / Existing infrastructure / Scaling issues / Key range partitioning / Shard growth / Shard allocation / Future work / Q&A
- 35. 35Scaling Box-Search: Gearing up for petabyte scale Future of Box Search What is enterprise search / Currently, routing is "implicit" / Integrating key range partitioning into SolrCloud / Extend SolrClouds DocRouter with the Lumos Controller for a new mode / Reuse SolrClouds zk coordinator / Create a Search Federation Service on top to: • Classify and expand queries • Route to multiple collections and verticals • ML ranking of results
- 36. 36Scaling Box-Search: Gearing up for petabyte scale Enterprise search pros and cons / Con: cannot crowdsource for human judgements due to customer document privacy / Con: users document set has little to no overlap, hence lack of query-doc consensus / Pro: semi closed loop, documents are previewed within our domain / Pro: signals beyond clicks such as comment, edit, download and share Data for training and features
- 37. 37Scaling Box-Search: Gearing up for petabyte scale Relevance pyramid Data for training and features Search resultant events to train a relevance model Train the below features using session logs and the above relevance model Term based features (BM25, Classifications, Query alterations, etc..) Relationship based features from Box Graph (Non search events)
- 38. 38Scaling Box-Search: Gearing up for petabyte scale Sue Bob File2 Joe Sue Bob File3 Explicit user interactionsImplicit user graph Joe File1 Box graph
- 39. 39Scaling Box-Search: Gearing up for petabyte scale Joe Sue Bob File1 File2 Joe Sue Bob File3 Explicit user interactionsImplicit user graph Box graph
- 40. 40Scaling Box-Search: Gearing up for petabyte scale Joe Sue Bob File1 File2 Joe Sue Bob File3 Explicit user interactionsImplicit user graph Box graph
- 41. 41Scaling Box-Search: Gearing up for petabyte scale Joe Sue Bob File1 File2 Joe Sue Bob File3 Explicit user interactionsImplicit user graph Box graph
- 42. 42Scaling Box-Search: Gearing up for petabyte scale Joe Sue Bob Joe Sue Bob File1 File2 File3 Explicit user interactionsImplicit user graph Box graph
- 43. 43Scaling Box-Search: Gearing up for petabyte scale Joe Sue Bob File1 File2 Joe Sue Bob File3 Explicit user interactionsImplicit user graph Box graph
- 44. Shubhro Roy sroy@box.com Anthony Urbanowicz aurbanowicz@box.com We are Hiring ! https://www.box.com/careers/engineering Thank you Q&A
Editor's Notes
- Welcome to our talk Interesting article: Human Perception of Realtime
- Visual Stimuli: 13 millisecond > 100ms is NOT instantaneous. > 2s user has disengaged Latency is critical for search NOT 10 blue links Additional processing: Query Expansion, Relevance Ranking, Collaborative Filtering etc.
- Before we begin: How many of you have used Box?
- What you will learn: Typical challenges with distributed search infra How to address them at scale
- Collaboration and content management platform Create and share content securely Content Discovery: find relevant conent 300K+ enterprises Millions of users
- Type-ahead style search Snippeting Highlighting Date / author / image recognition / video transcripts Api: Notes / Mobile / Platform
- - >60 billion files > 100 million files per day Need to take a long hard look at our infrastructure to ensure it scales
- Index Process Cluster => Machine => Shard Different clusters in different DCs => DR & HA Offline store + Reindex
- <FASTER> Machines go down all the time Systems should be agnostic to such failures
- Solution: Reduce query fanout - Shard by some natural grouping of documents
- - Shard by some natural grouping of documents Users access restricted to enterprise Only query eid shard Same eid similar content / same lang index compression Similar fqs Warmed up cache
- Cant have 1:1 mapping Large enterprises will blow up shards
- Few massively large enterprises Need more control on distribution of documents New enterprise issue Better to be dynamic than predictive
- Introduce key range partitioning
- Solves new enterprise case
- Entire sharding logic encapsulated by Lumos Hbase: Metastore Zookeeper: Co-ordination
- Reasons for using Hbase: Already used Sorted keys MR integrations
- Old Way: Uniform shard growth New Way: Enterprises grow differently
- - We cannot add new machines when other machines have more capacity
- Increase in query fanout optimal utilization of space
- Solving the Second problem= shard to node allocation
- Niave approach Some enterprises use box a lot more then others. Hence alot shard have a lot more query load then others. This causes hot spotting.
- Binpacking a type of scheduling to level out load over all nodes
- There are optimal and non optimal algorythms, We used a greedy algo of descending first fit, which is zipping up the least with the highest QPS shards. How does it work?
- Example of applied binpacking that decrease spikes.
- Over view of what will be added to infra. Add all that was said as a mode into solr cloud so you all can scale too. Beyond that - Adding relevance to search To effect relevance we need to have data – transition
- Diff between web search and enterprise search at box. Transition to ---- how to maximize user feedbacks effect on relevance.
- Top down No labeled sets, use a relavance model trained from end to end events to extract good clicks. Some results are good even without an event as they user might of just wanted a peiece of info from the document. Use this to extract query-doc pairs, and train all non search-event based features. These are BM25, document-query classifications, and the box graph.
- Explicit interactions, creates implicit connections between the users.
- If two users interact on a same content, they have common interests
- These interactions can lead to larger graphs
- Strength of the connection represents collaboration strength
- More common files you work stronger the collaboration strength is.
- These are what we are working on and commit and present in 2019.