Haystack LIVE! - 5 ways to increase result diversity at web-scale - Dmitry Kan & Daniel Wärnå
•Download as PPTX, PDF•
0 likes•304 views
Promoting diversity among items in a search result has been shown to increase user satisfaction, compared to relevancy only based ranking. In this talk, we'll present how we went about implementing search result diversification methods across different vertical search engines. Starting from zero with no diversification at all, exploring simple heuristic-based methods and moving onwards to more complex ones based on entropy and determinantal point processing. We'll also discuss evaluation methods and useful tooling around that. Presented by Dmitry Kan, Principal AI Scientist at Silo AI and Daniel Wärnå, AI Engineer, Silo AI. YouTube recording: https://www.youtube.com/watch?v=bri0C28mfl8 Code demoed: https://github.com/DmitryKey/bert-solr-search/tree/master/src/diversify
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Haystack LIVE! - 5 ways to increase result diversity at web-scale - Dmitry Kan & Daniel Wärnå
Similar to Haystack LIVE! - 5 ways to increase result diversity at web-scale - Dmitry Kan & Daniel Wärnå (20)
More from Dmitry Kan
More from Dmitry Kan (20)
Recently uploaded
Recently uploaded (20)
Haystack LIVE! - 5 ways to increase result diversity at web-scale - Dmitry Kan & Daniel Wärnå
- 1. Search diversity Daniel Wärnå, AI Engineer Dmitry Kan, Principal AI Researcher
- 2. Who we are What we do Trusted AI partner. We deliver AI-driven solutions and products to our clients by providing world-class expertise and tooling. Vision AI for people. A world with safe human-centric AI that frees the human mind for meaningful work. 200+ Experts 100+ PhDs Network of 500+ Nordics Helsinki, Turku, Tampere, Jyväskylä, Oulu, Stockholm, Copenhagen US Palo Alto Largest private AI lab from the Nordics DACH Switzerland UK London
- 3. Daniel ● First AI engineer at Silo.AI ● Full-stack engineer ● Practices circus :) Search diversity Silo.AI Dmitry ● Built a few search engines: vertical and web ● Hosts Vector Podcast ● Helps out with Quepid About us
- 4. A search application takes a user’s search query and attempts to return content that the user is satisfied with A search engine should: - Understand the query - Find relevant items - Show found items in a relevant order - But sometimes imprecise is more useful Search diversity Silo.AI
- 5. Related work Search diversity Silo.AI 1. Daniel Tunkelang 2. Jo Kristian Bergum (Vespa) 3. Eric Pugh
- 6. Thoughts on Search Result Diversity Search diversity Silo.AI ● https://dtunkelang.medium.com/thoughts-on-search-result- diversity-1df54cb5bf4a ● Diversity sometimes fits broad queries (“men shoes”), but not all broad queries (“shoes” -> show facets) ● Diversity never works for ambiguous queries (“mixer” - kitchen or an audio product item?) ● Diversity works best for aspects that aren’t constraints: style, color or material ● Balance between result desirability and result diversity ● Use stratified sampling, not necessarily with uniform distribution ● Use KL-divergence as penalty for how far off our distribution is from a desired one (greedy vs combinatorial explosion) Daniel Tunkelang
- 7. Result Diversification with Vespa Search diversity Silo.AI ● https://blog.vespa.ai/result-diversification-with-vespa/ ● Types of retrieval: keyword, filters and nearest neighbors ● Grouping method is used for diversification ● Group by a predefined category: say colour ● Complex rules for group reordering, e.g. order(- max(relevance())*count()) ● Scaling diversity is an important consideration Jo Kristian Bergum
- 12. Entropy = Importance x Uncertainty The more important an event in the system, the more uncertainty it can introduce into the system
- 14. Search Diversity Search diversity Silo.AI More precise More diverse Sweet spot
- 16. Search Diversity Search diversity Silo.AI Search engine Query Diversification Result
- 17. Search Diversity Search diversity Silo.AI In practice we need to: - Define what makes two items similar/dissimilar - Reorder a list of items so that the end result is more diverse
- 18. Diversification methods Search diversity Silo.AI ■ Randomization
- 19. Diversification methods Search diversity Silo.AI - Randomization ■ Rule based
- 20. Rule based diversity Search diversity Silo.AI Alternated between different sources For example, if we have sources S1, S2, S3 For K = 1, result [S1, S2, S3, S1, S2, S3, …. ] For K = 2, result [S1, S1, S2, S2, S3, S3, ….] The choice of selecting S1 could be based on the _score, weight
- 21. Diversification methods Search diversity Silo.AI - Randomization - Rule based ■ Entropy based
- 22. Entropy based diversity Search diversity Silo.AI Weighted and Entropy@K ■ One key takeaway from with Entropy based methods is: High Entropy -> Diversity ■ For a set of hits from a query, one can compute entropy given we have probabilities ■ Multiple ways to compute probabilities ■ By aggregating counts of a category ■ By normalizing the scores of the hits ■ Entropy@K ■ Given we have top 50 hits, we can generate new samples by reordering the indices ■ Compute the entropy of top (K=10) of each sample ■ Pick the sampled index with the highest entropy
- 23. Entropy based diversity Search diversity Silo.AI Weighted and Entropy@K ■ Weighted entropy ■ By weighting the document rank, one can force similar hits not to appear at the top of the results
- 24. Diversification methods Search diversity Silo.AI - Randomization - Rule based - Entropy based ■ K-Means clustering
- 25. K-Means clustering Search diversity Silo.AI ■ Clustering: machine learning tool for discovering group structure underlying the data ■ K-means clustering algorithm: creates the cluster allocations by computing the centroids/means of data-points, iteratively ■ Data points within the cluster are considered to be relevant ■ To bring diversity: ■ we can show data-points allocated to different clusters first and repeat till we exhaust all cluster allocations ■ Or by increasing the number of clusters we can just display different clusters groups Simplest clustering model
- 26. K-Means clustering Search diversity Silo.AI Simplest clustering based diversification Indices after k-means re-indexing Indices before re-indexing K-Means (K=2)
- 27. K-Means clustering Search diversity Silo.AI Simplest clustering based diversification Indices after k-means re-indexing Indices before re-indexing K-Means (K=2)
- 28. Diversification methods Search diversity Silo.AI - Randomization - Rule based - Entropy based - K-Means clustering ■ DPP - Determinantal point process
- 29. DPP - Determinantal Point Process Search diversity Silo.AI ■ DPP has connection to physics: thermal equilibrium, quantum physics ■ Also used in ML: image search, document / video summarization, product recommendation ■ Maximum a posteriori (MAP) is NP-hard ■ Greedy methods try to speed things up: O(M4)->O(M3) with approximations ■ We used Exact method that gives O(M3) ■ Runs much faster than approximate O(M3) in practice ■ Code: https://github.com/laming-chen/fast-map-dpp based on fast greedy MAP inference
- 30. How to integrate DPP/K-Means Search diversity Silo.AI ■ Both methods require some kind of similarity measure ■ Similarity can be derived from e.g: ■ Sentence embeddings ■ Encoding categorical data as an vector ■ Recommender system output ■ Over-querying results from the underlying search engine might produce more diverse results
- 31. Measuring results Search diversity Silo.AI - Manually evaluate query relevancy, did relevancy drop due to diversification? - User metrics, did click-through rate (or some other metric) increase or decrease?
- 33. Thank you! Search diversity Silo.AI - Daniel’s twitter: @danielwarna - Dmitry’s twitter: @DmitryKan - Vector Podcast: https://bit.ly/3HVsvcg
- 34. Sources and links Quepid: https://quepid.com/ Entropy tool: http://bit.ly/measure-diversity How to measure diversity talk : https://2021.berlinbuzzwords.de/session/how- measure-diversity-search-results https://towardsdatascience.com/what-is-shannons-entropy- 5ad1b5a83ce1 https://dtunkelang.medium.com/thoughts-on- search-result-diversity-1df54cb5bf4a https://blog.vespa.ai/result-diversification-with- vespa/ Search diversity Silo.AI
Editor's Notes
- From DT: Focus on demand more than on supply
- Given a set of hits from a query, one can compute its entropy given we have probabilities for each hit There are two ways to compute probabilities (can be more): Either by aggregation counts of a given category or by normalizing the scores of the hits Entropy@K, where K is 10, Given the top 50 hits, randomly generate new sample by reordering indices compute the entropy of top K=10 hits of each sample pick the sampled index with highest entropy
- And weighted entropy can be used as a measure of diversity By weighting the index locations, for example first entry gets high weight and subsequent entries get lower weights. Now if we sample the indices and compute the weighted entropies of each sample and pick a sample with high weighted entropy. we end up getting a diverse enough sample. Essentially penalizing hits with similar results being displayed at the top
- Pick items from clusters starting from centroids, then paginate Or show cluster after cluster, starting from most narrow cluster: it has higher diversity / sparsity in data
- The numbers mean the rank of an item on the screen