The need for sophistication in modern search engine implementations
•Download as ODP, PDF•
0 likes•36 views
The need for more sophisticated search implementations is often at odds with the limited feature set available in modern out of the box open source search engines. This presentation discusses the challenges associated with properly modeling information within a domain and why it's critically needed.
Recommended
More Related Content
What's hot
What's hot (14)
Similar to The need for sophistication in modern search engine implementations
Similar to The need for sophistication in modern search engine implementations (20)
Recently uploaded
Recently uploaded (20)
The need for sophistication in modern search engine implementations
- 1. A Case for Sophistication The modern requirements of search (Solr Focused)
- 2. Let’s get philosophical – for a moment.
- 3. George Gilder published a book in 1990 that made some fairly accurate predictions about the future of computing.
- 5. If wealth is knowledge, knowledge about our domain, and the knowledge to model it accurately could be said to be “value”. ”Value” is the proposition that drives users to engage with search
- 6. In computer graphics, 3d models are simplified for real- time applications (video games). Fidelity is preserved by applying a high-fidelity proxy to the lower-fidelity “real-time” representation. This process is called “baking”.
- 7. In machine learning, when we ‘train’ a model, we are ‘baking’ knowledge into a more efficient representation. The same is true for how we might enhance searches by using external datasets, query statistics, LTR, etc. Modeling a high-fidelity representation of data into a real-time, more efficient form is key to climbing the ladder of search sophistication.
- 8. Representing domain knowledge within our search platform so that it provides value to our users is how we achieve sophistication. This is perhaps the greatest challenge in building search products.
- 9. Our premise Intent IS accuracy, recall IS relevancy This may be controversial; recall vs accuracy is the wrong juxtaposition.
- 10. Our premise Perhaps the best way this relationship can be described is: - The fidelity of a domain model impacts recall - Accuracy is linked to our domain model - Relevancy is linked to accuracy - Accuracy is best modeled by understanding intent - Restrictive queries shouldn’t be presumed to be accurate. Accuracy exists independent of the percent of documents matched.
- 11. Our premise If accuracy is the ultimate goal, and ‘recall’ is apart of accuracy, how do we go about achieving this?
- 12. Our premise Intent Disambiguation Location Title (Known Item) Category Conceptual Modeling intent, allows you to have a conversation with your user.
- 13. Our premise recall entities synonymy hierarchy ontology conceptual proximity More sophisticated and higher resolution primitives within the index offers the opportunity for recall to be more useful and more accurate.
- 14. Modeling Knowledge Let’s return to discussing sophistication. Before we made the claim that knowledge is what provides value. We also said that modeling knowledge is difficult. Implementing maturity in our search platform is what allows us to model our domain knowledge.
- 16. Modeling Knowledge Observation… It’s really hard for most organizations to climb the sophistication ladder that was shown in the previous slide.
- 17. Out of the box Scorer (default similarity) Query Handler (Edismax) Import Handlers Analyzers / TokenFilters Boost Functions
- 18. What we need Query Classifiers ML Models Behavior Sampling / Ingestion Identity Awareness Secondary Data Sources (data connectors) Alternative forms of storage (inverted index) Integrations (Spark, Airflow, etc) Collections as “Containers” for behavior.
- 19. Modeling Knowledge When we model our domain we want to model “things”, that we can call “entities”. Modeling entities in any domain can be extremely valuable.
- 20. Modeling Knowledge - Entities 1.) disambiguation for free
- 21. Modeling Knowledge - Entities 1.) disambiguation for free 2.) fairly easy to generate candidates for any domain
- 22. Modeling Knowledge - Entities 1.) disambiguation for free 2.) fairly easy to generate candidates for any domain 3.) fairly well researched area of ML
- 23. Modeling Knowledge - Entities 1.) disambiguation for free 2.) fairly easy to generate candidates for any domain 3.) fairly well researched area of ML 4.) helps in the modeling of “conceptual” synonyms
- 24. Modeling Knowledge - Entities 1.) disambiguation for free 2.) fairly easy to generate candidates for any domain 3.) fairly well researched area of ML 4.) helps in the modeling of “conceptual” synonyms 5.) must be pruned by user feedback / behavior
- 25. Modeling Knowledge - Entities 1.) disambiguation for free 2.) fairly easy to generate candidates for any domain 3.) fairly well researched area of ML 4.) helps in the modeling of “conceptual” synonyms 5.) must be pruned by user feedback / behavior 6.) ground work for higher-level more sophisticated features.
- 26. Modeling Knowledge - Entities Ok, but why ?
- 27. Modeling Knowledge - Entities Ok, but WHY?
- 28. Modeling Knowledge - Entities Ok, but why ? In the previous slide we saw that 40% of Target Corporations searches are low-information, and they don’t know what they mean. Without modeling your corpus (the content you are searching) you won’t be able to reason about the behavior or relationship between searches, actions, and ultimately intent. It is extremely common for a good portion of searches (half) to not provided the necessary information to give relevant term-based search results. This is at the core of the case for sophistication. Term search simply can’t provide useful results for a large number of searches that your users are going to perform.
- 29. Modeling Knowledge - Entities
- 30. Modeling Knowledge – Truth Systems entity feature value plato isA philosopher socrates isA philosopher plato knew socrates socrates knew plato plato isA historical-figure socrates isA historical-figure
- 31. Modeling Knowledge - Similarity Socrates != Plato - Related, but not the same - One is not a subset of the other - Found in many of the same documents - Found in many of the same contexts - This is where automatic similarity methods, fall down a bit.
- 32. Modeling Knowledge - Ontologies Entities and ontologies can work together... - when building ontologies there are different types of relationships. - word2vec / phrase2vec, LSA, cannot be used by themselves. - ontologies can be pruned and reshaped by supervised learning. - ontologies can be reshaped by feature-systems (truth systems). - most useful ontologies are modeled for a specific feature (product titles). - query classifier can choose between similarity features / models.
- 33. Modeling Knowledge Corpus Domain Model
- 34. Modeling Knowledge We can’t simply rely on our corpus to provide us with the information necessary to model our domain. We must use auxiliary data sources. Fortunately there are many open data sources in the world that we can use to augment our understanding of our corpus.
- 35. Modeling Knowledge The Internet Domain Model Wikipedia StackExchange Amazon Merchant API
- 36. Modeling Knowledge - Entities Entities and ontologies can work together... - when building ontologies there are different types of relationships. - word2vec / phrase2vec cannot be used by themselves - ontologies can be pruned and reshaped by supervised learning - ontologies can be reshaped by feature-systems (truth systems)
- 37. Modeling Knowledge - Entities The
- 38. In the previous slides we saw entity mapping and grading of a job-search domain model. This was accomplished by building candidate phrases and then pruning them using an SVM trained from features from a known good data source with phrases and topics already labeled. Also shown was a query classifier that takes a lazy or poorly constructed query, groups the components of the query logically and expands part of the query based on information it knows about the index and availability and relatedness of terms. A model to classify queries can be built by understanding the relationship between search entities, and the entities and information contained within a document.
- 39. SHReC is a Java package implementing a hierarchical document clustering algorithm based on a statistical co-occurence measure called subsumption. The algorithm is particularly suited to the problem of on-line "search results" clustering, requiring little amounts of text data. - http://shrec.sourceforge.net/ Search Action Document SHReC along with an entity model can be used to prune, grade, and reorganize an ontology to better understand the types and accuracy of relationships. Algorithms used to cluster behavior with search terms are invaluable in modeling search intent and rewriting search queries.
- 40. The perfect combination of phrase boosting, multi-term synonyms, term position (proximity) and performance is a frequent question within the community.
- 41. Exact Phrase Matches → PhraseQuery / SpanQuery Proximity of Terms → SpanQuery Related Phrases → Payloads / Index Time Synonyms
- 42. Currently in Solr there is no built-in way to represent related entities efficiently. Query rewriting or expansion can be performed at query time, but not all relationships can be modeled at query time due to the complexity of the query. Different classifications of synonym within the index are an option, as well as payloads being used to assign relatedness scores to a given entity. All index-side synonym solutions are quite custom and are not easy to quickly implement. Better tools are needed to correctly model graphs of terms or entities and to create rules for how and when to rewrite search queries without using crude rule based systems.
- 43. Conclusion - Modeling the world through language is hard.
- 44. Conclusion - Modeling the world through language is hard. - Modeling phrases and entities makes life a little easier.
- 45. Conclusion - Modeling the world through language is hard. - Modeling phrases and entities makes life a little easier. - Phrases form the basis of relationships.
- 46. Conclusion - Modeling the world through language is hard. - Modeling phrases and entities makes life a little easier. - Phrases form the basis of relationships. - Accuracy should be proportional to confidence
Editor's Notes
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- With that said, There is a relationship between recall and accuracy, that’s not up for debate. What is often missed in the discussion of recall and its relationship to intent.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- Intent is domain specific, so we want to find easier ways to model it When we model intent we want to do all the things you do with search, test it, debug it, update it, reinforce it with judgements.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- To have “expanded” recall we must model our domain, to do this we need entities and to understand the relationship between them.
- Solr, Elastic Search provide good primitives out of the box Demands of modern search applications require more layered and sophisticated primitives
- So I’ve talked about modeling entities, but why do we need to do this… can’t be get most of the way there with traditional search, hasn’t it worked fine for most people until now?
- This is a slide from Target corp presentation A lot of their searches are long-tail with poor matches Also, when their model can’t match both words within a given category, they fall-back to search for only 1 term that has the most matches. This is a great example where understanding the relationship between searches and entities can be very important.
- So I’ve talked about modeling entities, but why do we need to do this… can’t be get most of the way there with traditional search, hasn’t it worked fine for most people until now?
- Single term searches are a huge issue in job search, and interpreting what they mean is a challenge. Modeling entities in our domain helps us begin to understand the relationship between term searches and the types of documents viewed. Entity approaches can also help us understand the individual searchers affinity within ambiguous contexts.
- You can imagine a search system in which we are modeling people. This might be useful for a library or research system.
- Which brings us to traditional challenges with similarity and where it can go off the rails.
- Ontologies are a huge subject, but really what we are describing is a graph database with edges that are informed based on what we know about a particular entity. One entity may be perfectly related to another. If we were building a library data-system we might have an ontology of historical persons. The ontology might form features to tell us if the person was an inventor or politician.
- Ontologies are a huge subject, but really what we are describing is a graph database with edges that are informed based on what we know about a particular entity. One entity may be perfectly related to another. If we were building a library data-system we might have an ontology of historical persons. The ontology might form features to tell us if the person was an inventor or politician.
- A simple ontology can be constructed for job titles from query logs and reviewed / pruned by hand. Supervised ML approaches can get you pretty close to this as well.
- Conceptual relationships from phrases are easier to model than words or simpler language
- - accuracy or broadness of a search query should be related to how well we understand what’s being searched for - In the absence of high confidence search should fall back to a default algorithm.