Presented at Open Source Connections Haystack Relevance Conference on 904Labs' "Interleaving: from Evaluation to Self-Learning". 904Labs is the first to commercialize "Online Learning to Rank" as a state-of-art for technical Self-learning Search Ranking that automatically takes into account your customers human behaviors for personalized search results.
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesMax Irwin
Presentation as given to the Haystack Conference, which outlines research and techniques for automatic extraction of keywords, concepts, and vocabularies from text corpora.
his talk will feature some of my recent research into the alternative uses for Solr facets and facet metadata. I will develop the idea that facets can be used to discover similarities between items and attributes in a search index, and show some interesting applications of this idea. A common takeaway is that using facets and facet metadata in non-conventional ways enables the semantic context of a query to be automatically tuned. This has important implications for user-centric and semantically focused relevance.
Crowdsourced query augmentation through the semantic discovery of domain spec...Trey Grainger
Talk Abstract: Most work in semantic search has thus far focused upon either manually building language-specific taxonomies/ontologies or upon automatic techniques such as clustering or dimensionality reduction to discover latent semantic links within the content that is being searched. The former is very labor intensive and is hard to maintain, while the latter is prone to noise and may be hard for a human to understand or to interact with directly. We believe that the links between similar user’s queries represent a largely untapped source for discovering latent semantic relationships between search terms. The proposed system is capable of mining user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free.
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesMax Irwin
Presentation as given to the Haystack Conference, which outlines research and techniques for automatic extraction of keywords, concepts, and vocabularies from text corpora.
his talk will feature some of my recent research into the alternative uses for Solr facets and facet metadata. I will develop the idea that facets can be used to discover similarities between items and attributes in a search index, and show some interesting applications of this idea. A common takeaway is that using facets and facet metadata in non-conventional ways enables the semantic context of a query to be automatically tuned. This has important implications for user-centric and semantically focused relevance.
Crowdsourced query augmentation through the semantic discovery of domain spec...Trey Grainger
Talk Abstract: Most work in semantic search has thus far focused upon either manually building language-specific taxonomies/ontologies or upon automatic techniques such as clustering or dimensionality reduction to discover latent semantic links within the content that is being searched. The former is very labor intensive and is hard to maintain, while the latter is prone to noise and may be hard for a human to understand or to interact with directly. We believe that the links between similar user’s queries represent a largely untapped source for discovering latent semantic relationships between search terms. The proposed system is capable of mining user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free.
Dice.com Bay Area Search - Beyond Learning to Rank TalkSimon Hughes
This talk describes how to implement conceptual search (semantic search) within a modern search engine using the word2vec algorithm to learn concepts. We also cover how to auto-tune the search engine parameters using black box optimization techniques, and the problems of feedback loops encountered when building machine learning systems that modify the user behavior used to train the system.
Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...Trey Grainger
Search engines frequently miss the mark when it comes to understanding user intent. This talk will walk through some of the key building blocks necessary to turn a search engine into a dynamically-learning "intent engine", able to interpret and search on meaning, not just keywords. We will walk through CareerBuilder's semantic search architecture, including semantic autocomplete, query and document interpretation, probabilistic query parsing, automatic taxonomy discovery, keyword disambiguation, and personalization based upon user context/behavior. We will also see how to leverage an inverted index (Lucene/Solr) as a knowledge graph that can be used as a dynamic ontology to extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships.
As an example, most search engines completely miss the mark at parsing a query like (Senior Java Developer Portland, OR Hadoop). We will show how to dynamically understand that "senior" designates an experience level, that "java developer" is a job title related to "software engineering", that "portland, or" is a city with a specific geographical boundary (as opposed to a keyword followed by a boolean operator), and that "hadoop" is the skill "Apache Hadoop", which is also related to other terms like "hbase", "hive", and "map/reduce". We will discuss how to train the search engine to parse the query into this intended understanding and how to reflect this understanding to the end user to provide an insightful, augmented search experience.
Topics: Semantic Search, Apache Solr, Finite State Transducers, Probabilistic Query Parsing, Bayes Theorem, Augmented Search, Recommendations, Query Disambiguation, NLP, Knowledge Graphs
Reflected Intelligence: Lucene/Solr as a self-learning data systemTrey Grainger
What if your search engine could automatically tune its own domain-specific relevancy model? What if it could learn the important phrases and topics within your domain, automatically identify alternate spellings (synonyms, acronyms, and related phrases) and disambiguate multiple meanings of those phrases, learn the conceptual relationships embedded within your documents, and even use machine-learned ranking to discover the relative importance of different features and then automatically optimize its own ranking algorithms for your domain?
In this presentation, you’ll learn you how to do just that - to evolving Lucene/Solr implementations into self-learning data systems which are able to accept user queries, deliver relevance-ranked results, and automatically learn from your users’ subsequent interactions to continually deliver a more relevant experience for each keyword, category, and group of users.
Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the relevance signals present in the collective feedback from every prior user interaction with the system. Come learn how to move beyond manual relevancy tuning and toward a closed-loop system leveraging both the embedded meaning within your content and the wisdom of the crowds to automatically generate search relevancy algorithms optimized for your domain.
South Big Data Hub: Text Data Analysis PanelTrey Grainger
Slides from Trey's opening presentation for the South Big Data Hub's Text Data Analysis Panel on December 8th, 2016. Trey provided a quick introduction to Apache Solr, described how companies are using Solr to power relevant search in industry, and provided a glimpse on where the industry is heading with regard to implementing more intelligent and relevant semantic search.
Enhancing relevancy through personalization & semantic searchTrey Grainger
Matching keywords is just step one in the effort to maximize the relevancy of your search platform. In this talk, you'll learn how to implement advanced relevancy techniques which enable your search platform to "learn" from your content and users' behavior. Topics will include automatic synonym discovery, latent semantic indexing, payload scoring, document-to-document searching, foreground vs. background corpus analysis for interesting term extraction, collaborative filtering, and mining user behavior to drive geographically and conceptually personalized search results. You'll learn how CareerBuilder has enhanced Solr (also utilizing Hadoop) to dynamically discover relationships between data and behavior, and how you can implement similar techniques to greatly enhance the relevancy of your search platform.
Leveraging Lucene/Solr as a Knowledge Graph and Intent EngineTrey Grainger
Search engines frequently miss the mark when it comes to understanding user intent. This talk will describe how to overcome this by leveraging Lucene/Solr to power a knowledge graph that can extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships. For example, if a user types in (Senior Java Developer Portland, OR Hadoop), you or I know that the term “senior” designates an experience level, that “java developer” is a job title related to “software engineering”, that “portland, or” is a city with a specific geographical boundary, and that “hadoop” is a technology related to terms like “hbase”, “hive”, and “map/reduce”. Out of the box, however, most search engines just parse this query as text:((senior AND java AND developer AND portland) OR (hadoop)), which is not at all what the user intended. We will discuss how to train the search engine to parse the query into this intended understanding, and how to reflect this understanding to the end user to provide an insightful, augmented search experience. Topics: Semantic Search, Finite State Transducers, Probabilistic Parsing, Bayes Theorem, Augmented Search, Recommendations, NLP, Knowledge Graphs
Lucene/Solr Revolution 2015: Where Search Meets Machine LearningJoaquin Delgado PhD.
Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.
Better Search Through Query Understanding
Presented as a Data Talk at Intuit on April 22, 2014
Search is a fundamental problem of our time — we use search engines daily to satisfy a variety of personal and professional information needs. But search engine development still feels stuck in an information retrieval paradigm that focuses on result ranking. In this talk, I’ll advocate an emphasis on query understanding. I’ll talk about how we implement query understanding at LinkedIn, and I’ll present examples from the broader web. Hopefully you’ll come out with a different perspective on search and share my appreciation for how we can improve search through query understanding.
About the Speaker
Daniel Tunkelang leads LinkedIn's efforts around query understanding. Before that, he led LinkedIn's product data science team. He previously led a local search quality team at Google and was a founding employee of Endeca (acquired by Oracle in 2011). He has written a textbook on faceted search, and is a recognized advocate of human-computer interaction and information retrieval (HCIR). He has a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.
Vectors in Search - Towards More Semantic MatchingSimon Hughes
With the advent of deep learning and algorithms like word2vec and doc2vec, vectors-based representations are increasingly being used in search to represent anything from documents to images and products. However, search engines work with documents made of tokens, and not vectors, and are typically not designed for fast vector matching out of the box. In this talk, I will give an overview of how vectors can be derived from documents to produce a semantic representation of a document that can be used to implement semantic / conceptual search without hurting performance. I will then I will describe a few different techniques for efficiently searching vector-based representations in an inverted index, such as learning sparse representations of vectors, clustering, and learning binary vectors. Finally, I will discuss some of the pitfalls of vector-based search, and how to get the best of both worlds by combining vector-based scoring with traditional relevancy metrics such as BM25.
The Apache Solr Semantic Knowledge GraphTrey Grainger
What if instead of a query returning documents, you could alternatively return other keywords most related to the query: i.e. given a search for "data science", return me back results like "machine learning", "predictive modeling", "artificial neural networks", etc.? Solr’s Semantic Knowledge Graph does just that. It leverages the inverted index to automatically model the significance of relationships between every term in the inverted index (even across multiple fields) allowing real-time traversal and ranking of any relationship within your documents. Use cases for the Semantic Knowledge Graph include disambiguation of multiple meanings of terms (does "driver" mean truck driver, printer driver, a type of golf club, etc.), searching on vectors of related keywords to form a conceptual search (versus just a text match), powering recommendation algorithms, ranking lists of keywords based upon conceptual cohesion to reduce noise, summarizing documents by extracting their most significant terms, and numerous other applications involving anomaly detection, significance/relationship discovery, and semantic search. In this talk, we'll do a deep dive into the internals of how the Semantic Knowledge Graph works and will walk you through how to get up and running with an example dataset to explore the meaningful relationships hidden within your data.
Intent Algorithms: The Data Science of Smart Information Retrieval SystemsTrey Grainger
Search engines, recommendation systems, advertising networks, and even data analytics tools all share the same end goal - to deliver the most relevant information possible to meet a given information need (usually in real-time). Perfecting these systems requires algorithms which can build a deep understanding of the domains represented by the underlying data, understand the nuanced ways in which words and phrases should be parsed and interpreted within different contexts, score the relationships between arbitrary phrases and concepts, continually learn from users' context and interactions to make the system smarter, and generate custom models of personalized tastes for each user of the system.
In this talk, we'll dive into both the philosophical questions associated with such systems ("how do you accurately represent and interpret the meaning of words?", "How do you prevent filter bubbles?", etc.), as well as look at practical examples of how these systems have been successfully implemented in production systems combining a variety of available commercial and open source components (inverted indexes, entity extraction, similarity scoring and machine-learned ranking, auto-generated knowledge graphs, phrase interpretation and concept expansion, etc.).
Building Search & Recommendation EnginesTrey Grainger
In this talk, you'll learn how to build your own search and recommendation engine based on the open source Apache Lucene/Solr project. We'll dive into some of the data science behind how search engines work, covering multi-lingual text analysis, natural language processing, relevancy ranking algorithms, knowledge graphs, reflected intelligence, collaborative filtering, and other machine learning techniques used to drive relevant results for free-text queries. We'll also demonstrate how to build a recommendation engine leveraging the same platform and techniques that power search for most of the world's top companies. You'll walk away from this presentation with the toolbox you need to go and implement your very own search-based product using your own data.
This presentation introduces text analytics, its applications and various tools/algorithms used for this process. Given below are some of the important tools:
- Decision trees
- SVM
- Naive-Bayes
- K-nearest neighbours
- Artificial Neural Networks
- Fuzzy C-Means
- Latent Dirichlet Allocation
Building multi billion ( dollars, users, documents ) search engines on open ...Andrei Lopatenko
How to use open source technologies to build search engines for billions of users, billions of revenue, billions of documents
Keynote talk at The 16th International Conference on Open Source Systems.
Dice.com Bay Area Search - Beyond Learning to Rank TalkSimon Hughes
This talk describes how to implement conceptual search (semantic search) within a modern search engine using the word2vec algorithm to learn concepts. We also cover how to auto-tune the search engine parameters using black box optimization techniques, and the problems of feedback loops encountered when building machine learning systems that modify the user behavior used to train the system.
Searching on Intent: Knowledge Graphs, Personalization, and Contextual Disamb...Trey Grainger
Search engines frequently miss the mark when it comes to understanding user intent. This talk will walk through some of the key building blocks necessary to turn a search engine into a dynamically-learning "intent engine", able to interpret and search on meaning, not just keywords. We will walk through CareerBuilder's semantic search architecture, including semantic autocomplete, query and document interpretation, probabilistic query parsing, automatic taxonomy discovery, keyword disambiguation, and personalization based upon user context/behavior. We will also see how to leverage an inverted index (Lucene/Solr) as a knowledge graph that can be used as a dynamic ontology to extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships.
As an example, most search engines completely miss the mark at parsing a query like (Senior Java Developer Portland, OR Hadoop). We will show how to dynamically understand that "senior" designates an experience level, that "java developer" is a job title related to "software engineering", that "portland, or" is a city with a specific geographical boundary (as opposed to a keyword followed by a boolean operator), and that "hadoop" is the skill "Apache Hadoop", which is also related to other terms like "hbase", "hive", and "map/reduce". We will discuss how to train the search engine to parse the query into this intended understanding and how to reflect this understanding to the end user to provide an insightful, augmented search experience.
Topics: Semantic Search, Apache Solr, Finite State Transducers, Probabilistic Query Parsing, Bayes Theorem, Augmented Search, Recommendations, Query Disambiguation, NLP, Knowledge Graphs
Reflected Intelligence: Lucene/Solr as a self-learning data systemTrey Grainger
What if your search engine could automatically tune its own domain-specific relevancy model? What if it could learn the important phrases and topics within your domain, automatically identify alternate spellings (synonyms, acronyms, and related phrases) and disambiguate multiple meanings of those phrases, learn the conceptual relationships embedded within your documents, and even use machine-learned ranking to discover the relative importance of different features and then automatically optimize its own ranking algorithms for your domain?
In this presentation, you’ll learn you how to do just that - to evolving Lucene/Solr implementations into self-learning data systems which are able to accept user queries, deliver relevance-ranked results, and automatically learn from your users’ subsequent interactions to continually deliver a more relevant experience for each keyword, category, and group of users.
Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the relevance signals present in the collective feedback from every prior user interaction with the system. Come learn how to move beyond manual relevancy tuning and toward a closed-loop system leveraging both the embedded meaning within your content and the wisdom of the crowds to automatically generate search relevancy algorithms optimized for your domain.
South Big Data Hub: Text Data Analysis PanelTrey Grainger
Slides from Trey's opening presentation for the South Big Data Hub's Text Data Analysis Panel on December 8th, 2016. Trey provided a quick introduction to Apache Solr, described how companies are using Solr to power relevant search in industry, and provided a glimpse on where the industry is heading with regard to implementing more intelligent and relevant semantic search.
Enhancing relevancy through personalization & semantic searchTrey Grainger
Matching keywords is just step one in the effort to maximize the relevancy of your search platform. In this talk, you'll learn how to implement advanced relevancy techniques which enable your search platform to "learn" from your content and users' behavior. Topics will include automatic synonym discovery, latent semantic indexing, payload scoring, document-to-document searching, foreground vs. background corpus analysis for interesting term extraction, collaborative filtering, and mining user behavior to drive geographically and conceptually personalized search results. You'll learn how CareerBuilder has enhanced Solr (also utilizing Hadoop) to dynamically discover relationships between data and behavior, and how you can implement similar techniques to greatly enhance the relevancy of your search platform.
Leveraging Lucene/Solr as a Knowledge Graph and Intent EngineTrey Grainger
Search engines frequently miss the mark when it comes to understanding user intent. This talk will describe how to overcome this by leveraging Lucene/Solr to power a knowledge graph that can extract phrases, understand and weight the semantic relationships between those phrases and known entities, and expand the query to include those additional conceptual relationships. For example, if a user types in (Senior Java Developer Portland, OR Hadoop), you or I know that the term “senior” designates an experience level, that “java developer” is a job title related to “software engineering”, that “portland, or” is a city with a specific geographical boundary, and that “hadoop” is a technology related to terms like “hbase”, “hive”, and “map/reduce”. Out of the box, however, most search engines just parse this query as text:((senior AND java AND developer AND portland) OR (hadoop)), which is not at all what the user intended. We will discuss how to train the search engine to parse the query into this intended understanding, and how to reflect this understanding to the end user to provide an insightful, augmented search experience. Topics: Semantic Search, Finite State Transducers, Probabilistic Parsing, Bayes Theorem, Augmented Search, Recommendations, NLP, Knowledge Graphs
Lucene/Solr Revolution 2015: Where Search Meets Machine LearningJoaquin Delgado PhD.
Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.
Better Search Through Query Understanding
Presented as a Data Talk at Intuit on April 22, 2014
Search is a fundamental problem of our time — we use search engines daily to satisfy a variety of personal and professional information needs. But search engine development still feels stuck in an information retrieval paradigm that focuses on result ranking. In this talk, I’ll advocate an emphasis on query understanding. I’ll talk about how we implement query understanding at LinkedIn, and I’ll present examples from the broader web. Hopefully you’ll come out with a different perspective on search and share my appreciation for how we can improve search through query understanding.
About the Speaker
Daniel Tunkelang leads LinkedIn's efforts around query understanding. Before that, he led LinkedIn's product data science team. He previously led a local search quality team at Google and was a founding employee of Endeca (acquired by Oracle in 2011). He has written a textbook on faceted search, and is a recognized advocate of human-computer interaction and information retrieval (HCIR). He has a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.
Vectors in Search - Towards More Semantic MatchingSimon Hughes
With the advent of deep learning and algorithms like word2vec and doc2vec, vectors-based representations are increasingly being used in search to represent anything from documents to images and products. However, search engines work with documents made of tokens, and not vectors, and are typically not designed for fast vector matching out of the box. In this talk, I will give an overview of how vectors can be derived from documents to produce a semantic representation of a document that can be used to implement semantic / conceptual search without hurting performance. I will then I will describe a few different techniques for efficiently searching vector-based representations in an inverted index, such as learning sparse representations of vectors, clustering, and learning binary vectors. Finally, I will discuss some of the pitfalls of vector-based search, and how to get the best of both worlds by combining vector-based scoring with traditional relevancy metrics such as BM25.
The Apache Solr Semantic Knowledge GraphTrey Grainger
What if instead of a query returning documents, you could alternatively return other keywords most related to the query: i.e. given a search for "data science", return me back results like "machine learning", "predictive modeling", "artificial neural networks", etc.? Solr’s Semantic Knowledge Graph does just that. It leverages the inverted index to automatically model the significance of relationships between every term in the inverted index (even across multiple fields) allowing real-time traversal and ranking of any relationship within your documents. Use cases for the Semantic Knowledge Graph include disambiguation of multiple meanings of terms (does "driver" mean truck driver, printer driver, a type of golf club, etc.), searching on vectors of related keywords to form a conceptual search (versus just a text match), powering recommendation algorithms, ranking lists of keywords based upon conceptual cohesion to reduce noise, summarizing documents by extracting their most significant terms, and numerous other applications involving anomaly detection, significance/relationship discovery, and semantic search. In this talk, we'll do a deep dive into the internals of how the Semantic Knowledge Graph works and will walk you through how to get up and running with an example dataset to explore the meaningful relationships hidden within your data.
Intent Algorithms: The Data Science of Smart Information Retrieval SystemsTrey Grainger
Search engines, recommendation systems, advertising networks, and even data analytics tools all share the same end goal - to deliver the most relevant information possible to meet a given information need (usually in real-time). Perfecting these systems requires algorithms which can build a deep understanding of the domains represented by the underlying data, understand the nuanced ways in which words and phrases should be parsed and interpreted within different contexts, score the relationships between arbitrary phrases and concepts, continually learn from users' context and interactions to make the system smarter, and generate custom models of personalized tastes for each user of the system.
In this talk, we'll dive into both the philosophical questions associated with such systems ("how do you accurately represent and interpret the meaning of words?", "How do you prevent filter bubbles?", etc.), as well as look at practical examples of how these systems have been successfully implemented in production systems combining a variety of available commercial and open source components (inverted indexes, entity extraction, similarity scoring and machine-learned ranking, auto-generated knowledge graphs, phrase interpretation and concept expansion, etc.).
Building Search & Recommendation EnginesTrey Grainger
In this talk, you'll learn how to build your own search and recommendation engine based on the open source Apache Lucene/Solr project. We'll dive into some of the data science behind how search engines work, covering multi-lingual text analysis, natural language processing, relevancy ranking algorithms, knowledge graphs, reflected intelligence, collaborative filtering, and other machine learning techniques used to drive relevant results for free-text queries. We'll also demonstrate how to build a recommendation engine leveraging the same platform and techniques that power search for most of the world's top companies. You'll walk away from this presentation with the toolbox you need to go and implement your very own search-based product using your own data.
This presentation introduces text analytics, its applications and various tools/algorithms used for this process. Given below are some of the important tools:
- Decision trees
- SVM
- Naive-Bayes
- K-nearest neighbours
- Artificial Neural Networks
- Fuzzy C-Means
- Latent Dirichlet Allocation
Building multi billion ( dollars, users, documents ) search engines on open ...Andrei Lopatenko
How to use open source technologies to build search engines for billions of users, billions of revenue, billions of documents
Keynote talk at The 16th International Conference on Open Source Systems.
Recommendation Engines are everywhere these days, telling us which products to buy on Amazon, which movies to watch on Netflix, which courses to take on Coursera, and on and on. This presentation is a description of the collaborative filtering and content-based recommendation engines at Jane.com, Inc magazine's fastest-growing e-commerce company of 2015.
professional fuzzy type-ahead rummage around in xml type-ahead search techni...Kumar Goud
Abstract – It is a research venture on the new information-access standard called type-ahead search, in which systems discover responds to a keyword query on-the-fly as users type in the uncertainty. In this paper we learn how to support fuzzy type-ahead search in XML. Underneath fuzzy search is important when users have limited knowledge about the exact representation of the entities they are looking for, such as people records in an online directory. We have developed and deployed several such systems, some of which have been used by many people on a daily basis. The systems received overwhelmingly positive feedbacks from users due to their friendly interfaces with the fuzzy-search feature. We describe the design and implementation of the systems, and demonstrate several such systems. We show that our efficient techniques can indeed allow this search paradigm to scale on large amounts of data.
Index Terms - type-ahead, large data set, server side, online directory, search technique.
Big Data Meets Learning Science: Keynote by Al EssaSpark Summit
How do we learn and how can we learn better? Educational technology is undergoing a revolution fueled by learning science and data science. The promise is to make a high-quality personalized education accessible and affordable by all. In this presentation Alfred will describe how Apache Spark and Databricks are at the center of the innovation pipeline at McGraw Hill for developing next-generation learner models and algorithms in support of millions of learners and instructors worldwide.
The Searchmaster's Toolbox - David Hawking, Funnelback SearchSquiz
David Hawking, pre-eminent information retrieval researcher and Funnelback's Chief Scientist, gave this talk on the need for a Search Master within all but the smallest organisations at a Funnelback Seminar in London on March 31st, 2010. Even if there isn't an individual with that specific job title, the responsibility for maintaining, improving and monitoring search needs to be prioritised and clearly assigned. David's presentation covers the reasons why search is so vitally important and the tools which can improve search results.
Radical Roadmapping - Creating Synchronized Agile Product and Technology Road...Matt Roberts
This session will discuss why a company would create and maintain three major artifacts - Innovation Roadmap, Infrastructure/ Platform Roadmap, and Operations/DevOps Roadmap - as well as the process to do so. Further, it will cover how to synchronize them in order to move away from making "OR" decisions to making "AND" decisions that will please all stakeholders. It will also discuss key cultural changes that must be present in order to achieve maximum benefit from this approach and challenges experienced along the way to making this a reality at Socialware, a SaaS product company. Finally, this session will include real world examples of the evolution of these roadmaps over 18 months that participants can take away and use as guidelines for their own situations.
This was presented at the Keep Austin Agile 2016 conference, the #AgileAustin Product SIG and Product Camp Austin 2016 (#PCATX17)
This is a presentation I delivered at Enterprise Data World 2018 to make the case for developing intelligent systems using a hybrid or blended approach combining statistical-based machine learning with knowledge-based approaches that involve ontologies, taxonomies or knowledge graphs.
The majority of SharePoint migration planning has little to do with the technical move, but is more about information architecture, data transformation, and other PM and BA skills. This presentation outlines 5 key areas of planning.
The need for sophistication in modern search engine implementationsBen DeMott
The need for more sophisticated search implementations is often at odds with the limited feature set available in modern out of the box open source search engines.
This presentation discusses the challenges associated with properly modeling information within a domain and why it's critically needed.
Polyglot Persistence vs Multi-Model DatabasesLuca Garulli
Many complex applications scale up by using several different databases, i.e. selecting the best DBMS for each use case. This tends to complicate modern architecture with many products by different vendors, no standards, and a lot of ETL which ultimately causes unpredictable results and a lot of headaches. Multi-Model DBMSs were created to make your life easier, giving you the option of using one NoSQL product with powerful multi-purpose engines capable of handling complex domains. Could one DBMS handle all your needs including speed and scalability in the times of Big Data? Luca will walk you through the benefits and trade-offs of multi-model DBMSs and will show you how easy it is to setup one open source database to handle many different use cases, saving you time and money.
Presented at Data Day Texas - Austin (TX) - USA
Similar to Interleaving, Evaluation to Self-learning Search @904Labs (20)
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
2. About myself and 904Labs
I’ve been in the search field for 15+ years starting with SQL Server Full-text
Search (FTS) in 1998 with roles in Tech. Support, Sales Engineering (FAST) &
Product Manager roles at HP, Lucidworks (Fusion 1.0) and recently at HPE.
While I currently work for Voyager Search, I’m at Haystack representing
904Labs.
904Labs is a Dutch search company founded by Manos Tsagkias and Wouter
Weerkamp, two former academic researchers in the field of Information
Retrieval. The company offers Online Learning to Rank as-a-Service
(OLtRaaS)
3. For decades people tried to come up with clever ways to model
“relevance”. In the early 70s, TF-IDF was introduced, relying on counting
word overlap between queries and documents. (main use case: early digital
library / card catalog)
In early 80’s, researchers came up with BM25 (used in early SharePoint
Search 2001), a parameterized version of TF-IDF. It wasn’t until 2015 that
Lucene/Solr changed it’s default ranking function to BM25.
So, today’s standard search relevance uses 40 year old ranking functions.
How to determine relevance?
5. Enter machine learning
Since a couple of years people have started to realize that search,
or modeling relevance, has become too complex to fit in BM25. A
paradigm shift is taking place, moving into the direction of
learning the ranking function from training data.
This paradigm shift is translated into learning to rank plugins for
Apache Solr and Elasticsearch, and is also apparent from the
many talks at Haystack about learning to rank.
6. Learning to rank is a batch process. Training data is collected,
features are extracted, and a model is trained using an objective
function. Every couple of hours/days/weeks, this process is repeated
and a new model is trained. This requires heavy data processing
infrastructure + required software + expert personnel to run.
7. So, what’s next?
Reinforcement learning: don’t retrain, but update the existing
model in real time using feedback on the ranking produced by the
current model. Think of this as stage 2 of Paradigm Shift
No need to retrain, no need for batch data processing. This
allows for us to easily launch new features, weights are learned
on the fly (or online) & this allows us to adapts to changing user
behavior almost immediately (in real time).
8. Online learning to rank uses a pre-trained model to generate an initial
ranking. The user interacts with the ranking, giving (implicit) feedback
on its quality. This feedback is used to update the current model, and
the updated model then becomes the active model. And repeat...
13. Interleaving
for evaluation (recap 1)
Two competing search Engines, A and B
1) Both generate results for the same query
2) Results are then interleaved into one final result list
3) The final result list is shown to the user
4) Clicks on results are mapped to the originating search
engine
5) Winner is the search engine that receives most clicks
14. Interleaving
for evaluation (recap 2)
Fast and low-risk evaluation method for algorithmic changes,
esp. compared to A/B test. It is... Always ongoing &
...faster because every user evaluates both search engines at the
same time.
...low-risk because every user always sees several results from
the current search engine, which has a known quality.
15. Interleaving
for online learning
Interleaving is about identifying the winning search engine in a
competition. We can run a competition with every query to get a
continuous learning cycle. (think Ranking Models in one Search Engine)
Search Model B is always a slight adaptation of the current model. In
case B wins the competition, the original model (A) is updated into the
direction of B. The updated model becomes search engine A for the
next query, and competes with a new B.
19. ~30%
Increase in revenue and conversion rate for three of eCommerce
Search customers using online learning to rank on top of Apache Solr.
Blog posts with improvements in revenue:
https://www.904labs.com/en/blog-eci-increases-revenue-substantially-with-ai-for-search.html
and
https://www.904labs.com/en/blog-self-learning-search-improves-revenue-for-e-commerce.html
20.
21. Is 904Labs Open Source?
904Labs’ online learning to rank system is SaaS. It is implemented
on top of a client’s (or customer’s) own Apache Solr or
Elasticsearch. The data remains at the client side, and if the client
wants to move away from 904Labs, they can do so, without
extensive vendor lockin!
Many other (SaaS) search solutions provide Solr/Elasticsearch as
core part of their solution. Moving away from these solutions
leaves clients without any search infrastructure.
22. Feature engineering. Which features are readily available?
Delayed feedback. How to update the model when feedback is
delayed until after another update has already happened?
Efficiency vs. effectiveness. How to balance the number of
queries to Solr and the extent of the candidate document set?
Exploration vs exploitation. We want to exploit the current best
model, but need to explore to keep learning. What is the best
way?
Some open issues (as time allows)
23. Take home message for 904Labs
Search has moved from modeling relevance to learning from
user behavior data. The next Paradigm Shift is to learn these
models in real time, allowing immediate adaptation to changes in
user behavior and removing the necessity of large-scale data
(pre)processing for batch learning.
Many open issues remain, so expect lots of cool research on
that.
27. Pointwise: Try to predict the relevance of one document at a
time.
Pairwise: For a pair of documents, predict which is more relevant.
Listwise: Try to optimize the full ranking using existing IR
metrics.
Approaches to learning to rank
Editor's Notes
In search one of the core problems is to determine the relevance of items for a given search query. Research in this field started in the 1950s and led to the introduction of the popular TF.IDF formula in the 70s. TF.IDF is simple word counting, yet it has been the most popular ranking function for many decades. A parameterized version of it, BM25, was invented in the 70s as well. It is important to note that state-of-the-art open source search technology like Lucene used TF.IDF until 2015, and then replaced it with BM25, almost 40 years after its introduction! No wonder that out-of-the-box ranking performance of systems like Apache Solr and Elasticsearch can be improved as next the paradigm..
But things are changing!
As early as 1992, researchers started thinking about learning the parameters of ranking functions from data. From the 2000s, research in this direction really took off, and between 2000 and 2010 most of the currently used learning to rank approaches were introduced. The idea behind learning to rank is rather simple: estimating relevance is too complex a task to solve with a naive model like BM25. Many more features play a role other than word counts. To capture all these relevance feature and their relative weights, we apply machine learning to learn this from usage data and/or relevance assessments.
The paradigm shift from trying to model relevance to learning it from actual data led to the introduction of learning to rank plugins for Apache Solr and Elasticsearch, making these methods available to a large audience. The popularity of these plugins is apparent from the number of talks here at Haystack about learning to rank.
A quick reminder on what learning to rank is all about, although everyone should know by now… We give a machine training data (queries and clicked documents for example), features that describe the query and the documents, and an objective function. The machine learn a ranking model, which is put into production to generate rankings.
Learning to rank is a batch process. The pipeline of collecting training data, extracting features, and training a model is repeated every couple of hours, days, or weeks, depending on the organization. This batch processing often requires quite an impressive data processing pipeline. The question is whether we really need this batch process...
Reinforcement learning from an ML perspective? We would not ask the question if the answer would not be “no”. So, what’s the next step? Rather than periodically retrain and produce a completely new model, we can apply reinforcement learning to update the existing model in real time based on feedback. There are numerous advantages, but the most important ones are that we move away from the need to retrain our model periodically. We do not have to have large data processing pipelines to feed training data to our learner every couple of hours to train a new model, but rather take real time feedback on the quality of the current model, interpret this feedback, and update the model slightly accordingly.
Secondly, since the system is learning continuously, it is easy to introduce new features on the fly. Assigning it some initial weight, the system will quickly learn to adjust this weight according to the new feature’s importance. Thirdly, the system can quickly adapt to changes in user behavior instead of having to wait for the next iteration of batch training.
What does online learning look like on a high level? It starts out much like regular learning to rank. A model is learned from training data, features, and an objective function. The initial model is used to generate a first ranking, which is shown to the user issuing the query. This user then interacts with the ranking, providing (implicit) feedback on the quality of the ranking. This feedback is immediately taken into account by the system, updating the model slightly to match the feedback. The updated model becomes the active model, which is used to generate the next ranking, etc.
This probably all sounds very nice, but how can we make such an online learning to rank, or self-learning search system? One of the methods that can be used to power such a system is interleaving. Different between theory vs. practice.
Interleaving started out as an evaluation method for comparing ranking algorithms. Currently, however, it is also used to power self-learning search engines. I’ll explain how interleaving is used for evaluation and how this can be translated into an online learning to rank setting.
A practical offline example using mutiple google search sessions for same search terms:
Screen 1: lillistrate Search Engine A in Red & B in Blue.
Let’s say we’re running a web search engine and we want to find out whether a new ranking algorithm works better than our current algorithm. We can run an interleaving experiment to find out. Our current algorithm is search session A, the new version is search session B.
Every query that is issued on our site is fired to both versions of the search engine. In this example, the query “online learning to rank” is issued to both A and B, and both engines return a list of results.
The next step is to actually interleave the results from A and B into one final result list. For the interleaving of results we can use a variety of methods, but to simplify things, we simply assume that we pick the first result from A, followed by the first result from B, the second result from A, etc. The final interleaved result list is the ranking that is shown to the user who issued the query. To this user there is no difference between results from A or B, they all look identical.
The user interacts with the final ranking, and clicks a result. In the backend we know that this result came from A, and thus A is the winner for this mini-competition between the two engines.
So, to summarize, if we want to compare two search algorithms, A and B, we turn it into a competition. Both engines generate results for the same query, these results are interleaved into one final result list. This list is shown to the user, who clicks the results she wants. These clicks are mapped to the engine that produced the particular result, and in the end the search engine with most clicks is the winner.
Why would we want to use interleaving to compare two algorithms, and not use the more common A/B test? There’s nothing wrong with doing an A/B test, but an interleaving experiment is faster to run and is low-risk.
Interleaving is faster than an A/B test because each user evaluates both engines at the same time, unlike with an A/B test, in which a user is assigned to only one version. At the same time, because each user is shown at least some results from the current search engine (A), the experiment has a low risk, especially compared to an A/B test in which a certain percentage of users get to see only results from the new engine (B), which might lead to a bad user experience.
Now that we know how we can use interleaving to evaluate search algorithms, we can turn it into a method for self-learning search engines.
The trick to get an online learning to rank system using interleaving is to continuously run a competition between the current best model and a slight adaptation of that model, and to immediately update the current model when it is beaten by the adapted version.
In other words, we have our current model A, and take a slight adaptation of that model to be version B. When a query comes in, we run an interleaving experiment with these two versions, just like before. When clicks come in for that query, we determine the winner right away (instead of waiting for more queries to come in). In case version B wins, we update the current model A in such a way that we move into the direction of B. This updated version of A then becomes the new current model (A), and we generate a new adaptation from that model to be the new B. When a new query comes in, the process repeats itself.
As always, the devil is in the details, but these are too complex to discuss here. Examples of such details are how exactly to do the updating of the model A. There’s a lot of scientific literature on this available.
So, what does online learning to rank look like in practice? Unfortunately, we do not yet have a demo of what is happening. We do have a couple of screenshots from one of our customers and some general numbers to report. When our demo becomes available, we’ll send it around on social media etc.
On the left are the results from Apache Solr with some manual tuning for the query “case”. Results show mainly tool cases at the top.
After adding (online) learning to rank on top of Apache Solr, results become like the right hand side: suitcases have moved to the top, with a slight preference for those on sale.
Another example comparing Apache Solr with manual tuning to (online) learning to rank, now for the query “kitchen”. Results from Solr show toy kitchens and some kitchen equipment at the top.
After learning from what users really want, we see actual kitchens “naturally” moving to the top. Again, we observe a preference for kitchens that are on sale.
We have compared our online learning to rank system to manually tuned Apache Solr instances for three of our customers using A/B tests. In general, we observe increases in revenue of about 30% when using online learning to rank on top of an Apache Solr index. We observe similar improvements for conversion rates. Note that these numbers could be achieved using “batch” learning to rank as well, but without the before mentioned benefits of online learning.
Orange denotes customer’s network, purple 904Labs’. (Explain the procedure.)
This type of architecture is great for shortcutting the integration process, however, it is prone to network latency.
904Labs’ core ranking algorithms are based on scientific publications, which are available to everyone. The software, however, is not open source. 904Labs does encourage the use of open source by its clients. In fact, the implementation assumes that the client is running her own Apache Solr or Elasticsearch index for search, but would like to improve the ranking quality. 904Labs acts as a middleware between the web app and the existing search index, and uses this index for its online learning to rank algorithms. There are plenty of advantages for clients, including easy implementation, data remaining at the client, and the lack of a vendor lock-in. If a client removes 904Labs services, she’s back at the original setup including her original search index.
In this way 904Labs is a UNIQUE Search-as-a-Service providers who provide open source search engines as core part of their service. When a client wants to move away from such a solution, she also loses all data stored in the search index, as that is part of the solution. It is an open-source supported vendor lock-in :-)
There are still a lot of open issues when it comes to online learning to rank. Feature engineering is one, although it also applies to batchL2R. Which features are available and which ones could actually add something? The benefit of an online learning system is that you can easily add new features to analyze their impact.
Delayed feedback is a technical issue related to the order in which feedback comes back into the system. What to do when feedback for a particular query comes in after feedback for a more recent query has already led to an update to the model? Should we ignore the feedback or still take it into account?
Feel free to volenteer and work with 904Labs on these open issues
Efficiency is a potential problem when trying to achieve maximum effectiveness. Learning to rank assumes that there is some initial seed set of documents to rerank. Ideally, this set contains all relevant documents, but we can only select a limited seed set. How to optimize this efficiency-effectiveness trade-off?
Finally, when we have learned a model, we would like to make use of it, or exploit it. But if we would only exploit, we can not learn anything anymore. For that, we need to explore as well. How do we balance between exploiting the current best model and exploring to allow for learning?
To summarize this talk, we observe that a paradigm shift in search is happening, moving from old-fashioned ranking functions to learning from user behavior data. Current state-of-the-art uses learning to rank, but online learning to rank is on its way and offers again certain advantages on top of ltr. As is the case with learning to rank, there are still quite a few open issues for online learning to rank, so keep an eye on the research community to come up with cool new methods!
If you want to know (much) more about these details about this technology, feel free to contact Manos or Wouter directly.
If you want to know (much) more about these details, you could for example check out this tutorial on interleaving.
Two pointers for online learning to rank.
There are three approaches to learning to rank. Pointwise tries to predict the relevance of each single document. Pairwise looks at two documents and tries to determine which one is more relevant. Listwise, finally, tries to optimize the full ranking in one go, using existing IR metrics like nDCG. Going from pointwise to listwise should lead to better effectiveness, but also to a decrease in efficiency; the usual efficiency-effectiveness trade-off.