Better Search Through Query Understanding

•Download as PPTX, PDF•

44 likes•31,649 views

This document discusses query understanding in search engines. It describes how query understanding involves identifying entities and tags in queries, predicting the user's intent or topic area, expanding queries using related terms, and incorporating spelling corrections. The key aspects of query understanding covered are tagging queries for entities like names, titles, companies; predicting the user's vertical intent like jobs, people or companies; and expanding queries using name synonyms, job title synonyms or signals from past user queries and clicks. The document also suggests giving users more transparency, guidance and control over the search process.

Technology Business

Recruiting SolutionsRecruiting SolutionsRecruiting Solutions
Daniel Tunkelang
Head, Query Understanding
better search through
query understanding

overview
 query understanding: what is it?
 how we do query understanding at LinkedIn
 some other thoughts from search in the wild
what I’m not going to cover:
2

Information need query select from results
rank using IR model
user:
system:
tf-idf PageRank
bird’s-eye view of how a search engine works
3

Information need query select from results
rank using IR model
user:
system:
tf-idf PageRank
query understanding
4

6
tag: skill OR title
related skills:
search, ranking, …
tag: company
id: 1337
industry: internet
verticals:
people, jobs
intent: exploratory

query understanding pipeline
7
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

query understanding pipeline
8
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

9
fix obvious typos
help users spell names
spelling correction

spelling out the details
10
PEOPLE NAMES
COMPANIES
TITLES
PAST QUERIES
n-grams
marissa => ma ar ri is ss sa
metaphone
mark/marc => MRK
co-occurrence counts
marissa:mayer = 1000
marisa meyer yahoo
marissa
marisa
meyer
mayer
yahoo

spelling out the details
11
problem: corpus as well as query logs contain many spelling errors
certain spelling errors are quite frequent
while genuine words (especially names) might be infrequent

spelling out the details
12
problem: corpus & query logs contain spelling errors
solution: use query chains to infer correct spelling
[product manger] [product manager] CLICK
[marissa mayer] CLICK

query understanding pipeline
13
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

query tagging: identifying entities in the query
14
TITLE CO GEO
TITLE-237
software engineer
software developer
programmer
…
CO-1441
Google Inc.
Industry: Internet
GEO-7583
Country: US
Lat: 42.3482 N
Long: 75.1890 W
(RECOGNIZED TAGS: NAME, TITLE, COMPANY, SCHOOL, GEO, SKILL )

query tagging: identifying entities in the query
15
TITLE CO GEO
MORE PRECISE MATCHING WITH DOCUMENTS

query tagging: sequential model
22
EMISSION PROBABILITIES
(learned from user profiles)
TRANSITION PROBABILITIES
(learned from query logs)
TRAINING

query tagging: sequential model
23
INFERENCE
given a query, find the most likely sequence of tags

query understanding pipeline
24
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

vertical intent prediction: distribution
25
JOBS
PEOPLE
COMPANIES
(probability distribution over verticals)

vertical intent prediction: relevance
26
[company]
[employees]
[jobs]
[name search]

query understanding pipeline
27
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

30
query expansion: signals
[jon] [jonathan] CLICK
trained using query chains:
[programmer] [developer] CLICK
symmetric but not transitive!
[francis] ⇔ [frank]
[franklin] ⇔ [frank]
[francis] ≠ [franklin]
[software engineer] [software developer] CLICK
context based!
[software engineer] => [software developer]
[civil engineer] ≠ [civil developer]

query understanding pipeline
31
spellcheck
query tagging
vertical intent prediction
query expansion
raw query
structured query
+
annotations

32
what else can we learn from search in the wild?

don’t guess when it’s better to ask
33
vs.

give users transparency, guidance, and control
35

think beyond individual search queries
36
Gene Golovchinsky, FXPAL

know when you don’t know
37
Claudia Hauff, Query Difficulty for Digital Libraries [2009]

38
Daniel Tunkelang
dtunkelang@linkedin.com
https://linkedin.com/in/dtunkelang

Find and Be Found: Information Retrieval at LinkedIn SIGIR 2013 Industry Track Presentation http://sigir2013.ie/industry_track.html LinkedIn has a unique data collection: the 200M+ members who use LinkedIn are also the most valuable entities in our corpus, which consists of people, companies, jobs, and a rich content ecosystem. Our members use LinkedIn to satisfy a diverse set of navigational and exploratory information needs, which we address by leveraging semi-structured and social content to understanding their query intent and deliver a personalized search experience. In this talk, we will discuss some of the unique challenges we face in building the LinkedIn search platform, the solutions we've developed so far, and the open problems we see ahead of us. Shakti Sinha heads LinkedIn's search relevance team, and has been making key contributions to LinkedIn's search products since 2010. He previously worked at Google as both a research intern and a software engineer. He has an MS in Computer Science from Stanford, as well as a BS degree from College of Engineering, Pune. Daniel Tunkelang leads LinkedIn's efforts around query understanding. Before that, he led LinkedIn's product data science team. He previously led a local search quality team at Google and was a founding employee of Endeca (acquired by Oracle in 2011). He has written a textbook on faceted search, and is a recognized advocate of human-computer interaction and information retrieval (HCIR). He has a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.

Instant search - A hands-on tutorial

Ganesh Venkataraman

Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...

Qi Guo

*** Please check out our LinkedIn Engineering blog post: https://engineering.linkedin.com/blog/2019/04/ai-behind-linkedin-recruiter-search-and-recommendation-systems *** LinkedIn Talent Solutions business contributes to around 65% of LinkedIn’s annual revenue, and provides tools for job providers to reach out to potential candidates and for job seekers to find suitable career opportunities. LinkedIn’s job ecosystem has been designed as a platform to connect job providers and job seekers, and to serve as a marketplace for efficient matching between potential candidates and job openings. A key mechanism to help achieve these goals is the LinkedIn Recruiter product, which enables recruiters to search for relevant candidates and obtain candidate recommendations for their job postings. We highlight a few unique information retrieval, system, and modeling challenges associated with talent search and recommendation systems. In this talk, we will present how we formulated and addressed the problems, the overall system design and architecture, the challenges encountered in practice, and the lessons learned from the production deployment of these systems at LinkedIn. By presenting our experiences of applying techniques at the intersection of recommender systems, information retrieval, machine learning, and statistical modeling in a large-scale industrial setting and highlighting the open problems, we hope to stimulate further research and collaborations within the SIGIR community.

How to Build your Training Set for a Learning To Rank Project

Sease

Learning to rank (LTR from now on) is the application of machine learning techniques, typically supervised, in the formulation of ranking models for information retrieval systems. With LTR becoming more and more popular (Apache Solr supports it from Jan 2017), organisations struggle with the problem of how to collect and structure relevance signals necessary to train their ranking models. This talk is a technical guide to explore and master various techniques to generate your training set(s) correctly and efficiently. Expect to learn how to : – model and collect the necessary feedback from the users (implicit or explicit) – calculate for each training sample a relevance label which is meaningful and not ambiguous (Click Through Rate, Sales Rate …) – transform the raw data collected in an effective training set (in the numerical vector format most of the LTR training library expect) Join us as we explore real world scenarios and dos and don’ts from the e-commerce industry.

Anatomy of an eCommerce Search Engine by Mayur Datar

Naresh Jain

Feed at LinkedIn (Quora Talk)

Shubham Gupta

E-commerce Query Tagging System Using Unsupervised Training Methods: Amazon is one of the world’s largest e-commerce sites and Amazon Search powers the majority of Amazon’s sales. A key component of Amazon Search is the query understanding pipeline, which extracts appropriate semantic information used to precisely display products for billions of queries everyday. In this talk, we will go through the primary building blocks of query understanding pipeline. Amazon Search enables users to search against structured products, hence it is necessary to extract information from queries in a format that is consistent with the structured information about the products. Query tagging is the task of semantically annotating query terms to pre-defined labels (such as brand, product-type and color). We propose a scalable system to train large-scale machine learning algorithms to solve this problem. Our system improved the precision over baseline, which is a dictionary lookup based tagger, by 10% and approximately doubled the recall.

Activity Ranking in LinkedIn FeedBodla Kumar

[CXL Live 16] How to Utilize Your Test Capacity? by Ton Wesseling

CXL

Learning to Rank Datasets for Search with Oscar Castaneda

Databricks

Learning to rank methods automatically learn from user interaction instead of relying on labeled data prepared manually. Learning to rank, also referred to as machine-learned ranking, is an application of reinforcement learning concerned with building ranking models for information retrieval. Learning to rank has been successfully applied in building intelligent search engines, but has yet to show up in dataset search. Dataset search is ripe for innovation with learning to rank specifically by automating the process of index construction. Oscar will recap previous presentations on dataset search and introduce learning to rank as a way to automate relevance scoring of dataset search results. He will also give a demo of a dataset search engine that makes use of an automatically constructed index using learning to rank on Elasticsearch and Spark. Oscar will explain the motivation and use case of learning to rank in dataset search focusing on why it is interesting to rank datasets through machine-learned relevance scoring and how to improve indexing efficiency by tapping into user interaction data from clicks. Dataset Search and Learning to Rank are IR and ML topics that should be of interest to Spark Summit attendees who are looking for use cases and new opportunities to organize and rank Datasets in Data Lakes to make them searchable and relevant to users. In preparation for this talk it is recommend that attendees watch previous two talks on dataset search from prior Spark Summit events as they build up to the present talk: [1] https://spark-summit.org/east-2017/events/building-a-dataset-search-engine-with-spark-and-elasticsearch/ [2] https://spark-summit.org/eu-2016/events/spark-cluster-with-elasticsearch-inside/

JoomlaDay Conference_September 2023 PDF.pdf

Oliver Brett

GTC 2021: Counterfactual Learning to Rank in E-commerce

GrubhubTech

Many ecommerce companies have extensive logs of user behavior such as clicks and conversions. However, if supervised learning is naively applied, then systems can suffer from poor performance due to bias and feedback loops. Using techniques from counterfactual learning we can leverage log data in a principled manner in order to model user behaviour and build personalized recommender systems. At Grubhub, a user journey begins with recommendations and the vast majority of conversions are powered by recommendations. Our recommender policies can drive user behavior to increase orders and/or profit. Accordingly, the ability to rapidly iterate and experiment is very important. Because of our powerful GPU workflows, we can iterate 200% more rapidly than with counterpart CPU workflows. Developers iterate ideas with notebooks powered by GPUs. Hyperparameter spaces are explored up to 8x faster with multi-GPUs Ray clusters. Solutions are shipped from notebooks to production in half the time with nbdev. With our accelerated DS workflows and Deep Learning on GPUs, we were able to deliver a +12.6% conversion boost in just a few months. In this talk we hope to present modern techniques for industrial recommender systems powered by GPU workflows. First a small background on counterfactual learning techniques, then followed by practical information and data from our industrial application. By Alex Egg, accepted to Nvidia GTC 2021 Conference

Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...

Sease

RRE is an open-source search quality evaluation tool that can be used to produce a set of reports about the quality of a system, iteration after iteration, and that can be integrated within a continuous integration infrastructure to monitor quality metrics after each release. Many aspects remained problematic though: – how to directly evaluate a middle layer search-API that communicates with Apache Solr or Elasticsearch? – how to easily generate explicit and implicit ratings without spending hours on tedious json files? – how to better explore the evaluation results? with nice widgets and interesting insights? Rated Ranking Evaluator Enterprise solves these problems and much more. Join us as we introduce the next generation of open-source search quality evaluation tools, exploring the internals and real-world scenarios!

Applied Machine Learning for Ranking Products in an Ecommerce Setting

Databricks

As a leading e-commerce company in fashion in the Netherlands, Wehkamp dedicates itself to provide a better shopping experience for the customers. Using Spark, the data science team is able to develop various machine-learning projects for this purpose based on the large scale data of products and customers. A major topic for the data science team is ranking products. If a visitor enters a search phrase, what are the best products that fit the search phrase and in what order should the products been shown? Ranking products is also important if a visitor enters a product overview page, where hundreds or even thousands of products of a certain article type are displayed. In this project, Spark is used in the whole pipeline: retrieving and processing the search phrases and their results, making click models, creating feature sets, training and evaluating ranking models, pushing the models to production using ElasticSearch and creating Tableau dashboarding. In this talk, we are going to demonstrate how we use Spark to build up the whole pipeline of ranking products and the challenges we faced along the way.

The step by step guide to SEO Website Audit

amandacerry

Recommendation at Netflix Scale

Justin Basilico

Building a Recommendation Engine - An example of a product recommendation engineNYC Predictive Analytics

Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020

Dawn Anderson MSc DigM

Ranking in Google Since The Advent of The Knowledge Graph

Bill Slawski

A Two Person Panel Discussion/Presentation by Bill Slawski and Barbara Starr On June 23, 2015 The Lotico Semantic Web of San Diego The SEO San Diego Meetup The SEM San Diego Meetup http://www.meetup.com/InternetMarketingSanDiego/events/222788495/ User experience drives search engines, and hence their results. Search Engine Result Presentation/Placements naturally follow that route. This means that search results are no longer exclusively based on just ranking criteria. Amongst other critical factors is understanding the notion of 'ordering vs ranking', the impact of context and many others.

Recommender system algorithm and architectureLiang Xiang

Semantic search Bill Slawski DEEP SEA Con

Bill Slawski

Website and seo audit checklist

360 Digi Wolrd - Digital Marketing Services

Search@flipkart

Umesh Prasad

Learn to Rank search results

Ganesh Venkataraman

Natural Language Processing and Search Intent Understanding C3 Conductor 2019...

Dawn Anderson MSc DigM

Deep Learning for Semantic Search in E-commerce

Somnath Banerjee

Learn how deep learning is used in incorporating semantic understanding to solve the complex and challenging problem of e-commerce search. Get informed about the deep learning-based query understanding, image understanding and embedding generation systems developed at Walmart Labs. Gain insights on several practical aspects of building and deploying DL models on production to serve large scale live traffic.

Hr analytics and graphs : job recommendations

Linkurious

Next generation linked in talent search

Ryan Wu

What's hot

Tanvi Motwani, Lead Data Scientist, Guided Search at A9.com at MLconf ATL 2016

MLconf

Activity Ranking in LinkedIn FeedBodla Kumar

[CXL Live 16] How to Utilize Your Test Capacity? by Ton Wesseling

CXL

Learning to Rank Datasets for Search with Oscar Castaneda

Databricks

JoomlaDay Conference_September 2023 PDF.pdf

Oliver Brett

GTC 2021: Counterfactual Learning to Rank in E-commerce

GrubhubTech

Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...

Sease

Applied Machine Learning for Ranking Products in an Ecommerce Setting

Databricks

The step by step guide to SEO Website Audit

amandacerry

Recommendation at Netflix Scale

Justin Basilico

Building a Recommendation Engine - An example of a product recommendation engineNYC Predictive Analytics

Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020

Dawn Anderson MSc DigM

Ranking in Google Since The Advent of The Knowledge Graph

Bill Slawski

Recommender system algorithm and architectureLiang Xiang

Semantic search Bill Slawski DEEP SEA Con

Bill Slawski

Website and seo audit checklist

360 Digi Wolrd - Digital Marketing Services

Search@flipkart

Umesh Prasad

Learn to Rank search results

Ganesh Venkataraman

Natural Language Processing and Search Intent Understanding C3 Conductor 2019...

Dawn Anderson MSc DigM

Deep Learning for Semantic Search in E-commerce

Somnath Banerjee

What's hot (20)

Tanvi Motwani, Lead Data Scientist, Guided Search at A9.com at MLconf ATL 2016

Activity Ranking in LinkedIn Feed

[CXL Live 16] How to Utilize Your Test Capacity? by Ton Wesseling

Learning to Rank Datasets for Search with Oscar Castaneda

JoomlaDay Conference_September 2023 PDF.pdf

GTC 2021: Counterfactual Learning to Rank in E-commerce

Rated Ranking Evaluator Enterprise: the next generation of free Search Qualit...

Applied Machine Learning for Ranking Products in an Ecommerce Setting

The step by step guide to SEO Website Audit

Recommendation at Netflix Scale

Building a Recommendation Engine - An example of a product recommendation engine

Disambiguating Equiprobability in SEO Dawn Anderson Friends of Search 2020

Ranking in Google Since The Advent of The Knowledge Graph

Recommender system algorithm and architecture

Semantic search Bill Slawski DEEP SEA Con

Website and seo audit checklist

Search@flipkart

Learn to Rank search results

Natural Language Processing and Search Intent Understanding C3 Conductor 2019...

Deep Learning for Semantic Search in E-commerce

Similar to Better Search Through Query Understanding

Hr analytics and graphs : job recommendations

Linkurious

Next generation linked in talent search

Ryan Wu

AI, Search, and the Disruption of Knowledge Management

Trey Grainger

Skills, Reputation, and Search

Peter Skomoroch

Keynote Peter Skomoroch - skills, reputation, and searchlucenerevolution

KEYNOTE: Skills, Reputation and Search

lucenerevolution

Neo4j: Data Engineering for RAG (retrieval augmented generation)

Neo4j

Path 101 Opportunitypath101

How to Build a Semantic Search System

Trey Grainger

Building a semantic search system - one that can correctly parse and interpret end-user intent and return the ideal results for users’ queries - is not an easy task. It requires semantically parsing the terms, phrases, and structure within queries, disambiguating polysemous terms, correcting misspellings, expanding to conceptually synonymous or related concepts, and rewriting queries in a way that maps the correct interpretation of each end user’s query into the ideal representation of features and weights that will return the best results for that user. Not only that, but the above must often be done within the confines of a very specific domain - ripe with its own jargon and linguistic and conceptual nuances. This talk will walk through the anatomy of a semantic search system and how each of the pieces described above fit together to deliver a final solution. We'll leverage several recently-released capabilities in Apache Solr (the Semantic Knowledge Graph, Solr Text Tagger, Statistical Phrase Identifier) and Lucidworks Fusion (query log mining, misspelling job, word2vec job, query pipelines, relevancy experiment backtesting) to show you an end-to-end working Semantic Search system that can automatically learn the nuances of any domain and deliver a substantially more relevant search experience.

Connecting the Dots for Information Discovery.pdf

Neo4j

In this presentation, delivered by ABK Andreas Kollegger at QCon London 2024, the focus was on Connecting the Dots for Information Discovery. The classic RAG application extends an LLM with private information, able to fetch answers to questions that are contained in a single chunk of text. What if the answer requires connecting the dots across multiple chunks that may not be directly similar to the question? That is information discovery with GraphRAG. You'll learn how to: - reconstruct chunks into the original context - meaningfully connect disparate chunks - expand unstructured text data with structured data - combine all this into a RAG workflow

Search Ranking Across Heterogeneous Information Sources

Viet Ha-Thuc

Alok das cv updated

Alok Das

Hello Sir/Madam, Please find enclosed my CV in application for the Digital Marketing Executive The nature of my degree course has prepared me for this position. It involved a great deal of independent research, requiring initiative, self-motivation and a wide range of skills. I am, an understanding of the online industry was essential. I found this subject very stimulating. Thank you for taking the time to consider this application and I look forward to hearing from you in the near future. Yours sincerely Alok Das Current CTC -3.35 lacs/year Expected CTC - 5 lacs/year Immediate Joining in 10 days

A Recruiter's Checklist (Strings Attached)

Maysoun Mohamed

Activate 2019 Opening Keynote, Will Hayes, CEO, Lucidworks

Lucidworks

Reflected intelligence evolving self-learning data systems

Trey Grainger

In this presentation, we’ll talk about evolving self-learning search and recommendation systems which are able to accept user queries, deliver relevance-ranked results, and iteratively learn from the users’ subsequent interactions to continually deliver a more relevant experience. Such a self-learning system leverages reflected intelligence to consistently improve its understanding of the content (documents and queries), the context of specific users, and the collective feedback from all prior user interactions with the system. Through iterative feedback loops, such a system can leverage user interactions to learn the meaning of important phrases and topics within a domain, identify alternate spellings and disambiguate multiple meanings of those phrases, learn the conceptual relationships between phrases, and even learn the relative importance of features to automatically optimize its own ranking algorithms on a per-query, per-category, or per-user/group basis.

Gary Conaway 6-21-11

Melanie Brandt

You Don't Know SEO

Michael King

Shall we search? Lviv.

Vira Povkh

RDBMS to Graph Webinar

Neo4j

Relational databases were conceived to digitize paper forms and automate well-structured business processes, and still have their uses. But, oftentimes with RDBMS, performance degrades with the increasing number and levels of data relationships and data size. A graph database like Neo4j naturally stores, manages, analyzes, and uses data within the context of connections meaning Neo4j provides faster query performance and vastly improved flexibility in handling complex hierarchies than SQL. This webinar explains why companies are shifting away from RDBMS towards graphs to unlock the business value in their data relationships.

Alok das resume

Alok Das

Similar to Better Search Through Query Understanding (20)

Hr analytics and graphs : job recommendations

Next generation linked in talent search

AI, Search, and the Disruption of Knowledge Management

Skills, Reputation, and Search

Keynote Peter Skomoroch - skills, reputation, and search

KEYNOTE: Skills, Reputation and Search

Neo4j: Data Engineering for RAG (retrieval augmented generation)

Path 101 Opportunity

How to Build a Semantic Search System

Connecting the Dots for Information Discovery.pdf

Search Ranking Across Heterogeneous Information Sources

Alok das cv updated

A Recruiter's Checklist (Strings Attached)

Activate 2019 Opening Keynote, Will Hayes, CEO, Lucidworks

Reflected intelligence evolving self-learning data systems

Gary Conaway 6-21-11

You Don't Know SEO

Shall we search? Lviv.

RDBMS to Graph Webinar

Alok das resume

More from Daniel Tunkelang

Query Understanding and Ecommerce

Better Search Through Query Understanding

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Better Search Through Query Understanding

Similar to Better Search Through Query Understanding (20)

More from Daniel Tunkelang

More from Daniel Tunkelang (20)

Recently uploaded

Recently uploaded (20)

Better Search Through Query Understanding