WG
Uploaded byWeiwei Guo
PPTX, PDF4,098 views

Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)

This document discusses deep learning techniques for natural language processing and their applications in search systems. It begins with an introduction to how natural language processing is used in search systems to understand queries, documents, and perform matching. It then provides an overview of popular deep learning models for natural language processing, including sequential models, representation learning techniques, and their uses for tasks like classification, sequence modeling, and ranking. Specific techniques discussed include RNNs, Transformers, BERT, and representation learning methods like Word2Vec, ELMO, and GPT. The document emphasizes how these deep learning NLP methods can help search systems understand language better.

Embed presentation

Downloaded 403 times
1 / 211
2 / 211
3 / 211
4 / 211
5 / 211
6 / 211
7 / 211
8 / 211
9 / 211
10 / 211
11 / 211
12 / 211
13 / 211
14 / 211
15 / 211
16 / 211
17 / 211
18 / 211
19 / 211
20 / 211
21 / 211
22 / 211
23 / 211
24 / 211
25 / 211
26 / 211
27 / 211
28 / 211
29 / 211
30 / 211
31 / 211
32 / 211
33 / 211
34 / 211
35 / 211
36 / 211
37 / 211
38 / 211
39 / 211
40 / 211
41 / 211
42 / 211
43 / 211
44 / 211
45 / 211
46 / 211
47 / 211
48 / 211
49 / 211
50 / 211
51 / 211
52 / 211
53 / 211
54 / 211
55 / 211
56 / 211
57 / 211
58 / 211
59 / 211
60 / 211
61 / 211
62 / 211
63 / 211
64 / 211
65 / 211
66 / 211
67 / 211
68 / 211
69 / 211
70 / 211
71 / 211
72 / 211
73 / 211
74 / 211
75 / 211
76 / 211
77 / 211
78 / 211
79 / 211
80 / 211
81 / 211
82 / 211
83 / 211
84 / 211
85 / 211
86 / 211
87 / 211
88 / 211
89 / 211
90 / 211
91 / 211
92 / 211
93 / 211
94 / 211
95 / 211
96 / 211
97 / 211
98 / 211
99 / 211
100 / 211
101 / 211
102 / 211
103 / 211
104 / 211
105 / 211
106 / 211
107 / 211
108 / 211
109 / 211
110 / 211
111 / 211
112 / 211
113 / 211
114 / 211
115 / 211
116 / 211
117 / 211
118 / 211
119 / 211
120 / 211
121 / 211
122 / 211
123 / 211
124 / 211
125 / 211
126 / 211
127 / 211
128 / 211
129 / 211
130 / 211
131 / 211
132 / 211
133 / 211
134 / 211
135 / 211
136 / 211
137 / 211
138 / 211
139 / 211
140 / 211
141 / 211
142 / 211
143 / 211
144 / 211
145 / 211
146 / 211
147 / 211
148 / 211
149 / 211
150 / 211
151 / 211
152 / 211
153 / 211
154 / 211
155 / 211
156 / 211
157 / 211
158 / 211
159 / 211
160 / 211
161 / 211
162 / 211
163 / 211
164 / 211
165 / 211
166 / 211
167 / 211
168 / 211
169 / 211
170 / 211
171 / 211
172 / 211
173 / 211
174 / 211
175 / 211
176 / 211
177 / 211
178 / 211
179 / 211
180 / 211
181 / 211
182 / 211
183 / 211
184 / 211
185 / 211
186 / 211
187 / 211
188 / 211
189 / 211
190 / 211
191 / 211
192 / 211
193 / 211
194 / 211
195 / 211
196 / 211
197 / 211
198 / 211
199 / 211
200 / 211
201 / 211
202 / 211
203 / 211
204 / 211
205 / 211
206 / 211
207 / 211
208 / 211
209 / 211
210 / 211
211 / 211
Deep Natural Language Processing for Search Systems Weiwei Guo Huiji Gao Jun Shi Bo Long
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
Introduction Huiji Gao
SIGIR 2019 5 Introduction - Search Systems Search is Everywhere
SIGIR 2019 6 Introduction - Search Systems Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs SIGIR 2019 InfoInfo Travel
SIGIR 2019 7 Introduction - Search Systems SIGIR 2019 Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs Info Shopping Food Arrive Paris
SIGIR 2019 8 Introduction - Search Systems SIGIR 2019 Travel FoodShopping Attractions Connect People Looking or Jobs Learning Materials Info Arrive Paris Attending SIGIR Attractions Learning Materials
SIGIR 2019 9 Introduction - Search Systems SIGIR 2019 Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs Info Arrive Paris Attending SIGIR Connect People Looking or Jobs
SIGIR 2019 10 Introduction - NLP in Search Systems ● Understand Searcher Intention ○ Search Queries ○ User Profiles Best Places to Shop in Paris Best Areas to Stay in Paris Safely Budget? Gender? Location? ... ● Understand Documents ○ Posts, Reviews, Comments ○ Synonyms, Facets, Semantics, etc. ● Matching ○ Query & Doc Matching for Retrieval & Ranking ● Search Assistant ○ Query Reformulation
SIGIR 2019 11 Introduction - NLP in Search Systems Generate Human Search Engine Generate Assistant Natural Language Understanding (NLU) Natural Language Generation (NLG) Understand Natural Language Data
SIGIR 2019 12 Natural Language Processing in Search Ecosystem Query Tagging Retrieval & Ranking Raw Query Query Understanding Query Intention Document Understanding Auto Completion Spell Check Query Suggestion Natural Language Generation for Search Assistant Natural Language Understanding Indexing Matching Text Processing Language Detection Tokenization ... ... ... ...
SIGIR 2019 13 Introduction - Deep Learning for NLP Figure source: Elvis, 2018, https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d
SIGIR 2019 14 Opportunities - Deep Learning for NLP in Search Systems Why Deep Learning? ● Deep Semantics from High Dimension and Sparse Data ○ Synonymous, Disambiguation, etc. ● Easy Feature Engineering ○ Hand Crafted Features V.S. Auto Feature Representation ● Model Flexibility ○ Model end-to-end process ○ Various NN components to model and cooperate systematically ● Multi-level Feature Representation ○ Hierarchical Representations character -> token -> word -> phrase -> sentence [Young et. al. 2018]
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
Deep Learning for Natural Language Processing Huiji Gao
SIGIR 2019 17 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 18 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
SIGIR 2019 19 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context Sequence of Words w_1 Sequential Modeling w_2 w_3 w_n…... Sequence Encoding (Feature Embedding) Tasks (Classification, etc.) Contains Essential Semantics of the Sequence
SIGIR 2019 20 am an NLP an NLP engineerI am an am an an NLP NLP EngineerI am am an NLP engineerI Sequential Modeling on Semantic Context I am an NLP engineer I am an NLP engineer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition I am an NLP engineer Je suis un ingénieur en NLP I am an NLP engineer Je suis un ingénieur en NLP Context Strengthening Transformer GAN Context Consistency Global Context I am an NLP engineer Je suis un ingénieur en NLP Je sont une ingénieur en NLP train test Encoding Encoding Encoding
SIGIR 2019 21 Multilayer Perceptron (MLP) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Figure source: https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn/
SIGIR 2019 22 Convolutional Neural Networks (CNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Zhang et. al. 2015]
SIGIR 2019 23 Recurrent Neural Networks (RNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context LSTM GRU [LeCun et. al. 2015] github: 2015-08-Understanding-LSTMs
SIGIR 2019 24 Sequence to Sequence (Encoder - Decoder) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
SIGIR 2019 25 Attention Mechanisms MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
SIGIR 2019 26 Transformer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Novelty: Eliminate Recurrence transformer http://jalammar.github.io/illustrated-transformer/
SIGIR 2019 27 Generative Adversarial Networks (GANs) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Lee, et al., KDD 2018]
SIGIR 2019 28 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 29 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
SIGIR 2019 30 ▪ Representation Learning Sequence of Words w_1 Representation Learning w_2 w_3 w_n…... T’_1 T’_2 T’_3 T’_n…... Word Representation (Embeddings) Deep Learning for Natural Language Processing
SIGIR 2019 31 Representation Learning Word2Vec Static Word Embedding [Mikolov et. al. 2013]
SIGIR 2019 32 Representation Learning Word2Vec ELMO Static Word Embedding Contextual Word Embedding - Bidirectional LSTM [Peters et. al. 2018]
SIGIR 2019 33 Representation Learning Word2Vec ELMO GPT Static Word Embedding Contextual Word Embedding Contextual Word Embedding - Bidirectional LSTM - One Direction Transformer [Radford et. al. 2018]
SIGIR 2019 34 Representation Learning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding BERT - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer [Devlin et. al. 2018]
SIGIR 2019 35 Representation Learning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding XLNET Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding XLNet - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer - Bidirectional, Learn correlations among masked tokens Pre-trained NLP Model [Yang et. al. 2018]
SIGIR 2019 36 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching MLP, CNN, RNN, Transfomer, ... RNN, Transfomer, ... Seq2seq, Transformer, GAN, ... CNN, RNN, ... Word2Vec, ELMO, GPT, BERT, XLNet, ...
SIGIR 2019 37 References - Deep Learning for Natural Language Processing ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [LeCun et. al, 2015] Deep learning, Nature, vol. 521, no. 7553, pp. 436–444, 2015. ▪ [Lee et. al, 2018] Rare query expansion through generative adversarial networks in search advertising, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ [Mikolov et. al. 2013] Distributed representations of words and phrases and their compositionality, in Advances in neural information processing systems, 2013 ▪ [Mikolov et. al. 2013] Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, 2013. ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Robinson, 2018] https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn, 2018. ▪ [Shi et. al, 2018] Neural Abstractive Text Summarization with Sequence-to-Sequence Models, arXiv preprint arXiv:1812.02303v2, 2018. ▪ [Yang et. al. 2019] XLNet: Generalized Autoregressive Pretraining for Language Understanding, arXiv preprint arXiv:1906.08237v1, 2019 ▪ [Young et. al. 2018] Recent Trends in Deep Learning Based Natural Language Processing, arXiv preprint arXiv:1708.02709v8, 2018 ▪ [Zhang et. al. 2015] A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, arXiv preprint arXiv:1510.03820, 2015.
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
Deep NLP in Search Systems - Language Understanding Jun Shi
SIGIR 2019 40 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
SIGIR 2019 41 Entity Tagging ▪ Problem statement ▪ Traditional statistical models – Hidden Markov Model – Maximum Entropy Markov Model – (Semi-Markov) Conditional Random Field ▪ Deep learning models – Input layer – Context layer – Decoder layer ▪ Training dataset ▪ Evaluation
SIGIR 2019 42 Entity Tagging - Problem Statement ▪ A named entity, a word or a phrase that clearly identifies one item from a set of other items that have similar attributes. [Li et. al. 2018] ▪ Entity tagging (Named Entity Recognition, NER), the process of locating and classifying named entities in text into predefined entity categories. Washington D.C. was named after George Washington. LOCATION PERSON
SIGIR 2019 43 Motivation ▪ Efficiency – Looking only for named entities can speed up search. ▪ Precision – Matching both named entity and tags can increase search precision. ▪ Quality – Ranking retrieved results by considering tags improves search quality.
SIGIR 2019 44 Traditional Statistical Models ▪ Generative model – Hidden Markov Model [Baum, et. al. 1966] ▪ Discriminative model – Maximum Entropy Markov Model [McCallum, et. al. 2000] – (Semi-Markov) Conditional Random Field [Lafferty, et. al. 2001]
SIGIR 2019 45 Hidden Markov Model Generative model, model joint probability Pr(T, W) instead of conditional probability Pr(T | W). T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
SIGIR 2019 46 Maximum Entropy Markov Model Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
SIGIR 2019 47 Conditional Random Field Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
SIGIR 2019 48 Semi-Markov Conditional Random Field Each tag can correspond to a variable-length phrase. T1 T2 W1 W2 W3 tags words T3 W4 W5 W6 T0
SIGIR 2019 49 Summary - Traditional Statistical Models Model Pros Cons Hidden Markov Model training is simple when states are observed difficult to include features Maximum Entropy Markov Model easy to include features suffer from “label bias problem” (prefer states with lower number of transitions) Conditional Random Field easy to include features, do not suffer from “label bias problem” training is relatively complex
SIGIR 2019 50 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
SIGIR 2019 51 Input Layer - Word Embedding [Yadav, et. al. 2018]
SIGIR 2019 52 Input Layer - Char Embedding [Yadav, et. al. 2018]
SIGIR 2019 53 Input Layer - Word and Char Embedding [Yadav, et. al. 2018]
SIGIR 2019 54 Summary - Input Layer word embedding char embedding external features, lexicon, etc. char CNN/RNN word
SIGIR 2019 55 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
SIGIR 2019 56 Context Layer - CNN embedding CNN P A D P A D Tag Decoder [Li et. al. 2018]
SIGIR 2019 57 Context Layer - RNN [Li et. al. 2018]
SIGIR 2019 58 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
SIGIR 2019 59 Tag Decoder - MLP+softmax [Li et. al. 2018]
SIGIR 2019 60 Tag Decoder - CRF [Li et. al. 2018]
SIGIR 2019 61 Tag Decoder - RNN [Li et. al. 2018]
SIGIR 2019 62 Pre-Training and Fine-Tuning BERT GPT ELMo [Devlin et. al. 2018] [Peters et. al. 2018][Radford et. al. 2018]
SIGIR 2019 63 Entity Tagging Evaluation Exact-match evaluation ▪ segment match ▪ tag match
SIGIR 2019 64 Entity Tagging Training Dataset [Li et. al. 2018]
SIGIR 2019 65 Entity Tagging on CoNLL03 English Source Method F1 score [Passos et al. 2014] CRF 90.90 [Huang et al. 2015] Bi-LSTM+CRF 84.26 [Collobert et al. 2011] Conv-CRF 89.59 [Kuru et al. 2016] char embedding 84.52 [Chiu and Nichols 2015] word + char embedding 91.62 [Devlin et. al. 2018] BERT Large 92.8
SIGIR 2019 66 References - Entity Tagging ▪ [Baum et. al. 1966] Baum, L. E.; Petrie, T. (1966). "Statistical Inference for Probabilistic Functions of Finite State Markov Chains". The Annals of Mathematical Statistics. 37 (6): 1554–1563. ▪ [Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 ▪ [Collins and Singer 1999] Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ▪ [Collobert et. al. 2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. ▪ [Collobert and Weston 2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [Huang et. al. 2015] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 ▪ [Kuru et. al. 2016] Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. Charner: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911–921 ▪ [Lafferty et. al. 2001] Lafferty, J., McCallum, A., Pereira, F. (2001). "Conditional random fields: Probabilistic models for segmenting and labeling sequence data". Proc. 18th International Conf. on Machine Learning. Morgan Kaufmann. pp. 282–289 ▪ [Li et. al. 2018] Jing Li, Aixin Sun, Jianglei Han, Chenliang Li, A Survey on Deep Learning for Named Entity Recognition, Dec. 2018, arXiv preprint, https://arxiv.org/abs/1812.09449 ▪ [McCallum et. al. 2000] McCallum, Andrew; Freitag, Dayne; Pereira, Fernando (2000). "Maximum Entropy Markov Models for Information Extraction and Segmentation" (PDF). Proc. ICML 2000. pp. 591–598
SIGIR 2019 67 References (continued) ▪ [Passos 2014] Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367 ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Yadav et. al. 2018] Vikas Yadav, Steven Bethard, A Survey on Recent Advances in Named Entity Recognition from Deep Learning models, Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158 Santa Fe, New Mexico, USA, August 20-26, 2018.
SIGIR 2019 68 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
SIGIR 2019 69 Entity Disambiguation ▪ Problem statement ▪ Motivation ▪ Challenges ▪ System architecture – Candidate Entity Generation – Candidate Entity Selection – Joint entity tagging and disambiguation ▪ Evaluation
SIGIR 2019 70 Entity Disambiguation - Problem Statement ▪ Also known as entity linking ▪ Resolves mentions to entities in a given knowledge base – Mentions are mostly named entities – Example of knowledge base: freebase, wikipedia ▪ Examples: Jaguar – The prey saw the jaguar cross the jungle. KB:Jaguar – The man saw a jaguar speed on the highway. KB:Jaguar_Cars
SIGIR 2019 71 Motivation Increase the quality of the retrieved results Example: Which Michael Jordan were you looking for?
SIGIR 2019 72 Challenges ▪ Name variation: New York vs Big Apple ▪ Ambiguity: Michael Jordan ▪ Metonymy: Beijing (city or Chinese government) ▪ Absence: no entries in knowledge base ▪ Evolving information: new company names, e.g. tic-tok.
SIGIR 2019 73 System Architecture Candidate Generation Candidate selection 1. Candidate entity generation a. Name dictionary based techniques. b. Surface form expansion from the local document. c. Methods based on search engines. 2. Candidate entity selection a. Graph-based methods b. Text-based methods knowledge base Documents Linked results
SIGIR 2019 74 Candidate Entity Generation ▪ Name dictionary based techniques – Entity pages – Redirect pages – Disambiguation pages – Bold phrases from the first paragraphs – Hyperlinks in Wikipedia articles ▪ Surface form expansion – Heuristic based methods – Supervised learning methods ▪ Methods based on search engines [Shen et. al. 2014]
SIGIR 2019 75 Candidate Entity Selection - Graph based models ▪ knowledge graph lends itself to graph based methods. ▪ Build a weighted undirected graph with mentions and candidate entities as nodes – weights on mention-entity edges. – weights on entity-entity edge. ▪ Compute an optimal subgraph: – optimality is model dependent. – contains all mentions – one entity per mention mention entity
SIGIR 2019 76 An Example Graph [Hoffart, et. al. 2011]
SIGIR 2019 77 Candidate Entity Selection - Text-Based Models ▪ Idea: find agreement (similarity) between entity and mention. – Example: computational intelligence by Michael Jordan and Stuart Russell from UC Berkeley increasing context ● wiki/Michael_Jordan ● wiki/Michael_I._Jordan▪ Models according to context. – Direct Models. ▪ use mention and candidate entities only. – Local Models ▪ use local context around the mention. – Coherence Models. ▪ use mentions and entities in a document. – Collaborative Models. ▪ use mentions and entities across related documents.
SIGIR 2019 78 Individual vs Joint Approach Two types of approaches according to how mentions are resolved. ▪ Individual approach – Handle one mention at a time. – Rank the candidate entities. – Relatively low complexity. ▪ Joint approach – Treat all mentions in a document as a sequence. – Jointly optimize the mention-entity pairs. – High complexity, usually resorting to approximations.
SIGIR 2019 79 Features Different features are used in candidate selection modeling. ▪ Traditional Features – entity popularity, entity type. – surface form similarity between entity and mention. – coherence between mapped entities ▪ Deep Features [Yamada et. al. 2016] – word embedding – entity embedding – similarity between words, context and candidate entities.
SIGIR 2019 80 Example: Entity Disambiguation with Attention training: max-margin loss [Genea et. al. 2017]
SIGIR 2019 81 Joint Entity Tagging and Disambiguation ▪ Previously, we find entities first, then link them. But if entities are identified wrong, disambiguation will likely fail too. – Over-split: Romeo and Juliet by Shakespeare – Under-split: Baby Romeo and Juliet were born hours apart. ▪ We could do entity tagging and disambiguation jointly. – Over-generate candidate mentions – Generate possible entities per mention – Score non-overlapping mention-entity pair jointly. [Sil, et. al.2013]
SIGIR 2019 82 A Joint Neural Entity Tagging and Disambiguation [Kolitsas, et. al. 2018] Select mentions that have at least one possible entity ● Trained on golden mention-entity pairs. ● Used max-margin loss. cosine similarity between current entity and average of other entities in the document
SIGIR 2019 83 Entity Disambiguation Evaluation ▪ Entity-tagging-style F1 score – A link is considered correct only if the mention matches the gold boundary and the linked entity is also correct ▪ Accuracy – another common metric, simpler than F1 score. ▪ TAC-KBP B-Cubed+ F1 score – not widely used
SIGIR 2019 84 References - Named Entity Disambiguation ▪ [Cucerzan 2007] Cucerzan, Silviu. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP- CoNLL): 708–716. ▪ [Ganea et. al. 2017] Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2609–2619. Association for Computational Linguistics. ▪ [Globerson et. al. 2016] Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In ACL (1). ▪ [Han et. al. 2011] Han, Xianpei; Sun, Le; Zhao, Jun. Collective Entity Linking in Web Text: A Graph-based Method. ▪ [Hoffart, et. al. 2011] Johannes Hoffart, Mohamed A. Yosef, Ilaria Bordino, Hagen F¨urstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Method in Natural Language Processing, pages 782–792. Association for Computational Linguistics. ▪ [Kolitsas et al., 2018] Nikolaos Kolitsas, Octavianeugen Ganea, and Thomas Hofmann. End-to-end neural entity linking. In CoNLL, 2018. ▪ [Kulkarni et. al. 2009] S. Kulkarni, A. Singh, G. Ramakrishnan, and S. Chakrabarti, Collective annotation of Wikipedia entities in web text, in SIGKDD, 2009, pp. 457–466 ▪ [Ling et al.2015] Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315–328. ▪ [Le et. al., 2018] Phong Le and Ivan Titov. Improving entity linking by modeling latent relations between mentions. In ACL, 2018. ▪ [Rao et. al. 2013] Rao, Delip; McNamee, Paul; Dredze, Mark (2013). Entity Linking: Finding Extracted Entities in a Knowledge Base. Multi-source, Multilingual Information Extraction and Summarization. Springer Berlin Heidelberg: 93–115
SIGIR 2019 85 References (continued) ▪ [Raiman et. al. 2018] Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual Entity Linking by Neural Type System Evolution. In Proc. of AAAI. ▪ [Shen et. al. 2014] Wei Shen, Jianyong Wang, and Jiawei Han. 2014. Entity linking with a knowledge base: Issues, techniques, and solutions. TKDE ▪ [Sil et. al. 2013] Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2369–2374. ACM ▪ [Sil et. al. 2016] Sil, A., and Florian, R. 2016. One for all: Towards language independent named entity linking. ACL. ▪ [Yamada et. al. 2016] Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. CoNLL 2016, page 250.
SIGIR 2019 86 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
SIGIR 2019 87 Intent Classification ▪ Problem statement ▪ Deep Learning Models – fastText – CNN – Bi-RNN + Attention
SIGIR 2019 88 Intent Classification - Problem Statement ▪ Web search intent can be classified into 3 class: [Broder 2002] – Navigational. The immediate intent is to reach a particular site. ▪ Greyhound Bus. Probable target http://www.greyhound.com – Informational. The intent is to acquire some information assumed to be present on one or more web pages. ▪ San Francisco – Transactional. The intent is to perform some web-mediated activity. ▪ Shopping activities ▪ In conversational AI, intent is usually task-specific, e.g. – I would like to book a flight from SFO to CDG: FlightBooking – What software can I use to view epub documents: SoftwareRecommendation. ▪ We focus on the latter in this tutorial.
SIGIR 2019 89 Intent Classification - Methods ▪ Traditional methods – Features: bag of words, n-gram, TF-IDF. – Models: logistic regression, naive Bayes, SVM, random forest. ▪ Deep learning methods – Word embedding + linear classifier (fastText) [Joulin et. al. 2016]. – Convolutional neural networks [Hashemi 2016] . – Bi-RNN + attention (joint slot filling) [Liu et al.2016] .
SIGIR 2019 90 Intent Classification - fastText word and N gram embedding features Average softmax [Joulin et. al. 2016]
SIGIR 2019 91 Intent Classification - CNN [Hashemi 2016]
SIGIR 2019 92 Intent Classification - Bi-RNN+Attention [Liu et. al. 2016]
SIGIR 2019 93 References - Intention Classification ▪ [Broder 2002] A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, 2002. ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Joulin et. al. 2016] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ▪ [Lai et al.2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267–2273 ▪ [Liu et al.2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 ▪ [Zhou et al.2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 207. ▪ [Kim et. al. 2016] Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In Proceedings of SLT. ▪ [Hashemi 2016] Homa B Hashemi, Amir Asiaee, and Reiner Kraft. 2016. Query intent detection using convolutional neural networks. In International Conference on Web Search and Data Mining, Workshop on Query Understanding. ▪ [Liu et. al. 2016]. Liu and I. Lane, “Attention-based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling,” in Interspeech, 2016. ▪ [Shi et. al. 2016] Y. Shi, K. Yao, L. Tian, and D. Jiang, “Deep lstm based feature mapping for query classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1501–1511.
Deep NLP in Search Systems - Document Retrieval & Ranking Jun Shi, Weiwei Guo
SIGIR 2019 95 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
SIGIR 2019 96 Document Retrieval and Ranking ▪ Efficient Candidate Retrieval ▪ Deep Neural Ranking
SIGIR 2019 97 Efficient Candidate Retrieval ▪ Syntactic retrieval – based on string matching – use inverted index – can include different fields (name, title, etc). ▪ Semantic retrieval – based on vector space representation. – approximate nearest neighbor search
SIGIR 2019 98 Syntactic Retrieval Query: mike software engineer mike Doc1, Doc2, Doc4, ... software Doc2, Doc3, Doc4, Doc7, ... engineer Doc1, Doc2, Doc4, Doc9, ... Inverted index Results: Doc2, Doc4 It won’t retrieve a document contains “mike is a software developer,...”
SIGIR 2019 99 Semantic Retrieval - Concept mike software engineer [0.3, 1.27, -3.4, ...] embedding mike software engineer The problem becomes nearest neighbor search [Mitra et. al. 2017]
SIGIR 2019 100 Semantic Retrieval - Vector Generation ▪ Bag of words – count based, TF-IDF vectors. ▪ Embedding vectors – word embedding: word2vec, Glove, BERT, etc. – sentence/document embedding ▪ universal sentence encoder [Cer et. al. 2018] ▪ Gaussian document representation [Giannis et. al. 2017] ▪ power mean concatenation [Rücklé et. al. 2018]
SIGIR 2019 101 Approximate Nearest Neighbor Search ▪ Exact nearest neighbor search – complexity is linear with the size of dataset, not suitable for large dataset. ▪ Approximate nearest neighbor search – allow bounded errors. – complexity is sublinear – many algorithms available ▪ product quantization [Jégou et. al. 2016] ▪ hierarchical navigable small world graphs [Malkov et. al. 2016]
SIGIR 2019 102 References - Efficient Candidate Retrieval ▪ [Cer et. al. 2018] D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, Y. Sung, B. Strope, and R. Kurzweil. Universal sentence encoder. CoRR, abs/1803.11175, 2018. URL http://arxiv.org/abs/1803.11175. ▪ [Giannis et. al. 2017] Giannis Nikolentzos, Polykarpos Meladianos, Franc¸ois Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. 2017. Multivariate gaussian document representation from word embeddings for text categorization. In EACL ▪ [Manning et. al. 2008] Christopher D. Manning Prabhakar Raghavan Hinrich Schütze, Introduction to Information Retrieval Cambridge University Press New York, NY, USA, 2008 ▪ [Mitra et. al. 2017] Mitra, B., & Craswell, N. (2017). Neural models for Information Retrieval. CoRR, abs/1705.01509. ▪ [Rücklé et. al. 2018] A. Rücklé, S. Eger, M. Peyrard, and I. Gurevych. Concatenated p-mean word embeddings as universal cross- lingual sentence representations. CoRR, abs/1803.01400, 2018. URL http://arxiv.org/abs/1803.01400. ▪ [Malkov et. al. 2016] Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. arXiv:1603.09320, 2016 ▪ [Jégou et. al. 2016] Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2011, 33 (1), pp.117- 128. ff10.1109/TPAMI.2010.57ff. ffinria-00514462v2
SIGIR 2019 103 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Auto Completion – Query Reformulation – Spell Correction
SIGIR 2019 104 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods q, d1 q, d2 s1 s2 learning-to-rank loss
SIGIR 2019 105 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks q d1 q d2q, d1 q, d2
SIGIR 2019 106 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks q q d1 d2
SIGIR 2019 107 Traditional Ranking Features ▪ Hand-crafted features – query/document matching features ▪ Cosine similarity between query and doc title ▪ Clickthrough rate from query to this doc based on search log ▪ ……. – Document alone ▪ popularity ▪ number of incoming links ▪ …...
SIGIR 2019 108 Learning to Rank ▪ Pointwise ranking – Logistic regression (Burges, 2010) ▪ Pairwise ranking ▪ Listwise ranking – Cross entropy
SIGIR 2019 109 Deep Neural Ranking ▪ Focus on compute query/document score s1 s2 learning-to-rank loss q q d1 d2 ▪ Two categories: – Siamese Networks – Interaction based Networks
SIGIR 2019 110 Deep Structured Semantic Model feed-forward (Huang et al., 2013) 30k 300 128 30k 300 128 high dimensional query/doc vector query doc1 sum of word embeddings cos sim
SIGIR 2019 111 Modeling Word Sequence by CNN CNN (Shen et al., 2014) d by n matrix 300 128 token embedding matrix d by n matrix 300 128 feed-forward cos sim
SIGIR 2019 112 Siamese Networks ▪ Pros: – Generalization; semantic matching – Efficient; doc embs can be precomputed ▪ Cons: – Lexical features lost: people/company names, rare words
SIGIR 2019 113 Interaction-based Networks (Guo et al., 2016) query: jobs at linkedin …... document Feed-forward z 1 z 2 z 3 s cosine sim between query words and doc words sim_vec1 sim_vec2 sim_vec3 histogram1 histogram2 histogram2 historgrams based on sim vectors g1 g2 g3 0.4 0.1 0.5
SIGIR 2019 114 Deep Neural Ranking Summary ▪ Only focus on query/doc scoring ▪ End-to-end models ▪ Two popular architectures Siamese Network Interaction Network Match topical matches lexical matches Latency small large
SIGIR 2019 115 References - Deep Neural Ranking ▪ Huang, Po-Sen, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. "Learning deep structured semantic models for web search using clickthrough data." In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338. ACM, 2013. ▪ Shen, Yelong, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. "A latent semantic model with convolutional-pooling structure for information retrieval." In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp. 101-110. ACM, 2014. ▪ Rodrigo Nogueira, Kyunghyun Cho, Passage re-ranking with BERT, 2019 ▪ Burges. "From ranknet to lambdarank to lambdamart: An overview." Learning. 2010. ▪ Guo, Jiafeng, Yixing Fan, Qingyao Ai, and W. Bruce Croft. "A deep relevance matching model for ad-hoc retrieval." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55-64. ACM, 2016. ▪ Mitra, Bhaskar, Fernando Diaz, and Nick Craswell. "Learning to match using local and distributed representations of text for web search." In Proceedings of the 26th International Conference on World Wide Web, pp. 1291-1299. International World Wide Web Conferences Steering Committee, 2017.
Deep NLP in Search Systems - Language Generation for Search Assistance Weiwei Guo
SIGIR 2019 117 Deep NLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Auto Completion – Query Reformulation – Spell Correction
SIGIR 2019 118 Language Generation for Search Assistance ▪ Goal: improve user experience by interacting with users ▪ NLP: Language generation Common Difference ▪ Character/word modeling ▪ Generation models: – language modeling, seq2seq ▪ Auto Completion ▪ Query Reformulation ▪ Spell Correction
SIGIR 2019 119 Language Generation for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation – word-level seq to seq ▪ Spell Correction – character-level seq to seq
SIGIR 2019 120 Query Auto-Completion ▪ Problem statement: save user keystrokes by predicting the entire query
SIGIR 2019 121 Challenges ▪ Limited Context – Hard to extract features due to limited words in a query ▪ Latency – The search component with most strict requirement on latency
SIGIR 2019 122 ▪ Traditional approach: 2-step approach Candidate Generation and Ranking soft softbank softball softmax software engineer …... software engineer salary soft skills softball softbank …... candidate generation candidate ranking
SIGIR 2019 123 Agenda Candidate Generation Candidate Ranking Context aware Personalization Traditional methods
SIGIR 2019 124 Candidate Generation ▪ Collect completed queries and associated frequency from search log ▪ Efficiently retrieve most frequent queries starting with the prefix – Using a trie data structure soft softbank 100 softball 50 softmax 40 software engineer 35 …... candidate generation
SIGIR 2019 125 Candidate Generation for Rare Prefix (Mitra & Craswell, 2015) ▪ No such prefix in search log – “cheapest flights from seattle to” “cheapest flights from seattle to” “to” to dc to sfo to airport to seattle …... 100 50 40 35
SIGIR 2019 126 ▪ Challenge: very few features can be extracted Candidate Ranking soft softbank softball softmax software engineer …... software engineer salary soft skills softball softbank …... candidate generation candidate ranking frequency 100 50 20 15
SIGIR 2019 127 Context-Aware ( Bar-Yossef & Kraus, 2011) ▪ Key idea: identify the candidates most similar to previous queries infant n (candidates) infant nike shoes infant nutrition …... infant, nike, shoe, adidas, clothes... infant, nutrition, baby, eat, food... …... (previous queries) baby eating disorder …... baby, eating, disorder, nutrition, food... …... sim score 0.1 0.8
SIGIR 2019 128 Personalization (Shokouhi, 2013) Feature list Same age Same gender Same region …... query prefix is “i”
SIGIR 2019 129 Agenda Candidate Generation Candidate Ranking Context aware Personalization Deep NLP methods CNN for Ranking End-to-end: Language Model Personalization Traditional methods
SIGIR 2019 130 Apply Deep Models in Ranking ▪ Measuring the semantic coherence between prefix and suffix embsize tri-letter length Query Prefix Query Suffix CNN conv and max-pooling text emb cos sim Cosine similarity (Mitra & Craswell, 2015)
SIGIR 2019 131 president barack obaman Language Modeling for Auto-Completion ‘ ’ softmax over chars ‘b’ LSTM char emb ‘t’ char emb ‘ ’ (Park & Chiba, 2017) ▪ Training: Character level language model word emb <INC> word emb `president` + word embedding
SIGIR 2019 132 char emb ‘t’ (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time 0.05 ‘e’ ‘b’ ‘a’ 0.4 0.3 ‘e’ ‘b’ ‘a’ 0.1 0.5 0.1 ‘a’ 0.4 Language Modeling for Auto-Completion
SIGIR 2019 133 (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time ▪ Greedy vs beam search ‘b’ ‘a’ ‘c’ ‘aa’ ‘ab’ ‘ac’ ‘ba’ ‘bb’ ‘bc’ ‘ca’ ‘cb’ ‘cc’ 0.01 0.3 0.3 0.1 0.01 0.01 0.26 0.01 0.01 ‘ab’ ‘ac’ ‘ca’ Language Modeling for Auto-Completion
SIGIR 2019 134 Personalization ▪ Embeddings for User Ids LSTM softmax over chars char emb ‘t’ word emb <INC> (Fiorin & Lu, 2018) user id emb
SIGIR 2019 135 Query Auto-Completion: Summary ▪ Traditional methods: hard to extract features ▪ Deep language model framework: – Very flexible to incorporate personalized/contextualized information – An end-to-end solution – Train: all parameters are optimized together – Test: generation and ranking at the same time – Cons – Time-consuming – May generate wrong words
SIGIR 2019 136 Reference ▪ Mitra, Bhaskar, and Nick Craswell. "Query auto-completion for rare prefixes." In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 1755-1758. ACM, 2015. ▪ Park, Dae Hoon, and Rikio Chiba. "A neural language model for query auto-completion." In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189-1192. ACM, 2017. ▪ Bar-Yossef, Ziv, and Naama Kraus. "Context-sensitive query auto-completion." In Proceedings of the 20th international conference on World wide web, pp. 107-116. ACM, 2011. ▪ Shokouhi, Milad. "Learning to personalize query auto-completion." In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pp. 103-112. ACM, 2013. ▪ Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018). ▪ Fiorini, Nicolas, and Zhiyong Lu. "Personalized neural language models for real-world query auto completion." arXiv preprint arXiv:1804.06439 (2018).
SIGIR 2019 137 Language Generation for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation – word-level seq to seq ▪ Spell Correction – character-level seq to seq
SIGIR 2019 138 Query Reformulation ▪ Problem Statement: automatically reformulate the previous query Search results for “fb developer” Reformulated queries for “fb developer”
SIGIR 2019 139 Agenda collaborative filtering Traditional methods
SIGIR 2019 140 Traditional Approach: Collaborative Filtering (Rida et al., 2012) ▪ Collect query pairs issued by the same user from search log ▪ Treat each query as an ID, and build a query-to-query matrix – Value in the matrix is TF-IDF
SIGIR 2019 141 Agenda collaborative filtering Traditional methods seq2seq conversational seq2seq GAN for seq2seq Deep NLP methods
SIGIR 2019 142 Query Reformulation as a Translation Task ▪ Sequence-to-sequence modeling (He et al, 2016) decoderencoder facebook developer <s> facebook software facebook software development </s> development ▪ Directly modeling the words in a query
SIGIR 2019 143 Conversational Query Reformulation ▪ Conversational queries (Ren et al, 2018) first query (q1) second query (q2) when was California founded? who is its governor? California population in 1990 how tall is kobe bryant? what about Lebron James? when was the last summer Olympics? and the winter one? summarized query (q3) who is California’s governor? population of California in 1990 how tall is Lebron James? when was the last winter Olympics? ▪ Goal: summarize the conversation in one query
SIGIR 2019 144 encoder decoder w1 w2 <s> w1 w2 w1 w2 </s> SEP w1 w2 Conversational Query Reformulation ▪ 1st: concatenated seq2seq (Ren et al, 2018)
SIGIR 2019 145 Conversational Query Reformulation ▪ 2nd: Attention over attention (Ren et al, 2018) 2nd query h1 h2 h3 c1 c2 c h1 h2 h31st query 3rd query (decoder) ht h0
SIGIR 2019 146 GAN for seq2seq ▪ Motivation: Discrepancy in decoder of seq2seq (Lee et al, 2018) decoder (training) <s> w1 w2 w1 w2 </s> – Training: inputs are the gold-standard words – Testing: inputs are the previous predicted words ▪ Goal: Resolve training/testing discrepancy with GAN decoder (testing) <s> w’1 w’2 </s>
SIGIR 2019 147 Discriminator Generator GAN for seq2seq (Lee et al, 2018) <s> software ui </s> <s> software engineer 1/0generated sequence software ui <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
SIGIR 2019 148 Query Reformulation: Summary ▪ seq2seq framework: – Directly modeling the words – Very flexible to incorporate session information – Achieves great performance (no character modeling)
SIGIR 2019 149 Reference ▪ Reda, Azarias, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah. "Metaphor: a system for related search recommendations." In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 664- 673. ACM, 2012. ▪ He, Yunlong, Jiliang Tang, Hua Ouyang, Changsung Kang, Dawei Yin, and Yi Chang. "Learning to rewrite queries." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1443-1452. ACM, 2016. ▪ Ren, Gary, Xiaochuan Ni, Manish Malik, and Qifa Ke. "Conversational query understanding using sequence to sequence modeling." In Proceedings of the 2018 World Wide Web Conference, pp. 1715-1724. International World Wide Web Conferences Steering Committee, 2018. ▪ Lee, Mu-Chu, Bin Gao, and Ruofei Zhang. "Rare query expansion through generative adversarial networks in search advertising." In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018.
SIGIR 2019 150 Language Generation for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation: – word-level seq to seq ▪ Spell Correction – character-level seq to seq
SIGIR 2019 151 Spell Correction
SIGIR 2019 152 Spell Correction ▪ Why spell correction: – Reduce the no results ▪ Challenge: – Many rare words (people/company names) look like spell errors query similar query has error? tumblr tumble No tumblw tumble Yes galaxy s10e galaxy s10 No galaxy s10d galaxy s10 Yes – Modeling characters and words at the same time
SIGIR 2019 153 Agenda Candidate Generation for a word Candidate generation for a query Candidate ranking Traditional methods
SIGIR 2019 154 Candidate Generation for a Word (Whitelaw et al 2009) ▪ Goal: – given “tumblw”, suggest “tumblr”, “tumble”... – given “tumblr”, suggest “tumble”, but not “tumblw” ▪ Key challenge: what is a legit word? (“tumblr” 👍 “tumblw” 👎) – Coverage of a dictionary is not enough ▪ Solution: use statistics in web noisy data – Correct words appear more frequent than incorrect words
SIGIR 2019 155 Candidate Generation for a Word (Whitelaw et al 2009) tumblr tumblw …... <tumblr, tumblw> <tumblw, tumblr> …... edit distance <tumblr, tumblw, 2> <tumblw, tumblr, 50> …... Filter by frequency of context term list confusion word pairs <word, candidate, count> ▪ For the context “social media X”: – freq(“social media tumblw”) < freq(“social media tumblr”)
SIGIR 2019 156 Candidate Generation for a Query (Chen et al 2007) earr pian kids ear, era, ... pain, pan, ... kids, kid, ... ear pain kids, era pan kid, ear pin kid, ….... ▪ Problem: query candidate size grows exponentially with # of words ▪ Solution: prune with language model – ear pian kids: 0.8 – era pian kids: 0.1 – earr pain kids: 0.9 – earr pan kids: 0.7 – earr pien kids: 0.2 – earr pian kid: 0.8
SIGIR 2019 157 Candidate Ranking (Li et al 2006, Chen et al 2007) Feature Types Examples similarity Edit distance Frequency of user reformulation …... Likelihood Language model score of the candidate Frequency of candidate terms appearing in the page titles …... querycandidate
SIGIR 2019 158 Agenda Candidate Generation for a word Candidate generation for a query Candidate ranking Traditional methods seq2seq for spell correction word-to-word char-to-char char-to-word Deep NLP methods
SIGIR 2019 159 Seq2seq for Spell Correction ▪ From subwords to subwords (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder ear #r <s> ear pain ear pain </s> pi #an
SIGIR 2019 160 Seq2seq for Spell Correction ▪ From subwords to subwords ▪ Issue: subword is designed for texts without errors (Ghosh and Kristensson, 2017, Zhou et al., 2017) Normal texts Example hunter → hunt #er Subword semantics relevant Spell errors hunetr → hu #net #r irrelevant
SIGIR 2019 161 Seq2seq for Spell Correction ▪ From characters to characters (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> e a e a r r r r </s> ▪ Issue: on the decoder, no word information ▪ Might produce words with wrong spelling
SIGIR 2019 162 Seq2seq for Spell Correction ▪ From characters to words – Most popular structure – Can leverage pretrained language model (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> ear r r ear </s>
SIGIR 2019 163 Reference ▪ Hasan, Saša, Carmen Heger, and Saab Mansour. "Spelling correction of user search queries through statistical machine translation." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 451-460. 2015. ▪ Yingbo Zhou, Utkarsh Porwal, Roberto Konow. "Spelling Correction as a Foreign Language." arXiv. 2017. ▪ Ghosh, Shaona, and Per Ola Kristensson. "Neural networks for text correction and completion in keyboard decoding." arXiv preprint arXiv:1709.06429 (2017). ▪ Chen, Qing, Mu Li, and Ming Zhou. "Improving query spelling correction using web search results." In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP- CoNLL), pp. 181-189. 2007. ▪ Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. "Exploring distributional similarity based models for query spelling correction." In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 1025-1032. Association for Computational Linguistics, 2006. ▪ Whitelaw, Casey, Ben Hutchinson, Grace Y. Chung, and Gerard Ellis. "Using the web for language independent spellchecking and autocorrection." In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pp. 890-899. Association for Computational Linguistics, 2009.
SIGIR 2019 164 Language Generation for Search Assistance: Summary Traditional methods deep NLP methods Candidate generation Rule based End-to-end solution Language modeling Candidate ranking Few features Latency Low High
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
Deep NLP in Search Systems - Real World Examples Huiji Gao
SIGIR 2019 167 Natural Language Data in LinkedIn Search Systems Query Job Posts Content Articles Learning Courses Member Profiles
SIGIR 2019 168 LinkedIn Search Ecosystem Auto Complete Spell Check Language Detection Tokenization Query Tagging Retrieval Ranking Normalization/ Accent Handling Raw Query Query Reformulation Query Processing Query Understanding Retrieval & Ranking Query Suggestion Query Intention Instant Results (TYH Name Search, etc.) Conversational Search
SIGIR 2019 169 NLP in LinkedIn Search: Challenges ● Data Ambiguity ○ Short Query Text ■ “abc” ○ No Strict Syntax ■ “bing search engineer” ○ Strong Correlation to the Searcher ■ “looking for new jobs” ● Deep Semantics ○ Representations for query & document w.r.t. search intent, entities, topics ABC News? ABC Stores? “Bing Search, Engineer” “Bing, Search Engineer” Job Seeker looks for jobs ■ “Engineering Openings” -> Job Posts Recruiter looks for candidates
SIGIR 2019 170 Deep NLP in LinkedIn Search: Challenges ● Complicated Search Ecosystem ○ Query suggestion affects both recall and precision in downstream retrieval and ranking. ○ Query tagging needs to be compatible with indexing and align with ranking features. ● Product Oriented Model Design ○ Design deep NLP algorithms for specific search components ○ Consider business rules, post filters, results blender, user experience, etc ● Online Latency ○ Serving deep NLP models with product latency restriction
SIGIR 2019 171 Applying Deep NLP in LinkedIn Search ● Feature Driven ○ Representation Learning ■ Using features generated from deep learning models e.g., word embedding ● Model Driven ○ Power product features directly with deep learning models ■ CNN/LSTM/Seq2seq/GAN/BERT based deep NLP models
SIGIR 2019 172 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 173 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 174 Query Intention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Search Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
SIGIR 2019 175 Query Intention Model: Task Oriented Intention Prediction People from Company X in Title Y Video on Topic X from Company Y Company with Name X in Geo Y Help in Topic X on Platform Y People from Geo X in Degree Y with Title Z Article from Author X with Topic Y Learning Course from Author X in Skill Y Professional with Skill X in Geo Y …… …… …... Search Tasks …… …… …...
SIGIR 2019 176 Query Intention: Member Search Footprint People Search Job Search Company Content Jobs Video Learning People Feed/Ads
SIGIR 2019 177 Query Intention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
SIGIR 2019 178 Query Intention on Search Blending Primary result Secondary cluster Primary result Search verticals Goal: ▪ To understand the vertical preference of user on LinkedIn search
SIGIR 2019 179 Query Intention Model: Challenges ▪ Complicated Semantics Job? Job Job!
SIGIR 2019 180 Query Intention Model: Challenges ▪ Personalization ● Query: Blockchain Tech ● Job seeker looks for Blockchain Technology job ● Company named Blockchain Tech ● Learning course on Blockchain Tech ● Video about blockchain technology ● Recruiter looks for candidates with Blockchain tech skill ● Content on Blockchain Technology ● …...
SIGIR 2019 181 CNN Based Query Intention Model CNN for Semantic Feature Extraction ▪ Word/query representations ▪ Generalization power ▪ Word n-gram patterns Personalization ▪ Member-level Features ...
SIGIR 2019 182 Query Intent - Experiment Results ▪ Offline Results Overall Accuracy F1 on PEOPLE F1 on JOB Baseline (ML Model) - - - CNN +2.9% +11.9% +1.7% ▪ Online Results – +0.65% JOB Ctr At 1 Serp – +0.90% Overall Cluster Ctr, +4.03% Cluster Ctr Via Entity Click
SIGIR 2019 183 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 184 Query Tagger at LinkedIn ● LinkedIn Search Engine Query: Mike LinkedIn Software Engineer Relevant Results Query Results Pool Recall Precision Retrieving (Indexing) Ranking (Ranker) Query Query Tagger Irrelevant Results
SIGIR 2019 185 Search at LinkedIn ● Understanding Queries with Query Tagger Query: Mike LinkedIn Software Engineer Mike LinkedIn Software Engineer FN CN T T ○ Query Tagger for Retrieval FN:{mike} Doc1, Doc2, Doc4, ... CN:{linkedin} Doc2, Doc4, Doc5, Doc6, ... T:{Software} Doc2, Doc3, Doc4, Doc7, ... T:{Engineer} Doc1, Doc2, Doc4, Doc9, ... Index Query Doc IsFnMatch IsLnMatch IsTitleMatch etc. ○ Ranking Features CN: company name​ FN: first name​ T: title​
SIGIR 2019 186 Natural Language Understanding: Query Tagger LinkedIn software engineer data scientist jobs​ CN​ T​ T​ T​ T​ O​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ B-CN: beginning of a company name​ I-CN: Inside of a company name​ B-T: beginning of a job title I-T: Inside of a job title O: Not an entity​ B-PN: beginning of person name …
SIGIR 2019 187 Query Tagger: Logistic Regression LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ N way classification sparse ftr vec ftr 0: whether the current word is “linkedin” ftr 1: whether the current word is “facebook” … ftr n: whether the next word is “software” ftr n+1: whether the next word is “linkedin” ….
SIGIR 2019 188 Query Tagger: CRF LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ CRF Layer sparse ftr vec
SIGIR 2019 189 Query Tagger: CRF + LSTM LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ CRF Layer LSTM hidden state Word embedding
SIGIR 2019 190 Query Tagger Performance SCRF CRF CRF + Bi-LSTM Bi-LSTM HMM Deep & Wide Modeling Structure for Entity Tagging
SIGIR 2019 191 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 192 Natural Language Generation: Query Suggestion ▪ Save User Effort of Rephrasing Informative Queries ▪ Capture Users’ Search Intention ▪ Automatic Query Reformulation
SIGIR 2019 193 Natural Language Generation: Query Suggestion ▪ Traditional Frequency Based Methods – Collect <q1, q2> pairs from search log – Save the frequent pairs in a key-value store ▪ Lack of Generalization – Purely string matching – Cannot handle unseen queries, rare words ▪ Seq2seq: Model Query Reformulation Behavior
SIGIR 2019 194 encoder Query Suggestion: Reformulate to Related Queries ■ Training: the 2nd query is given ■ Maximize linkedin engineering <s> linkedin software linkedin software development </s> development decoder
SIGIR 2019 195 decoderencoder Query Suggestion: Reformulate to Related Queries linkedin engineering <s> linkedin software development </s> ■ Inference: the 2nd query is unknown ■ Beam search instead of greedy search
SIGIR 2019 196 Query Suggestion: How to Handle Online Latency ▪ Latency is strictly constrained for one query – Make it parallel with search ranking Query Generate Ranking Results Generate Suggested Queries Merge for Serving
SIGIR 2019 197 Online Performance ▪ English Market – Coverage: +80% Impressions, +26% CTR – +1% Total job application ▪ I18n Market – +3.2% Successful Searches – +28.6% First Order Actions
SIGIR 2019 198 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 199 Natural Language Generation: Auto-Completion ▪ Given a prefix, predict the completed query, rather than the completed word
SIGIR 2019 200 Auto-completion Challenges ▪ Short Context – How to enrich semantic features ▪ Personalization ▪ Latency Restriction – Have to adopt simple and fast models
SIGIR 2019 201 A Two-step Approach: Generation and Ranking ▪ Candidate Generation – Collect query frequency from search log ▪ Candidate Ranking – Neural Language Model serves as a scoring function
SIGIR 2019 202 Auto-Completion: Neural Language Model as Scoring/Ranking ▪ Achieved 6x speedup with optimization
SIGIR 2019 203 Auto-Completion Online Experiments Neural Auto-completion vs Frequency Based Methods ▪ People Search English Market – +88.75% autocomplete clicks – +0.81% CTR@1 in SERP ▪ Job Search German Market – +3.24% Results clicked – -6.24% Raw searches w/o results – +3.24% Entity view
SIGIR 2019 204 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
SIGIR 2019 205 Natural Language Processing: Document Ranking
SIGIR 2019 206 Neural Ranking Feature Processing Layer
SIGIR 2019 207 Experiments People Search Ranking (NDCG@10) Wide Features CNN +1.32% BERT (google pretrained) +1.52% BERT (linkedin pretrained) +1.96% Offline ▪ Help center ranking (BERT vs CNN) – +9.5% search CTR – +7% search Clicks Online
SIGIR 2019 208 Lessons & Future Trends ● Ground Truth Data Availability ○ Human Labelled Data (Crowdsourcing) ○ Behavior Data ○ Mixture Data ○ Automatic Data Generation with Generative Models (e.g., GANs) ● Model Debugging in Complicated Search Systems ○ Model Interpretation ○ Model Reduction
SIGIR 2019 209 Lessons & Future Trends (Cont’d) ● Internationalization ○ Transfer Learning ○ Multilingual Learning ● Model Generalization ○ Pre-trained Model for Multiple Products ● Efficient Modeling Training and Serving ○ Automatic Hyperparameter Tuning & Structure Learning ○ Memory and Latency Optimization for Efficient Online Serving
SIGIR 2019 210 Feedback ● Your Feedback is Greatly Appreciated Slides will be Available on Website
SIGIR 2019 211 Thank You!

More Related Content

PDF
Deep Learning for Personalized Search and Recommender Systems
byBenjamin Le
 
PPTX
Introduction to Transformer Model
byNuwan Sriyantha Bandara
 
PDF
Deep Natural Language Processing for Search and Recommender Systems
byHuiji Gao
 
PDF
Natural Language Processing NLP (Transformers)
byHichem Felouat
 
PDF
Vector databases and neural search
byDmitry Kan
 
PDF
Building real time analytics applications using pinot : A LinkedIn case study
byKishore Gopalakrishna
 
PDF
Homepage Personalization at Spotify
byOguz Semerci
 
PDF
【論文紹介】Seq2Seq (NIPS 2014)
byTomoyuki Hioki
 
Deep Learning for Personalized Search and Recommender Systems
byBenjamin Le
 
Introduction to Transformer Model
byNuwan Sriyantha Bandara
 
Deep Natural Language Processing for Search and Recommender Systems
byHuiji Gao
 
Natural Language Processing NLP (Transformers)
byHichem Felouat
 
Vector databases and neural search
byDmitry Kan
 
Building real time analytics applications using pinot : A LinkedIn case study
byKishore Gopalakrishna
 
Homepage Personalization at Spotify
byOguz Semerci
 
【論文紹介】Seq2Seq (NIPS 2014)
byTomoyuki Hioki
 

What's hot

PPTX
NLP
byGirish Khanzode
 
PPTX
Word embedding
byShivaniChoudhary74
 
PDF
Transformer Introduction (Seminar Material)
byYuta Niki
 
PPTX
Transformers AI PPT.pptx
byRahulKumar854607
 
PPTX
Serving BERT Models in Production with TorchServe
byNidhin Pattaniyil
 
PDF
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
byMLconf
 
PDF
Deep Learning for Recommender Systems RecSys2017 Tutorial
byAlexandros Karatzoglou
 
PPTX
Learning a Personalized Homepage
byJustin Basilico
 
PDF
Building NLP applications with Transformers
byJulien SIMON
 
PDF
An introduction to the Transformers architecture and BERT
bySuman Debnath
 
PDF
NLP using transformers
byArvind Devaraj
 
PPTX
Introduction to CNN
byShuai Zhang
 
PDF
An introduction to Elasticsearch's advanced relevance ranking toolbox
byElasticsearch
 
PPTX
A Simple Introduction to Word Embeddings
byBhaskar Mitra
 
PDF
Interactive Recommender Systems
byRoelof van Zwol
 
PDF
Natural language processing (NLP) introduction
byRobert Lujo
 
PDF
Query Facet Mapping and its Applications in Streaming Services: The Netflix C...
bySudeep Das, Ph.D.
 
PPTX
Neural Text Embeddings for Information Retrieval (WSDM 2017)
byBhaskar Mitra
 
PDF
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
byAlexandros Karatzoglou
 
PDF
Attention is All You Need (Transformer)
byJeong-Gwan Lee
 
NLP
byGirish Khanzode
 
Word embedding
byShivaniChoudhary74
 
Transformer Introduction (Seminar Material)
byYuta Niki
 
Transformers AI PPT.pptx
byRahulKumar854607
 
Serving BERT Models in Production with TorchServe
byNidhin Pattaniyil
 
Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...
byMLconf
 
Deep Learning for Recommender Systems RecSys2017 Tutorial
byAlexandros Karatzoglou
 
Learning a Personalized Homepage
byJustin Basilico
 
Building NLP applications with Transformers
byJulien SIMON
 
An introduction to the Transformers architecture and BERT
bySuman Debnath
 
NLP using transformers
byArvind Devaraj
 
Introduction to CNN
byShuai Zhang
 
An introduction to Elasticsearch's advanced relevance ranking toolbox
byElasticsearch
 
A Simple Introduction to Word Embeddings
byBhaskar Mitra
 
Interactive Recommender Systems
byRoelof van Zwol
 
Natural language processing (NLP) introduction
byRobert Lujo
 
Query Facet Mapping and its Applications in Streaming Services: The Netflix C...
bySudeep Das, Ph.D.
 
Neural Text Embeddings for Information Retrieval (WSDM 2017)
byBhaskar Mitra
 
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
byAlexandros Karatzoglou
 
Attention is All You Need (Transformer)
byJeong-Gwan Lee
 

Similar to Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)

PPTX
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
byresming1
 
PPTX
Natural Language Processing Advancements By Deep Learning: A Survey
byRimzim Thube
 
PDF
Should we be afraid of Transformers?
byDominik Seisser
 
PDF
Deep Learning Approaches to Text Production 1st Edition Shashi Narayan
bypqxcjbpx6997
 
PDF
CSCE181 Big ideas in NLP
byInsoo Chung
 
PPTX
Short story presentation
byStutiAgarwal36
 
PDF
Frontiers of Natural Language Processing
bySebastian Ruder
 
PDF
Natural Language Processing (NLP)
byYuriy Guts
 
PDF
企業における自然言語処理技術利用の最先端
byYuya Unno
 
PDF
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
PDF
Beyond the Symbols: A 30-minute Overview of NLP
byMENGSAYLOEM1
 
PPTX
Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
byZachary S. Brown
 
PPTX
History of deep learning
byayatan2
 
PDF
ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications
byForward Gradient
 
PPTX
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
PDF
NLP Meetup 2023
byGabiMaeztu
 
PDF
大規模データ時代に求められる自然言語処理 -言語情報から世界を捉える-
byYuya Unno
 
PDF
Deep Learning for Natural Language Processing: Word Embeddings
byRoelof Pieters
 
PPTX
Discover How Scientific Data is Used for the Public Good with Natural Languag...
byBaoTramDuong2
 
PPTX
Natural language processing
byBirger Moell
 
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
byresming1
 
Natural Language Processing Advancements By Deep Learning: A Survey
byRimzim Thube
 
Should we be afraid of Transformers?
byDominik Seisser
 
Deep Learning Approaches to Text Production 1st Edition Shashi Narayan
bypqxcjbpx6997
 
CSCE181 Big ideas in NLP
byInsoo Chung
 
Short story presentation
byStutiAgarwal36
 
Frontiers of Natural Language Processing
bySebastian Ruder
 
Natural Language Processing (NLP)
byYuriy Guts
 
企業における自然言語処理技術利用の最先端
byYuya Unno
 
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
Beyond the Symbols: A 30-minute Overview of NLP
byMENGSAYLOEM1
 
Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
byZachary S. Brown
 
History of deep learning
byayatan2
 
ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications
byForward Gradient
 
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
NLP Meetup 2023
byGabiMaeztu
 
大規模データ時代に求められる自然言語処理 -言語情報から世界を捉える-
byYuya Unno
 
Deep Learning for Natural Language Processing: Word Embeddings
byRoelof Pieters
 
Discover How Scientific Data is Used for the Public Good with Natural Languag...
byBaoTramDuong2
 
Natural language processing
byBirger Moell
 

Recently uploaded

PDF
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
PDF
What is Context Engineering for Agentforce Developer and Architect
byParis Salesforce Developer Group
 
PPTX
structured training deck for beginners Power BI Training
bysomashekharkedar
 
PDF
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
PDF
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
PDF
SF User Group - Account-Contact Relationship Feb 2026
byLukas Lunow
 
PPTX
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
PPTX
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
PPTX
Reading and Writing Skills - Critical Reading ppt.pptx
byMargieJavier2
 
PPTX
Organizational Structure and Design Topic 4
bydavenieralayaay
 
PDF
Machine_Psychology_2050_Strategic_Scenarios.pdf
byKambs Consulting
 
PPTX
250106204341-c238c4fadonebeforelastyear.pptx
byAbduRouf3
 
PDF
Data Science vs Machine Learning vs AI | Industry Skills Explained
bymuhsinam9074
 
PDF
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
PPTX
inbound4709748695955476507 Deformable bodies.pptx
byaltheabautistam
 
PDF
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
PDF
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
PPTX
GENPHYSICS-2.pptx this about general physics. Grade 11 lesson
byGwynMorata
 
PPT
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
PDF
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
What is Context Engineering for Agentforce Developer and Architect
byParis Salesforce Developer Group
 
structured training deck for beginners Power BI Training
bysomashekharkedar
 
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
SF User Group - Account-Contact Relationship Feb 2026
byLukas Lunow
 
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
Reading and Writing Skills - Critical Reading ppt.pptx
byMargieJavier2
 
Organizational Structure and Design Topic 4
bydavenieralayaay
 
Machine_Psychology_2050_Strategic_Scenarios.pdf
byKambs Consulting
 
250106204341-c238c4fadonebeforelastyear.pptx
byAbduRouf3
 
Data Science vs Machine Learning vs AI | Industry Skills Explained
bymuhsinam9074
 
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
inbound4709748695955476507 Deformable bodies.pptx
byaltheabautistam
 
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
GENPHYSICS-2.pptx this about general physics. Grade 11 lesson
byGwynMorata
 
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 

Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)

  • 1.
    Deep Natural LanguageProcessing for Search Systems Weiwei Guo Huiji Gao Jun Shi Bo Long
  • 2.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
  • 3.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
  • 4.
  • 5.
    SIGIR 2019 5 Introduction- Search Systems Search is Everywhere
  • 6.
    SIGIR 2019 6 Introduction- Search Systems Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs SIGIR 2019 InfoInfo Travel
  • 7.
    SIGIR 2019 7 Introduction- Search Systems SIGIR 2019 Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs Info Shopping Food Arrive Paris
  • 8.
    SIGIR 2019 8 Introduction- Search Systems SIGIR 2019 Travel FoodShopping Attractions Connect People Looking or Jobs Learning Materials Info Arrive Paris Attending SIGIR Attractions Learning Materials
  • 9.
    SIGIR 2019 9 Introduction- Search Systems SIGIR 2019 Travel FoodShopping Attractions Learning Materials Connect People Looking or Jobs Info Arrive Paris Attending SIGIR Connect People Looking or Jobs
  • 10.
    SIGIR 2019 10 Introduction- NLP in Search Systems ● Understand Searcher Intention ○ Search Queries ○ User Profiles Best Places to Shop in Paris Best Areas to Stay in Paris Safely Budget? Gender? Location? ... ● Understand Documents ○ Posts, Reviews, Comments ○ Synonyms, Facets, Semantics, etc. ● Matching ○ Query & Doc Matching for Retrieval & Ranking ● Search Assistant ○ Query Reformulation
  • 11.
    SIGIR 2019 11 Introduction- NLP in Search Systems Generate Human Search Engine Generate Assistant Natural Language Understanding (NLU) Natural Language Generation (NLG) Understand Natural Language Data
  • 12.
    SIGIR 2019 12 NaturalLanguage Processing in Search Ecosystem Query Tagging Retrieval & Ranking Raw Query Query Understanding Query Intention Document Understanding Auto Completion Spell Check Query Suggestion Natural Language Generation for Search Assistant Natural Language Understanding Indexing Matching Text Processing Language Detection Tokenization ... ... ... ...
  • 13.
    SIGIR 2019 13 Introduction- Deep Learning for NLP Figure source: Elvis, 2018, https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d
  • 14.
    SIGIR 2019 14 Opportunities- Deep Learning for NLP in Search Systems Why Deep Learning? ● Deep Semantics from High Dimension and Sparse Data ○ Synonymous, Disambiguation, etc. ● Easy Feature Engineering ○ Hand Crafted Features V.S. Auto Feature Representation ● Model Flexibility ○ Model end-to-end process ○ Various NN components to model and cooperate systematically ● Multi-level Feature Representation ○ Hierarchical Representations character -> token -> word -> phrase -> sentence [Young et. al. 2018]
  • 15.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
  • 16.
    Deep Learning for NaturalLanguage Processing Huiji Gao
  • 17.
    SIGIR 2019 17 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 18.
    SIGIR 2019 18 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
  • 19.
    SIGIR 2019 19 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context Sequence of Words w_1 Sequential Modeling w_2 w_3 w_n…... Sequence Encoding (Feature Embedding) Tasks (Classification, etc.) Contains Essential Semantics of the Sequence
  • 20.
    SIGIR 2019 20 aman NLP an NLP engineerI am an am an an NLP NLP EngineerI am am an NLP engineerI Sequential Modeling on Semantic Context I am an NLP engineer I am an NLP engineer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition I am an NLP engineer Je suis un ingénieur en NLP I am an NLP engineer Je suis un ingénieur en NLP Context Strengthening Transformer GAN Context Consistency Global Context I am an NLP engineer Je suis un ingénieur en NLP Je sont une ingénieur en NLP train test Encoding Encoding Encoding
  • 21.
    SIGIR 2019 21 MultilayerPerceptron (MLP) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Figure source: https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn/
  • 22.
    SIGIR 2019 22 ConvolutionalNeural Networks (CNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Zhang et. al. 2015]
  • 23.
    SIGIR 2019 23 RecurrentNeural Networks (RNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context LSTM GRU [LeCun et. al. 2015] github: 2015-08-Understanding-LSTMs
  • 24.
    SIGIR 2019 24 Sequenceto Sequence (Encoder - Decoder) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
  • 25.
    SIGIR 2019 25 AttentionMechanisms MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
  • 26.
    SIGIR 2019 26 Transformer MLP CNN UnigramContext RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Novelty: Eliminate Recurrence transformer http://jalammar.github.io/illustrated-transformer/
  • 27.
    SIGIR 2019 27 GenerativeAdversarial Networks (GANs) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Lee, et al., KDD 2018]
  • 28.
    SIGIR 2019 28 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 29.
    SIGIR 2019 29 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
  • 30.
    SIGIR 2019 30 ▪Representation Learning Sequence of Words w_1 Representation Learning w_2 w_3 w_n…... T’_1 T’_2 T’_3 T’_n…... Word Representation (Embeddings) Deep Learning for Natural Language Processing
  • 31.
    SIGIR 2019 31 RepresentationLearning Word2Vec Static Word Embedding [Mikolov et. al. 2013]
  • 32.
    SIGIR 2019 32 RepresentationLearning Word2Vec ELMO Static Word Embedding Contextual Word Embedding - Bidirectional LSTM [Peters et. al. 2018]
  • 33.
    SIGIR 2019 33 RepresentationLearning Word2Vec ELMO GPT Static Word Embedding Contextual Word Embedding Contextual Word Embedding - Bidirectional LSTM - One Direction Transformer [Radford et. al. 2018]
  • 34.
    SIGIR 2019 34 RepresentationLearning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding BERT - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer [Devlin et. al. 2018]
  • 35.
    SIGIR 2019 35 RepresentationLearning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding XLNET Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding XLNet - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer - Bidirectional, Learn correlations among masked tokens Pre-trained NLP Model [Yang et. al. 2018]
  • 36.
    SIGIR 2019 36 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching MLP, CNN, RNN, Transfomer, ... RNN, Transfomer, ... Seq2seq, Transformer, GAN, ... CNN, RNN, ... Word2Vec, ELMO, GPT, BERT, XLNet, ...
  • 37.
    SIGIR 2019 37 References- Deep Learning for Natural Language Processing ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [LeCun et. al, 2015] Deep learning, Nature, vol. 521, no. 7553, pp. 436–444, 2015. ▪ [Lee et. al, 2018] Rare query expansion through generative adversarial networks in search advertising, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ [Mikolov et. al. 2013] Distributed representations of words and phrases and their compositionality, in Advances in neural information processing systems, 2013 ▪ [Mikolov et. al. 2013] Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, 2013. ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Robinson, 2018] https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn, 2018. ▪ [Shi et. al, 2018] Neural Abstractive Text Summarization with Sequence-to-Sequence Models, arXiv preprint arXiv:1812.02303v2, 2018. ▪ [Yang et. al. 2019] XLNet: Generalized Autoregressive Pretraining for Language Understanding, arXiv preprint arXiv:1906.08237v1, 2019 ▪ [Young et. al. 2018] Recent Trends in Deep Learning Based Natural Language Processing, arXiv preprint arXiv:1708.02709v8, 2018 ▪ [Zhang et. al. 2015] A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, arXiv preprint arXiv:1510.03820, 2015.
  • 38.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
  • 39.
    Deep NLP inSearch Systems - Language Understanding Jun Shi
  • 40.
    SIGIR 2019 40 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
  • 41.
    SIGIR 2019 41 EntityTagging ▪ Problem statement ▪ Traditional statistical models – Hidden Markov Model – Maximum Entropy Markov Model – (Semi-Markov) Conditional Random Field ▪ Deep learning models – Input layer – Context layer – Decoder layer ▪ Training dataset ▪ Evaluation
  • 42.
    SIGIR 2019 42 EntityTagging - Problem Statement ▪ A named entity, a word or a phrase that clearly identifies one item from a set of other items that have similar attributes. [Li et. al. 2018] ▪ Entity tagging (Named Entity Recognition, NER), the process of locating and classifying named entities in text into predefined entity categories. Washington D.C. was named after George Washington. LOCATION PERSON
  • 43.
    SIGIR 2019 43 Motivation ▪Efficiency – Looking only for named entities can speed up search. ▪ Precision – Matching both named entity and tags can increase search precision. ▪ Quality – Ranking retrieved results by considering tags improves search quality.
  • 44.
    SIGIR 2019 44 TraditionalStatistical Models ▪ Generative model – Hidden Markov Model [Baum, et. al. 1966] ▪ Discriminative model – Maximum Entropy Markov Model [McCallum, et. al. 2000] – (Semi-Markov) Conditional Random Field [Lafferty, et. al. 2001]
  • 45.
    SIGIR 2019 45 HiddenMarkov Model Generative model, model joint probability Pr(T, W) instead of conditional probability Pr(T | W). T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 46.
    SIGIR 2019 46 MaximumEntropy Markov Model Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 47.
    SIGIR 2019 47 ConditionalRandom Field Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 48.
    SIGIR 2019 48 Semi-MarkovConditional Random Field Each tag can correspond to a variable-length phrase. T1 T2 W1 W2 W3 tags words T3 W4 W5 W6 T0
  • 49.
    SIGIR 2019 49 Summary- Traditional Statistical Models Model Pros Cons Hidden Markov Model training is simple when states are observed difficult to include features Maximum Entropy Markov Model easy to include features suffer from “label bias problem” (prefer states with lower number of transitions) Conditional Random Field easy to include features, do not suffer from “label bias problem” training is relatively complex
  • 50.
    SIGIR 2019 50 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 51.
    SIGIR 2019 51 InputLayer - Word Embedding [Yadav, et. al. 2018]
  • 52.
    SIGIR 2019 52 InputLayer - Char Embedding [Yadav, et. al. 2018]
  • 53.
    SIGIR 2019 53 InputLayer - Word and Char Embedding [Yadav, et. al. 2018]
  • 54.
    SIGIR 2019 54 Summary- Input Layer word embedding char embedding external features, lexicon, etc. char CNN/RNN word
  • 55.
    SIGIR 2019 55 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 56.
    SIGIR 2019 56 ContextLayer - CNN embedding CNN P A D P A D Tag Decoder [Li et. al. 2018]
  • 57.
    SIGIR 2019 57 ContextLayer - RNN [Li et. al. 2018]
  • 58.
    SIGIR 2019 58 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 59.
    SIGIR 2019 59 TagDecoder - MLP+softmax [Li et. al. 2018]
  • 60.
    SIGIR 2019 60 TagDecoder - CRF [Li et. al. 2018]
  • 61.
    SIGIR 2019 61 TagDecoder - RNN [Li et. al. 2018]
  • 62.
    SIGIR 2019 62 Pre-Trainingand Fine-Tuning BERT GPT ELMo [Devlin et. al. 2018] [Peters et. al. 2018][Radford et. al. 2018]
  • 63.
    SIGIR 2019 63 EntityTagging Evaluation Exact-match evaluation ▪ segment match ▪ tag match
  • 64.
    SIGIR 2019 64 EntityTagging Training Dataset [Li et. al. 2018]
  • 65.
    SIGIR 2019 65 EntityTagging on CoNLL03 English Source Method F1 score [Passos et al. 2014] CRF 90.90 [Huang et al. 2015] Bi-LSTM+CRF 84.26 [Collobert et al. 2011] Conv-CRF 89.59 [Kuru et al. 2016] char embedding 84.52 [Chiu and Nichols 2015] word + char embedding 91.62 [Devlin et. al. 2018] BERT Large 92.8
  • 66.
    SIGIR 2019 66 References- Entity Tagging ▪ [Baum et. al. 1966] Baum, L. E.; Petrie, T. (1966). "Statistical Inference for Probabilistic Functions of Finite State Markov Chains". The Annals of Mathematical Statistics. 37 (6): 1554–1563. ▪ [Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 ▪ [Collins and Singer 1999] Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ▪ [Collobert et. al. 2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. ▪ [Collobert and Weston 2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [Huang et. al. 2015] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 ▪ [Kuru et. al. 2016] Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. Charner: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911–921 ▪ [Lafferty et. al. 2001] Lafferty, J., McCallum, A., Pereira, F. (2001). "Conditional random fields: Probabilistic models for segmenting and labeling sequence data". Proc. 18th International Conf. on Machine Learning. Morgan Kaufmann. pp. 282–289 ▪ [Li et. al. 2018] Jing Li, Aixin Sun, Jianglei Han, Chenliang Li, A Survey on Deep Learning for Named Entity Recognition, Dec. 2018, arXiv preprint, https://arxiv.org/abs/1812.09449 ▪ [McCallum et. al. 2000] McCallum, Andrew; Freitag, Dayne; Pereira, Fernando (2000). "Maximum Entropy Markov Models for Information Extraction and Segmentation" (PDF). Proc. ICML 2000. pp. 591–598
  • 67.
    SIGIR 2019 67 References(continued) ▪ [Passos 2014] Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367 ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Yadav et. al. 2018] Vikas Yadav, Steven Bethard, A Survey on Recent Advances in Named Entity Recognition from Deep Learning models, Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158 Santa Fe, New Mexico, USA, August 20-26, 2018.
  • 68.
    SIGIR 2019 68 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
  • 69.
    SIGIR 2019 69 EntityDisambiguation ▪ Problem statement ▪ Motivation ▪ Challenges ▪ System architecture – Candidate Entity Generation – Candidate Entity Selection – Joint entity tagging and disambiguation ▪ Evaluation
  • 70.
    SIGIR 2019 70 EntityDisambiguation - Problem Statement ▪ Also known as entity linking ▪ Resolves mentions to entities in a given knowledge base – Mentions are mostly named entities – Example of knowledge base: freebase, wikipedia ▪ Examples: Jaguar – The prey saw the jaguar cross the jungle. KB:Jaguar – The man saw a jaguar speed on the highway. KB:Jaguar_Cars
  • 71.
    SIGIR 2019 71 Motivation Increasethe quality of the retrieved results Example: Which Michael Jordan were you looking for?
  • 72.
    SIGIR 2019 72 Challenges ▪Name variation: New York vs Big Apple ▪ Ambiguity: Michael Jordan ▪ Metonymy: Beijing (city or Chinese government) ▪ Absence: no entries in knowledge base ▪ Evolving information: new company names, e.g. tic-tok.
  • 73.
    SIGIR 2019 73 SystemArchitecture Candidate Generation Candidate selection 1. Candidate entity generation a. Name dictionary based techniques. b. Surface form expansion from the local document. c. Methods based on search engines. 2. Candidate entity selection a. Graph-based methods b. Text-based methods knowledge base Documents Linked results
  • 74.
    SIGIR 2019 74 CandidateEntity Generation ▪ Name dictionary based techniques – Entity pages – Redirect pages – Disambiguation pages – Bold phrases from the first paragraphs – Hyperlinks in Wikipedia articles ▪ Surface form expansion – Heuristic based methods – Supervised learning methods ▪ Methods based on search engines [Shen et. al. 2014]
  • 75.
    SIGIR 2019 75 CandidateEntity Selection - Graph based models ▪ knowledge graph lends itself to graph based methods. ▪ Build a weighted undirected graph with mentions and candidate entities as nodes – weights on mention-entity edges. – weights on entity-entity edge. ▪ Compute an optimal subgraph: – optimality is model dependent. – contains all mentions – one entity per mention mention entity
  • 76.
    SIGIR 2019 76 AnExample Graph [Hoffart, et. al. 2011]
  • 77.
    SIGIR 2019 77 CandidateEntity Selection - Text-Based Models ▪ Idea: find agreement (similarity) between entity and mention. – Example: computational intelligence by Michael Jordan and Stuart Russell from UC Berkeley increasing context ● wiki/Michael_Jordan ● wiki/Michael_I._Jordan▪ Models according to context. – Direct Models. ▪ use mention and candidate entities only. – Local Models ▪ use local context around the mention. – Coherence Models. ▪ use mentions and entities in a document. – Collaborative Models. ▪ use mentions and entities across related documents.
  • 78.
    SIGIR 2019 78 Individualvs Joint Approach Two types of approaches according to how mentions are resolved. ▪ Individual approach – Handle one mention at a time. – Rank the candidate entities. – Relatively low complexity. ▪ Joint approach – Treat all mentions in a document as a sequence. – Jointly optimize the mention-entity pairs. – High complexity, usually resorting to approximations.
  • 79.
    SIGIR 2019 79 Features Differentfeatures are used in candidate selection modeling. ▪ Traditional Features – entity popularity, entity type. – surface form similarity between entity and mention. – coherence between mapped entities ▪ Deep Features [Yamada et. al. 2016] – word embedding – entity embedding – similarity between words, context and candidate entities.
  • 80.
    SIGIR 2019 80 Example:Entity Disambiguation with Attention training: max-margin loss [Genea et. al. 2017]
  • 81.
    SIGIR 2019 81 JointEntity Tagging and Disambiguation ▪ Previously, we find entities first, then link them. But if entities are identified wrong, disambiguation will likely fail too. – Over-split: Romeo and Juliet by Shakespeare – Under-split: Baby Romeo and Juliet were born hours apart. ▪ We could do entity tagging and disambiguation jointly. – Over-generate candidate mentions – Generate possible entities per mention – Score non-overlapping mention-entity pair jointly. [Sil, et. al.2013]
  • 82.
    SIGIR 2019 82 AJoint Neural Entity Tagging and Disambiguation [Kolitsas, et. al. 2018] Select mentions that have at least one possible entity ● Trained on golden mention-entity pairs. ● Used max-margin loss. cosine similarity between current entity and average of other entities in the document
  • 83.
    SIGIR 2019 83 EntityDisambiguation Evaluation ▪ Entity-tagging-style F1 score – A link is considered correct only if the mention matches the gold boundary and the linked entity is also correct ▪ Accuracy – another common metric, simpler than F1 score. ▪ TAC-KBP B-Cubed+ F1 score – not widely used
  • 84.
    SIGIR 2019 84 References- Named Entity Disambiguation ▪ [Cucerzan 2007] Cucerzan, Silviu. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP- CoNLL): 708–716. ▪ [Ganea et. al. 2017] Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2609–2619. Association for Computational Linguistics. ▪ [Globerson et. al. 2016] Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In ACL (1). ▪ [Han et. al. 2011] Han, Xianpei; Sun, Le; Zhao, Jun. Collective Entity Linking in Web Text: A Graph-based Method. ▪ [Hoffart, et. al. 2011] Johannes Hoffart, Mohamed A. Yosef, Ilaria Bordino, Hagen F¨urstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Method in Natural Language Processing, pages 782–792. Association for Computational Linguistics. ▪ [Kolitsas et al., 2018] Nikolaos Kolitsas, Octavianeugen Ganea, and Thomas Hofmann. End-to-end neural entity linking. In CoNLL, 2018. ▪ [Kulkarni et. al. 2009] S. Kulkarni, A. Singh, G. Ramakrishnan, and S. Chakrabarti, Collective annotation of Wikipedia entities in web text, in SIGKDD, 2009, pp. 457–466 ▪ [Ling et al.2015] Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315–328. ▪ [Le et. al., 2018] Phong Le and Ivan Titov. Improving entity linking by modeling latent relations between mentions. In ACL, 2018. ▪ [Rao et. al. 2013] Rao, Delip; McNamee, Paul; Dredze, Mark (2013). Entity Linking: Finding Extracted Entities in a Knowledge Base. Multi-source, Multilingual Information Extraction and Summarization. Springer Berlin Heidelberg: 93–115
  • 85.
    SIGIR 2019 85 References(continued) ▪ [Raiman et. al. 2018] Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual Entity Linking by Neural Type System Evolution. In Proc. of AAAI. ▪ [Shen et. al. 2014] Wei Shen, Jianyong Wang, and Jiawei Han. 2014. Entity linking with a knowledge base: Issues, techniques, and solutions. TKDE ▪ [Sil et. al. 2013] Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2369–2374. ACM ▪ [Sil et. al. 2016] Sil, A., and Florian, R. 2016. One for all: Towards language independent named entity linking. ACL. ▪ [Yamada et. al. 2016] Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. CoNLL 2016, page 250.
  • 86.
    SIGIR 2019 86 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
  • 87.
    SIGIR 2019 87 IntentClassification ▪ Problem statement ▪ Deep Learning Models – fastText – CNN – Bi-RNN + Attention
  • 88.
    SIGIR 2019 88 IntentClassification - Problem Statement ▪ Web search intent can be classified into 3 class: [Broder 2002] – Navigational. The immediate intent is to reach a particular site. ▪ Greyhound Bus. Probable target http://www.greyhound.com – Informational. The intent is to acquire some information assumed to be present on one or more web pages. ▪ San Francisco – Transactional. The intent is to perform some web-mediated activity. ▪ Shopping activities ▪ In conversational AI, intent is usually task-specific, e.g. – I would like to book a flight from SFO to CDG: FlightBooking – What software can I use to view epub documents: SoftwareRecommendation. ▪ We focus on the latter in this tutorial.
  • 89.
    SIGIR 2019 89 IntentClassification - Methods ▪ Traditional methods – Features: bag of words, n-gram, TF-IDF. – Models: logistic regression, naive Bayes, SVM, random forest. ▪ Deep learning methods – Word embedding + linear classifier (fastText) [Joulin et. al. 2016]. – Convolutional neural networks [Hashemi 2016] . – Bi-RNN + attention (joint slot filling) [Liu et al.2016] .
  • 90.
    SIGIR 2019 90 IntentClassification - fastText word and N gram embedding features Average softmax [Joulin et. al. 2016]
  • 91.
    SIGIR 2019 91 IntentClassification - CNN [Hashemi 2016]
  • 92.
    SIGIR 2019 92 IntentClassification - Bi-RNN+Attention [Liu et. al. 2016]
  • 93.
    SIGIR 2019 93 References- Intention Classification ▪ [Broder 2002] A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, 2002. ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Joulin et. al. 2016] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ▪ [Lai et al.2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267–2273 ▪ [Liu et al.2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 ▪ [Zhou et al.2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 207. ▪ [Kim et. al. 2016] Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In Proceedings of SLT. ▪ [Hashemi 2016] Homa B Hashemi, Amir Asiaee, and Reiner Kraft. 2016. Query intent detection using convolutional neural networks. In International Conference on Web Search and Data Mining, Workshop on Query Understanding. ▪ [Liu et. al. 2016]. Liu and I. Lane, “Attention-based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling,” in Interspeech, 2016. ▪ [Shi et. al. 2016] Y. Shi, K. Yao, L. Tian, and D. Jiang, “Deep lstm based feature mapping for query classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1501–1511.
  • 94.
    Deep NLP inSearch Systems - Document Retrieval & Ranking Jun Shi, Weiwei Guo
  • 95.
    SIGIR 2019 95 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Query Suggestion: word-level sequence to sequence – Spell Correction: character-level sequence to sequence – Auto Complete: partial sequence to sequence
  • 96.
    SIGIR 2019 96 DocumentRetrieval and Ranking ▪ Efficient Candidate Retrieval ▪ Deep Neural Ranking
  • 97.
    SIGIR 2019 97 EfficientCandidate Retrieval ▪ Syntactic retrieval – based on string matching – use inverted index – can include different fields (name, title, etc). ▪ Semantic retrieval – based on vector space representation. – approximate nearest neighbor search
  • 98.
    SIGIR 2019 98 SyntacticRetrieval Query: mike software engineer mike Doc1, Doc2, Doc4, ... software Doc2, Doc3, Doc4, Doc7, ... engineer Doc1, Doc2, Doc4, Doc9, ... Inverted index Results: Doc2, Doc4 It won’t retrieve a document contains “mike is a software developer,...”
  • 99.
    SIGIR 2019 99 SemanticRetrieval - Concept mike software engineer [0.3, 1.27, -3.4, ...] embedding mike software engineer The problem becomes nearest neighbor search [Mitra et. al. 2017]
  • 100.
    SIGIR 2019 100 SemanticRetrieval - Vector Generation ▪ Bag of words – count based, TF-IDF vectors. ▪ Embedding vectors – word embedding: word2vec, Glove, BERT, etc. – sentence/document embedding ▪ universal sentence encoder [Cer et. al. 2018] ▪ Gaussian document representation [Giannis et. al. 2017] ▪ power mean concatenation [Rücklé et. al. 2018]
  • 101.
    SIGIR 2019 101 ApproximateNearest Neighbor Search ▪ Exact nearest neighbor search – complexity is linear with the size of dataset, not suitable for large dataset. ▪ Approximate nearest neighbor search – allow bounded errors. – complexity is sublinear – many algorithms available ▪ product quantization [Jégou et. al. 2016] ▪ hierarchical navigable small world graphs [Malkov et. al. 2016]
  • 102.
    SIGIR 2019 102 References- Efficient Candidate Retrieval ▪ [Cer et. al. 2018] D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, Y. Sung, B. Strope, and R. Kurzweil. Universal sentence encoder. CoRR, abs/1803.11175, 2018. URL http://arxiv.org/abs/1803.11175. ▪ [Giannis et. al. 2017] Giannis Nikolentzos, Polykarpos Meladianos, Franc¸ois Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. 2017. Multivariate gaussian document representation from word embeddings for text categorization. In EACL ▪ [Manning et. al. 2008] Christopher D. Manning Prabhakar Raghavan Hinrich Schütze, Introduction to Information Retrieval Cambridge University Press New York, NY, USA, 2008 ▪ [Mitra et. al. 2017] Mitra, B., & Craswell, N. (2017). Neural models for Information Retrieval. CoRR, abs/1705.01509. ▪ [Rücklé et. al. 2018] A. Rücklé, S. Eger, M. Peyrard, and I. Gurevych. Concatenated p-mean word embeddings as universal cross- lingual sentence representations. CoRR, abs/1803.01400, 2018. URL http://arxiv.org/abs/1803.01400. ▪ [Malkov et. al. 2016] Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. arXiv:1603.09320, 2016 ▪ [Jégou et. al. 2016] Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2011, 33 (1), pp.117- 128. ff10.1109/TPAMI.2010.57ff. ffinria-00514462v2
  • 103.
    SIGIR 2019 103 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Auto Completion – Query Reformulation – Spell Correction
  • 104.
    SIGIR 2019 104 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods q, d1 q, d2 s1 s2 learning-to-rank loss
  • 105.
    SIGIR 2019 105 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks q d1 q d2q, d1 q, d2
  • 106.
    SIGIR 2019 106 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks q q d1 d2
  • 107.
    SIGIR 2019 107 TraditionalRanking Features ▪ Hand-crafted features – query/document matching features ▪ Cosine similarity between query and doc title ▪ Clickthrough rate from query to this doc based on search log ▪ ……. – Document alone ▪ popularity ▪ number of incoming links ▪ …...
  • 108.
    SIGIR 2019 108 Learningto Rank ▪ Pointwise ranking – Logistic regression (Burges, 2010) ▪ Pairwise ranking ▪ Listwise ranking – Cross entropy
  • 109.
    SIGIR 2019 109 DeepNeural Ranking ▪ Focus on compute query/document score s1 s2 learning-to-rank loss q q d1 d2 ▪ Two categories: – Siamese Networks – Interaction based Networks
  • 110.
    SIGIR 2019 110 DeepStructured Semantic Model feed-forward (Huang et al., 2013) 30k 300 128 30k 300 128 high dimensional query/doc vector query doc1 sum of word embeddings cos sim
  • 111.
    SIGIR 2019 111 ModelingWord Sequence by CNN CNN (Shen et al., 2014) d by n matrix 300 128 token embedding matrix d by n matrix 300 128 feed-forward cos sim
  • 112.
    SIGIR 2019 112 SiameseNetworks ▪ Pros: – Generalization; semantic matching – Efficient; doc embs can be precomputed ▪ Cons: – Lexical features lost: people/company names, rare words
  • 113.
    SIGIR 2019 113 Interaction-basedNetworks (Guo et al., 2016) query: jobs at linkedin …... document Feed-forward z 1 z 2 z 3 s cosine sim between query words and doc words sim_vec1 sim_vec2 sim_vec3 histogram1 histogram2 histogram2 historgrams based on sim vectors g1 g2 g3 0.4 0.1 0.5
  • 114.
    SIGIR 2019 114 DeepNeural Ranking Summary ▪ Only focus on query/doc scoring ▪ End-to-end models ▪ Two popular architectures Siamese Network Interaction Network Match topical matches lexical matches Latency small large
  • 115.
    SIGIR 2019 115 References- Deep Neural Ranking ▪ Huang, Po-Sen, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. "Learning deep structured semantic models for web search using clickthrough data." In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338. ACM, 2013. ▪ Shen, Yelong, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. "A latent semantic model with convolutional-pooling structure for information retrieval." In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp. 101-110. ACM, 2014. ▪ Rodrigo Nogueira, Kyunghyun Cho, Passage re-ranking with BERT, 2019 ▪ Burges. "From ranknet to lambdarank to lambdamart: An overview." Learning. 2010. ▪ Guo, Jiafeng, Yixing Fan, Qingyao Ai, and W. Bruce Croft. "A deep relevance matching model for ad-hoc retrieval." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55-64. ACM, 2016. ▪ Mitra, Bhaskar, Fernando Diaz, and Nick Craswell. "Learning to match using local and distributed representations of text for web search." In Proceedings of the 26th International Conference on World Wide Web, pp. 1291-1299. International World Wide Web Conferences Steering Committee, 2017.
  • 116.
    Deep NLP inSearch Systems - Language Generation for Search Assistance Weiwei Guo
  • 117.
    SIGIR 2019 117 DeepNLP in Search Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Search Assistance – Auto Completion – Query Reformulation – Spell Correction
  • 118.
    SIGIR 2019 118 LanguageGeneration for Search Assistance ▪ Goal: improve user experience by interacting with users ▪ NLP: Language generation Common Difference ▪ Character/word modeling ▪ Generation models: – language modeling, seq2seq ▪ Auto Completion ▪ Query Reformulation ▪ Spell Correction
  • 119.
    SIGIR 2019 119 LanguageGeneration for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation – word-level seq to seq ▪ Spell Correction – character-level seq to seq
  • 120.
    SIGIR 2019 120 QueryAuto-Completion ▪ Problem statement: save user keystrokes by predicting the entire query
  • 121.
    SIGIR 2019 121 Challenges ▪Limited Context – Hard to extract features due to limited words in a query ▪ Latency – The search component with most strict requirement on latency
  • 122.
    SIGIR 2019 122 ▪Traditional approach: 2-step approach Candidate Generation and Ranking soft softbank softball softmax software engineer …... software engineer salary soft skills softball softbank …... candidate generation candidate ranking
  • 123.
  • 124.
    SIGIR 2019 124 CandidateGeneration ▪ Collect completed queries and associated frequency from search log ▪ Efficiently retrieve most frequent queries starting with the prefix – Using a trie data structure soft softbank 100 softball 50 softmax 40 software engineer 35 …... candidate generation
  • 125.
    SIGIR 2019 125 CandidateGeneration for Rare Prefix (Mitra & Craswell, 2015) ▪ No such prefix in search log – “cheapest flights from seattle to” “cheapest flights from seattle to” “to” to dc to sfo to airport to seattle …... 100 50 40 35
  • 126.
    SIGIR 2019 126 ▪Challenge: very few features can be extracted Candidate Ranking soft softbank softball softmax software engineer …... software engineer salary soft skills softball softbank …... candidate generation candidate ranking frequency 100 50 20 15
  • 127.
    SIGIR 2019 127 Context-Aware( Bar-Yossef & Kraus, 2011) ▪ Key idea: identify the candidates most similar to previous queries infant n (candidates) infant nike shoes infant nutrition …... infant, nike, shoe, adidas, clothes... infant, nutrition, baby, eat, food... …... (previous queries) baby eating disorder …... baby, eating, disorder, nutrition, food... …... sim score 0.1 0.8
  • 128.
    SIGIR 2019 128 Personalization(Shokouhi, 2013) Feature list Same age Same gender Same region …... query prefix is “i”
  • 129.
    SIGIR 2019 129 Agenda Candidate Generation Candidate Ranking Context aware Personalization DeepNLP methods CNN for Ranking End-to-end: Language Model Personalization Traditional methods
  • 130.
    SIGIR 2019 130 ApplyDeep Models in Ranking ▪ Measuring the semantic coherence between prefix and suffix embsize tri-letter length Query Prefix Query Suffix CNN conv and max-pooling text emb cos sim Cosine similarity (Mitra & Craswell, 2015)
  • 131.
    SIGIR 2019 131 presidentbarack obaman Language Modeling for Auto-Completion ‘ ’ softmax over chars ‘b’ LSTM char emb ‘t’ char emb ‘ ’ (Park & Chiba, 2017) ▪ Training: Character level language model word emb <INC> word emb `president` + word embedding
  • 132.
    SIGIR 2019 132 charemb ‘t’ (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time 0.05 ‘e’ ‘b’ ‘a’ 0.4 0.3 ‘e’ ‘b’ ‘a’ 0.1 0.5 0.1 ‘a’ 0.4 Language Modeling for Auto-Completion
  • 133.
    SIGIR 2019 133 (Park& Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time ▪ Greedy vs beam search ‘b’ ‘a’ ‘c’ ‘aa’ ‘ab’ ‘ac’ ‘ba’ ‘bb’ ‘bc’ ‘ca’ ‘cb’ ‘cc’ 0.01 0.3 0.3 0.1 0.01 0.01 0.26 0.01 0.01 ‘ab’ ‘ac’ ‘ca’ Language Modeling for Auto-Completion
  • 134.
    SIGIR 2019 134 Personalization ▪Embeddings for User Ids LSTM softmax over chars char emb ‘t’ word emb <INC> (Fiorin & Lu, 2018) user id emb
  • 135.
    SIGIR 2019 135 QueryAuto-Completion: Summary ▪ Traditional methods: hard to extract features ▪ Deep language model framework: – Very flexible to incorporate personalized/contextualized information – An end-to-end solution – Train: all parameters are optimized together – Test: generation and ranking at the same time – Cons – Time-consuming – May generate wrong words
  • 136.
    SIGIR 2019 136 Reference ▪Mitra, Bhaskar, and Nick Craswell. "Query auto-completion for rare prefixes." In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 1755-1758. ACM, 2015. ▪ Park, Dae Hoon, and Rikio Chiba. "A neural language model for query auto-completion." In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189-1192. ACM, 2017. ▪ Bar-Yossef, Ziv, and Naama Kraus. "Context-sensitive query auto-completion." In Proceedings of the 20th international conference on World wide web, pp. 107-116. ACM, 2011. ▪ Shokouhi, Milad. "Learning to personalize query auto-completion." In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pp. 103-112. ACM, 2013. ▪ Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018). ▪ Fiorini, Nicolas, and Zhiyong Lu. "Personalized neural language models for real-world query auto completion." arXiv preprint arXiv:1804.06439 (2018).
  • 137.
    SIGIR 2019 137 LanguageGeneration for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation – word-level seq to seq ▪ Spell Correction – character-level seq to seq
  • 138.
    SIGIR 2019 138 QueryReformulation ▪ Problem Statement: automatically reformulate the previous query Search results for “fb developer” Reformulated queries for “fb developer”
  • 139.
  • 140.
    SIGIR 2019 140 TraditionalApproach: Collaborative Filtering (Rida et al., 2012) ▪ Collect query pairs issued by the same user from search log ▪ Treat each query as an ID, and build a query-to-query matrix – Value in the matrix is TF-IDF
  • 141.
    SIGIR 2019 141 Agenda collaborative filtering Traditionalmethods seq2seq conversational seq2seq GAN for seq2seq Deep NLP methods
  • 142.
    SIGIR 2019 142 QueryReformulation as a Translation Task ▪ Sequence-to-sequence modeling (He et al, 2016) decoderencoder facebook developer <s> facebook software facebook software development </s> development ▪ Directly modeling the words in a query
  • 143.
    SIGIR 2019 143 ConversationalQuery Reformulation ▪ Conversational queries (Ren et al, 2018) first query (q1) second query (q2) when was California founded? who is its governor? California population in 1990 how tall is kobe bryant? what about Lebron James? when was the last summer Olympics? and the winter one? summarized query (q3) who is California’s governor? population of California in 1990 how tall is Lebron James? when was the last winter Olympics? ▪ Goal: summarize the conversation in one query
  • 144.
    SIGIR 2019 144 encoderdecoder w1 w2 <s> w1 w2 w1 w2 </s> SEP w1 w2 Conversational Query Reformulation ▪ 1st: concatenated seq2seq (Ren et al, 2018)
  • 145.
    SIGIR 2019 145 ConversationalQuery Reformulation ▪ 2nd: Attention over attention (Ren et al, 2018) 2nd query h1 h2 h3 c1 c2 c h1 h2 h31st query 3rd query (decoder) ht h0
  • 146.
    SIGIR 2019 146 GANfor seq2seq ▪ Motivation: Discrepancy in decoder of seq2seq (Lee et al, 2018) decoder (training) <s> w1 w2 w1 w2 </s> – Training: inputs are the gold-standard words – Testing: inputs are the previous predicted words ▪ Goal: Resolve training/testing discrepancy with GAN decoder (testing) <s> w’1 w’2 </s>
  • 147.
    SIGIR 2019 147 Discriminator Generator GANfor seq2seq (Lee et al, 2018) <s> software ui </s> <s> software engineer 1/0generated sequence software ui <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
  • 148.
    SIGIR 2019 148 QueryReformulation: Summary ▪ seq2seq framework: – Directly modeling the words – Very flexible to incorporate session information – Achieves great performance (no character modeling)
  • 149.
    SIGIR 2019 149 Reference ▪Reda, Azarias, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah. "Metaphor: a system for related search recommendations." In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 664- 673. ACM, 2012. ▪ He, Yunlong, Jiliang Tang, Hua Ouyang, Changsung Kang, Dawei Yin, and Yi Chang. "Learning to rewrite queries." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1443-1452. ACM, 2016. ▪ Ren, Gary, Xiaochuan Ni, Manish Malik, and Qifa Ke. "Conversational query understanding using sequence to sequence modeling." In Proceedings of the 2018 World Wide Web Conference, pp. 1715-1724. International World Wide Web Conferences Steering Committee, 2018. ▪ Lee, Mu-Chu, Bin Gao, and Ruofei Zhang. "Rare query expansion through generative adversarial networks in search advertising." In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018.
  • 150.
    SIGIR 2019 150 LanguageGeneration for Search Assistance ▪ Auto Completion – partial seq to seq ▪ Query Reformulation: – word-level seq to seq ▪ Spell Correction – character-level seq to seq
  • 151.
  • 152.
    SIGIR 2019 152 SpellCorrection ▪ Why spell correction: – Reduce the no results ▪ Challenge: – Many rare words (people/company names) look like spell errors query similar query has error? tumblr tumble No tumblw tumble Yes galaxy s10e galaxy s10 No galaxy s10d galaxy s10 Yes – Modeling characters and words at the same time
  • 153.
    SIGIR 2019 153 Agenda Candidate Generation fora word Candidate generation for a query Candidate ranking Traditional methods
  • 154.
    SIGIR 2019 154 CandidateGeneration for a Word (Whitelaw et al 2009) ▪ Goal: – given “tumblw”, suggest “tumblr”, “tumble”... – given “tumblr”, suggest “tumble”, but not “tumblw” ▪ Key challenge: what is a legit word? (“tumblr” 👍 “tumblw” 👎) – Coverage of a dictionary is not enough ▪ Solution: use statistics in web noisy data – Correct words appear more frequent than incorrect words
  • 155.
    SIGIR 2019 155 CandidateGeneration for a Word (Whitelaw et al 2009) tumblr tumblw …... <tumblr, tumblw> <tumblw, tumblr> …... edit distance <tumblr, tumblw, 2> <tumblw, tumblr, 50> …... Filter by frequency of context term list confusion word pairs <word, candidate, count> ▪ For the context “social media X”: – freq(“social media tumblw”) < freq(“social media tumblr”)
  • 156.
    SIGIR 2019 156 CandidateGeneration for a Query (Chen et al 2007) earr pian kids ear, era, ... pain, pan, ... kids, kid, ... ear pain kids, era pan kid, ear pin kid, ….... ▪ Problem: query candidate size grows exponentially with # of words ▪ Solution: prune with language model – ear pian kids: 0.8 – era pian kids: 0.1 – earr pain kids: 0.9 – earr pan kids: 0.7 – earr pien kids: 0.2 – earr pian kid: 0.8
  • 157.
    SIGIR 2019 157 CandidateRanking (Li et al 2006, Chen et al 2007) Feature Types Examples similarity Edit distance Frequency of user reformulation …... Likelihood Language model score of the candidate Frequency of candidate terms appearing in the page titles …... querycandidate
  • 158.
    SIGIR 2019 158 Agenda Candidate Generation fora word Candidate generation for a query Candidate ranking Traditional methods seq2seq for spell correction word-to-word char-to-char char-to-word Deep NLP methods
  • 159.
    SIGIR 2019 159 Seq2seqfor Spell Correction ▪ From subwords to subwords (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder ear #r <s> ear pain ear pain </s> pi #an
  • 160.
    SIGIR 2019 160 Seq2seqfor Spell Correction ▪ From subwords to subwords ▪ Issue: subword is designed for texts without errors (Ghosh and Kristensson, 2017, Zhou et al., 2017) Normal texts Example hunter → hunt #er Subword semantics relevant Spell errors hunetr → hu #net #r irrelevant
  • 161.
    SIGIR 2019 161 Seq2seqfor Spell Correction ▪ From characters to characters (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> e a e a r r r r </s> ▪ Issue: on the decoder, no word information ▪ Might produce words with wrong spelling
  • 162.
    SIGIR 2019 162 Seq2seqfor Spell Correction ▪ From characters to words – Most popular structure – Can leverage pretrained language model (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> ear r r ear </s>
  • 163.
    SIGIR 2019 163 Reference ▪Hasan, Saša, Carmen Heger, and Saab Mansour. "Spelling correction of user search queries through statistical machine translation." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 451-460. 2015. ▪ Yingbo Zhou, Utkarsh Porwal, Roberto Konow. "Spelling Correction as a Foreign Language." arXiv. 2017. ▪ Ghosh, Shaona, and Per Ola Kristensson. "Neural networks for text correction and completion in keyboard decoding." arXiv preprint arXiv:1709.06429 (2017). ▪ Chen, Qing, Mu Li, and Ming Zhou. "Improving query spelling correction using web search results." In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP- CoNLL), pp. 181-189. 2007. ▪ Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. "Exploring distributional similarity based models for query spelling correction." In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 1025-1032. Association for Computational Linguistics, 2006. ▪ Whitelaw, Casey, Ben Hutchinson, Grace Y. Chung, and Gerard Ellis. "Using the web for language independent spellchecking and autocorrection." In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pp. 890-899. Association for Computational Linguistics, 2009.
  • 164.
    SIGIR 2019 164 LanguageGeneration for Search Assistance: Summary Traditional methods deep NLP methods Candidate generation Rule based End-to-end solution Language modeling Candidate ranking Few features Latency Low High
  • 165.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search Systems 4 Real World Examples
  • 166.
    Deep NLP inSearch Systems - Real World Examples Huiji Gao
  • 167.
    SIGIR 2019 167 NaturalLanguage Data in LinkedIn Search Systems Query Job Posts Content Articles Learning Courses Member Profiles
  • 168.
    SIGIR 2019 168 LinkedInSearch Ecosystem Auto Complete Spell Check Language Detection Tokenization Query Tagging Retrieval Ranking Normalization/ Accent Handling Raw Query Query Reformulation Query Processing Query Understanding Retrieval & Ranking Query Suggestion Query Intention Instant Results (TYH Name Search, etc.) Conversational Search
  • 169.
    SIGIR 2019 169 NLPin LinkedIn Search: Challenges ● Data Ambiguity ○ Short Query Text ■ “abc” ○ No Strict Syntax ■ “bing search engineer” ○ Strong Correlation to the Searcher ■ “looking for new jobs” ● Deep Semantics ○ Representations for query & document w.r.t. search intent, entities, topics ABC News? ABC Stores? “Bing Search, Engineer” “Bing, Search Engineer” Job Seeker looks for jobs ■ “Engineering Openings” -> Job Posts Recruiter looks for candidates
  • 170.
    SIGIR 2019 170 DeepNLP in LinkedIn Search: Challenges ● Complicated Search Ecosystem ○ Query suggestion affects both recall and precision in downstream retrieval and ranking. ○ Query tagging needs to be compatible with indexing and align with ranking features. ● Product Oriented Model Design ○ Design deep NLP algorithms for specific search components ○ Consider business rules, post filters, results blender, user experience, etc ● Online Latency ○ Serving deep NLP models with product latency restriction
  • 171.
    SIGIR 2019 171 ApplyingDeep NLP in LinkedIn Search ● Feature Driven ○ Representation Learning ■ Using features generated from deep learning models e.g., word embedding ● Model Driven ○ Power product features directly with deep learning models ■ CNN/LSTM/Seq2seq/GAN/BERT based deep NLP models
  • 172.
    SIGIR 2019 172 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 173.
    SIGIR 2019 173 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 174.
    SIGIR 2019 174 QueryIntention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Search Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
  • 175.
    SIGIR 2019 175 QueryIntention Model: Task Oriented Intention Prediction People from Company X in Title Y Video on Topic X from Company Y Company with Name X in Geo Y Help in Topic X on Platform Y People from Geo X in Degree Y with Title Z Article from Author X with Topic Y Learning Course from Author X in Skill Y Professional with Skill X in Geo Y …… …… …... Search Tasks …… …… …...
  • 176.
    SIGIR 2019 176 QueryIntention: Member Search Footprint People Search Job Search Company Content Jobs Video Learning People Feed/Ads
  • 177.
    SIGIR 2019 177 QueryIntention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
  • 178.
    SIGIR 2019 178 QueryIntention on Search Blending Primary result Secondary cluster Primary result Search verticals Goal: ▪ To understand the vertical preference of user on LinkedIn search
  • 179.
    SIGIR 2019 179 QueryIntention Model: Challenges ▪ Complicated Semantics Job? Job Job!
  • 180.
    SIGIR 2019 180 QueryIntention Model: Challenges ▪ Personalization ● Query: Blockchain Tech ● Job seeker looks for Blockchain Technology job ● Company named Blockchain Tech ● Learning course on Blockchain Tech ● Video about blockchain technology ● Recruiter looks for candidates with Blockchain tech skill ● Content on Blockchain Technology ● …...
  • 181.
    SIGIR 2019 181 CNNBased Query Intention Model CNN for Semantic Feature Extraction ▪ Word/query representations ▪ Generalization power ▪ Word n-gram patterns Personalization ▪ Member-level Features ...
  • 182.
    SIGIR 2019 182 QueryIntent - Experiment Results ▪ Offline Results Overall Accuracy F1 on PEOPLE F1 on JOB Baseline (ML Model) - - - CNN +2.9% +11.9% +1.7% ▪ Online Results – +0.65% JOB Ctr At 1 Serp – +0.90% Overall Cluster Ctr, +4.03% Cluster Ctr Via Entity Click
  • 183.
    SIGIR 2019 183 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 184.
    SIGIR 2019 184 QueryTagger at LinkedIn ● LinkedIn Search Engine Query: Mike LinkedIn Software Engineer Relevant Results Query Results Pool Recall Precision Retrieving (Indexing) Ranking (Ranker) Query Query Tagger Irrelevant Results
  • 185.
    SIGIR 2019 185 Searchat LinkedIn ● Understanding Queries with Query Tagger Query: Mike LinkedIn Software Engineer Mike LinkedIn Software Engineer FN CN T T ○ Query Tagger for Retrieval FN:{mike} Doc1, Doc2, Doc4, ... CN:{linkedin} Doc2, Doc4, Doc5, Doc6, ... T:{Software} Doc2, Doc3, Doc4, Doc7, ... T:{Engineer} Doc1, Doc2, Doc4, Doc9, ... Index Query Doc IsFnMatch IsLnMatch IsTitleMatch etc. ○ Ranking Features CN: company name​ FN: first name​ T: title​
  • 186.
    SIGIR 2019 186 NaturalLanguage Understanding: Query Tagger LinkedIn software engineer data scientist jobs​ CN​ T​ T​ T​ T​ O​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ B-CN: beginning of a company name​ I-CN: Inside of a company name​ B-T: beginning of a job title I-T: Inside of a job title O: Not an entity​ B-PN: beginning of person name …
  • 187.
    SIGIR 2019 187 QueryTagger: Logistic Regression LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ N way classification sparse ftr vec ftr 0: whether the current word is “linkedin” ftr 1: whether the current word is “facebook” … ftr n: whether the next word is “software” ftr n+1: whether the next word is “linkedin” ….
  • 188.
    SIGIR 2019 188 QueryTagger: CRF LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ CRF Layer sparse ftr vec
  • 189.
    SIGIR 2019 189 QueryTagger: CRF + LSTM LinkedIn software engineer data scientist jobs​ B-CN​ B-T​ I-T​ B-T​ I-T​ O​ CRF Layer LSTM hidden state Word embedding
  • 190.
    SIGIR 2019 190 QueryTagger Performance SCRF CRF CRF + Bi-LSTM Bi-LSTM HMM Deep & Wide Modeling Structure for Entity Tagging
  • 191.
    SIGIR 2019 191 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 192.
    SIGIR 2019 192 NaturalLanguage Generation: Query Suggestion ▪ Save User Effort of Rephrasing Informative Queries ▪ Capture Users’ Search Intention ▪ Automatic Query Reformulation
  • 193.
    SIGIR 2019 193 NaturalLanguage Generation: Query Suggestion ▪ Traditional Frequency Based Methods – Collect <q1, q2> pairs from search log – Save the frequent pairs in a key-value store ▪ Lack of Generalization – Purely string matching – Cannot handle unseen queries, rare words ▪ Seq2seq: Model Query Reformulation Behavior
  • 194.
    SIGIR 2019 194 encoder QuerySuggestion: Reformulate to Related Queries ■ Training: the 2nd query is given ■ Maximize linkedin engineering <s> linkedin software linkedin software development </s> development decoder
  • 195.
    SIGIR 2019 195 decoderencoder QuerySuggestion: Reformulate to Related Queries linkedin engineering <s> linkedin software development </s> ■ Inference: the 2nd query is unknown ■ Beam search instead of greedy search
  • 196.
    SIGIR 2019 196 QuerySuggestion: How to Handle Online Latency ▪ Latency is strictly constrained for one query – Make it parallel with search ranking Query Generate Ranking Results Generate Suggested Queries Merge for Serving
  • 197.
    SIGIR 2019 197 OnlinePerformance ▪ English Market – Coverage: +80% Impressions, +26% CTR – +1% Total job application ▪ I18n Market – +3.2% Successful Searches – +28.6% First Order Actions
  • 198.
    SIGIR 2019 198 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 199.
    SIGIR 2019 199 NaturalLanguage Generation: Auto-Completion ▪ Given a prefix, predict the completed query, rather than the completed word
  • 200.
    SIGIR 2019 200 Auto-completionChallenges ▪ Short Context – How to enrich semantic features ▪ Personalization ▪ Latency Restriction – Have to adopt simple and fast models
  • 201.
    SIGIR 2019 201 ATwo-step Approach: Generation and Ranking ▪ Candidate Generation – Collect query frequency from search log ▪ Candidate Ranking – Neural Language Model serves as a scoring function
  • 202.
    SIGIR 2019 202 Auto-Completion:Neural Language Model as Scoring/Ranking ▪ Achieved 6x speedup with optimization
  • 203.
    SIGIR 2019 203 Auto-CompletionOnline Experiments Neural Auto-completion vs Frequency Based Methods ▪ People Search English Market – +88.75% autocomplete clicks – +0.81% CTR@1 in SERP ▪ Job Search German Market – +3.24% Results clicked – -6.24% Raw searches w/o results – +3.24% Entity view
  • 204.
    SIGIR 2019 204 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 205.
    SIGIR 2019 205 NaturalLanguage Processing: Document Ranking
  • 206.
    SIGIR 2019 206 NeuralRanking Feature Processing Layer
  • 207.
    SIGIR 2019 207 Experiments PeopleSearch Ranking (NDCG@10) Wide Features CNN +1.32% BERT (google pretrained) +1.52% BERT (linkedin pretrained) +1.96% Offline ▪ Help center ranking (BERT vs CNN) – +9.5% search CTR – +7% search Clicks Online
  • 208.
    SIGIR 2019 208 Lessons& Future Trends ● Ground Truth Data Availability ○ Human Labelled Data (Crowdsourcing) ○ Behavior Data ○ Mixture Data ○ Automatic Data Generation with Generative Models (e.g., GANs) ● Model Debugging in Complicated Search Systems ○ Model Interpretation ○ Model Reduction
  • 209.
    SIGIR 2019 209 Lessons& Future Trends (Cont’d) ● Internationalization ○ Transfer Learning ○ Multilingual Learning ● Model Generalization ○ Pre-trained Model for Multiple Products ● Efficient Modeling Training and Serving ○ Automatic Hyperparameter Tuning & Structure Learning ○ Memory and Latency Optimization for Efficient Online Serving
  • 210.
    SIGIR 2019 210 Feedback ●Your Feedback is Greatly Appreciated Slides will be Available on Website
  • 211.