Huiji Gao, profile picture
Uploaded byHuiji Gao
4,743 views

Deep Natural Language Processing for Search and Recommender Systems

The document discusses the application of deep natural language processing (NLP) techniques in search and recommender systems. It covers the capabilities of deep learning for understanding user intentions, document retrieval, and ranking, as well as language generation for assistance. Several models and real-world examples are mentioned to illustrate the effectiveness of deep NLP in enhancing search and recommendation functionalities.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Downloaded 734 times
1 / 272
2 / 272
Most read
3 / 272
4 / 272
5 / 272
Most read
6 / 272
Most read
7 / 272
8 / 272
9 / 272
10 / 272
11 / 272
12 / 272
13 / 272
14 / 272
15 / 272
16 / 272
17 / 272
18 / 272
19 / 272
20 / 272
21 / 272
22 / 272
23 / 272
24 / 272
25 / 272
26 / 272
27 / 272
28 / 272
29 / 272
30 / 272
31 / 272
32 / 272
33 / 272
34 / 272
35 / 272
36 / 272
37 / 272
38 / 272
39 / 272
40 / 272
41 / 272
42 / 272
43 / 272
44 / 272
45 / 272
46 / 272
47 / 272
48 / 272
49 / 272
50 / 272
51 / 272
52 / 272
53 / 272
54 / 272
55 / 272
56 / 272
57 / 272
58 / 272
59 / 272
60 / 272
61 / 272
62 / 272
63 / 272
64 / 272
65 / 272
66 / 272
67 / 272
68 / 272
69 / 272
70 / 272
71 / 272
72 / 272
73 / 272
74 / 272
75 / 272
76 / 272
77 / 272
78 / 272
79 / 272
80 / 272
81 / 272
82 / 272
83 / 272
84 / 272
85 / 272
86 / 272
87 / 272
88 / 272
89 / 272
90 / 272
91 / 272
92 / 272
93 / 272
94 / 272
95 / 272
96 / 272
97 / 272
98 / 272
99 / 272
100 / 272
101 / 272
102 / 272
103 / 272
104 / 272
105 / 272
106 / 272
107 / 272
108 / 272
109 / 272
110 / 272
111 / 272
112 / 272
113 / 272
114 / 272
115 / 272
116 / 272
117 / 272
118 / 272
119 / 272
120 / 272
121 / 272
122 / 272
123 / 272
124 / 272
125 / 272
126 / 272
127 / 272
128 / 272
129 / 272
130 / 272
131 / 272
132 / 272
133 / 272
134 / 272
135 / 272
136 / 272
137 / 272
138 / 272
139 / 272
140 / 272
141 / 272
142 / 272
143 / 272
144 / 272
145 / 272
146 / 272
147 / 272
148 / 272
149 / 272
150 / 272
151 / 272
152 / 272
153 / 272
154 / 272
155 / 272
156 / 272
157 / 272
158 / 272
159 / 272
160 / 272
161 / 272
162 / 272
163 / 272
164 / 272
165 / 272
166 / 272
167 / 272
168 / 272
169 / 272
170 / 272
171 / 272
172 / 272
173 / 272
174 / 272
175 / 272
176 / 272
177 / 272
178 / 272
179 / 272
180 / 272
181 / 272
182 / 272
183 / 272
184 / 272
185 / 272
186 / 272
187 / 272
188 / 272
189 / 272
190 / 272
191 / 272
192 / 272
193 / 272
194 / 272
195 / 272
196 / 272
197 / 272
198 / 272
199 / 272
200 / 272
201 / 272
202 / 272
203 / 272
204 / 272
205 / 272
206 / 272
207 / 272
208 / 272
209 / 272
210 / 272
211 / 272
212 / 272
213 / 272
214 / 272
215 / 272
216 / 272
217 / 272
218 / 272
219 / 272
220 / 272
221 / 272
222 / 272
223 / 272
224 / 272
225 / 272
226 / 272
227 / 272
228 / 272
229 / 272
230 / 272
231 / 272
232 / 272
233 / 272
234 / 272
235 / 272
236 / 272
237 / 272
238 / 272
239 / 272
240 / 272
241 / 272
242 / 272
243 / 272
244 / 272
245 / 272
246 / 272
247 / 272
248 / 272
249 / 272
250 / 272
251 / 272
252 / 272
253 / 272
254 / 272
255 / 272
256 / 272
257 / 272
258 / 272
259 / 272
260 / 272
261 / 272
262 / 272
263 / 272
264 / 272
265 / 272
266 / 272
267 / 272
268 / 272
269 / 272
270 / 272
271 / 272
272 / 272
Deep Natural Language Processing for Search and Recommender Systems Weiwei Guo Huiji Gao Jun Shi Bo Long Liang Zhang Bee-Chung Chen Deepak Agarwal
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
Introduction Bo Long
KDD 2019 5 Introduction - Search and Recommender Systems Search and Recommendation are Everywhere
KDD 2019 6 Introduction - Search and Recommender Systems Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs KDD 2019 InfoInfo Travel
KDD 2019 7 KDD 2019 Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs Info Sightseeing Food Arrive Anchorage Introduction - Search and Recommender Systems
KDD 2019 8 KDD 2019 Travel FoodSightseeing Attractions Connect People Looking or Jobs Learning Materials Info Attending KDD Attractions Learning Materials Introduction - Search and Recommender Systems Arrive Anchorage
KDD 2019 9 KDD 2019 Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs Info Attending KDD Connect People Looking or Jobs Introduction - Search and Recommender Systems Arrive Anchorage
KDD 2019 10 Introduction - NLP in Search and Recommender Systems ● Understand User Intention ○ Search Queries (search) Best Places for Bear Viewing Best Places for Bear Viewing Safely Budget? Location? ... ● Understand Documents ○ Posts, Reviews, Comments ○ Synonyms, Facets, Semantics, etc. ● Matching ○ Query & Doc Matching for Retrieval & Ranking ● Online Assistance ○ Query Reformulation ○ User Profiles (recommendation)
KDD 2019 11 Introduction - NLP in Search and Recommender Systems Generate Human Search & Recommender Systems Generate Assistance Natural Language Understanding (NLU) Natural Language Generation (NLG) Understand Natural Language Data
KDD 2019 12 Natural Language Processing in Recommendation Ecosystem Retrieval & Ranking Text Processing Member Profile Member Behavior Document Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing
KDD 2019 13 Natural Language Processing in Search Ecosystem Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation
KDD 2019 14 Natural Language Processing in Search and Recommendation Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation Text Processing Member Behavior Document Understanding Recommender Systems Search Systems
KDD 2019 15 Natural Language Processing in Search and Recommendation Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation ... Query Tagging Query Intention ... Auto Completion Query Suggestion Spell Check ... Natural Language Understanding Entity Tagging Language Detection ... Tokenization Topic Modeling Sentiment Analysis Natural Language Understanding Natural Language Generation
KDD 2019 16 Introduction - Deep Learning for NLP Figure source: Elvis, 2018, https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d
KDD 2019 17 Opportunities - Deep Learning for NLP in Search and Recommender Systems Why Deep Learning? ● Deep Semantics from High Dimension and Sparse Data ○ Synonymous, Disambiguation, etc. ● Easy Feature Engineering ○ Hand Crafted Features V.S. Auto Feature Representation ● Model Flexibility ○ Model end-to-end process ○ Various NN components to model and cooperate systematically ● Multi-level Feature Representation ○ Hierarchical Representations character -> token -> word -> phrase -> sentence [Young et. al. 2018]
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
Deep Learning for Natural Language Processing Bo Long
KDD 2019 20 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 21 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
KDD 2019 22 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context Sequence of Words w_1 Sequential Modeling w_2 w_3 w_n…... Sequence Encoding (Feature Embedding) Tasks (Classification, etc.) Contains Essential Semantics of the Sequence
KDD 2019 23 am an NLP an NLP engineerI am an am an an NLP NLP EngineerI am am an NLP engineerI Sequential Modeling on Semantic Context I am an NLP engineer I am an NLP engineer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition I am an NLP engineer Je suis un ingénieur en NLP I am an NLP engineer Je suis un ingénieur en NLP Context Strengthening Transformer GAN Context Consistency Global Context I am an NLP engineer Je suis un ingénieur en NLP Je sont une ingénieur en NLP train test Encoding Encoding Encoding
KDD 2019 24 Multilayer Perceptron (MLP) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Figure source: https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn/
KDD 2019 25 Convolutional Neural Networks (CNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Zhang et. al. 2015]
KDD 2019 26 Recurrent Neural Networks (RNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context LSTM GRU [LeCun et. al. 2015] github: 2015-08-Understanding-LSTMs
KDD 2019 27 Sequence to Sequence (Encoder - Decoder) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
KDD 2019 28 Attention Mechanisms MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
KDD 2019 29 Transformer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Novelty: Eliminate Recurrence transformer http://jalammar.github.io/illustrated-transformer/
KDD 2019 30 Generative Adversarial Networks (GANs) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Lee, et al., KDD 2018]
KDD 2019 31 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 32 Deep Learning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
KDD 2019 33 ▪ Representation Learning Sequence of Words w_1 Representation Learning w_2 w_3 w_n…... T’_1 T’_2 T’_3 T’_n…... Word Representation (Embeddings) Deep Learning for Natural Language Processing
KDD 2019 34 Representation Learning Word2Vec Static Word Embedding [Mikolov et. al. 2013]
KDD 2019 35 Representation Learning Word2Vec ELMO Static Word Embedding Contextual Word Embedding - Bidirectional LSTM [Peters et. al. 2018]
KDD 2019 36 Representation Learning Word2Vec ELMO GPT Static Word Embedding Contextual Word Embedding Contextual Word Embedding - Bidirectional LSTM - One Direction Transformer [Radford et. al. 2018]
KDD 2019 37 Representation Learning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding BERT - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer [Devlin et. al. 2018]
KDD 2019 38 Representation Learning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding XLNET Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding XLNet - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer - Bidirectional, Learn correlations among masked tokens Pre-trained NLP Model [Yang et. al. 2019]
KDD 2019 39 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching MLP, CNN, RNN, Transfomer, ... RNN, Transfomer, ... Seq2seq, Transformer, GAN, ... CNN, RNN, ... Word2Vec, ELMO, GPT, BERT, XLNet, ... MLP, CNN, RNN, seq2seq, attention,transformer, GAN
KDD 2019 40 References - Deep Learning for Natural Language Processing ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [LeCun et. al, 2015] Deep learning, Nature, vol. 521, no. 7553, pp. 436–444, 2015. ▪ [Lee et. al, 2018] Rare query expansion through generative adversarial networks in search advertising, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ [Mikolov et. al. 2013] Distributed representations of words and phrases and their compositionality, in Advances in neural information processing systems, 2013 ▪ [Mikolov et. al. 2013] Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, 2013. ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Robinson, 2018] https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn, 2018. ▪ [Shi et. al, 2018] Neural Abstractive Text Summarization with Sequence-to-Sequence Models, arXiv preprint arXiv:1812.02303v2, 2018. ▪ [Yang et. al. 2019] XLNet: Generalized Autoregressive Pretraining for Language Understanding, arXiv preprint arXiv:1906.08237v1, 2019 ▪ [Young et. al. 2018] Recent Trends in Deep Learning Based Natural Language Processing, arXiv preprint arXiv:1708.02709v8, 2018 ▪ [Zhang et. al. 2015] A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, arXiv preprint arXiv:1510.03820, 2015.
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
Deep NLP in Search and Recommender Systems - Language Understanding Jun Shi
KDD 2019 43 Deep NLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
KDD 2019 44 Entity Tagging ▪ Problem statement ▪ Traditional statistical models – Hidden Markov Model – Maximum Entropy Markov Model – (Semi-Markov) Conditional Random Field ▪ Deep learning models – Input layer – Context layer – Decoder layer ▪ Training dataset ▪ Evaluation
KDD 2019 45 Entity Tagging - Problem Statement ▪ A named entity, a word or a phrase that clearly identifies one item from a set of other items that have similar attributes. [Li et. al. 2018] ▪ Entity tagging (Named Entity Recognition, NER), the process of locating and classifying named entities in text into predefined entity categories. Washington D.C. was named after George Washington. LOCATION PERSON
KDD 2019 46 Motivation ▪ Efficiency – Looking only for named entities can speed up search. ▪ Precision – Matching both named entity and tags can increase search precision. ▪ Quality – Ranking retrieved results by considering tags improves search quality.
KDD 2019 47 Traditional Statistical Models ▪ Generative model – Hidden Markov Model [Baum, et. al. 1966] ▪ Discriminative model – Maximum Entropy Markov Model [McCallum, et. al. 2000] – (Semi-Markov) Conditional Random Field [Lafferty, et. al. 2001]
KDD 2019 48 Hidden Markov Model Generative model, model joint probability Pr(T, W) instead of conditional probability Pr(T | W). T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
KDD 2019 49 Maximum Entropy Markov Model Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
KDD 2019 50 Conditional Random Field Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
KDD 2019 51 Semi-Markov Conditional Random Field Each tag can correspond to a variable-length phrase. T1 T2 W1 W2 W3 tags words T3 W4 W5 W6 T0
KDD 2019 52 Summary - Traditional Statistical Models Model Pros Cons Hidden Markov Model training is simple when states are observed difficult to include features Maximum Entropy Markov Model easy to include features suffer from “label bias problem” (prefer states with lower number of transitions) Conditional Random Field easy to include features, do not suffer from “label bias problem” training is relatively complex
KDD 2019 53 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
KDD 2019 54 Input Layer - Word Embedding [Yadav, et. al. 2018]
KDD 2019 55 Input Layer - Char Embedding [Yadav, et. al. 2018]
KDD 2019 56 Input Layer - Word and Char Embedding [Yadav, et. al. 2018]
KDD 2019 57 Summary - Input Layer word embedding char embedding external features, lexicon, etc. char CNN/RNN word
KDD 2019 58 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
KDD 2019 59 Context Layer - CNN embedding CNN P A D P A D Tag Decoder [Li et. al. 2018]
KDD 2019 60 Context Layer - RNN [Li et. al. 2018]
KDD 2019 61 Deep Learning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
KDD 2019 62 Tag Decoder - MLP+softmax [Li et. al. 2018]
KDD 2019 63 Tag Decoder - CRF [Li et. al. 2018]
KDD 2019 64 Tag Decoder - RNN [Li et. al. 2018]
KDD 2019 65 Pre-Training and Fine-Tuning BERT GPT ELMo [Devlin et. al. 2018] [Peters et. al. 2018][Radford et. al. 2018]
KDD 2019 66 Entity Tagging Evaluation Exact-match evaluation ▪ segment match ▪ tag match
KDD 2019 67 Entity Tagging Training Dataset [Li et. al. 2018]
KDD 2019 68 Entity Tagging on CoNLL03 English Source Method F1 score [Passos et al. 2014] CRF 90.90 [Huang et al. 2015] Bi-LSTM+CRF 84.26 [Collobert et al. 2011] Conv-CRF 89.59 [Kuru et al. 2016] char embedding 84.52 [Chiu and Nichols 2015] word + char embedding 91.62 [Devlin et. al. 2018] BERT Large 92.8
KDD 2019 69 References - Entity Tagging ▪ [Baum et. al. 1966] Baum, L. E.; Petrie, T. (1966). "Statistical Inference for Probabilistic Functions of Finite State Markov Chains". The Annals of Mathematical Statistics. 37 (6): 1554–1563. ▪ [Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 ▪ [Collins and Singer 1999] Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ▪ [Collobert et. al. 2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. ▪ [Collobert and Weston 2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [Huang et. al. 2015] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 ▪ [Kuru et. al. 2016] Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. Charner: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911–921 ▪ [Lafferty et. al. 2001] Lafferty, J., McCallum, A., Pereira, F. (2001). "Conditional random fields: Probabilistic models for segmenting and labeling sequence data". Proc. 18th International Conf. on Machine Learning. Morgan Kaufmann. pp. 282–289 ▪ [Li et. al. 2018] Jing Li, Aixin Sun, Jianglei Han, Chenliang Li, A Survey on Deep Learning for Named Entity Recognition, Dec. 2018, arXiv preprint, https://arxiv.org/abs/1812.09449 ▪ [McCallum et. al. 2000] McCallum, Andrew; Freitag, Dayne; Pereira, Fernando (2000). "Maximum Entropy Markov Models for Information Extraction and Segmentation" (PDF). Proc. ICML 2000. pp. 591–598
KDD 2019 70 References (continued) ▪ [Passos 2014] Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367 ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Yadav et. al. 2018] Vikas Yadav, Steven Bethard, A Survey on Recent Advances in Named Entity Recognition from Deep Learning models, Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158 Santa Fe, New Mexico, USA, August 20-26, 2018.
KDD 2019 71 Deep NLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation
KDD 2019 72 Entity Disambiguation ▪ Problem statement ▪ Motivation ▪ Challenges ▪ System architecture – Candidate Entity Generation – Candidate Entity Selection – Joint entity tagging and disambiguation ▪ Evaluation
KDD 2019 73 Entity Disambiguation - Problem Statement ▪ Also known as entity linking ▪ Resolves mentions to entities in a given knowledge base – Mentions are mostly named entities – Example of knowledge base: freebase, wikipedia ▪ Examples: Jaguar – The prey saw the jaguar cross the jungle. KB:Jaguar – The man saw a jaguar speed on the highway. KB:Jaguar_Cars
KDD 2019 74 Motivation Increase the quality of the retrieved results Example: Which Michael Jordan were you looking for?
KDD 2019 75 Challenges ▪ Name variation: New York vs Big Apple ▪ Ambiguity: Michael Jordan ▪ Metonymy: Beijing (city or Chinese government) ▪ Absence: no entries in knowledge base ▪ Evolving information: new company names, e.g. tic-tok.
KDD 2019 76 System Architecture Candidate Generation Candidate selection 1. Candidate entity generation a. Name dictionary based techniques. b. Surface form expansion from the local document. c. Methods based on search engines. 2. Candidate entity selection a. Graph-based methods b. Text-based methods knowledge base Documents Linked results
KDD 2019 77 Candidate Entity Generation ▪ Name dictionary based techniques – Entity pages – Redirect pages – Disambiguation pages – Bold phrases from the first paragraphs – Hyperlinks in Wikipedia articles ▪ Surface form expansion – Heuristic based methods – Supervised learning methods ▪ Methods based on search engines [Shen et. al. 2014]
KDD 2019 78 Candidate Entity Selection - Graph based models ▪ knowledge graph lends itself to graph based methods. ▪ Build a weighted undirected graph with mentions and candidate entities as nodes – weights on mention-entity edges. – weights on entity-entity edge. ▪ Compute an optimal subgraph: – optimality is model dependent. – contains all mentions – one entity per mention mention entity
KDD 2019 79 An Example Graph [Hoffart, et. al. 2011]
KDD 2019 80 Candidate Entity Selection - Text-Based Models ▪ Idea: find agreement (similarity) between entity and mention. – Example: computational intelligence by Michael Jordan and Stuart Russell from UC Berkeley increasing context ● wiki/Michael_Jordan ● wiki/Michael_I._Jordan▪ Models according to context. – Direct Models. ▪ use mention and candidate entities only. – Local Models ▪ use local context around the mention. – Coherence Models. ▪ use mentions and entities in a document. – Collaborative Models. ▪ use mentions and entities across related documents.
KDD 2019 81 Individual vs Joint Approach Two types of approaches according to how mentions are resolved. ▪ Individual approach – Handle one mention at a time. – Rank the candidate entities. – Relatively low complexity. ▪ Joint approach – Treat all mentions in a document as a sequence. – Jointly optimize the mention-entity pairs. – High complexity, usually resorting to approximations.
KDD 2019 82 Features Different features are used in candidate selection modeling. ▪ Traditional Features – entity popularity, entity type. – surface form similarity between entity and mention. – coherence between mapped entities ▪ Deep Features [Yamada et. al. 2016] – word embedding – entity embedding – similarity between words, context and candidate entities.
KDD 2019 83 Example: Entity Disambiguation with Attention training: max-margin loss [Genea et. al. 2017]
KDD 2019 84 Joint Entity Tagging and Disambiguation ▪ Previously, we find entities first, then link them. But if entities are identified wrong, disambiguation will likely fail too. – Over-split: Romeo and Juliet by Shakespeare – Under-split: Baby Romeo and Juliet were born hours apart. ▪ We could do entity tagging and disambiguation jointly. – Over-generate candidate mentions – Generate possible entities per mention – Score non-overlapping mention-entity pair jointly. [Sil, et. al.2013]
KDD 2019 85 A Joint Neural Entity Tagging and Disambiguation [Kolitsas, et. al. 2018] Select mentions that have at least one possible entity ● Trained on golden mention-entity pairs. ● Used max-margin loss. cosine similarity between current entity and average of other entities in the document
KDD 2019 86 Entity Disambiguation Evaluation ▪ Entity-tagging-style F1 score – A link is considered correct only if the mention matches the gold boundary and the linked entity is also correct ▪ Accuracy – another common metric, simpler than F1 score. ▪ TAC-KBP B-Cubed+ F1 score – not widely used
KDD 2019 87 References - Named Entity Disambiguation ▪ [Cucerzan 2007] Cucerzan, Silviu. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL): 708–716. ▪ [Ganea et. al. 2017] Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2609–2619. Association for Computational Linguistics. ▪ [Globerson et. al. 2016] Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In ACL (1). ▪ [Han et. al. 2011] Han, Xianpei; Sun, Le; Zhao, Jun. Collective Entity Linking in Web Text: A Graph-based Method. ▪ [Hoffart, et. al. 2011] Johannes Hoffart, Mohamed A. Yosef, Ilaria Bordino, Hagen F¨urstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Method in Natural Language Processing, pages 782–792. Association for Computational Linguistics. ▪ [Kolitsas et al., 2018] Nikolaos Kolitsas, Octavianeugen Ganea, and Thomas Hofmann. End-to-end neural entity linking. In CoNLL, 2018. ▪ [Kulkarni et. al. 2009] S. Kulkarni, A. Singh, G. Ramakrishnan, and S. Chakrabarti, Collective annotation of Wikipedia entities in web text, in SIGKDD, 2009, pp. 457–466 ▪ [Ling et al.2015] Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315–328. ▪ [Le et. al., 2018] Phong Le and Ivan Titov. Improving entity linking by modeling latent relations between mentions. In ACL, 2018. ▪ [Rao et. al. 2013] Rao, Delip; McNamee, Paul; Dredze, Mark (2013). Entity Linking: Finding Extracted Entities in a Knowledge Base. Multi-source, Multilingual Information Extraction and Summarization. Springer Berlin Heidelberg: 93–115
KDD 2019 88 References (continued) ▪ [Raiman et. al. 2018] Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual Entity Linking by Neural Type System Evolution. In Proc. of AAAI. ▪ [Shen et. al. 2014] Wei Shen, Jianyong Wang, and Jiawei Han. 2014. Entity linking with a knowledge base: Issues, techniques, and solutions. TKDE ▪ [Sil et. al. 2013] Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2369–2374. ACM ▪ [Sil et. al. 2016] Sil, A., and Florian, R. 2016. One for all: Towards language independent named entity linking. ACL. ▪ [Yamada et. al. 2016] Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. CoNLL 2016, page 250.
KDD 2019 89 Deep NLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
KDD 2019 90 Intent Classification ▪ Problem statement ▪ Deep Learning Models – fastText – CNN – Bi-RNN + Attention
KDD 2019 91 Intent Classification - Problem Statement ▪ Web search intent can be classified into 3 class: [Broder 2002] – Navigational. The immediate intent is to reach a particular site. ▪ Greyhound Bus. Probable target http://www.greyhound.com – Informational. The intent is to acquire some information assumed to be present on one or more web pages. ▪ San Francisco – Transactional. The intent is to perform some web-mediated activity. ▪ Shopping activities ▪ In conversational AI, intent is usually task-specific, e.g. – I would like to book a flight from SFO to ANC: FlightBooking – What software can I use to view epub documents: SoftwareRecommendation. ▪ We focus on the latter in this tutorial.
KDD 2019 92 Intent Classification - Methods ▪ Traditional methods – Features: bag of words, n-gram, TF-IDF. – Models: logistic regression, naive Bayes, SVM, random forest. ▪ Deep learning methods – Word embedding + linear classifier (fastText) [Joulin et. al. 2016]. – Convolutional neural networks [Hashemi 2016] . – Bi-RNN + attention (joint slot filling) [Liu et al.2016] .
KDD 2019 93 Intent Classification - fastText word and N gram embedding features Average softmax [Joulin et. al. 2016]
KDD 2019 94 Intent Classification - CNN [Hashemi 2016]
KDD 2019 95 Intent Classification - Bi-RNN+Attention [Liu et. al. 2016]
KDD 2019 96 References - Intention Classification ▪ [Broder 2002] A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, 2002. ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Joulin et. al. 2016] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ▪ [Lai et al.2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267–2273 ▪ [Liu et al.2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 ▪ [Zhou et al.2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 207. ▪ [Kim et. al. 2016] Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In Proceedings of SLT. ▪ [Hashemi 2016] Homa B Hashemi, Amir Asiaee, and Reiner Kraft. 2016. Query intent detection using convolutional neural networks. In International Conference on Web Search and Data Mining, Workshop on Query Understanding. ▪ [Liu et. al. 2016]. Liu and I. Lane, “Attention-based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling,” in Interspeech, 2016. ▪ [Shi et. al. 2016] Y. Shi, K. Yao, L. Tian, and D. Jiang, “Deep lstm based feature mapping for query classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1501–1511.
KDD 2019 97 Deep NLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation
KDD 2019 98 Sentiment Analysis and Opinion Mining ▪ Problem Statement ▪ Challenges ▪ Traditional methods ▪ Deep learning methods
KDD 2019 99 Sentiment Analysis - Problem Statement ▪ Interchangable with Opinion Mining in this tutorial. ▪ sentiment analysis mainly studies opinions which express or imply positive or negative sentiments [Liu 2012] ▪ Granularity: – Document level – Sentence level – Aspect level Posted by: John Smith Date: September 10, 2011 “(1) I bought a Canon G12 camera six months ago. (2) I simply love it. (3) The picture quality is amazing. (4) The battery life is also long. (5) However, my wife thinks it is too heavy for her.”
KDD 2019 100 Sentiment Analysis - Challenges ▪ Word ambiguity – This camera sucks. – This vacuum cleaner really sucks. ▪ Sentiment-less sentiment words – Can you tell me which camera is good? ▪ Sarcasm – What a great car! It stopped working in two days. ▪ Implicit sentiment – This washer uses a lot of water.
KDD 2019 101 Traditional Methods ▪ Document/Sentence Sentiment Analysis – unsupervised: rule based, sentiment orientation – supervised: naive Bayes, logistic regression, SVM ▪ Aspect-based Sentiment Analysis – Aspect extraction ▪ Frequent nouns and noun phrases ▪ Opinion and target relations ▪ Sequential tagging: HMM, CRF ▪ Topic modeling – Aspect sentiment classification ▪ Similar methods in document/sentence analysis. ▪ limited to aspect dependent features The screen is very clear but the battery life is too short.
KDD 2019 102 Deep Learning Approaches ▪ Document/Sentence Sentiment Analysis – Use embedding alone – Use CNN/RNN ▪ Aspect-based Sentiment Analysis – Aspect extraction ▪ CNN or RNN – Aspect sentiment analysis ▪ attention-based RNN, memory networks The screen is very clear but the battery life is too short.
KDD 2019 103 Document Sentiment Analysis - Paragraph Vector [Le et. al. 2014]
KDD 2019 104 Document Sentiment Analysis - RNN word level sentence level [Yang et. al. 2014]
KDD 2019 105 Sentence Sentiment Analysis - CNN [Kim 2014]
KDD 2019 106 Aspect and Opinion Extraction - RNN [Wang et. al. 2016] Labels: ● BA (beginning of aspect) ● IA (inside of aspect), ● BO (beginning of opinion) ● IO (inside of opinion) ● O (others)
KDD 2019 107 Aspect Level Sentiment Analysis - Memory Networks shared parameters [Tang et. al. 2016]
KDD 2019 108 Aspect Level Sentiment Analysis - Hierarchical RNN ● one aspect per sentence ● aspects are given ● Utilize the relationship between aspects [Ruder et. al. 2016]
KDD 2019 109 References - Sentiment Analysis and Opinion Mining ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Le et. al. 2014] Le Q, Mikolov T. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML 2014), 2014. ▪ [Liu 2015] Liu B. Sentiment analysis: mining opinions, sentiments, and emotions. The Cambridge University Press, 2015. ▪ [Liu 2012] Liu B. Sentiment analysis and opinion mining (introduction and survey), Morgan & Claypool, May 2012. ▪ [Ruder et al. 2016] Sebastian Ruder, Parsa Ghaffari, and John G Breslin. A hierarchical model of reviews for aspect-based sentiment analysis. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, November, 2016. ▪ [Schouten et. al. 2016] Kim Schouten and Flavius Frasincar. 2016. Survey on Aspect-Level Sentiment Analysis. IEEE Trans. Knowledge and Data Engineering, 28(3):813-830 ▪ [Tang et. al. 2015] Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432 ▪ [Tang et. al. 2016] Tang D, Qin B, and Liu T. Aspect-level sentiment classification with deep memory network. arXiv preprint ▪ arXiv:1605.08900, 2016. ▪ [Xu et. al. 2016] Xu J, Chen D, Qiu X, and Huang X. Cached long short-term memory neural networks for document-level sentiment classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), 2016. ▪ [Yang et. al. 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics ▪ [Zhang et. al. 2018] Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. In Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, page e1253. Wiley Online Library
KDD 2019 110 References (Continued) ▪ [Poria et al.2016] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016a. Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems, 108:42–49. ▪ [Wang et. al. 2016] Wang W, Pan SJ, Dahlmeier D, and Xiao X. Recursive neural conditional random fields for aspect-based sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), 2016.
Jun Shi, Weiwei Guo Deep NLP in Search and Recommender Systems - Document Retrieval & Ranking
KDD 2019 112 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 113 Document Retrieval and Ranking ▪ Efficient Candidate Retrieval ▪ Deep Neural Ranking
KDD 2019 114 Efficient Candidate Retrieval ▪ Syntactic retrieval – based on string matching – use inverted index – can include different fields (name, title, etc). ▪ Semantic retrieval – based on vector space representation. – approximate nearest neighbor search
KDD 2019 115 Syntactic Retrieval Query: mike software engineer mike Doc1, Doc2, Doc4, ... software Doc2, Doc3, Doc4, Doc7, ... engineer Doc1, Doc2, Doc4, Doc9, ... Inverted index Results: Doc2, Doc4 It won’t retrieve a document contains “mike is a software developer,...”
KDD 2019 116 Semantic Retrieval - Concept mike software engineer [0.3, 1.27, -3.4, ...] embedding mike software engineer The problem becomes nearest neighbor search [Mitra et. al. 2017]
KDD 2019 117 Semantic Retrieval - Vector Generation ▪ Bag of words – count based, TF-IDF vectors. ▪ Embedding vectors – word embedding: word2vec, Glove, BERT, etc. – sentence/document embedding ▪ universal sentence encoder [Cer et. al. 2018] ▪ Gaussian document representation [Giannis et. al. 2017] ▪ power mean concatenation [Rücklé et. al. 2018]
KDD 2019 118 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 119 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods q, d1 q, d2 s1 s2 learning-to-rank loss
KDD 2019 120 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks q d1 q d2 q, d1 q, d2
KDD 2019 121 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks q q d1 d2
KDD 2019 122 Deep Neural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks ▪ Wide & Deep Model
KDD 2019 123 Traditional Ranking Features ▪ Hand-crafted features – Query/document matching features ▪ Cosine similarity between query and doc title ▪ Entity matching ▪ Clickthrough rate from query to this doc based on search log ▪ ……. – Document alone ▪ popularity ▪ number of incoming links ▪ …...
KDD 2019 124 Learning to Rank ▪ Pointwise ranking – Logistic regression (Burges, 2010) ▪ Pairwise ranking ▪ Listwise ranking – Cross entropy
KDD 2019 125 Deep Neural Ranking ▪ Focus on computing query/document score s1 s2 learning-to-rank loss q q d1 d2 ▪ Two categories: – Siamese Networks – Interaction based Networks
KDD 2019 126 Deep Structured Semantic Model feed-forward (Huang et al., 2013) 30k 300 128 30k 300 128 high dimensional query/doc vector query doc1 sum of word embeddings cos sim
KDD 2019 127 Modeling Word Sequence by CNN CNN (Shen et al., 2014) d by n matrix 300 128 token embedding matrix d by n matrix 300 128 feed-forward cos sim
KDD 2019 128 Siamese Networks ▪ Pros: – Generalization; semantic matching – Efficient; doc embs can be precomputed ▪ Cons: – Lexical features lost: people/company names, rare words
KDD 2019 129 Interaction-based Networks (Guo et al., 2016) query: jobs at linkedin …... document Feed-forward z1 z2 z3 s cosine sim between query words and doc words sim_vec1 sim_vec2 sim_vec3 histogram1 histogram2 histogram2 historgrams based on sim vectors g1 g2 g3 0.4 0.1 0.5
KDD 2019 130 Search Ranking Neural Networks Summary ▪ Only focus on query/doc scoring ▪ End-to-end models ▪ Two popular architectures Siamese Network Interaction Network Match topical matches lexical matches Latency small large
KDD 2019 131 Recommender System: Wide & Deep Model (Cheng et al., 2016) wide ftrs user_last_doc_topic=sports user_last_doc_topic=sports && doc_id=100
KDD 2019 132 Recommender System: Wide & Deep Model (Cheng et al., 2016) user_last_doc_topic=sports user_last_doc_topic=sports && doc_id=100 sports emb doc #100 emb ... emb hidden layer wide ftrs
KDD 2019 133 References - Deep Neural Ranking ▪ Huang, Po-Sen, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. "Learning deep structured semantic models for web search using clickthrough data." In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338. ACM, 2013. ▪ Shen, Yelong, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. "A latent semantic model with convolutional-pooling structure for information retrieval." In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp. 101-110. ACM, 2014. ▪ Rodrigo Nogueira, Kyunghyun Cho, Passage re-ranking with BERT, 2019 ▪ Burges. "From ranknet to lambdarank to lambdamart: An overview." Learning. 2010. ▪ Guo, Jiafeng, Yixing Fan, Qingyao Ai, and W. Bruce Croft. "A deep relevance matching model for ad-hoc retrieval." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55-64. ACM, 2016. ▪ Mitra, Bhaskar, Fernando Diaz, and Nick Craswell. "Learning to match using local and distributed representations of text for web search." In Proceedings of the 26th International Conference on World Wide Web, pp. 1291-1299. International World Wide Web Conferences Steering Committee, 2017. ▪ Cheng, Heng-Tze, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson et al. "Wide & deep learning for recommender systems." In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10. ACM, 2016. ▪ Xiong, Chenyan, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. "End-to-end neural ad-hoc ranking with kernel pooling." In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pp. 55-64. ACM, 2017. ▪ Dai, Zhuyun, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. "Convolutional neural networks for soft-matching n-grams in ad-hoc search." In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 126-134. ACM, 2018.
Weiwei Guo Deep NLP in Search and Recommender Systems - Language Generation for Assistance
KDD 2019 135 Deep NLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
KDD 2019 136 Language Generation for Assistance ▪ Goal: improve user experience ▪ How: interact with users ▪ NLP: Language generation Common Difference ▪ Character/word modeling ▪ Generation models: – language modeling, seq2seq ▪ Auto Completion ▪ Query Reformulation ▪ Spell Correction ▪ Conversational Recommendation
KDD 2019 137 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 138 Query Auto-Completion ▪ Problem statement: save user keystrokes by predicting the entire query
KDD 2019 139 Challenges ▪ Limited Context – Hard to extract features due to limited words in a query ▪ Latency – The search component with most strict requirement on latency
KDD 2019 140 Agenda Candidate Generation Candidate Ranking Context aware Personalization Traditional methods
KDD 2019 141 Candidate Generation ▪ Collect completed queries and associated frequency from search log ▪ Efficiently retrieve most frequent queries starting with the prefix – Using a trie data structure soft software engineer 100 softball 50 softmax 40 softbank 35 …... candidate generation
KDD 2019 142 Candidate Generation for Rare Prefix (Mitra & Craswell, 2015) ▪ No such prefix in search log – “cheapest flights from seattle to” “cheapest flights from seattle to” “to” to dc to sfo to airport to seattle …... 100 50 40 35 ▪ Another trie on suffixes – “distance from seattle to dc” → “from seattle to dc”, “seattle to dc”, “to dc”
KDD 2019 143 ▪ Challenge: very few features can be extracted Candidate Ranking soft softbank softball softmax software engineer …... software engineer soft skills softball softbank …... candidate generation candidate ranking frequency 100 50 20 15
KDD 2019 144 Context-Aware ( Bar-Yossef & Kraus, 2011) ▪ Key idea: identify the candidates most similar to previous queries infant n (candidates) infant nike shoes infant nutrition …... infant, nike, shoe, adidas, clothes... infant, nutrition, baby, eat, food... …... (previous queries) baby eating disorder …... baby, eating, disorder, nutrition, food... …... sim score 0.1 0.8
KDD 2019 145 Personalization (Shokouhi, 2013) Feature list Same age Same gender Same region …... query prefix is “i”
KDD 2019 146 Agenda Candidate Generation Candidate Ranking Context aware Personalization Deep NLP methods CNN for Ranking End-to-end: Language Model Personalization Traditional methods
KDD 2019 147 Apply Deep Models in Ranking ▪ Measuring the semantic coherence between prefix and suffix embsize tri-letter length Query Prefix Query Suffix CNN conv and max-pooling text emb cos sim Cosine similarity (Mitra & Craswell, 2015)
KDD 2019 148 president barack obaman Language Modeling for Auto-Completion ‘ ’ softmax over chars ‘b’ LSTM char emb ‘t’ char emb ‘ ’ (Park & Chiba, 2017) ▪ Training: Character level language model word emb <INC> word emb `president` + word embedding
KDD 2019 149 char emb ‘b’ (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time 0.05 c ‘b’ ‘a’ 0.4 0.3 c ‘b’ ‘a’ 0.1 0.5 0.3 ‘a’ 0.4 Language Modeling for Auto-Completion
KDD 2019 150 (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time ▪ Greedy vs beam search ‘b’ ‘a’ ‘c’ ‘aa’ ‘ab’ ‘ac’ ‘ba’ ‘bb’ ‘bc’ ‘ca’ ‘cb’ ‘cc’ 0.01 0.3 0.3 0.1 0.01 0.01 0.26 0.01 0.01 ‘ab’ ‘ac’ ‘ca’ Language Modeling for Auto-Completion
KDD 2019 151 Personalization ▪ Embeddings for User Ids LSTM softmax over chars char emb ‘t’ word emb <INC> (Fiorin & Lu, 2018) user id emb
KDD 2019 152 Query Auto-Completion: Summary ▪ Traditional methods: hard to extract features ▪ Deep language model framework: – Very flexible to incorporate personalized/contextualized information – An end-to-end solution – Train: all parameters are optimized together – Test: generation and ranking are performed at the same time – Cons – Time-consuming – May generate wrong words
KDD 2019 153 Reference ▪ Mitra, Bhaskar, and Nick Craswell. "Query auto-completion for rare prefixes." In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 1755-1758. ACM, 2015. ▪ Park, Dae Hoon, and Rikio Chiba. "A neural language model for query auto-completion." In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189-1192. ACM, 2017. ▪ Bar-Yossef, Ziv, and Naama Kraus. "Context-sensitive query auto-completion." In Proceedings of the 20th international conference on World wide web, pp. 107-116. ACM, 2011. ▪ Shokouhi, Milad. "Learning to personalize query auto-completion." In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pp. 103-112. ACM, 2013. ▪ Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018). ▪ Fiorini, Nicolas, and Zhiyong Lu. "Personalized neural language models for real-world query auto completion." arXiv preprint arXiv:1804.06439 (2018).
KDD 2019 154 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 155 Query Reformulation ▪ Problem Statement: automatically reformulate the previous query Search results for “fb developer” Reformulated queries for “fb developer”
KDD 2019 156 Agenda collaborative filtering Traditional methods
KDD 2019 157 Traditional Approach: Collaborative Filtering (Rida et al., 2012) ▪ Collect query pairs issued by the same user from search log ▪ Treat each query as an ID, and build a query-to-query matrix – Value in the matrix is TF-IDF ▪ Issues: – No understanding of words in the query – Data sparsity – Cannot generate new queries
KDD 2019 158 Agenda collaborative filtering Traditional methods seq2seq conversational seq2seq GAN for seq2seq Deep NLP methods
KDD 2019 159 Query Reformulation as a Translation Task ▪ Sequence-to-sequence modeling decoderencoder facebook developer <s> facebook software facebook software development </s> development ▪ Directly modeling the words in a query (Sordoni et al. 2015, He et al., 2016)
KDD 2019 160 Conversational Query Reformulation ▪ Conversational queries (Ren et al, 2018) first query (q1) second query (q2) when was California founded? who is its governor? California population in 1990 how tall is kobe bryant? what about Lebron James? when was the last summer Olympics? and the winter one? summarized query (q3) who is California’s governor? population of California in 1990 how tall is Lebron James? when was the last winter Olympics? ▪ Goal: summarize q1 & q2 into q3
KDD 2019 161 encoder decoder w1 w2 <s> w1 w2 w1 w2 </s> SEP w1 w2 Conversational Query Reformulation ▪ 1st: concatenated seq2seq (Ren et al, 2018)
KDD 2019 162 Conversational Query Reformulation ▪ 2nd: Attention over attention (Ren et al, 2018) 2nd query h1 h2 h3 c1 c2 c h1 h2 h31st query 3rd query (decoder) ht h0
KDD 2019 163 GAN for seq2seq ▪ Motivation: Training/testing discrepancy in decoder of seq2seq (Lee et al, 2018) decoder (training) <s> w1 w2 w1 w2 </s> – Training: inputs are the groundtruth words – Testing: inputs are the previous predicted words ▪ Goal: Resolve training/testing discrepancy with GAN decoder (testing) <s> w’1 w’2 </s>
KDD 2019 164 Discriminator Generator GAN for seq2seq (Lee et al, 2018) <s> software ui </s> <s> software engineer 0generated sequence software ui <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
KDD 2019 165 Discriminator Generator GAN for seq2seq (Lee et al, 2018) <s> software developer </s> <s> software engineer 1generated sequence software developer <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
KDD 2019 166 Query Reformulation: Summary ▪ seq2seq framework: – Directly modeling the words – Very flexible to incorporate session information – Achieves great performance (no character modeling)
KDD 2019 167 Reference ▪ Reda, Azarias, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah. "Metaphor: a system for related search recommendations." In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 664-673. ACM, 2012. ▪ He, Yunlong, Jiliang Tang, Hua Ouyang, Changsung Kang, Dawei Yin, and Yi Chang. "Learning to rewrite queries." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1443-1452. ACM, 2016. ▪ Ren, Gary, Xiaochuan Ni, Manish Malik, and Qifa Ke. "Conversational query understanding using sequence to sequence modeling." In Proceedings of the 2018 World Wide Web Conference, pp. 1715-1724. International World Wide Web Conferences Steering Committee, 2018. ▪ Lee, Mu-Chu, Bin Gao, and Ruofei Zhang. "Rare query expansion through generative adversarial networks in search advertising." In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ Cao, Huanhuan, Daxin Jiang, Jian Pei, Qi He, Zhen Liao, Enhong Chen, and Hang Li. "Context-aware query suggestion by mining click-through and session data." In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 875-883. ACM, 2008. ▪ Sordoni, Alessandro, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. "A hierarchical recurrent encoder-decoder for generative context-aware query suggestion." In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 553-562. ACM, 2015. ▪ Mei, Qiaozhu, Dengyong Zhou, and Kenneth Church. "Query suggestion using hitting time." In Proceedings of the 17th ACM conference on Information and knowledge management, pp. 469-478. ACM, 2008.
KDD 2019 168 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 169 Spell Correction
KDD 2019 170 Spell Correction ▪ Why spell correction: – Reduce the no results ▪ Challenges: – Many rare words (people/company names) look like misspelled words query similar query has error? tumblr tumble No tumblw tumble Yes galaxy s10e galaxy s10 No galaxy s10d galaxy s10 Yes – Modeling characters and words at the same time
KDD 2019 171 Agenda Candidate Generation for a word Candidate generation for a query Candidate ranking Traditional methods
KDD 2019 172 Candidate Generation for a Word (Whitelaw et al 2009) ▪ Goal: – given “tumblw”, suggest “tumblr”, “tumble”... – given “tumblr”, suggest “tumble”, but not “tumblw” ▪ Key challenge: what is a legit word? (“tumblr” 👍 “tumblw” 👎) – Coverage of a dictionary is not enough ▪ Solution: use statistics in web noisy data – Correct words appear more frequent than incorrect words
KDD 2019 173 Candidate Generation for a Word (Whitelaw et al 2009) tumblr tumblw …... <tumblr, tumblw> <tumblw, tumblr> …... edit distance <tumblr, tumblw, 2> <tumblw, tumblr, 50> …... Filter by frequency of context term list confusion word pairs <word, candidate, count> ▪ For the context “social media X”: – freq(“social media tumblw”) < freq(“social media tumblr”)
KDD 2019 174 Candidate Generation for a Query (Chen et al 2007) earr pian kids ear, era, ... pain, pan, ... kids, kid, ... ear pain kids, era pan kid, ear pin kid, ….... ▪ Problem: query candidate size grows exponentially with # of words ▪ Solution: prune with language model – ear pian kids: 0.8 – era pian kids: 0.1 – earr pain kids: 0.9 – earr pan kids: 0.7 – earr pien kids: 0.2 – earr pian kid: 0.8
KDD 2019 175 Candidate Ranking (Li et al 2006, Chen et al 2007) Feature Types Examples similarity Edit distance Frequency of user reformulation …... Likelihood Language model score of the candidate Frequency of candidate terms appearing in the page titles …... querycandidate
KDD 2019 176 Agenda Candidate Generation for a word Candidate generation for a query Candidate ranking Traditional methods seq2seq for spell correction word-to-word char-to-char char-to-word Deep NLP methods
KDD 2019 177 Seq2seq for Spell Correction ▪ From subwords to subwords (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder ear #r <s> ear pain ear pain </s> pi #an
KDD 2019 178 Seq2seq for Spell Correction ▪ From subwords to subwords ▪ Issue: subword is designed for texts without errors (Ghosh and Kristensson, 2017, Zhou et al., 2017) Normal texts Example hunter → hunt #er Subword semantics relevant Spell errors hunetr → hu #net #r irrelevant
KDD 2019 179 Seq2seq for Spell Correction ▪ From characters to characters (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> e a e a r r r r </s> ▪ Issue: on the decoder, no word information ▪ Might produce words with wrong spelling
KDD 2019 180 Seq2seq for Spell Correction ▪ From characters to words – Most popular structure – Can leverage pretrained language model (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> ear r r ear </s>
KDD 2019 181 Reference ▪ Hasan, Saša, Carmen Heger, and Saab Mansour. "Spelling correction of user search queries through statistical machine translation." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 451-460. 2015. ▪ Yingbo Zhou, Utkarsh Porwal, Roberto Konow. "Spelling Correction as a Foreign Language." arXiv. 2017. ▪ Ghosh, Shaona, and Per Ola Kristensson. "Neural networks for text correction and completion in keyboard decoding." arXiv preprint arXiv:1709.06429 (2017). ▪ Chen, Qing, Mu Li, and Ming Zhou. "Improving query spelling correction using web search results." In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 181-189. 2007. ▪ Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. "Exploring distributional similarity based models for query spelling correction." In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 1025-1032. Association for Computational Linguistics, 2006. ▪ Whitelaw, Casey, Ben Hutchinson, Grace Y. Chung, and Gerard Ellis. "Using the web for language independent spellchecking and autocorrection." In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pp. 890-899. Association for Computational Linguistics, 2009.
KDD 2019 182 ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
KDD 2019 183 Conversational recommender systems I watched “Forrest Gump” before Do you like it? I love it! You can try “Saving Private Ryan” today ▪ Motivation: – Actively gather more information or resolve ambiguity – Better user experience ▪ Challenges: – What to ask to the user – Ask vs Rec
KDD 2019 184 Traditional Methods: Collaborative Filtering (Christakopoulou et al., 2016) Recommender Question Generation with Template do you prefer x1 or x2? User answer update Make recommendation ● Issues: ○ Questions are generated by a fixed template ○ No modeling of ask vs rec x3
KDD 2019 185 Seq2seq with Recommender Systems (Li et al., 2018) encoder: I watched “forrest gump” before w1 w2 w3 w4 decoder: do you like it? <s> w1 w2 w1 w2 w3 w3 w4 Recommender forrest gump fight club sentiment analysis: neural
KDD 2019 186 Seq2seq with Recommender Systems (Li et al., 2018) encoder: I love it! w1 w2 w3 w4 decoder: you can try “saving private ryan” today <s> w1 w2 w1 w2 w3 w3 w4 ▪ Issue: only sentiment is encoded; not applicable for facets, such as directed-by, release date Recommender saving private ryan sentiment analysis: positive forrest gump
KDD 2019 187 Recommendation with Belief Tracker (Dhingra et al., 2016, Sun & Zhang, 2018) Hi, I want to find a Chinese restaurant. Belief Tracker Which city are you in? Policy Network Recommender type Chinese price - location - time - ... ... actions: recommendation vs. ask_for_price vs. ask_for_location...
KDD 2019 188 Recommendation with Belief Tracker policy network belief tracker ...... user answer LSTM1 LSTMn + ... Recommender ++ ▪ Reinforcement learning to consider future actions (Dhingra et al., 2016, Sun & Zhang, 2018) rec location → rec food_type → location → rec
KDD 2019 189 Reference ▪ Sun, Yueming, and Yi Zhang. "Conversational recommender system." In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 235-244. ACM, 2018. ▪ Li, Raymond, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. "Towards deep conversational recommendations." In Advances in Neural Information Processing Systems, pp. 9725-9735. 2018. ▪ Christakopoulou, Konstantina, Filip Radlinski, and Katja Hofmann. "Towards conversational recommender systems." In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 815-824. ACM, 2016. ▪ Dhingra, Bhuwan, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. "Towards end-to-end reinforcement learning of dialogue agents for information access." arXiv preprint arXiv:1609.00777 (2016). ▪ Christakopoulou, Konstantina, Filip Radlinski, and Katja Hofmann. "Towards conversational recommender systems." In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 815-824. ACM, 2016.
KDD 2019 190 Language Generation for Search: Summary Traditional methods Candidate generation Rule based Candidate ranking Few features Latency Low deep NLP methods End-to-end solution Language modeling High
Agenda 1 Introduction 2 Deep Learning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
Deep NLP in Search and Recommender Systems - Real World Examples at LinkedIn Huiji Gao
KDD 2019 193 LinkedIn Search and Recommender Systems Job Search Job Recommendation Rank job posts for LinkedIn member to help the user to find jobs that he/she would like to apply for (then, click the “apply” button)
KDD 2019 194 LinkedIn Search and Recommender Systems People Search People RecommendationRecruiter Search Lead Search
KDD 2019 195 Content Recommendation Ads LinkedIn Search and Recommender Systems Content Search
KDD 2019 196 Natural Language Data at LinkedIn Query Job Posts Content Articles Learning Courses Member Profiles
KDD 2019 197 LinkedIn Search and Recommendation Ecosystem Auto Complete Spell Check Language Detection Tokenization Ranking Normalization/ Accent Handling Raw Query Online Assistance Query Understanding Query Suggestion Conversational AI Member Profile Retrieval Search Recommendation Targeting Profile Understanding User Intention Understanding Query Tagging Query Intention Text Processing Document Understanding Candidate Selection
KDD 2019 198 NLP in LinkedIn: Challenges ● Data Ambiguity ○ Short Query Text ■ “abc” ○ No Strict Syntax ■ “bing search engineer” ○ Personalization ■ “looking for new jobs” ● Deep Semantics ○ Representations of documents w.r.t. member intent ABC News? ABC Stores? “Bing Search, Engineer” “Bing, Search Engineer” Job Seeker looks for jobs ■ “engineering openings” -> Job Posts Recruiter looks for candidates ■ “developer proficient at cloud computing” -> “Azure, AWS ...”
KDD 2019 199 Deep NLP in LinkedIn Search & Recommendation: Challenges ● Product Oriented Model Design ○ Design deep NLP algorithms for specific search & recommendation components ○ Consider business rules, post filters, results blender, user experience, etc ● Online Latency ○ Serving deep NLP models with product latency restriction ● Model Debugging ○ Understand model for online performance investigation
KDD 2019 200 Applying Deep NLP in LinkedIn Search & Recommendation ● Feature Driven ○ Representation Learning ■ Using features generated from deep learning models e.g., word embedding ● Model Driven ○ Power product features directly with deep learning models ■ CNN/LSTM/Seq2seq/GAN/BERT based deep NLP models
KDD 2019 201 LinkedIn Search and Recommendation Ecosystem Auto Complete Spell Check Language Detection Tokenization Ranking Normalization/ Accent Handling Raw Query Online Assistance Query Understanding Query Suggestion Conversational AI Member Profile Retrieval Search Recommendation Targeting Profile Understanding User Intention Understanding Query Tagging Query Intention Text Processing Document Understanding Candidate Selection NLG NLU NLU Matching
KDD 2019 202 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 203 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 204 Query Intention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Search Vertical Prediction ■ People, Job Posts, Articles ... People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
KDD 2019 205 Query Intention Model: Task Oriented Intention Prediction People from Company X in Title Y Video on Topic X from Company Y Company with Name X in Geo Y Help in Topic X on Platform Y People from Geo X in Degree Y with Title Z Article from Author X with Topic Y Learning Course from Author X in Skill Y Professional with Skill X in Geo Y …… …… …... Search Tasks …… …… …...
KDD 2019 206 Query Intention: Member Search Footprint People Search Job Search Company Content Jobs Video Learning People Feed/Ads Bridging the Gap between Search and Recommendation
KDD 2019 207 Query Intention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
KDD 2019 208 Query Intention on Search Blending Primary result Secondary cluster Primary result Search verticals Goal: ▪ To understand the vertical preference of user on LinkedIn search
KDD 2019 209 Query Intention Model: Challenges ▪ Complicated Semantics Job? Job Job!
KDD 2019 210 Query Intention Model: Challenges ▪ Personalization ● Query: Blockchain Tech ● Job seeker looks for Blockchain Technology job ● Company named Blockchain Tech ● Learning course on Blockchain Tech ● Video about blockchain technology ● Recruiter looks for candidates with Blockchain tech skill ● Content on Blockchain Technology ● …...
KDD 2019 211 CNN Based Query Intention Model CNN for Semantic Feature Extraction ▪ Word/query representations ▪ Generalization power ▪ Word n-gram patterns Personalization ▪ Member-level Features ...
KDD 2019 212 Query Intent - Experiment Results ▪ Offline Results Overall Accuracy F1 on PEOPLE F1 on JOB Baseline (ML Model) - - - CNN +2.9% +11.9% +1.7% ▪ Online Results – +0.65% Job CTR@1 – +0.90% Overall Cluster CTR, +4.03% Cluster CTR via Entity Click
KDD 2019 213 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 214 Entity Tagging at LinkedIn ● Search Systems Query: Mike LinkedIn Software Engineer Relevant Results Query Results Pool Recall Precision Retrieving (Indexing) Ranking (Ranker) Query Query Tagger Irrelevant Results
KDD 2019 215 ● Understanding Queries & Documents with Entity Tagger Query: Mike LinkedIn Software Engineer Mike LinkedIn Software Engineer FN CN T T ○ Query Tagger for Retrieval FN:{mike} Doc1, Doc2, Doc4, ... CN:{linkedin} Doc2, Doc4, Doc5, Doc6, ... T:{software} Doc2, Doc3, Doc4, Doc7, ... T:{engineer} Doc1, Doc2, Doc4, Doc9, ... Index Query Doc IsFnMatch IsLnMatch IsTitleMatch etc. ○ Ranking Features CN: company name FN: first name T: title Entity Tagging at LinkedIn
KDD 2019 216 Entity Tagging at LinkedIn ● Recommender Systems Ads Recommendation Advertiser Targeting Member Profile Entity Tagging
KDD 2019 217 Natural Language Understanding: Entity Tagging LinkedIn software engineer data scientist jobs CN T T T T O B-CN B-T I-T B-T I-T O B-CN: beginning of a company name I-CN: Inside of a company name B-T: beginning of a job title I-T: Inside of a job title O: Not an entity B-PN: beginning of person name …
KDD 2019 218 Query Tagger: Logistic Regression LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O N way classification sparse ftr vec ftr 0: whether the current word is “linkedin” ftr 1: whether the current word is “facebook” … ftr n: whether the next word is “software” ftr n+1: whether the next word is “linkedin” ….
KDD 2019 219 Query Tagger: CRF LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O CRF Layer sparse ftr vec
KDD 2019 220 Query Tagger: CRF + LSTM LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O CRF Layer LSTM hidden state Word embedding
KDD 2019 221 Query Tagger Performance SCRF CRF CRF + Bi-LSTM Bi-LSTM HMM Deep & Wide Modeling Structure for Entity Tagging
KDD 2019 222 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 223 Natural Language Generation: Query Suggestion ▪ Save User Effort of Rephrasing Informative Queries ▪ Capture Users’ Search Intention ▪ Personalization
KDD 2019 224 Natural Language Generation: Query Suggestion ▪ Traditional Frequency Based Methods – Collect <q1, q2> pairs from search log – Save the frequent pairs in a key-value store ▪ Lack of Generalization – Purely string matching – Cannot handle unseen queries, rare words ▪ Seq2seq: Model Query Reformulation Behavior
KDD 2019 225 encoder Query Suggestion: Reformulate to Related Queries ■ Training: the 2nd query is given ■ Maximize linkedin engineering <s> linkedin software linkedin software development </s> development decoder
KDD 2019 226 decoderencoder Query Suggestion: Reformulate to Related Queries linkedin engineering <s> linkedin software development </s> ■ Inference: the 2nd query is unknown ■ Beam search instead of greedy search
KDD 2019 227 Query Suggestion: Personalization Software engineer Accountant
KDD 2019 228 encoder Query Suggestion: Personalization linkedin engineering<industry_23> decoder <s> linkedin software development </s>
KDD 2019 229 Query Suggestion: How to Handle Online Latency ▪ Latency is strictly constrained for one query – Make it parallel with search ranking Query Generate Ranking Results Generate Suggested Queries Merge for Serving
KDD 2019 230 Online Performance English Market ▪ No Personalization – Coverage: +80% Impressions, +26% CTR – +1% Total job application ▪ With Personalization – +1.18% Session Success for passive job seekers
KDD 2019 231 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 232 Natural Language Generation: Auto-Completion ▪ Given a prefix, predict the completed query, rather than the completed word
KDD 2019 233 Auto-completion Challenges ▪ Short Context – How to enrich semantic features ▪ Personalization ▪ Latency Restriction – Have to adopt simple and fast models
KDD 2019 234 A Two-step Approach: Generation and Ranking ▪ Candidate Generation – Collect query frequency from search log ▪ Candidate Ranking – Neural Language Model serves as a scoring function
KDD 2019 235 Auto-Completion: Neural Language Model as Scoring/Ranking ▪ Achieved 6x speedup with optimization
KDD 2019 236 Auto-Completion Online Experiments Neural Auto-completion vs Frequency Based Methods ▪ People Search English Market – +88.75% autocomplete clicks – +0.81% CTR@1 in SERP ▪ Job Search German Market – +3.24% Results clicked – -6.24% Raw searches w/o results – +3.24% Entity view
KDD 2019 237 Sequential Modeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
KDD 2019 238 Natural Language Processing: Document Ranking Job Search Ads Recommendation
KDD 2019 239 Natural Language Processing: Document Ranking Query Search Systems Recommender Systems P (match | user, query, doc1) P (match | user, doc1) P (match | user, query, doc2) P (match | user, query, doc3) P (match | user, query, docN) …... P (match | user, doc2) P (match | user, doc3) P (match | user, docN) …... Rank Rank
KDD 2019 240 Natural Language Processing: Document Ranking f( , , )User Feature Query Feature Doc FeatureMatching Score = Deep Neural Network, e.g., CNN / BERT User Feature Query Feature Doc Feature User Feature Query Feature Doc Feature Wide Features Combination Layer Deep Features
KDD 2019 241 User Feature Query Feature Doc Feature User Network Tower e.g., BERT Query Network Tower e.g., BERT Doc Network Tower e.g., BERT User Embedding Query Embedding Doc Embedding Interaction network Deep Networks Shallow Network Deep Neural Network e.g., BERT User Feature Query Feature Doc Feature Multi-Tower NetworkOne Big Network Natural Language Processing: Document Ranking
KDD 2019 242 Neural Ranking
KDD 2019 243 Experiments People Search (NDCG@10) Wide Features - CNN +1.32% LinkedIn BERT +1.98% Job Search (NDCG@10) - +2.36% +5.14% Ads Click-through Rate (AUC) - +3.43% - ● Offline ● Online (CNN) ○ Job Search ■ Job apply clicks: +1.62% ■ Onsite paid applicants: +1.38% ○ People Search ■ Satisfied clicks: +0.78% ■ Successful sessions: +0.37% Ads Conversion (AUC) - +1.37% -
Deep NLP in Search and Recommender Systems - Question Answering at LinkedIn Liang Zhang
Traditional Web Search ● User Input: Keyword query ● Output: A list of documents ● A classic way of obtaining information from machines What can be done better: ● Understand user intent and semantic meaning of the query ● Directly show the answer ● Interact with the users when the results do not satisfy the user’s need
Modern Web Search with Question Answering
Modern Web Search with Question Answering Knowledge Card
How Question Answering Can Help LinkedIn LinkedIn Knowledge Base How should I hire for this AI engineer position? Recruiter Assistant Job Seeker Assistant … urgency ...… skills ...which location ... other considerations These are good candidates ... … career goals ...… skill gaps ...which location...other considerationsThese are good job positions for you ... Find good jobs for me.
LinkedIn Help Center • Input: Question in Natural Language • Traditional keyword search does not work well • Our solution: ○ Question-answering through Search ○ Chatbot (Demo)
Question Answering Through Search
LinkedIn Help Center Search • Input: Question as a Query • Output: Article / Section of the Article that contains the answer ○ Few thousands candidates ○ New candidates get added regularly
Our Data • A random sample of user click-through data in 01/2017 - 04/2018 • 200K <query, list of articles> samples, click / non-click as responses for each <query, article> • Only contains queries with > 4 words (where traditional keyword search generally fails!)
Our Modeling Approach • Consider each candidate article as an intent • Problem: Natural language Query => Intent • Need to handle regularly added new articles without too-frequently retraining the model
Our Modeling Approach -- CNN • Independent CNN layers for Question & article title, keywords, content ○ Word2Vec Embedding to initialize • Final layers: Sim<Question Embedding, Article Embedding> => Softmax over all candidate questions • For new articles in test data: Use embedding similarity instead of retraining the model
Deep + Wide Model with Better Performance • Pure CNN Model did not work well for out-of-vocabulary queries • Traditional IR-based Keyword-match features added to complement (wide component) Wide Component (keyword matches)
Our Modeling Approach -- BERT + Wide
Offline Evaluation Model Lift % in Precision@1 vs Control (IR-based) Deep Model (CNN) +26% CNN + Wide Model +32% BERT + Wide Model +36%
Online A/B Test Performance Metric CNN vs Control BERT vs CNN Happy Path Rate +15% +11.3% Clicked Session +5.7% +5.2% ● Happy Path: Users who clicked only one search result and then left help center without creating a case
More Examples...
Interactive Search (Demo)
ChatBot for LinkedIn Help Center
Problems to Solve ● Out-of-scope questions ○ “How are you?” ○ “Will you marry me?” ● Ambiguity ○ Incomplete questions (e.g. verb / object missing) ■ e.g. “premium” => “cancel premium”? “be premium”? ○ Question is complete (verb + object) but ambiguous ■ e.g. “Cancel account” => “cancel premium subscription”? “closing LinkedIn account?” ● Actions to take instead of just article link
Ask for user input User input a question Ask for completion Find the matching help center article User input no yes yesIs question out-of-scope? Is this question complete? Exractable action-to-take sections? Tell user to ask relevant question Question ambiguous (multiple answers)? no Present the answer to user no Ask follow-up question (e.g. device) User input end User satisfied with the answer High-level Workflow Clarify with user User input yes yes User paraphrases the question no
Out-of-Scope Detection Model ● “Who is abraham lincoln?” → Not help center query ● Current: A binary classification approach ○ Positive: help center queries ○ Negative: LinkedIn search queries, movie dialogue, quora questions ● Future: ○ Being able to respond to other intents, e.g. ■ Greetings ■ Ask for speaking to an agent ■ Looking up personal account info (has my premium account expired?)
Incomplete Question Handling ● Incomplete Questions: ○ Object (noun) missing ■ e.g. “cancel” -> what to cancel? ○ Verb missing ■ e.g. “Profile” → what to do with profile? ● We assume the question must contain nouns (objects) and verbs ○ If not, explicitly ask for it ● Standard POS Tagging model from openNLP ● If incomplete: Suggest most related & frequent complete queries from query logs ● Future work ○ Use Entity Tagging in help center context to detect question completeness
Question-Answering Model (BERT + Wide)
Actions-to-take Section Extraction & Ambiguity Handling ● Ambiguity Handling ○ Clarify with user if scores among top 3 articles are too close ● Extract Action-to-take Sections from Top-ranked Article ○ Pattern mining (e.g. ios / android section heads) + Human labeling ○ Follow up with user to pick the right section with actions to take (e.g. desktop / ios / android)
Demo
Conclusion
KDD 2019 270 Lessons & Future Trends ● Ground Truth Data ○ Human Labelled Data (Crowdsourcing) ○ User Behavior Data ○ Automatic Data Generation with Generative Models (e.g., GANs) ● Efficient Modeling Training and Serving ○ Pre-trained Model for Multiple Products ○ Automatic Hyperparameter Tuning & Structure Learning ○ Memory and Latency Optimization for Efficient Online Serving
KDD 2019 271 Lessons & Future Trends (Cont’d) ● Internationalization ○ Transfer Learning ○ Multilingual Learning ● Reinforcement Learning ○ Long-term reward optimization in near-real-time user feedback loop ○ Multi-turn question answering ● Model Debugging ○ Model Interpretation ○ Model Fairness & Privacy
KDD 2019 272 Thank You!

More Related Content

PDF
Beyond the Symbols: A 30-minute Overview of NLP
byMENGSAYLOEM1
 
PPTX
Introduction to natural language processing (NLP)
byAlia Hamwi
 
PDF
NLP Bootcamp 2018 : Representation Learning of text for NLP
byAnuj Gupta
 
PDF
A Review of Deep Contextualized Word Representations (Peters+, 2018)
byShuntaro Yada
 
PDF
Neural Machine Translation (D3L4 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
PPTX
NLP.pptx
byRahul Borate
 
PPTX
Tutorial on Question Answering Systems
bySaeedeh Shekarpour
 
PPTX
Sequence to Sequence Learning with Neural Networks
byNguyen Quang
 
Beyond the Symbols: A 30-minute Overview of NLP
byMENGSAYLOEM1
 
Introduction to natural language processing (NLP)
byAlia Hamwi
 
NLP Bootcamp 2018 : Representation Learning of text for NLP
byAnuj Gupta
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
byShuntaro Yada
 
Neural Machine Translation (D3L4 Deep Learning for Speech and Language UPC 2017)
byUniversitat Politècnica de Catalunya
 
NLP.pptx
byRahul Borate
 
Tutorial on Question Answering Systems
bySaeedeh Shekarpour
 
Sequence to Sequence Learning with Neural Networks
byNguyen Quang
 

What's hot

PDF
Transformer Seq2Sqe Models: Concepts, Trends & Limitations (DLI)
byDeep Learning Italia
 
PDF
NLP using transformers
byArvind Devaraj
 
PDF
Seq2Seq (encoder decoder) model
by佳蓉 倪
 
PDF
Deep Learning for Recommender Systems RecSys2017 Tutorial
byAlexandros Karatzoglou
 
PDF
Convolutional Neural Networks and Natural Language Processing
byThomas Delteil
 
PPTX
Word embedding
byShivaniChoudhary74
 
PDF
Transformer Introduction (Seminar Material)
byYuta Niki
 
PDF
Word2Vec
byhyunyoung Lee
 
PDF
GAN - Theory and Applications
byEmanuele Ghelfi
 
PDF
Word Embeddings - Introduction
byChristian Perone
 
PPTX
Natural Language Processing: Parsing
byRushdi Shams
 
PPTX
Natural language processing
byYogendra Tamang
 
PDF
Deep Learning for Personalized Search and Recommender Systems
byBenjamin Le
 
PPTX
Natural Language processing
bySanzid Kawsar
 
PDF
Deep Learning for Recommender Systems
byJustin Basilico
 
PPT
Big Data and Natural Language Processing
byMichel Bruley
 
PPT
Introduction to Natural Language Processing
byrohitnayak
 
PPTX
Natural Language Processing
bySaurabh Kaushik
 
PPTX
Deep learning lecture - part 1 (basics, CNN)
bySungminYou
 
PPTX
Word embeddings, RNN, GRU and LSTM
byDivya Gera
 
Transformer Seq2Sqe Models: Concepts, Trends & Limitations (DLI)
byDeep Learning Italia
 
NLP using transformers
byArvind Devaraj
 
Seq2Seq (encoder decoder) model
by佳蓉 倪
 
Deep Learning for Recommender Systems RecSys2017 Tutorial
byAlexandros Karatzoglou
 
Convolutional Neural Networks and Natural Language Processing
byThomas Delteil
 
Word embedding
byShivaniChoudhary74
 
Transformer Introduction (Seminar Material)
byYuta Niki
 
Word2Vec
byhyunyoung Lee
 
GAN - Theory and Applications
byEmanuele Ghelfi
 
Word Embeddings - Introduction
byChristian Perone
 
Natural Language Processing: Parsing
byRushdi Shams
 
Natural language processing
byYogendra Tamang
 
Deep Learning for Personalized Search and Recommender Systems
byBenjamin Le
 
Natural Language processing
bySanzid Kawsar
 
Deep Learning for Recommender Systems
byJustin Basilico
 
Big Data and Natural Language Processing
byMichel Bruley
 
Introduction to Natural Language Processing
byrohitnayak
 
Natural Language Processing
bySaurabh Kaushik
 
Deep learning lecture - part 1 (basics, CNN)
bySungminYou
 
Word embeddings, RNN, GRU and LSTM
byDivya Gera
 

Similar to Deep Natural Language Processing for Search and Recommender Systems

PDF
Natural Language Processing (NLP)
byYuriy Guts
 
PPTX
Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)
byWeiwei Guo
 
PPTX
History of deep learning
byayatan2
 
PDF
Deep Learning book by Palash Goyal.
byParagPawar23
 
PPTX
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
byresming1
 
PPTX
Deep natural language processing in search systems
byBill Liu
 
PPTX
Natural Language Processing Advancements By Deep Learning: A Survey
byRimzim Thube
 
PPTX
Discover How Scientific Data is Used for the Public Good with Natural Languag...
byBaoTramDuong2
 
PPTX
Deep Learning for Unified Personalized Search and Recommendations - Jake Mann...
byLucidworks
 
PDF
Nlp and Neural Networks workshop
byQuantUniversity
 
PDF
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
PDF
ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications
byForward Gradient
 
PPTX
Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
byZachary S. Brown
 
PPTX
Building NLP solutions for Davidson ML Group
bybotsplash.com
 
PPTX
Recent nlp trends
byprabu palanisamy
 
PPTX
No more bad news!
bySimon Lia-Jonassen
 
PDF
Big Data Analytics course: Named Entities and Deep Learning for NLP
byChristian Morbidoni
 
PDF
CSCE181 Big ideas in NLP
byInsoo Chung
 
PPTX
Deep Learning and Watson Studio
bySasha Lazarevic
 
PDF
Deep Learning for Search: Personalization and Deep Tokenization
byJake Mannix
 
Natural Language Processing (NLP)
byYuriy Guts
 
Deep Natural Language Processing for Search Systems (sigir 2019 tutorial)
byWeiwei Guo
 
History of deep learning
byayatan2
 
Deep Learning book by Palash Goyal.
byParagPawar23
 
Deep Learning for Natural Language Processing_FDP on 16 June 2025 MITS.pptx
byresming1
 
Deep natural language processing in search systems
byBill Liu
 
Natural Language Processing Advancements By Deep Learning: A Survey
byRimzim Thube
 
Discover How Scientific Data is Used for the Public Good with Natural Languag...
byBaoTramDuong2
 
Deep Learning for Unified Personalized Search and Recommendations - Jake Mann...
byLucidworks
 
Nlp and Neural Networks workshop
byQuantUniversity
 
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
ICDM 2019 Tutorial: Speech and Language Processing: New Tools and Applications
byForward Gradient
 
Deep Learning and Modern Natural Language Processing (AnacondaCon2019)
byZachary S. Brown
 
Building NLP solutions for Davidson ML Group
bybotsplash.com
 
Recent nlp trends
byprabu palanisamy
 
No more bad news!
bySimon Lia-Jonassen
 
Big Data Analytics course: Named Entities and Deep Learning for NLP
byChristian Morbidoni
 
CSCE181 Big ideas in NLP
byInsoo Chung
 
Deep Learning and Watson Studio
bySasha Lazarevic
 
Deep Learning for Search: Personalization and Deep Tokenization
byJake Mannix
 

Recently uploaded

PDF
Enterprise Data Services_ A Development Guide with Use Cases, Trends and Impl...
by2601harsh23
 
PDF
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PPTX
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
PPTX
6G and Non-Terrestrial Networks (NTN) Testing.pptx
byXRComm
 
PDF
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
PPTX
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
PPTX
Coded Agents – with UiPath SDK + LlamaIndex.pptx
bysuhanisingh58689
 
PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PPTX
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
PDF
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
PPTX
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
PPTX
Dell poweredge-customer-presentation.pptx
byhungungphu123
 
PDF
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
PDF
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
PDF
CISO_2027_Playbook: Sovereign AI Resilience & Quantum-Proof Identity
bySaraDavis91
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PPTX
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
PPTX
Advance Computer Architecture Lecture 4.pptx
byBottshikanBottshikan
 
Enterprise Data Services_ A Development Guide with Use Cases, Trends and Impl...
by2601harsh23
 
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
6G and Non-Terrestrial Networks (NTN) Testing.pptx
byXRComm
 
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
SOFTWARE DEVELOPMENT PROCESS - INTRODUCTION
byParithi Thamizh
 
Coded Agents – with UiPath SDK + LlamaIndex.pptx
bysuhanisingh58689
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
Dell poweredge-customer-presentation.pptx
byhungungphu123
 
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
CISO_2027_Playbook: Sovereign AI Resilience & Quantum-Proof Identity
bySaraDavis91
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
Advance Computer Architecture Lecture 4.pptx
byBottshikanBottshikan
 

Deep Natural Language Processing for Search and Recommender Systems

  • 1.
    Deep Natural LanguageProcessing for Search and Recommender Systems Weiwei Guo Huiji Gao Jun Shi Bo Long Liang Zhang Bee-Chung Chen Deepak Agarwal
  • 2.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
  • 3.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
  • 4.
  • 5.
    KDD 2019 5 Introduction- Search and Recommender Systems Search and Recommendation are Everywhere
  • 6.
    KDD 2019 6 Introduction- Search and Recommender Systems Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs KDD 2019 InfoInfo Travel
  • 7.
    KDD 2019 7 KDD2019 Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs Info Sightseeing Food Arrive Anchorage Introduction - Search and Recommender Systems
  • 8.
    KDD 2019 8 KDD2019 Travel FoodSightseeing Attractions Connect People Looking or Jobs Learning Materials Info Attending KDD Attractions Learning Materials Introduction - Search and Recommender Systems Arrive Anchorage
  • 9.
    KDD 2019 9 KDD2019 Travel FoodSightseeing Attractions Learning Materials Connect People Looking or Jobs Info Attending KDD Connect People Looking or Jobs Introduction - Search and Recommender Systems Arrive Anchorage
  • 10.
    KDD 2019 10 Introduction- NLP in Search and Recommender Systems ● Understand User Intention ○ Search Queries (search) Best Places for Bear Viewing Best Places for Bear Viewing Safely Budget? Location? ... ● Understand Documents ○ Posts, Reviews, Comments ○ Synonyms, Facets, Semantics, etc. ● Matching ○ Query & Doc Matching for Retrieval & Ranking ● Online Assistance ○ Query Reformulation ○ User Profiles (recommendation)
  • 11.
    KDD 2019 11 Introduction- NLP in Search and Recommender Systems Generate Human Search & Recommender Systems Generate Assistance Natural Language Understanding (NLU) Natural Language Generation (NLG) Understand Natural Language Data
  • 12.
    KDD 2019 12 NaturalLanguage Processing in Recommendation Ecosystem Retrieval & Ranking Text Processing Member Profile Member Behavior Document Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing
  • 13.
    KDD 2019 13 NaturalLanguage Processing in Search Ecosystem Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation
  • 14.
    KDD 2019 14 NaturalLanguage Processing in Search and Recommendation Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation Text Processing Member Behavior Document Understanding Recommender Systems Search Systems
  • 15.
    KDD 2019 15 NaturalLanguage Processing in Search and Recommendation Retrieval & Ranking Text Processing Member Profile Raw Query Query Understanding Document Understanding Documents Text Processing Profile Understanding Text Processing Query Reformulation ... Query Tagging Query Intention ... Auto Completion Query Suggestion Spell Check ... Natural Language Understanding Entity Tagging Language Detection ... Tokenization Topic Modeling Sentiment Analysis Natural Language Understanding Natural Language Generation
  • 16.
    KDD 2019 16 Introduction- Deep Learning for NLP Figure source: Elvis, 2018, https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-trends-d0d8f40a776d
  • 17.
    KDD 2019 17 Opportunities- Deep Learning for NLP in Search and Recommender Systems Why Deep Learning? ● Deep Semantics from High Dimension and Sparse Data ○ Synonymous, Disambiguation, etc. ● Easy Feature Engineering ○ Hand Crafted Features V.S. Auto Feature Representation ● Model Flexibility ○ Model end-to-end process ○ Various NN components to model and cooperate systematically ● Multi-level Feature Representation ○ Hierarchical Representations character -> token -> word -> phrase -> sentence [Young et. al. 2018]
  • 18.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
  • 19.
    Deep Learning for NaturalLanguage Processing Bo Long
  • 20.
    KDD 2019 20 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 21.
    KDD 2019 21 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
  • 22.
    KDD 2019 22 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context Sequence of Words w_1 Sequential Modeling w_2 w_3 w_n…... Sequence Encoding (Feature Embedding) Tasks (Classification, etc.) Contains Essential Semantics of the Sequence
  • 23.
    KDD 2019 23 aman NLP an NLP engineerI am an am an an NLP NLP EngineerI am am an NLP engineerI Sequential Modeling on Semantic Context I am an NLP engineer I am an NLP engineer MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition I am an NLP engineer Je suis un ingénieur en NLP I am an NLP engineer Je suis un ingénieur en NLP Context Strengthening Transformer GAN Context Consistency Global Context I am an NLP engineer Je suis un ingénieur en NLP Je sont une ingénieur en NLP train test Encoding Encoding Encoding
  • 24.
    KDD 2019 24 MultilayerPerceptron (MLP) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Figure source: https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn/
  • 25.
    KDD 2019 25 ConvolutionalNeural Networks (CNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Zhang et. al. 2015]
  • 26.
    KDD 2019 26 RecurrentNeural Networks (RNN) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context LSTM GRU [LeCun et. al. 2015] github: 2015-08-Understanding-LSTMs
  • 27.
    KDD 2019 27 Sequenceto Sequence (Encoder - Decoder) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
  • 28.
    KDD 2019 28 AttentionMechanisms MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Shi et. al. 2018]
  • 29.
    KDD 2019 29 Transformer MLP CNN UnigramContext RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context Novelty: Eliminate Recurrence transformer http://jalammar.github.io/illustrated-transformer/
  • 30.
    KDD 2019 30 GenerativeAdversarial Networks (GANs) MLP CNN Unigram Context RNN Local (n-gram) Context Accumulated Context Seq2seq Attention Context Transition Context Strengthening Transformer GAN Context Consistency Global Context [Lee, et al., KDD 2018]
  • 31.
    KDD 2019 31 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 32.
    KDD 2019 32 DeepLearning for Natural Language Processing ▪ Sequential Modeling on Semantic Context ▪ Representation Learning for Data Processing
  • 33.
    KDD 2019 33 ▪Representation Learning Sequence of Words w_1 Representation Learning w_2 w_3 w_n…... T’_1 T’_2 T’_3 T’_n…... Word Representation (Embeddings) Deep Learning for Natural Language Processing
  • 34.
    KDD 2019 34 RepresentationLearning Word2Vec Static Word Embedding [Mikolov et. al. 2013]
  • 35.
    KDD 2019 35 RepresentationLearning Word2Vec ELMO Static Word Embedding Contextual Word Embedding - Bidirectional LSTM [Peters et. al. 2018]
  • 36.
    KDD 2019 36 RepresentationLearning Word2Vec ELMO GPT Static Word Embedding Contextual Word Embedding Contextual Word Embedding - Bidirectional LSTM - One Direction Transformer [Radford et. al. 2018]
  • 37.
    KDD 2019 37 RepresentationLearning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding BERT - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer [Devlin et. al. 2018]
  • 38.
    KDD 2019 38 RepresentationLearning Word2Vec ELMO GPT BERT Static Word Embedding Contextual Word Embedding XLNET Contextual Word Embedding Contextual Word Embedding Contextual Word Embedding XLNet - Bidirectional LSTM - Bidirectional Transformer with Multitask Learning - One Direction Transformer - Bidirectional, Learn correlations among masked tokens Pre-trained NLP Model [Yang et. al. 2019]
  • 39.
    KDD 2019 39 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching MLP, CNN, RNN, Transfomer, ... RNN, Transfomer, ... Seq2seq, Transformer, GAN, ... CNN, RNN, ... Word2Vec, ELMO, GPT, BERT, XLNet, ... MLP, CNN, RNN, seq2seq, attention,transformer, GAN
  • 40.
    KDD 2019 40 References- Deep Learning for Natural Language Processing ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [LeCun et. al, 2015] Deep learning, Nature, vol. 521, no. 7553, pp. 436–444, 2015. ▪ [Lee et. al, 2018] Rare query expansion through generative adversarial networks in search advertising, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ [Mikolov et. al. 2013] Distributed representations of words and phrases and their compositionality, in Advances in neural information processing systems, 2013 ▪ [Mikolov et. al. 2013] Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781, 2013. ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Robinson, 2018] https://stackabuse.com/introduction-to-neural-networks-with-scikit-learn, 2018. ▪ [Shi et. al, 2018] Neural Abstractive Text Summarization with Sequence-to-Sequence Models, arXiv preprint arXiv:1812.02303v2, 2018. ▪ [Yang et. al. 2019] XLNet: Generalized Autoregressive Pretraining for Language Understanding, arXiv preprint arXiv:1906.08237v1, 2019 ▪ [Young et. al. 2018] Recent Trends in Deep Learning Based Natural Language Processing, arXiv preprint arXiv:1708.02709v8, 2018 ▪ [Zhang et. al. 2015] A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification, arXiv preprint arXiv:1510.03820, 2015.
  • 41.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
  • 42.
    Deep NLP inSearch and Recommender Systems - Language Understanding Jun Shi
  • 43.
    KDD 2019 43 DeepNLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
  • 44.
    KDD 2019 44 EntityTagging ▪ Problem statement ▪ Traditional statistical models – Hidden Markov Model – Maximum Entropy Markov Model – (Semi-Markov) Conditional Random Field ▪ Deep learning models – Input layer – Context layer – Decoder layer ▪ Training dataset ▪ Evaluation
  • 45.
    KDD 2019 45 EntityTagging - Problem Statement ▪ A named entity, a word or a phrase that clearly identifies one item from a set of other items that have similar attributes. [Li et. al. 2018] ▪ Entity tagging (Named Entity Recognition, NER), the process of locating and classifying named entities in text into predefined entity categories. Washington D.C. was named after George Washington. LOCATION PERSON
  • 46.
    KDD 2019 46 Motivation ▪Efficiency – Looking only for named entities can speed up search. ▪ Precision – Matching both named entity and tags can increase search precision. ▪ Quality – Ranking retrieved results by considering tags improves search quality.
  • 47.
    KDD 2019 47 TraditionalStatistical Models ▪ Generative model – Hidden Markov Model [Baum, et. al. 1966] ▪ Discriminative model – Maximum Entropy Markov Model [McCallum, et. al. 2000] – (Semi-Markov) Conditional Random Field [Lafferty, et. al. 2001]
  • 48.
    KDD 2019 48 HiddenMarkov Model Generative model, model joint probability Pr(T, W) instead of conditional probability Pr(T | W). T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 49.
    KDD 2019 49 MaximumEntropy Markov Model Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 50.
    KDD 2019 50 ConditionalRandom Field Discriminative model, model conditional probability Pr(T | W) directly. T1 T2 T3 W1 W2 W3 tags words T4 T5 W4 W5 is a dummy start state. T0
  • 51.
    KDD 2019 51 Semi-MarkovConditional Random Field Each tag can correspond to a variable-length phrase. T1 T2 W1 W2 W3 tags words T3 W4 W5 W6 T0
  • 52.
    KDD 2019 52 Summary- Traditional Statistical Models Model Pros Cons Hidden Markov Model training is simple when states are observed difficult to include features Maximum Entropy Markov Model easy to include features suffer from “label bias problem” (prefer states with lower number of transitions) Conditional Random Field easy to include features, do not suffer from “label bias problem” training is relatively complex
  • 53.
    KDD 2019 53 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 54.
    KDD 2019 54 InputLayer - Word Embedding [Yadav, et. al. 2018]
  • 55.
    KDD 2019 55 InputLayer - Char Embedding [Yadav, et. al. 2018]
  • 56.
    KDD 2019 56 InputLayer - Word and Char Embedding [Yadav, et. al. 2018]
  • 57.
    KDD 2019 57 Summary- Input Layer word embedding char embedding external features, lexicon, etc. char CNN/RNN word
  • 58.
    KDD 2019 58 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 59.
    KDD 2019 59 ContextLayer - CNN embedding CNN P A D P A D Tag Decoder [Li et. al. 2018]
  • 60.
    KDD 2019 60 ContextLayer - RNN [Li et. al. 2018]
  • 61.
    KDD 2019 61 DeepLearning Tagger Architecture Word embedding, Character embedding, Lexicon features, POS tags,... CNN, RNN, LSTM, Transformer,... Soft-max, CRF, SCRF,... input layer context layer decoder layer [Li, et. al. 2018]
  • 62.
    KDD 2019 62 TagDecoder - MLP+softmax [Li et. al. 2018]
  • 63.
    KDD 2019 63 TagDecoder - CRF [Li et. al. 2018]
  • 64.
    KDD 2019 64 TagDecoder - RNN [Li et. al. 2018]
  • 65.
    KDD 2019 65 Pre-Trainingand Fine-Tuning BERT GPT ELMo [Devlin et. al. 2018] [Peters et. al. 2018][Radford et. al. 2018]
  • 66.
    KDD 2019 66 EntityTagging Evaluation Exact-match evaluation ▪ segment match ▪ tag match
  • 67.
    KDD 2019 67 EntityTagging Training Dataset [Li et. al. 2018]
  • 68.
    KDD 2019 68 EntityTagging on CoNLL03 English Source Method F1 score [Passos et al. 2014] CRF 90.90 [Huang et al. 2015] Bi-LSTM+CRF 84.26 [Collobert et al. 2011] Conv-CRF 89.59 [Kuru et al. 2016] char embedding 84.52 [Chiu and Nichols 2015] word + char embedding 91.62 [Devlin et. al. 2018] BERT Large 92.8
  • 69.
    KDD 2019 69 References- Entity Tagging ▪ [Baum et. al. 1966] Baum, L. E.; Petrie, T. (1966). "Statistical Inference for Probabilistic Functions of Finite State Markov Chains". The Annals of Mathematical Statistics. 37 (6): 1554–1563. ▪ [Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 ▪ [Collins and Singer 1999] Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ▪ [Collobert et. al. 2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. ▪ [Collobert and Weston 2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. ▪ [Devlin et. al. 2018] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint arXiv:1810.04805, 2018 ▪ [Huang et. al. 2015] Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 ▪ [Kuru et. al. 2016] Onur Kuru, Ozan Arkan Can, and Deniz Yuret. 2016. Charner: Character-level named entity recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 911–921 ▪ [Lafferty et. al. 2001] Lafferty, J., McCallum, A., Pereira, F. (2001). "Conditional random fields: Probabilistic models for segmenting and labeling sequence data". Proc. 18th International Conf. on Machine Learning. Morgan Kaufmann. pp. 282–289 ▪ [Li et. al. 2018] Jing Li, Aixin Sun, Jianglei Han, Chenliang Li, A Survey on Deep Learning for Named Entity Recognition, Dec. 2018, arXiv preprint, https://arxiv.org/abs/1812.09449 ▪ [McCallum et. al. 2000] McCallum, Andrew; Freitag, Dayne; Pereira, Fernando (2000). "Maximum Entropy Markov Models for Information Extraction and Segmentation" (PDF). Proc. ICML 2000. pp. 591–598
  • 70.
    KDD 2019 70 References(continued) ▪ [Passos 2014] Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367 ▪ [Peters et. al. 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. ▪ [Radford et.al. 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. ▪ [Yadav et. al. 2018] Vikas Yadav, Steven Bethard, A Survey on Recent Advances in Named Entity Recognition from Deep Learning models, Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158 Santa Fe, New Mexico, USA, August 20-26, 2018.
  • 71.
    KDD 2019 71 DeepNLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation
  • 72.
    KDD 2019 72 EntityDisambiguation ▪ Problem statement ▪ Motivation ▪ Challenges ▪ System architecture – Candidate Entity Generation – Candidate Entity Selection – Joint entity tagging and disambiguation ▪ Evaluation
  • 73.
    KDD 2019 73 EntityDisambiguation - Problem Statement ▪ Also known as entity linking ▪ Resolves mentions to entities in a given knowledge base – Mentions are mostly named entities – Example of knowledge base: freebase, wikipedia ▪ Examples: Jaguar – The prey saw the jaguar cross the jungle. KB:Jaguar – The man saw a jaguar speed on the highway. KB:Jaguar_Cars
  • 74.
    KDD 2019 74 Motivation Increasethe quality of the retrieved results Example: Which Michael Jordan were you looking for?
  • 75.
    KDD 2019 75 Challenges ▪Name variation: New York vs Big Apple ▪ Ambiguity: Michael Jordan ▪ Metonymy: Beijing (city or Chinese government) ▪ Absence: no entries in knowledge base ▪ Evolving information: new company names, e.g. tic-tok.
  • 76.
    KDD 2019 76 SystemArchitecture Candidate Generation Candidate selection 1. Candidate entity generation a. Name dictionary based techniques. b. Surface form expansion from the local document. c. Methods based on search engines. 2. Candidate entity selection a. Graph-based methods b. Text-based methods knowledge base Documents Linked results
  • 77.
    KDD 2019 77 CandidateEntity Generation ▪ Name dictionary based techniques – Entity pages – Redirect pages – Disambiguation pages – Bold phrases from the first paragraphs – Hyperlinks in Wikipedia articles ▪ Surface form expansion – Heuristic based methods – Supervised learning methods ▪ Methods based on search engines [Shen et. al. 2014]
  • 78.
    KDD 2019 78 CandidateEntity Selection - Graph based models ▪ knowledge graph lends itself to graph based methods. ▪ Build a weighted undirected graph with mentions and candidate entities as nodes – weights on mention-entity edges. – weights on entity-entity edge. ▪ Compute an optimal subgraph: – optimality is model dependent. – contains all mentions – one entity per mention mention entity
  • 79.
    KDD 2019 79 AnExample Graph [Hoffart, et. al. 2011]
  • 80.
    KDD 2019 80 CandidateEntity Selection - Text-Based Models ▪ Idea: find agreement (similarity) between entity and mention. – Example: computational intelligence by Michael Jordan and Stuart Russell from UC Berkeley increasing context ● wiki/Michael_Jordan ● wiki/Michael_I._Jordan▪ Models according to context. – Direct Models. ▪ use mention and candidate entities only. – Local Models ▪ use local context around the mention. – Coherence Models. ▪ use mentions and entities in a document. – Collaborative Models. ▪ use mentions and entities across related documents.
  • 81.
    KDD 2019 81 Individualvs Joint Approach Two types of approaches according to how mentions are resolved. ▪ Individual approach – Handle one mention at a time. – Rank the candidate entities. – Relatively low complexity. ▪ Joint approach – Treat all mentions in a document as a sequence. – Jointly optimize the mention-entity pairs. – High complexity, usually resorting to approximations.
  • 82.
    KDD 2019 82 Features Differentfeatures are used in candidate selection modeling. ▪ Traditional Features – entity popularity, entity type. – surface form similarity between entity and mention. – coherence between mapped entities ▪ Deep Features [Yamada et. al. 2016] – word embedding – entity embedding – similarity between words, context and candidate entities.
  • 83.
    KDD 2019 83 Example:Entity Disambiguation with Attention training: max-margin loss [Genea et. al. 2017]
  • 84.
    KDD 2019 84 JointEntity Tagging and Disambiguation ▪ Previously, we find entities first, then link them. But if entities are identified wrong, disambiguation will likely fail too. – Over-split: Romeo and Juliet by Shakespeare – Under-split: Baby Romeo and Juliet were born hours apart. ▪ We could do entity tagging and disambiguation jointly. – Over-generate candidate mentions – Generate possible entities per mention – Score non-overlapping mention-entity pair jointly. [Sil, et. al.2013]
  • 85.
    KDD 2019 85 AJoint Neural Entity Tagging and Disambiguation [Kolitsas, et. al. 2018] Select mentions that have at least one possible entity ● Trained on golden mention-entity pairs. ● Used max-margin loss. cosine similarity between current entity and average of other entities in the document
  • 86.
    KDD 2019 86 EntityDisambiguation Evaluation ▪ Entity-tagging-style F1 score – A link is considered correct only if the mention matches the gold boundary and the linked entity is also correct ▪ Accuracy – another common metric, simpler than F1 score. ▪ TAC-KBP B-Cubed+ F1 score – not widely used
  • 87.
    KDD 2019 87 References- Named Entity Disambiguation ▪ [Cucerzan 2007] Cucerzan, Silviu. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL): 708–716. ▪ [Ganea et. al. 2017] Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2609–2619. Association for Computational Linguistics. ▪ [Globerson et. al. 2016] Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In ACL (1). ▪ [Han et. al. 2011] Han, Xianpei; Sun, Le; Zhao, Jun. Collective Entity Linking in Web Text: A Graph-based Method. ▪ [Hoffart, et. al. 2011] Johannes Hoffart, Mohamed A. Yosef, Ilaria Bordino, Hagen F¨urstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Method in Natural Language Processing, pages 782–792. Association for Computational Linguistics. ▪ [Kolitsas et al., 2018] Nikolaos Kolitsas, Octavianeugen Ganea, and Thomas Hofmann. End-to-end neural entity linking. In CoNLL, 2018. ▪ [Kulkarni et. al. 2009] S. Kulkarni, A. Singh, G. Ramakrishnan, and S. Chakrabarti, Collective annotation of Wikipedia entities in web text, in SIGKDD, 2009, pp. 457–466 ▪ [Ling et al.2015] Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315–328. ▪ [Le et. al., 2018] Phong Le and Ivan Titov. Improving entity linking by modeling latent relations between mentions. In ACL, 2018. ▪ [Rao et. al. 2013] Rao, Delip; McNamee, Paul; Dredze, Mark (2013). Entity Linking: Finding Extracted Entities in a Knowledge Base. Multi-source, Multilingual Information Extraction and Summarization. Springer Berlin Heidelberg: 93–115
  • 88.
    KDD 2019 88 References(continued) ▪ [Raiman et. al. 2018] Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual Entity Linking by Neural Type System Evolution. In Proc. of AAAI. ▪ [Shen et. al. 2014] Wei Shen, Jianyong Wang, and Jiawei Han. 2014. Entity linking with a knowledge base: Issues, techniques, and solutions. TKDE ▪ [Sil et. al. 2013] Avirup Sil and Alexander Yates. 2013. Re-ranking for joint named-entity recognition and linking. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2369–2374. ACM ▪ [Sil et. al. 2016] Sil, A., and Florian, R. 2016. One for all: Towards language independent named entity linking. ACL. ▪ [Yamada et. al. 2016] Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. CoNLL 2016, page 250.
  • 89.
    KDD 2019 89 DeepNLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
  • 90.
    KDD 2019 90 IntentClassification ▪ Problem statement ▪ Deep Learning Models – fastText – CNN – Bi-RNN + Attention
  • 91.
    KDD 2019 91 IntentClassification - Problem Statement ▪ Web search intent can be classified into 3 class: [Broder 2002] – Navigational. The immediate intent is to reach a particular site. ▪ Greyhound Bus. Probable target http://www.greyhound.com – Informational. The intent is to acquire some information assumed to be present on one or more web pages. ▪ San Francisco – Transactional. The intent is to perform some web-mediated activity. ▪ Shopping activities ▪ In conversational AI, intent is usually task-specific, e.g. – I would like to book a flight from SFO to ANC: FlightBooking – What software can I use to view epub documents: SoftwareRecommendation. ▪ We focus on the latter in this tutorial.
  • 92.
    KDD 2019 92 IntentClassification - Methods ▪ Traditional methods – Features: bag of words, n-gram, TF-IDF. – Models: logistic regression, naive Bayes, SVM, random forest. ▪ Deep learning methods – Word embedding + linear classifier (fastText) [Joulin et. al. 2016]. – Convolutional neural networks [Hashemi 2016] . – Bi-RNN + attention (joint slot filling) [Liu et al.2016] .
  • 93.
    KDD 2019 93 IntentClassification - fastText word and N gram embedding features Average softmax [Joulin et. al. 2016]
  • 94.
    KDD 2019 94 IntentClassification - CNN [Hashemi 2016]
  • 95.
    KDD 2019 95 IntentClassification - Bi-RNN+Attention [Liu et. al. 2016]
  • 96.
    KDD 2019 96 References- Intention Classification ▪ [Broder 2002] A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3–10, 2002. ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Joulin et. al. 2016] Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ▪ [Lai et al.2015] Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI, pages 2267–2273 ▪ [Liu et al.2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 ▪ [Zhou et al.2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 207. ▪ [Kim et. al. 2016] Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In Proceedings of SLT. ▪ [Hashemi 2016] Homa B Hashemi, Amir Asiaee, and Reiner Kraft. 2016. Query intent detection using convolutional neural networks. In International Conference on Web Search and Data Mining, Workshop on Query Understanding. ▪ [Liu et. al. 2016]. Liu and I. Lane, “Attention-based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling,” in Interspeech, 2016. ▪ [Shi et. al. 2016] Y. Shi, K. Yao, L. Tian, and D. Jiang, “Deep lstm based feature mapping for query classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1501–1511.
  • 97.
    KDD 2019 97 DeepNLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation
  • 98.
    KDD 2019 98 SentimentAnalysis and Opinion Mining ▪ Problem Statement ▪ Challenges ▪ Traditional methods ▪ Deep learning methods
  • 99.
    KDD 2019 99 SentimentAnalysis - Problem Statement ▪ Interchangable with Opinion Mining in this tutorial. ▪ sentiment analysis mainly studies opinions which express or imply positive or negative sentiments [Liu 2012] ▪ Granularity: – Document level – Sentence level – Aspect level Posted by: John Smith Date: September 10, 2011 “(1) I bought a Canon G12 camera six months ago. (2) I simply love it. (3) The picture quality is amazing. (4) The battery life is also long. (5) However, my wife thinks it is too heavy for her.”
  • 100.
    KDD 2019 100 SentimentAnalysis - Challenges ▪ Word ambiguity – This camera sucks. – This vacuum cleaner really sucks. ▪ Sentiment-less sentiment words – Can you tell me which camera is good? ▪ Sarcasm – What a great car! It stopped working in two days. ▪ Implicit sentiment – This washer uses a lot of water.
  • 101.
    KDD 2019 101 TraditionalMethods ▪ Document/Sentence Sentiment Analysis – unsupervised: rule based, sentiment orientation – supervised: naive Bayes, logistic regression, SVM ▪ Aspect-based Sentiment Analysis – Aspect extraction ▪ Frequent nouns and noun phrases ▪ Opinion and target relations ▪ Sequential tagging: HMM, CRF ▪ Topic modeling – Aspect sentiment classification ▪ Similar methods in document/sentence analysis. ▪ limited to aspect dependent features The screen is very clear but the battery life is too short.
  • 102.
    KDD 2019 102 DeepLearning Approaches ▪ Document/Sentence Sentiment Analysis – Use embedding alone – Use CNN/RNN ▪ Aspect-based Sentiment Analysis – Aspect extraction ▪ CNN or RNN – Aspect sentiment analysis ▪ attention-based RNN, memory networks The screen is very clear but the battery life is too short.
  • 103.
    KDD 2019 103 DocumentSentiment Analysis - Paragraph Vector [Le et. al. 2014]
  • 104.
    KDD 2019 104 DocumentSentiment Analysis - RNN word level sentence level [Yang et. al. 2014]
  • 105.
    KDD 2019 105 SentenceSentiment Analysis - CNN [Kim 2014]
  • 106.
    KDD 2019 106 Aspectand Opinion Extraction - RNN [Wang et. al. 2016] Labels: ● BA (beginning of aspect) ● IA (inside of aspect), ● BO (beginning of opinion) ● IO (inside of opinion) ● O (others)
  • 107.
    KDD 2019 107 AspectLevel Sentiment Analysis - Memory Networks shared parameters [Tang et. al. 2016]
  • 108.
    KDD 2019 108 AspectLevel Sentiment Analysis - Hierarchical RNN ● one aspect per sentence ● aspects are given ● Utilize the relationship between aspects [Ruder et. al. 2016]
  • 109.
    KDD 2019 109 References- Sentiment Analysis and Opinion Mining ▪ [Kim 2014] Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October 2014. Association for Computational Linguistics. ▪ [Le et. al. 2014] Le Q, Mikolov T. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML 2014), 2014. ▪ [Liu 2015] Liu B. Sentiment analysis: mining opinions, sentiments, and emotions. The Cambridge University Press, 2015. ▪ [Liu 2012] Liu B. Sentiment analysis and opinion mining (introduction and survey), Morgan & Claypool, May 2012. ▪ [Ruder et al. 2016] Sebastian Ruder, Parsa Ghaffari, and John G Breslin. A hierarchical model of reviews for aspect-based sentiment analysis. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, November, 2016. ▪ [Schouten et. al. 2016] Kim Schouten and Flavius Frasincar. 2016. Survey on Aspect-Level Sentiment Analysis. IEEE Trans. Knowledge and Data Engineering, 28(3):813-830 ▪ [Tang et. al. 2015] Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422–1432 ▪ [Tang et. al. 2016] Tang D, Qin B, and Liu T. Aspect-level sentiment classification with deep memory network. arXiv preprint ▪ arXiv:1605.08900, 2016. ▪ [Xu et. al. 2016] Xu J, Chen D, Qiu X, and Huang X. Cached long short-term memory neural networks for document-level sentiment classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), 2016. ▪ [Yang et. al. 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics ▪ [Zhang et. al. 2018] Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. In Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, page e1253. Wiley Online Library
  • 110.
    KDD 2019 110 References(Continued) ▪ [Poria et al.2016] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016a. Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems, 108:42–49. ▪ [Wang et. al. 2016] Wang W, Pan SJ, Dahlmeier D, and Xiao X. Recursive neural conditional random fields for aspect-based sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), 2016.
  • 111.
    Jun Shi, WeiweiGuo Deep NLP in Search and Recommender Systems - Document Retrieval & Ranking
  • 112.
    KDD 2019 112 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 113.
    KDD 2019 113 DocumentRetrieval and Ranking ▪ Efficient Candidate Retrieval ▪ Deep Neural Ranking
  • 114.
    KDD 2019 114 EfficientCandidate Retrieval ▪ Syntactic retrieval – based on string matching – use inverted index – can include different fields (name, title, etc). ▪ Semantic retrieval – based on vector space representation. – approximate nearest neighbor search
  • 115.
    KDD 2019 115 SyntacticRetrieval Query: mike software engineer mike Doc1, Doc2, Doc4, ... software Doc2, Doc3, Doc4, Doc7, ... engineer Doc1, Doc2, Doc4, Doc9, ... Inverted index Results: Doc2, Doc4 It won’t retrieve a document contains “mike is a software developer,...”
  • 116.
    KDD 2019 116 SemanticRetrieval - Concept mike software engineer [0.3, 1.27, -3.4, ...] embedding mike software engineer The problem becomes nearest neighbor search [Mitra et. al. 2017]
  • 117.
    KDD 2019 117 SemanticRetrieval - Vector Generation ▪ Bag of words – count based, TF-IDF vectors. ▪ Embedding vectors – word embedding: word2vec, Glove, BERT, etc. – sentence/document embedding ▪ universal sentence encoder [Cer et. al. 2018] ▪ Gaussian document representation [Giannis et. al. 2017] ▪ power mean concatenation [Rücklé et. al. 2018]
  • 118.
    KDD 2019 118 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Suggestion – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 119.
    KDD 2019 119 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods q, d1 q, d2 s1 s2 learning-to-rank loss
  • 120.
    KDD 2019 120 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks q d1 q d2 q, d1 q, d2
  • 121.
    KDD 2019 121 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods s1 s2 learning-to-rank loss ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks q q d1 d2
  • 122.
    KDD 2019 122 DeepNeural Ranking - Agenda ▪ Ranking features ▪ Learning to rank ▪ Traditional methods ▪ Deep neural ranking ▪ Siamese Networks ▪ Interaction-based Networks ▪ Wide & Deep Model
  • 123.
    KDD 2019 123 TraditionalRanking Features ▪ Hand-crafted features – Query/document matching features ▪ Cosine similarity between query and doc title ▪ Entity matching ▪ Clickthrough rate from query to this doc based on search log ▪ ……. – Document alone ▪ popularity ▪ number of incoming links ▪ …...
  • 124.
    KDD 2019 124 Learningto Rank ▪ Pointwise ranking – Logistic regression (Burges, 2010) ▪ Pairwise ranking ▪ Listwise ranking – Cross entropy
  • 125.
    KDD 2019 125 DeepNeural Ranking ▪ Focus on computing query/document score s1 s2 learning-to-rank loss q q d1 d2 ▪ Two categories: – Siamese Networks – Interaction based Networks
  • 126.
    KDD 2019 126 DeepStructured Semantic Model feed-forward (Huang et al., 2013) 30k 300 128 30k 300 128 high dimensional query/doc vector query doc1 sum of word embeddings cos sim
  • 127.
    KDD 2019 127 ModelingWord Sequence by CNN CNN (Shen et al., 2014) d by n matrix 300 128 token embedding matrix d by n matrix 300 128 feed-forward cos sim
  • 128.
    KDD 2019 128 SiameseNetworks ▪ Pros: – Generalization; semantic matching – Efficient; doc embs can be precomputed ▪ Cons: – Lexical features lost: people/company names, rare words
  • 129.
    KDD 2019 129 Interaction-basedNetworks (Guo et al., 2016) query: jobs at linkedin …... document Feed-forward z1 z2 z3 s cosine sim between query words and doc words sim_vec1 sim_vec2 sim_vec3 histogram1 histogram2 histogram2 historgrams based on sim vectors g1 g2 g3 0.4 0.1 0.5
  • 130.
    KDD 2019 130 SearchRanking Neural Networks Summary ▪ Only focus on query/doc scoring ▪ End-to-end models ▪ Two popular architectures Siamese Network Interaction Network Match topical matches lexical matches Latency small large
  • 131.
    KDD 2019 131 RecommenderSystem: Wide & Deep Model (Cheng et al., 2016) wide ftrs user_last_doc_topic=sports user_last_doc_topic=sports && doc_id=100
  • 132.
    KDD 2019 132 RecommenderSystem: Wide & Deep Model (Cheng et al., 2016) user_last_doc_topic=sports user_last_doc_topic=sports && doc_id=100 sports emb doc #100 emb ... emb hidden layer wide ftrs
  • 133.
    KDD 2019 133 References- Deep Neural Ranking ▪ Huang, Po-Sen, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. "Learning deep structured semantic models for web search using clickthrough data." In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338. ACM, 2013. ▪ Shen, Yelong, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. "A latent semantic model with convolutional-pooling structure for information retrieval." In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp. 101-110. ACM, 2014. ▪ Rodrigo Nogueira, Kyunghyun Cho, Passage re-ranking with BERT, 2019 ▪ Burges. "From ranknet to lambdarank to lambdamart: An overview." Learning. 2010. ▪ Guo, Jiafeng, Yixing Fan, Qingyao Ai, and W. Bruce Croft. "A deep relevance matching model for ad-hoc retrieval." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55-64. ACM, 2016. ▪ Mitra, Bhaskar, Fernando Diaz, and Nick Craswell. "Learning to match using local and distributed representations of text for web search." In Proceedings of the 26th International Conference on World Wide Web, pp. 1291-1299. International World Wide Web Conferences Steering Committee, 2017. ▪ Cheng, Heng-Tze, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson et al. "Wide & deep learning for recommender systems." In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10. ACM, 2016. ▪ Xiong, Chenyan, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. "End-to-end neural ad-hoc ranking with kernel pooling." In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pp. 55-64. ACM, 2017. ▪ Dai, Zhuyun, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. "Convolutional neural networks for soft-matching n-grams in ad-hoc search." In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 126-134. ACM, 2018.
  • 134.
    Weiwei Guo Deep NLPin Search and Recommender Systems - Language Generation for Assistance
  • 135.
    KDD 2019 135 DeepNLP in Search and Recommender Systems ▪ Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation
  • 136.
    KDD 2019 136 LanguageGeneration for Assistance ▪ Goal: improve user experience ▪ How: interact with users ▪ NLP: Language generation Common Difference ▪ Character/word modeling ▪ Generation models: – language modeling, seq2seq ▪ Auto Completion ▪ Query Reformulation ▪ Spell Correction ▪ Conversational Recommendation
  • 137.
    KDD 2019 137 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 138.
    KDD 2019 138 QueryAuto-Completion ▪ Problem statement: save user keystrokes by predicting the entire query
  • 139.
    KDD 2019 139 Challenges ▪Limited Context – Hard to extract features due to limited words in a query ▪ Latency – The search component with most strict requirement on latency
  • 140.
  • 141.
    KDD 2019 141 CandidateGeneration ▪ Collect completed queries and associated frequency from search log ▪ Efficiently retrieve most frequent queries starting with the prefix – Using a trie data structure soft software engineer 100 softball 50 softmax 40 softbank 35 …... candidate generation
  • 142.
    KDD 2019 142 CandidateGeneration for Rare Prefix (Mitra & Craswell, 2015) ▪ No such prefix in search log – “cheapest flights from seattle to” “cheapest flights from seattle to” “to” to dc to sfo to airport to seattle …... 100 50 40 35 ▪ Another trie on suffixes – “distance from seattle to dc” → “from seattle to dc”, “seattle to dc”, “to dc”
  • 143.
    KDD 2019 143 ▪Challenge: very few features can be extracted Candidate Ranking soft softbank softball softmax software engineer …... software engineer soft skills softball softbank …... candidate generation candidate ranking frequency 100 50 20 15
  • 144.
    KDD 2019 144 Context-Aware( Bar-Yossef & Kraus, 2011) ▪ Key idea: identify the candidates most similar to previous queries infant n (candidates) infant nike shoes infant nutrition …... infant, nike, shoe, adidas, clothes... infant, nutrition, baby, eat, food... …... (previous queries) baby eating disorder …... baby, eating, disorder, nutrition, food... …... sim score 0.1 0.8
  • 145.
    KDD 2019 145 Personalization(Shokouhi, 2013) Feature list Same age Same gender Same region …... query prefix is “i”
  • 146.
    KDD 2019 146 Agenda Candidate Generation Candidate Ranking Context aware Personalization DeepNLP methods CNN for Ranking End-to-end: Language Model Personalization Traditional methods
  • 147.
    KDD 2019 147 ApplyDeep Models in Ranking ▪ Measuring the semantic coherence between prefix and suffix embsize tri-letter length Query Prefix Query Suffix CNN conv and max-pooling text emb cos sim Cosine similarity (Mitra & Craswell, 2015)
  • 148.
    KDD 2019 148 presidentbarack obaman Language Modeling for Auto-Completion ‘ ’ softmax over chars ‘b’ LSTM char emb ‘t’ char emb ‘ ’ (Park & Chiba, 2017) ▪ Training: Character level language model word emb <INC> word emb `president` + word embedding
  • 149.
    KDD 2019 149 charemb ‘b’ (Park & Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time 0.05 c ‘b’ ‘a’ 0.4 0.3 c ‘b’ ‘a’ 0.1 0.5 0.3 ‘a’ 0.4 Language Modeling for Auto-Completion
  • 150.
    KDD 2019 150 (Park& Chiba, 2017) ▪ Testing: Generating and ranking candidates at the same time ▪ Greedy vs beam search ‘b’ ‘a’ ‘c’ ‘aa’ ‘ab’ ‘ac’ ‘ba’ ‘bb’ ‘bc’ ‘ca’ ‘cb’ ‘cc’ 0.01 0.3 0.3 0.1 0.01 0.01 0.26 0.01 0.01 ‘ab’ ‘ac’ ‘ca’ Language Modeling for Auto-Completion
  • 151.
    KDD 2019 151 Personalization ▪Embeddings for User Ids LSTM softmax over chars char emb ‘t’ word emb <INC> (Fiorin & Lu, 2018) user id emb
  • 152.
    KDD 2019 152 QueryAuto-Completion: Summary ▪ Traditional methods: hard to extract features ▪ Deep language model framework: – Very flexible to incorporate personalized/contextualized information – An end-to-end solution – Train: all parameters are optimized together – Test: generation and ranking are performed at the same time – Cons – Time-consuming – May generate wrong words
  • 153.
    KDD 2019 153 Reference ▪Mitra, Bhaskar, and Nick Craswell. "Query auto-completion for rare prefixes." In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 1755-1758. ACM, 2015. ▪ Park, Dae Hoon, and Rikio Chiba. "A neural language model for query auto-completion." In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189-1192. ACM, 2017. ▪ Bar-Yossef, Ziv, and Naama Kraus. "Context-sensitive query auto-completion." In Proceedings of the 20th international conference on World wide web, pp. 107-116. ACM, 2011. ▪ Shokouhi, Milad. "Learning to personalize query auto-completion." In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pp. 103-112. ACM, 2013. ▪ Jaech, Aaron, and Mari Ostendorf. "Personalized language model for query auto-completion." arXiv preprint arXiv:1804.09661 (2018). ▪ Fiorini, Nicolas, and Zhiyong Lu. "Personalized neural language models for real-world query auto completion." arXiv preprint arXiv:1804.06439 (2018).
  • 154.
    KDD 2019 154 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 155.
    KDD 2019 155 QueryReformulation ▪ Problem Statement: automatically reformulate the previous query Search results for “fb developer” Reformulated queries for “fb developer”
  • 156.
  • 157.
    KDD 2019 157 TraditionalApproach: Collaborative Filtering (Rida et al., 2012) ▪ Collect query pairs issued by the same user from search log ▪ Treat each query as an ID, and build a query-to-query matrix – Value in the matrix is TF-IDF ▪ Issues: – No understanding of words in the query – Data sparsity – Cannot generate new queries
  • 158.
    KDD 2019 158 Agenda collaborative filtering Traditionalmethods seq2seq conversational seq2seq GAN for seq2seq Deep NLP methods
  • 159.
    KDD 2019 159 QueryReformulation as a Translation Task ▪ Sequence-to-sequence modeling decoderencoder facebook developer <s> facebook software facebook software development </s> development ▪ Directly modeling the words in a query (Sordoni et al. 2015, He et al., 2016)
  • 160.
    KDD 2019 160 ConversationalQuery Reformulation ▪ Conversational queries (Ren et al, 2018) first query (q1) second query (q2) when was California founded? who is its governor? California population in 1990 how tall is kobe bryant? what about Lebron James? when was the last summer Olympics? and the winter one? summarized query (q3) who is California’s governor? population of California in 1990 how tall is Lebron James? when was the last winter Olympics? ▪ Goal: summarize q1 & q2 into q3
  • 161.
    KDD 2019 161 encoderdecoder w1 w2 <s> w1 w2 w1 w2 </s> SEP w1 w2 Conversational Query Reformulation ▪ 1st: concatenated seq2seq (Ren et al, 2018)
  • 162.
    KDD 2019 162 ConversationalQuery Reformulation ▪ 2nd: Attention over attention (Ren et al, 2018) 2nd query h1 h2 h3 c1 c2 c h1 h2 h31st query 3rd query (decoder) ht h0
  • 163.
    KDD 2019 163 GANfor seq2seq ▪ Motivation: Training/testing discrepancy in decoder of seq2seq (Lee et al, 2018) decoder (training) <s> w1 w2 w1 w2 </s> – Training: inputs are the groundtruth words – Testing: inputs are the previous predicted words ▪ Goal: Resolve training/testing discrepancy with GAN decoder (testing) <s> w’1 w’2 </s>
  • 164.
    KDD 2019 164 Discriminator Generator GANfor seq2seq (Lee et al, 2018) <s> software ui </s> <s> software engineer 0generated sequence software ui <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
  • 165.
    KDD 2019 165 Discriminator Generator GANfor seq2seq (Lee et al, 2018) <s> software developer </s> <s> software engineer 1generated sequence software developer <s/> 1st query <s> software engineer Original query pair: “software engineer” → “software developer”
  • 166.
    KDD 2019 166 QueryReformulation: Summary ▪ seq2seq framework: – Directly modeling the words – Very flexible to incorporate session information – Achieves great performance (no character modeling)
  • 167.
    KDD 2019 167 Reference ▪Reda, Azarias, Yubin Park, Mitul Tiwari, Christian Posse, and Sam Shah. "Metaphor: a system for related search recommendations." In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 664-673. ACM, 2012. ▪ He, Yunlong, Jiliang Tang, Hua Ouyang, Changsung Kang, Dawei Yin, and Yi Chang. "Learning to rewrite queries." In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1443-1452. ACM, 2016. ▪ Ren, Gary, Xiaochuan Ni, Manish Malik, and Qifa Ke. "Conversational query understanding using sequence to sequence modeling." In Proceedings of the 2018 World Wide Web Conference, pp. 1715-1724. International World Wide Web Conferences Steering Committee, 2018. ▪ Lee, Mu-Chu, Bin Gao, and Ruofei Zhang. "Rare query expansion through generative adversarial networks in search advertising." In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 500-508. ACM, 2018. ▪ Cao, Huanhuan, Daxin Jiang, Jian Pei, Qi He, Zhen Liao, Enhong Chen, and Hang Li. "Context-aware query suggestion by mining click-through and session data." In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 875-883. ACM, 2008. ▪ Sordoni, Alessandro, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. "A hierarchical recurrent encoder-decoder for generative context-aware query suggestion." In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 553-562. ACM, 2015. ▪ Mei, Qiaozhu, Dengyong Zhou, and Kenneth Church. "Query suggestion using hitting time." In Proceedings of the 17th ACM conference on Information and knowledge management, pp. 469-478. ACM, 2008.
  • 168.
    KDD 2019 168 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 169.
  • 170.
    KDD 2019 170 SpellCorrection ▪ Why spell correction: – Reduce the no results ▪ Challenges: – Many rare words (people/company names) look like misspelled words query similar query has error? tumblr tumble No tumblw tumble Yes galaxy s10e galaxy s10 No galaxy s10d galaxy s10 Yes – Modeling characters and words at the same time
  • 171.
    KDD 2019 171 Agenda Candidate Generation fora word Candidate generation for a query Candidate ranking Traditional methods
  • 172.
    KDD 2019 172 CandidateGeneration for a Word (Whitelaw et al 2009) ▪ Goal: – given “tumblw”, suggest “tumblr”, “tumble”... – given “tumblr”, suggest “tumble”, but not “tumblw” ▪ Key challenge: what is a legit word? (“tumblr” 👍 “tumblw” 👎) – Coverage of a dictionary is not enough ▪ Solution: use statistics in web noisy data – Correct words appear more frequent than incorrect words
  • 173.
    KDD 2019 173 CandidateGeneration for a Word (Whitelaw et al 2009) tumblr tumblw …... <tumblr, tumblw> <tumblw, tumblr> …... edit distance <tumblr, tumblw, 2> <tumblw, tumblr, 50> …... Filter by frequency of context term list confusion word pairs <word, candidate, count> ▪ For the context “social media X”: – freq(“social media tumblw”) < freq(“social media tumblr”)
  • 174.
    KDD 2019 174 CandidateGeneration for a Query (Chen et al 2007) earr pian kids ear, era, ... pain, pan, ... kids, kid, ... ear pain kids, era pan kid, ear pin kid, ….... ▪ Problem: query candidate size grows exponentially with # of words ▪ Solution: prune with language model – ear pian kids: 0.8 – era pian kids: 0.1 – earr pain kids: 0.9 – earr pan kids: 0.7 – earr pien kids: 0.2 – earr pian kid: 0.8
  • 175.
    KDD 2019 175 CandidateRanking (Li et al 2006, Chen et al 2007) Feature Types Examples similarity Edit distance Frequency of user reformulation …... Likelihood Language model score of the candidate Frequency of candidate terms appearing in the page titles …... querycandidate
  • 176.
    KDD 2019 176 Agenda Candidate Generation fora word Candidate generation for a query Candidate ranking Traditional methods seq2seq for spell correction word-to-word char-to-char char-to-word Deep NLP methods
  • 177.
    KDD 2019 177 Seq2seqfor Spell Correction ▪ From subwords to subwords (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder ear #r <s> ear pain ear pain </s> pi #an
  • 178.
    KDD 2019 178 Seq2seqfor Spell Correction ▪ From subwords to subwords ▪ Issue: subword is designed for texts without errors (Ghosh and Kristensson, 2017, Zhou et al., 2017) Normal texts Example hunter → hunt #er Subword semantics relevant Spell errors hunetr → hu #net #r irrelevant
  • 179.
    KDD 2019 179 Seq2seqfor Spell Correction ▪ From characters to characters (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> e a e a r r r r </s> ▪ Issue: on the decoder, no word information ▪ Might produce words with wrong spelling
  • 180.
    KDD 2019 180 Seq2seqfor Spell Correction ▪ From characters to words – Most popular structure – Can leverage pretrained language model (Ghosh and Kristensson, 2017, Zhou et al., 2017) encoder decoder e a <s> ear r r ear </s>
  • 181.
    KDD 2019 181 Reference ▪Hasan, Saša, Carmen Heger, and Saab Mansour. "Spelling correction of user search queries through statistical machine translation." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 451-460. 2015. ▪ Yingbo Zhou, Utkarsh Porwal, Roberto Konow. "Spelling Correction as a Foreign Language." arXiv. 2017. ▪ Ghosh, Shaona, and Per Ola Kristensson. "Neural networks for text correction and completion in keyboard decoding." arXiv preprint arXiv:1709.06429 (2017). ▪ Chen, Qing, Mu Li, and Ming Zhou. "Improving query spelling correction using web search results." In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 181-189. 2007. ▪ Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. "Exploring distributional similarity based models for query spelling correction." In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 1025-1032. Association for Computational Linguistics, 2006. ▪ Whitelaw, Casey, Ben Hutchinson, Grace Y. Chung, and Gerard Ellis. "Using the web for language independent spellchecking and autocorrection." In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pp. 890-899. Association for Computational Linguistics, 2009.
  • 182.
    KDD 2019 182 ▪Language Understanding – Entity Tagging: word level prediction – Entity Disambiguation: knowledge base entity prediction – Intent Classification: sentence level prediction – Sentiment Analysis and Opinion Mining ▪ Document Retrieval and Ranking – Efficient Candidate Retrieval – Deep Ranking Models ▪ Language Generation for Assistance – Auto Completion – Query Reformulation – Spell Correction – Conversational Recommendation Deep NLP in Search and Recommender Systems
  • 183.
    KDD 2019 183 Conversationalrecommender systems I watched “Forrest Gump” before Do you like it? I love it! You can try “Saving Private Ryan” today ▪ Motivation: – Actively gather more information or resolve ambiguity – Better user experience ▪ Challenges: – What to ask to the user – Ask vs Rec
  • 184.
    KDD 2019 184 TraditionalMethods: Collaborative Filtering (Christakopoulou et al., 2016) Recommender Question Generation with Template do you prefer x1 or x2? User answer update Make recommendation ● Issues: ○ Questions are generated by a fixed template ○ No modeling of ask vs rec x3
  • 185.
    KDD 2019 185 Seq2seqwith Recommender Systems (Li et al., 2018) encoder: I watched “forrest gump” before w1 w2 w3 w4 decoder: do you like it? <s> w1 w2 w1 w2 w3 w3 w4 Recommender forrest gump fight club sentiment analysis: neural
  • 186.
    KDD 2019 186 Seq2seqwith Recommender Systems (Li et al., 2018) encoder: I love it! w1 w2 w3 w4 decoder: you can try “saving private ryan” today <s> w1 w2 w1 w2 w3 w3 w4 ▪ Issue: only sentiment is encoded; not applicable for facets, such as directed-by, release date Recommender saving private ryan sentiment analysis: positive forrest gump
  • 187.
    KDD 2019 187 Recommendationwith Belief Tracker (Dhingra et al., 2016, Sun & Zhang, 2018) Hi, I want to find a Chinese restaurant. Belief Tracker Which city are you in? Policy Network Recommender type Chinese price - location - time - ... ... actions: recommendation vs. ask_for_price vs. ask_for_location...
  • 188.
    KDD 2019 188 Recommendationwith Belief Tracker policy network belief tracker ...... user answer LSTM1 LSTMn + ... Recommender ++ ▪ Reinforcement learning to consider future actions (Dhingra et al., 2016, Sun & Zhang, 2018) rec location → rec food_type → location → rec
  • 189.
    KDD 2019 189 Reference ▪Sun, Yueming, and Yi Zhang. "Conversational recommender system." In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 235-244. ACM, 2018. ▪ Li, Raymond, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. "Towards deep conversational recommendations." In Advances in Neural Information Processing Systems, pp. 9725-9735. 2018. ▪ Christakopoulou, Konstantina, Filip Radlinski, and Katja Hofmann. "Towards conversational recommender systems." In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 815-824. ACM, 2016. ▪ Dhingra, Bhuwan, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. "Towards end-to-end reinforcement learning of dialogue agents for information access." arXiv preprint arXiv:1609.00777 (2016). ▪ Christakopoulou, Konstantina, Filip Radlinski, and Katja Hofmann. "Towards conversational recommender systems." In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 815-824. ACM, 2016.
  • 190.
    KDD 2019 190 LanguageGeneration for Search: Summary Traditional methods Candidate generation Rule based Candidate ranking Few features Latency Low deep NLP methods End-to-end solution Language modeling High
  • 191.
    Agenda 1 Introduction 2 DeepLearning for Natural Language Processing 3 Deep NLP in Search and Recommender Systems 4 Real World Examples
  • 192.
    Deep NLP inSearch and Recommender Systems - Real World Examples at LinkedIn Huiji Gao
  • 193.
    KDD 2019 193 LinkedInSearch and Recommender Systems Job Search Job Recommendation Rank job posts for LinkedIn member to help the user to find jobs that he/she would like to apply for (then, click the “apply” button)
  • 194.
    KDD 2019 194 LinkedInSearch and Recommender Systems People Search People RecommendationRecruiter Search Lead Search
  • 195.
    KDD 2019 195 ContentRecommendation Ads LinkedIn Search and Recommender Systems Content Search
  • 196.
    KDD 2019 196 NaturalLanguage Data at LinkedIn Query Job Posts Content Articles Learning Courses Member Profiles
  • 197.
    KDD 2019 197 LinkedInSearch and Recommendation Ecosystem Auto Complete Spell Check Language Detection Tokenization Ranking Normalization/ Accent Handling Raw Query Online Assistance Query Understanding Query Suggestion Conversational AI Member Profile Retrieval Search Recommendation Targeting Profile Understanding User Intention Understanding Query Tagging Query Intention Text Processing Document Understanding Candidate Selection
  • 198.
    KDD 2019 198 NLPin LinkedIn: Challenges ● Data Ambiguity ○ Short Query Text ■ “abc” ○ No Strict Syntax ■ “bing search engineer” ○ Personalization ■ “looking for new jobs” ● Deep Semantics ○ Representations of documents w.r.t. member intent ABC News? ABC Stores? “Bing Search, Engineer” “Bing, Search Engineer” Job Seeker looks for jobs ■ “engineering openings” -> Job Posts Recruiter looks for candidates ■ “developer proficient at cloud computing” -> “Azure, AWS ...”
  • 199.
    KDD 2019 199 DeepNLP in LinkedIn Search & Recommendation: Challenges ● Product Oriented Model Design ○ Design deep NLP algorithms for specific search & recommendation components ○ Consider business rules, post filters, results blender, user experience, etc ● Online Latency ○ Serving deep NLP models with product latency restriction ● Model Debugging ○ Understand model for online performance investigation
  • 200.
    KDD 2019 200 ApplyingDeep NLP in LinkedIn Search & Recommendation ● Feature Driven ○ Representation Learning ■ Using features generated from deep learning models e.g., word embedding ● Model Driven ○ Power product features directly with deep learning models ■ CNN/LSTM/Seq2seq/GAN/BERT based deep NLP models
  • 201.
    KDD 2019 201 LinkedInSearch and Recommendation Ecosystem Auto Complete Spell Check Language Detection Tokenization Ranking Normalization/ Accent Handling Raw Query Online Assistance Query Understanding Query Suggestion Conversational AI Member Profile Retrieval Search Recommendation Targeting Profile Understanding User Intention Understanding Query Tagging Query Intention Text Processing Document Understanding Candidate Selection NLG NLU NLU Matching
  • 202.
    KDD 2019 202 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 203.
    KDD 2019 203 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 204.
    KDD 2019 204 QueryIntention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Search Vertical Prediction ■ People, Job Posts, Articles ... People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
  • 205.
    KDD 2019 205 QueryIntention Model: Task Oriented Intention Prediction People from Company X in Title Y Video on Topic X from Company Y Company with Name X in Geo Y Help in Topic X on Platform Y People from Geo X in Degree Y with Title Z Article from Author X with Topic Y Learning Course from Author X in Skill Y Professional with Skill X in Geo Y …… …… …... Search Tasks …… …… …...
  • 206.
    KDD 2019 206 QueryIntention: Member Search Footprint People Search Job Search Company Content Jobs Video Learning People Feed/Ads Bridging the Gap between Search and Recommendation
  • 207.
    KDD 2019 207 QueryIntention Model: Goal …... The searcher is looking for: 0.99 0.06 0.03 ● Output of Query Intention Model ○ Vertical Prediction ■ People, Job Posts, Articles ... Query: LinkedIn Software Engineer People from “LinkedIn” as a “Software Engineer” Job Post from “LinkedIn” on “Software Engineer” position Article from “LinkedIn” on “Software Engineer” topic ○ Properties Associated with the Vertical ■ Extracted from raw query OR Inferred from other information
  • 208.
    KDD 2019 208 QueryIntention on Search Blending Primary result Secondary cluster Primary result Search verticals Goal: ▪ To understand the vertical preference of user on LinkedIn search
  • 209.
    KDD 2019 209 QueryIntention Model: Challenges ▪ Complicated Semantics Job? Job Job!
  • 210.
    KDD 2019 210 QueryIntention Model: Challenges ▪ Personalization ● Query: Blockchain Tech ● Job seeker looks for Blockchain Technology job ● Company named Blockchain Tech ● Learning course on Blockchain Tech ● Video about blockchain technology ● Recruiter looks for candidates with Blockchain tech skill ● Content on Blockchain Technology ● …...
  • 211.
    KDD 2019 211 CNNBased Query Intention Model CNN for Semantic Feature Extraction ▪ Word/query representations ▪ Generalization power ▪ Word n-gram patterns Personalization ▪ Member-level Features ...
  • 212.
    KDD 2019 212 QueryIntent - Experiment Results ▪ Offline Results Overall Accuracy F1 on PEOPLE F1 on JOB Baseline (ML Model) - - - CNN +2.9% +11.9% +1.7% ▪ Online Results – +0.65% Job CTR@1 – +0.90% Overall Cluster CTR, +4.03% Cluster CTR via Entity Click
  • 213.
    KDD 2019 213 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 214.
    KDD 2019 214 EntityTagging at LinkedIn ● Search Systems Query: Mike LinkedIn Software Engineer Relevant Results Query Results Pool Recall Precision Retrieving (Indexing) Ranking (Ranker) Query Query Tagger Irrelevant Results
  • 215.
    KDD 2019 215 ●Understanding Queries & Documents with Entity Tagger Query: Mike LinkedIn Software Engineer Mike LinkedIn Software Engineer FN CN T T ○ Query Tagger for Retrieval FN:{mike} Doc1, Doc2, Doc4, ... CN:{linkedin} Doc2, Doc4, Doc5, Doc6, ... T:{software} Doc2, Doc3, Doc4, Doc7, ... T:{engineer} Doc1, Doc2, Doc4, Doc9, ... Index Query Doc IsFnMatch IsLnMatch IsTitleMatch etc. ○ Ranking Features CN: company name FN: first name T: title Entity Tagging at LinkedIn
  • 216.
    KDD 2019 216 EntityTagging at LinkedIn ● Recommender Systems Ads Recommendation Advertiser Targeting Member Profile Entity Tagging
  • 217.
    KDD 2019 217 NaturalLanguage Understanding: Entity Tagging LinkedIn software engineer data scientist jobs CN T T T T O B-CN B-T I-T B-T I-T O B-CN: beginning of a company name I-CN: Inside of a company name B-T: beginning of a job title I-T: Inside of a job title O: Not an entity B-PN: beginning of person name …
  • 218.
    KDD 2019 218 QueryTagger: Logistic Regression LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O N way classification sparse ftr vec ftr 0: whether the current word is “linkedin” ftr 1: whether the current word is “facebook” … ftr n: whether the next word is “software” ftr n+1: whether the next word is “linkedin” ….
  • 219.
    KDD 2019 219 QueryTagger: CRF LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O CRF Layer sparse ftr vec
  • 220.
    KDD 2019 220 QueryTagger: CRF + LSTM LinkedIn software engineer data scientist jobs B-CN B-T I-T B-T I-T O CRF Layer LSTM hidden state Word embedding
  • 221.
    KDD 2019 221 QueryTagger Performance SCRF CRF CRF + Bi-LSTM Bi-LSTM HMM Deep & Wide Modeling Structure for Entity Tagging
  • 222.
    KDD 2019 222 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 223.
    KDD 2019 223 NaturalLanguage Generation: Query Suggestion ▪ Save User Effort of Rephrasing Informative Queries ▪ Capture Users’ Search Intention ▪ Personalization
  • 224.
    KDD 2019 224 NaturalLanguage Generation: Query Suggestion ▪ Traditional Frequency Based Methods – Collect <q1, q2> pairs from search log – Save the frequent pairs in a key-value store ▪ Lack of Generalization – Purely string matching – Cannot handle unseen queries, rare words ▪ Seq2seq: Model Query Reformulation Behavior
  • 225.
    KDD 2019 225 encoder QuerySuggestion: Reformulate to Related Queries ■ Training: the 2nd query is given ■ Maximize linkedin engineering <s> linkedin software linkedin software development </s> development decoder
  • 226.
    KDD 2019 226 decoderencoder QuerySuggestion: Reformulate to Related Queries linkedin engineering <s> linkedin software development </s> ■ Inference: the 2nd query is unknown ■ Beam search instead of greedy search
  • 227.
    KDD 2019 227 QuerySuggestion: Personalization Software engineer Accountant
  • 228.
    KDD 2019 228 encoder QuerySuggestion: Personalization linkedin engineering<industry_23> decoder <s> linkedin software development </s>
  • 229.
    KDD 2019 229 QuerySuggestion: How to Handle Online Latency ▪ Latency is strictly constrained for one query – Make it parallel with search ranking Query Generate Ranking Results Generate Suggested Queries Merge for Serving
  • 230.
    KDD 2019 230 OnlinePerformance English Market ▪ No Personalization – Coverage: +80% Impressions, +26% CTR – +1% Total job application ▪ With Personalization – +1.18% Session Success for passive job seekers
  • 231.
    KDD 2019 231 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 232.
    KDD 2019 232 NaturalLanguage Generation: Auto-Completion ▪ Given a prefix, predict the completed query, rather than the completed word
  • 233.
    KDD 2019 233 Auto-completionChallenges ▪ Short Context – How to enrich semantic features ▪ Personalization ▪ Latency Restriction – Have to adopt simple and fast models
  • 234.
    KDD 2019 234 ATwo-step Approach: Generation and Ranking ▪ Candidate Generation – Collect query frequency from search log ▪ Candidate Ranking – Neural Language Model serves as a scoring function
  • 235.
    KDD 2019 235 Auto-Completion:Neural Language Model as Scoring/Ranking ▪ Achieved 6x speedup with optimization
  • 236.
    KDD 2019 236 Auto-CompletionOnline Experiments Neural Auto-completion vs Frequency Based Methods ▪ People Search English Market – +88.75% autocomplete clicks – +0.81% CTR@1 in SERP ▪ Job Search German Market – +3.24% Results clicked – -6.24% Raw searches w/o results – +3.24% Entity view
  • 237.
    KDD 2019 237 SequentialModeling Classification (Intention Model) Representation Learning (Data Representation) to sequence label to documents ranking Encoder-Decoder (Query Expansion)to next sequence Sequence Complete (Auto Completion) to completed sequence Deep Learning for Natural Language Processing Sequential Tagging (NER) to word labels Neural Ranker (Document Ranking) NLU NLG Matching
  • 238.
    KDD 2019 238 NaturalLanguage Processing: Document Ranking Job Search Ads Recommendation
  • 239.
    KDD 2019 239 NaturalLanguage Processing: Document Ranking Query Search Systems Recommender Systems P (match | user, query, doc1) P (match | user, doc1) P (match | user, query, doc2) P (match | user, query, doc3) P (match | user, query, docN) …... P (match | user, doc2) P (match | user, doc3) P (match | user, docN) …... Rank Rank
  • 240.
    KDD 2019 240 NaturalLanguage Processing: Document Ranking f( , , )User Feature Query Feature Doc FeatureMatching Score = Deep Neural Network, e.g., CNN / BERT User Feature Query Feature Doc Feature User Feature Query Feature Doc Feature Wide Features Combination Layer Deep Features
  • 241.
    KDD 2019 241 User Feature Query Feature Doc Feature User Network Tower e.g., BERT Query Network Tower e.g., BERT Doc Network Tower e.g., BERT User Embedding Query Embedding Doc Embedding Interactionnetwork Deep Networks Shallow Network Deep Neural Network e.g., BERT User Feature Query Feature Doc Feature Multi-Tower NetworkOne Big Network Natural Language Processing: Document Ranking
  • 242.
  • 243.
    KDD 2019 243 Experiments PeopleSearch (NDCG@10) Wide Features - CNN +1.32% LinkedIn BERT +1.98% Job Search (NDCG@10) - +2.36% +5.14% Ads Click-through Rate (AUC) - +3.43% - ● Offline ● Online (CNN) ○ Job Search ■ Job apply clicks: +1.62% ■ Onsite paid applicants: +1.38% ○ People Search ■ Satisfied clicks: +0.78% ■ Successful sessions: +0.37% Ads Conversion (AUC) - +1.37% -
  • 244.
    Deep NLP inSearch and Recommender Systems - Question Answering at LinkedIn Liang Zhang
  • 245.
    Traditional Web Search ●User Input: Keyword query ● Output: A list of documents ● A classic way of obtaining information from machines What can be done better: ● Understand user intent and semantic meaning of the query ● Directly show the answer ● Interact with the users when the results do not satisfy the user’s need
  • 246.
    Modern Web Searchwith Question Answering
  • 247.
    Modern Web Searchwith Question Answering Knowledge Card
  • 248.
    How Question AnsweringCan Help LinkedIn LinkedIn Knowledge Base How should I hire for this AI engineer position? Recruiter Assistant Job Seeker Assistant … urgency ...… skills ...which location ... other considerations These are good candidates ... … career goals ...… skill gaps ...which location...other considerationsThese are good job positions for you ... Find good jobs for me.
  • 249.
    LinkedIn Help Center •Input: Question in Natural Language • Traditional keyword search does not work well • Our solution: ○ Question-answering through Search ○ Chatbot (Demo)
  • 250.
  • 251.
    LinkedIn Help CenterSearch • Input: Question as a Query • Output: Article / Section of the Article that contains the answer ○ Few thousands candidates ○ New candidates get added regularly
  • 252.
    Our Data • Arandom sample of user click-through data in 01/2017 - 04/2018 • 200K <query, list of articles> samples, click / non-click as responses for each <query, article> • Only contains queries with > 4 words (where traditional keyword search generally fails!)
  • 253.
    Our Modeling Approach •Consider each candidate article as an intent • Problem: Natural language Query => Intent • Need to handle regularly added new articles without too-frequently retraining the model
  • 254.
    Our Modeling Approach-- CNN • Independent CNN layers for Question & article title, keywords, content ○ Word2Vec Embedding to initialize • Final layers: Sim<Question Embedding, Article Embedding> => Softmax over all candidate questions • For new articles in test data: Use embedding similarity instead of retraining the model
  • 255.
    Deep + WideModel with Better Performance • Pure CNN Model did not work well for out-of-vocabulary queries • Traditional IR-based Keyword-match features added to complement (wide component) Wide Component (keyword matches)
  • 256.
    Our Modeling Approach-- BERT + Wide
  • 257.
    Offline Evaluation Model Lift% in Precision@1 vs Control (IR-based) Deep Model (CNN) +26% CNN + Wide Model +32% BERT + Wide Model +36%
  • 258.
    Online A/B TestPerformance Metric CNN vs Control BERT vs CNN Happy Path Rate +15% +11.3% Clicked Session +5.7% +5.2% ● Happy Path: Users who clicked only one search result and then left help center without creating a case
  • 259.
  • 260.
  • 261.
  • 262.
    Problems to Solve ●Out-of-scope questions ○ “How are you?” ○ “Will you marry me?” ● Ambiguity ○ Incomplete questions (e.g. verb / object missing) ■ e.g. “premium” => “cancel premium”? “be premium”? ○ Question is complete (verb + object) but ambiguous ■ e.g. “Cancel account” => “cancel premium subscription”? “closing LinkedIn account?” ● Actions to take instead of just article link
  • 263.
    Ask for user input Userinput a question Ask for completion Find the matching help center article User input no yes yesIs question out-of-scope? Is this question complete? Exractable action-to-take sections? Tell user to ask relevant question Question ambiguous (multiple answers)? no Present the answer to user no Ask follow-up question (e.g. device) User input end User satisfied with the answer High-level Workflow Clarify with user User input yes yes User paraphrases the question no
  • 264.
    Out-of-Scope Detection Model ●“Who is abraham lincoln?” → Not help center query ● Current: A binary classification approach ○ Positive: help center queries ○ Negative: LinkedIn search queries, movie dialogue, quora questions ● Future: ○ Being able to respond to other intents, e.g. ■ Greetings ■ Ask for speaking to an agent ■ Looking up personal account info (has my premium account expired?)
  • 265.
    Incomplete Question Handling ●Incomplete Questions: ○ Object (noun) missing ■ e.g. “cancel” -> what to cancel? ○ Verb missing ■ e.g. “Profile” → what to do with profile? ● We assume the question must contain nouns (objects) and verbs ○ If not, explicitly ask for it ● Standard POS Tagging model from openNLP ● If incomplete: Suggest most related & frequent complete queries from query logs ● Future work ○ Use Entity Tagging in help center context to detect question completeness
  • 266.
  • 267.
    Actions-to-take Section Extraction& Ambiguity Handling ● Ambiguity Handling ○ Clarify with user if scores among top 3 articles are too close ● Extract Action-to-take Sections from Top-ranked Article ○ Pattern mining (e.g. ios / android section heads) + Human labeling ○ Follow up with user to pick the right section with actions to take (e.g. desktop / ios / android)
  • 268.
  • 269.
  • 270.
    KDD 2019 270 Lessons& Future Trends ● Ground Truth Data ○ Human Labelled Data (Crowdsourcing) ○ User Behavior Data ○ Automatic Data Generation with Generative Models (e.g., GANs) ● Efficient Modeling Training and Serving ○ Pre-trained Model for Multiple Products ○ Automatic Hyperparameter Tuning & Structure Learning ○ Memory and Latency Optimization for Efficient Online Serving
  • 271.
    KDD 2019 271 Lessons& Future Trends (Cont’d) ● Internationalization ○ Transfer Learning ○ Multilingual Learning ● Reinforcement Learning ○ Long-term reward optimization in near-real-time user feedback loop ○ Multi-turn question answering ● Model Debugging ○ Model Interpretation ○ Model Fairness & Privacy
  • 272.