Roelof Pieters, profile picture
Uploaded byRoelof Pieters
PDF, PPTX10,511 views

Deep Learning for Information Retrieval

This document provides an overview of deep learning for information retrieval. It begins with background on the speaker and discusses how the data landscape is changing with increasing amounts of diverse data types. It then introduces neural networks and how deep learning can learn hierarchical representations from data. Key aspects of deep learning that help with natural language processing tasks like word embeddings and modeling compositionality are discussed. Several influential papers that advanced word embeddings and recursive neural networks are also summarized.

Embed presentation

Download as PDF, PPTX
@graphific Roelof Pieters Guest  Lecture:  Deep  Learning   for  Informa8on  Retrieval 28  April  2015 www.csc.kth.se/~roelof/ roelof@kth.se roelof@graph-technologies.com Gve Systems Graph Technologies R&D DD2476 Search Engines and Information Retrieval Systems https://www.kth.se/social/course/DD2476/ slides  online  at  
 h4p://www.slideshare.net/roelofp/deep-­‐learning-­‐for-­‐informa=on-­‐retrieval  
2 About Me • (-10y) CS dropout (Amsterdam Technical Univ.) • (2y) Msc Social Anthropology, Stockholm University • Current: PhD candidate at KTH/CSC with focus on: • Deep Learning for Natural Language Processing (Distributed Semantics) • Graph-based approaches for Knowledge Representation • Multi-modal models • Current: Data Science Consultant at Graph Technologies RD & Gve-Systems • Recommender Systems • Deep Learning • Realtime Graph-based Search Engines
3 Information Retrieval (IR) - Hedvig Kjellström, lecture 1
4 Data landscape is changing 1. Amount of digital data is growing at increasing rate (IOT, digitalization, wearables, phones/ tablets) 2. Data types are shifting as well: 1. from text to audio-visual 2. from professional to personal/social (social media) 3. from semi-structured to unstructured
[Jussi Karlgren, NLP Sthlm Meetup 2014]
6 Data landscape is changing Triple V’s of Big Data: 1. Volume 2. Velocity 3. Variety
7 Making sense of Data Typical ML Regression
8 Making sense of Data Neural NetTypical ML Regression
Degrees of Complexity 9 perceptron demo
Neural Net 10 (figure from Lior Rokach, Ben-Gurion University)
Neural Net 11 (figure from Lior Rokach, Ben-Gurion University)
Neural Net 12 (figure from Lior Rokach, Ben-Gurion University)
Neural Net 13 (figure from Lior Rokach, Ben-Gurion University)
Neural Net 14 (figure from Lior Rokach, Ben-Gurion University)
Neural Net 15 multilayer nn demo
Deep Learning ?? 16
Deep Learning ?? 17 • Learning multiple layers • “Back propagation” • Can “theoretically” learn any function! Prior to 2006: • Very slow and inefficient • SVMs, random forests, etc. SOTA
18 2006+: the 3 Deep Learning Conspirators
19
20
— Andrew Ng “I’ve worked all my life in Machine Learning, and I’ve never seen one algorithm knock over benchmarks like Deep Learning” Deep Learning: Why? 21
Different Levels of Abstraction 22
Hierarchical Learning • Natural progression from low level to high level structure as seen in natural complexity Different Levels of Abstraction Feature Representation 23
Hierarchical Learning • Natural progression from low level to high level structure as seen in natural complexity • Easier to monitor what is being learnt and to guide the machine to better subspaces Different Levels of Abstraction Feature Representation 24
Hierarchical Learning • Natural progression from low level to high level structure as seen in natural complexity • Easier to monitor what is being learnt and to guide the machine to better subspaces • A good lower level representation can be used for many distinct tasks Different Levels of Abstraction Feature Representation 25
Hierarchical Learning • Natural progression from low level to high level structure as seen in natural complexity Different Levels of Abstraction Feature Representation 2626 • Easier to monitor what is being learnt and to guide the machine to better subspaces • A good lower level representation can be used for many distinct tasks
Classic Deep Architecture Input layer Hidden layers Output layer 27
Modern Deep Architecture Input layer Hidden layers Output layer movie time: http://www.cs.toronto.edu/~hinton/adi/index.htm 28
[Kudos to Richard Socher, for this eloquent summary :) ] • Manually designed features are often over-specified, incomplete and take a long time to design and validate • Learned Features are easy to adapt, fast to learn • Deep learning provides a very flexible, (almost?) universal, learnable framework for representing world, visual and linguistic information. • Deep learning can learn unsupervised (from raw text/audio/ images/whatever content) and supervised (with specific labels like positive/negative) Why Deep Learning ? 29
Word Embeddings 30
31 What about NLP ? 1. Language is ambiguous:
 Every sentence has many possible interpretations. 2. Language is productive:
 We will always encounter new words or new constructions 3. Language is culturally specific Some of the challenges in Language Understanding:
• NLP treats words mainly (rule-based/statistical approaches at least) as atomic symbols:
 • or in vector space:
 • also known as “one hot” representation. • Its problem ? Language Representation Love Candy Store [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] Candy [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] AND Store [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 …] = 0 ! 32
Language Representation 33 - Johan Boye, lecture 2 Term-document matrix = Sparse!
Distributional representations “You shall know a word by the company it keeps”
 (J. R. Firth 1957) One of the most successful ideas of modern statistical NLP! these words represent banking • Hard (class based) clustering models • Soft clustering models 34
Distributional hypothesis He filled the wampimuk, passed it around and we all drunk some We found a little, hairy wampimuk sleeping behind the tree (McDonald & Ramscar 2001) 35
Distributional semantics Landauer and Dumais (1997), Turney and Pantel (2010), … 36
Distributional semantics Distributional meaning as co-occurrence vector: 37
Distributional representations • Taking it further: • Continuous word embeddings • Combine vector space semantics with the prediction of probabilistic models • Words are represented as a dense vector: Candy = 38
Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: 39
Word Embeddings: SocherVector Space Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born 40
• Can theoretically (given enough units) approximate “any” function • and fit to “any” kind of data • Efficient for NLP: hidden layers can be used as word lookup tables • Dense distributed word vectors + efficient NN training algorithms: • Can scale to billions of words ! Why Neural Networks for NLP? 41
Word Embeddings: SocherVector Space Model Figure (edited) from Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born ? … 42
Compositionality Principle of compositionality: the “meaning (vector) of a complex expression (sentence) is determined by: — Gottlob Frege 
 (1848 - 1925) - the meanings of its constituent expressions (words) and - the rules (grammar) used to combine them” 43
• How do we handle the compositionality of language in our models? 44 Compositionality
• How do we handle the compositionality of language in our models? • Recursion :
 the same operator (same parameters) is applied repeatedly on different components 45 Compositionality
• How do we handle the compositionality of language in our models? • Option 1: Recurrent Neural Networks (RNN) 46 RNN 1: Recurrent Neural Networks (we ignore recurrent NN’s for this talk)
• How do we handle the compositionality of language in our models? • Option 2: Recursive Neural Networks (also sometimes called RNN) 47 RNN 2: Recursive Neural Networks
Recursive Neural Tensor Network 48
Recursive Neural Tensor Network 49 code & info: http://www.socher.org/index.php/Main/ ParsingNaturalScenesAndNaturalLanguageWithRecursiveNeuralNetworks Socher, R., Liu, C.C., NG, A.Y., Manning, C.D. (2011) 
 Parsing Natural Scenes and Natural Language with Recursive Neural Networks
NP PP/IN NP DT NN PRP$ NN Parse Tree Recurrent NN for Vector Space 50
NP PP/IN NP DT NN PRP$ NN Parse Tree INDT NN PRP NN Compositionality 51 Recurrent NN: CompositionalityRecurrent NN for Vector Space
NP IN NP PRP NN Parse Tree DT NN Compositionality 52 Recurrent NN: CompositionalityRecurrent NN for Vector Space
NP IN NP DT NN PRP NN PP NP (S / ROOT) “rules” “meanings” Compositionality 53 Recurrent NN: CompositionalityRecurrent NN for Vector Space
Vector Space + Word Embeddings: Socher 54 Recurrent NN: CompositionalityRecurrent NN for Vector Space
Vector Space + Word Embeddings: Socher 55 Recurrent NN for Vector Space
Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/56
Word Embeddings: Turian (2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/ 57
Word Embeddings: Demo
Word Embeddings: Collobert & Weston (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P. (2011) . Natural Language Processing (almost) from Scratch 59
Polysemous-embeddings: Stanford (2012) Eric H. Huang, Richard Socher, Christopher D. Manning, Andrew Y. Ng (2012)
 Improving Word Representations via Global Context and Multiple Word Prototypes 60
Linguistic Regularities: Mikolov (2013) code & info: https://code.google.com/p/word2vec/ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations 61
Word Embeddings for MT: Mikolov (2013) Mikolov, T., Le, V. L., Sutskever, I. (2013) . 
 Exploiting Similarities among Languages for Machine Translation 62
Word Embeddings for MT: Kiros (2014) Kiros, R., Zemel, R. S., Salakhutdinov, R. (2014) . 
 A Multiplicative Model for Learning Distributed Text-Based Attribute Representations 63
Recursive Deep Models & Sentiment: Socher (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J.,Manning, C., Ng, A., Potts, C. (2013) 
 Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. code & demo: http://nlp.stanford.edu/sentiment/index.html 64
Paragraph Vectors: Le & Mikolov (2014) Le, Q., Mikolov,. T. (2014) Distributed Representations of Sentences and Documents 65 • add context (sentence, paragraph, document) to word vectors during training ! Results on Stanford Sentiment 
 Treebank dataset:
Paragraph Vectors: Dai et al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 66
Paragraph Vectors: Dai et al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 67
Paragraph Vectors: Dai et al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 68 Nearest neighbours to the machine learning paper “Distributed Representations of Sentences and Documents” in arXiv.
Joint Image-Word Embeddings 69
1. Multimodal representation learning 2. Generating descriptions of images 3. Ranking images and captions (“image-sentence ranking”) Some Current Approaches 70
Bags of Visual Words 71 Source credit : K. Grauman, B. Leibe
Bags of Visual Words (Sivic & Zisserman 2003) standard BoW issues however What we get: But we want: • visual word order/relations • location • scale/viewpoint invariance • … 72
Zero-shot Learning • skip-gram text model on wikipedia corpus of 5.7 million documents (5.4 billion words) - approach from (Mikolov et al. ICLR 2013) 73 Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., Ranzato, M.A. (2013) 
 Devise: A deep visual-semantic embedding model DeViSE model
Encoder: A deep convolutional network (CNN) and long short- term memory recurrent network (LSTM) for learning a joint image-sentence embedding. Decoder: A new neural language model that combines structure and content vectors for generating words one at a time in sequence. Encoder-Decoder pipeline 74 Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (Kiros et al 2014)
• captures Multimodal linguistic regularities Encoder-Decoder pipeline 75
• captures Multimodal linguistic regularities Encoder-Decoder pipeline 76 (PCA projection of (300-dimensional) word and image representations)
77 Vinyals, O., Toshev, A., Bengio, S., Erhan. D. (2015) 
 Show and Tell: A Neural Image Caption Generator Joint Visual-Semantic embedding Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions CNN+LSTM CNN+RNN
78 Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions Joint Visual-Semantic embedding
79 Joint Visual-Semantic embedding Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions
80 Joint Visual-Semantic embedding Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions
Joint Visual-Semantic embedding 81 Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions demo
Any Questions?
Download example code samples from https://github.com/graphific/DL-Meetup-intro 83 git clone --recursive https://github.com/graphific/ DL-Meetup-intro.git Wanna Play ? Code! (more at http://deeplearning.net/ )
• Theano - CPU/GPU symbolic expression compiler in python (from LISA lab at University of Montreal). http://deeplearning.net/software/theano/ • Pylearn2 - library designed to make machine learning research easy. http://deeplearning.net/software/ pylearn2/ • Torch - Matlab-like environment for state-of-the-art machine learning algorithms in lua (from Ronan Collobert, Clement Farabet and Koray Kavukcuoglu) http://torch.ch/ • more info: http://deeplearning.net/software links/ Wanna Play ? Wanna Play ? General Deep Learning 84
• RNNLM (Mikolov)
 http://rnnlm.org • NB-SVM
 https://github.com/mesnilgr/nbsvm • Word2Vec (skipgrams/cbow)
 https://code.google.com/p/word2vec/ (original)
 http://radimrehurek.com/gensim/models/word2vec.html (python) • GloVe
 http://nlp.stanford.edu/projects/glove/ (original)
 https://github.com/maciejkula/glove-python (python) • Socher et al / Stanford RNN Sentiment code:
 http://nlp.stanford.edu/sentiment/code.html • Deep Learning without Magic Tutorial:
 http://nlp.stanford.edu/courses/NAACL2013/ Wanna Play ? NLP 85
• cuda-convnet2 (Alex Krizhevsky, Toronto) (c++/ CUDA, optimized for GTX 580) 
 https://code.google.com/p/cuda-convnet2/ • Caffe (Berkeley) (Cuda/OpenCL, Theano, Python)
 http://caffe.berkeleyvision.org/ • OverFeat (NYU) 
 http://cilvr.nyu.edu/doku.php?id=code:start Wanna Play ? Computer Vision 86
87
Impact on Computer Vision 88
Impact on Computer Vision (from Clarifai)89
Impact on Audio Processing Speech Recognition 90
Impact on Audio Processing TIMIT Speech Recognition (from: Clarifai)91
C&W 2011 Impact on Natural Language Processing Pos: Toutanova et al.
 2003) Ner: Ando & Zhang 
 2005 C&W 2011 92
Impact on Natural Language Processing Named Entity Recognition: 93

More Related Content

PDF
Multi modal retrieval and generation with deep distributed models
byRoelof Pieters
 
PDF
Deep learning for natural language embeddings
byRoelof Pieters
 
PDF
Deep learning for nlp
byViet-Trung TRAN
 
PDF
Visual-Semantic Embeddings: some thoughts on Language
byRoelof Pieters
 
PDF
Deep Learning for NLP: An Introduction to Neural Word Embeddings
byRoelof Pieters
 
PDF
Deep Learning for Natural Language Processing: Word Embeddings
byRoelof Pieters
 
PDF
Deep Learning and Text Mining
byWill Stanton
 
PDF
Zero shot learning through cross-modal transfer
byRoelof Pieters
 
Multi modal retrieval and generation with deep distributed models
byRoelof Pieters
 
Deep learning for natural language embeddings
byRoelof Pieters
 
Deep learning for nlp
byViet-Trung TRAN
 
Visual-Semantic Embeddings: some thoughts on Language
byRoelof Pieters
 
Deep Learning for NLP: An Introduction to Neural Word Embeddings
byRoelof Pieters
 
Deep Learning for Natural Language Processing: Word Embeddings
byRoelof Pieters
 
Deep Learning and Text Mining
byWill Stanton
 
Zero shot learning through cross-modal transfer
byRoelof Pieters
 

What's hot

PDF
Engineering Intelligent NLP Applications Using Deep Learning – Part 1
bySaurabh Kaushik
 
PDF
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
bySaurabh Kaushik
 
PPTX
Neural Text Embeddings for Information Retrieval (WSDM 2017)
byBhaskar Mitra
 
PDF
Representation Learning of Vectors of Words and Phrases
byFelipe Moraes
 
PDF
Information Retrieval with Deep Learning
byAdam Gibson
 
PDF
Anthiil Inside workshop on NLP
bySatyam Saxena
 
PDF
Word Embeddings, why the hype ?
byHady Elsahar
 
PDF
(Deep) Neural Networks在 NLP 和 Text Mining 总结
by君 廖
 
PDF
Learning to understand phrases by embedding the dictionary
byRoelof Pieters
 
PDF
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
byDaniele Di Mitri
 
PDF
Deep Learning, an interactive introduction for NLP-ers
byRoelof Pieters
 
PPTX
Tomáš Mikolov - Distributed Representations for NLP
byMachine Learning Prague
 
PPTX
Using Text Embeddings for Information Retrieval
byBhaskar Mitra
 
PPTX
NLP Bootcamp
byAnuj Gupta
 
PDF
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
PPTX
Recurrent networks and beyond by Tomas Mikolov
byBhaskar Mitra
 
PPTX
Word2vec slide(lab seminar)
byJinpyo Lee
 
PPTX
Talk from NVidia Developer Connect
byAnuj Gupta
 
PPTX
A Simple Introduction to Word Embeddings
byBhaskar Mitra
 
PDF
NLP Bootcamp 2018 : Representation Learning of text for NLP
byAnuj Gupta
 
Engineering Intelligent NLP Applications Using Deep Learning – Part 1
bySaurabh Kaushik
 
Engineering Intelligent NLP Applications Using Deep Learning – Part 2
bySaurabh Kaushik
 
Neural Text Embeddings for Information Retrieval (WSDM 2017)
byBhaskar Mitra
 
Representation Learning of Vectors of Words and Phrases
byFelipe Moraes
 
Information Retrieval with Deep Learning
byAdam Gibson
 
Anthiil Inside workshop on NLP
bySatyam Saxena
 
Word Embeddings, why the hype ?
byHady Elsahar
 
(Deep) Neural Networks在 NLP 和 Text Mining 总结
by君 廖
 
Learning to understand phrases by embedding the dictionary
byRoelof Pieters
 
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
byDaniele Di Mitri
 
Deep Learning, an interactive introduction for NLP-ers
byRoelof Pieters
 
Tomáš Mikolov - Distributed Representations for NLP
byMachine Learning Prague
 
Using Text Embeddings for Information Retrieval
byBhaskar Mitra
 
NLP Bootcamp
byAnuj Gupta
 
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)
byMárton Miháltz
 
Recurrent networks and beyond by Tomas Mikolov
byBhaskar Mitra
 
Word2vec slide(lab seminar)
byJinpyo Lee
 
Talk from NVidia Developer Connect
byAnuj Gupta
 
A Simple Introduction to Word Embeddings
byBhaskar Mitra
 
NLP Bootcamp 2018 : Representation Learning of text for NLP
byAnuj Gupta
 

Similar to Deep Learning for Information Retrieval

PDF
AINL 2016: Nikolenko
byLidia Pivovarova
 
PDF
Deep Learning for NLP (without Magic) - Richard Socher and Christopher Manning
byBigDataCloud
 
PDF
Representation Learning of Text for NLP
byAnuj Gupta
 
PDF
Word2vec: From intuition to practice using gensim
byEdgar Marca
 
PPTX
A Simple Introduction to Neural Information Retrieval
byBhaskar Mitra
 
PDF
Text Representations for Deep learning
byZachary S. Brown
 
PDF
Natural Language Processing (NLP)
byYuriy Guts
 
PDF
Recursive Neural Networks
bySangwoo Mo
 
PPTX
What is word2vec?
byTraian Rebedea
 
PDF
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
byMatthew Lease
 
PDF
Deep Learning & NLP: Graphs to the Rescue!
byRoelof Pieters
 
PDF
Contemporary Models of Natural Language Processing
byKaterina Vylomova
 
PDF
IRJET- Survey on Deep Learning Approaches for Phrase Structure Identification...
byIRJET Journal
 
PPTX
Deep Learning for Natural Language Processing
byParrotAI
 
PDF
Cheatsheet recurrent-neural-networks
bySteve Nouri
 
PPTX
Deep Learning for Natural Language Processing
byJonathan Mugan
 
PDF
Word2vec and Friends
byBruno Gonçalves
 
PPTX
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
PDF
Week 3 Deep Learning And POS Tagging Hands-On
bySARCCOM
 
PPTX
Introduction to Neural Information Retrieval and Large Language Models
bysadjadeb
 
AINL 2016: Nikolenko
byLidia Pivovarova
 
Deep Learning for NLP (without Magic) - Richard Socher and Christopher Manning
byBigDataCloud
 
Representation Learning of Text for NLP
byAnuj Gupta
 
Word2vec: From intuition to practice using gensim
byEdgar Marca
 
A Simple Introduction to Neural Information Retrieval
byBhaskar Mitra
 
Text Representations for Deep learning
byZachary S. Brown
 
Natural Language Processing (NLP)
byYuriy Guts
 
Recursive Neural Networks
bySangwoo Mo
 
What is word2vec?
byTraian Rebedea
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
byMatthew Lease
 
Deep Learning & NLP: Graphs to the Rescue!
byRoelof Pieters
 
Contemporary Models of Natural Language Processing
byKaterina Vylomova
 
IRJET- Survey on Deep Learning Approaches for Phrase Structure Identification...
byIRJET Journal
 
Deep Learning for Natural Language Processing
byParrotAI
 
Cheatsheet recurrent-neural-networks
bySteve Nouri
 
Deep Learning for Natural Language Processing
byJonathan Mugan
 
Word2vec and Friends
byBruno Gonçalves
 
Deep Neural Methods for Retrieval
byBhaskar Mitra
 
Week 3 Deep Learning And POS Tagging Hands-On
bySARCCOM
 
Introduction to Neural Information Retrieval and Large Language Models
bysadjadeb
 

More from Roelof Pieters

PDF
Speculations in anthropology and tech for an uncertain future
byRoelof Pieters
 
PDF
AI assisted creativity
byRoelof Pieters
 
PDF
Creativity and AI: 
Deep Neural Nets "Going Wild"
byRoelof Pieters
 
PDF
Deep Neural Networks 
that talk (Back)… with style
byRoelof Pieters
 
PDF
Building a Deep Learning (Dream) Machine
byRoelof Pieters
 
PDF
Multi-modal embeddings: from discriminative to generative models and creative ai
byRoelof Pieters
 
PDF
Creative AI & multimodality: looking ahead
byRoelof Pieters
 
PDF
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
byRoelof Pieters
 
PDF
Explore Data: Data Science + Visualization
byRoelof Pieters
 
PDF
Deep Learning as a Cat/Dog Detector
byRoelof Pieters
 
PDF
Graph, Data-science, and Deep Learning
byRoelof Pieters
 
PDF
Deep Learning: a birds eye view
byRoelof Pieters
 
PDF
Recommender Systems, Matrices and Graphs
byRoelof Pieters
 
PDF
Hackathon 2014 NLP Hack
byRoelof Pieters
 
Speculations in anthropology and tech for an uncertain future
byRoelof Pieters
 
AI assisted creativity
byRoelof Pieters
 
Creativity and AI: 
Deep Neural Nets "Going Wild"
byRoelof Pieters
 
Deep Neural Networks 
that talk (Back)… with style
byRoelof Pieters
 
Building a Deep Learning (Dream) Machine
byRoelof Pieters
 
Multi-modal embeddings: from discriminative to generative models and creative ai
byRoelof Pieters
 
Creative AI & multimodality: looking ahead
byRoelof Pieters
 
Python for Image Understanding: Deep Learning with Convolutional Neural Nets
byRoelof Pieters
 
Explore Data: Data Science + Visualization
byRoelof Pieters
 
Deep Learning as a Cat/Dog Detector
byRoelof Pieters
 
Graph, Data-science, and Deep Learning
byRoelof Pieters
 
Deep Learning: a birds eye view
byRoelof Pieters
 
Recommender Systems, Matrices and Graphs
byRoelof Pieters
 
Hackathon 2014 NLP Hack
byRoelof Pieters
 

Recently uploaded

PPTX
Complete and Detailed Overview of Odoo 18 Contact
byCeline George
 
PPTX
DISCHARGE FROM HOSPITAL (JHASKETAN )(1).pptx
byJK NURSING VIBES ONLY JHASKETAN KUANAR
 
PPTX
Redox Titration - Oxidation and Reduction
byKhushbu Shah
 
PPTX
Report for Prepare Time Management in Odoo 19 POS
byCeline George
 
PDF
2.India-and-Her-Neighbours ppt.pdf/Social science Grade 7 part -2 By:K SANDE...
bySandeep Swamy
 
PPTX
MELA QUIZ 2026 | Silchar Medical College & Hospital | 11-01-2026
byUltimatewinner0342
 
PPTX
INSTAGRAM: Where Moment Turns Into Stories.pptx
byandriemark51
 
PPTX
Prepositions of Place : English Grammar Presentation
byPintoo Dhillon
 
PPTX
CHAPTER NO. 09 HCP BY GG LABORATORY CLINICAL TEST
byGanu Gavade
 
PPTX
Cash on Delivery (COD) in Odoo 19 website
byCeline George
 
PPTX
what are the Talent pool in Odoo 19 Recruitment
byCeline George
 
PPTX
DIGITAL TRANSFORMATION MODULE KEY POINTS
byLisa Harris
 
PDF
5.India-A-Home-to-Many.pdf/Social science Grade 7 part/:K SANDEEP SWAMY(M.Sc,...
bySandeep Swamy
 
PPTX
CHAPTER NO.6 PLANT FIBRES USED AS SURGICAL DRESSINGS.pptx
byGanu Gavade
 
PPTX
Palta Utsav Open to All Quiz Final Set.pptx
bySourav Kr Podder
 
PDF
Master3 Realms and Yoga All students by LDMMIA
by©LDMMIA, ©Reiki Yoga
 
PPTX
OverView of AI in Products in Odoo 19
byCeline George
 
PPTX
Standard Development Principles in Assessing the Quality of Opinion Editorial...
byastairelyxx
 
PPTX
PLATELET RICH PLASMA in regenerative med
byDr SMRUTI REKHA HOTA
 
PPTX
ANTI- RHEUMATICS CHAPTER NO.05 PHARMACOGNOSY
byGanu Gavade
 
Complete and Detailed Overview of Odoo 18 Contact
byCeline George
 
DISCHARGE FROM HOSPITAL (JHASKETAN )(1).pptx
byJK NURSING VIBES ONLY JHASKETAN KUANAR
 
Redox Titration - Oxidation and Reduction
byKhushbu Shah
 
Report for Prepare Time Management in Odoo 19 POS
byCeline George
 
2.India-and-Her-Neighbours ppt.pdf/Social science Grade 7 part -2 By:K SANDE...
bySandeep Swamy
 
MELA QUIZ 2026 | Silchar Medical College & Hospital | 11-01-2026
byUltimatewinner0342
 
INSTAGRAM: Where Moment Turns Into Stories.pptx
byandriemark51
 
Prepositions of Place : English Grammar Presentation
byPintoo Dhillon
 
CHAPTER NO. 09 HCP BY GG LABORATORY CLINICAL TEST
byGanu Gavade
 
Cash on Delivery (COD) in Odoo 19 website
byCeline George
 
what are the Talent pool in Odoo 19 Recruitment
byCeline George
 
DIGITAL TRANSFORMATION MODULE KEY POINTS
byLisa Harris
 
5.India-A-Home-to-Many.pdf/Social science Grade 7 part/:K SANDEEP SWAMY(M.Sc,...
bySandeep Swamy
 
CHAPTER NO.6 PLANT FIBRES USED AS SURGICAL DRESSINGS.pptx
byGanu Gavade
 
Palta Utsav Open to All Quiz Final Set.pptx
bySourav Kr Podder
 
Master3 Realms and Yoga All students by LDMMIA
by©LDMMIA, ©Reiki Yoga
 
OverView of AI in Products in Odoo 19
byCeline George
 
Standard Development Principles in Assessing the Quality of Opinion Editorial...
byastairelyxx
 
PLATELET RICH PLASMA in regenerative med
byDr SMRUTI REKHA HOTA
 
ANTI- RHEUMATICS CHAPTER NO.05 PHARMACOGNOSY
byGanu Gavade
 

Deep Learning for Information Retrieval

  • 1.
    @graphific Roelof Pieters Guest  Lecture:  Deep  Learning   for  Informa8on  Retrieval 28  April  2015 www.csc.kth.se/~roelof/ roelof@kth.se roelof@graph-technologies.com Gve Systems Graph Technologies R&D DD2476 Search Engines and Information Retrieval Systems https://www.kth.se/social/course/DD2476/ slides  online  at  
 h4p://www.slideshare.net/roelofp/deep-­‐learning-­‐for-­‐informa=on-­‐retrieval  
  • 2.
    2 About Me • (-10y)CS dropout (Amsterdam Technical Univ.) • (2y) Msc Social Anthropology, Stockholm University • Current: PhD candidate at KTH/CSC with focus on: • Deep Learning for Natural Language Processing (Distributed Semantics) • Graph-based approaches for Knowledge Representation • Multi-modal models • Current: Data Science Consultant at Graph Technologies RD & Gve-Systems • Recommender Systems • Deep Learning • Realtime Graph-based Search Engines
  • 3.
    3 Information Retrieval (IR) -Hedvig Kjellström, lecture 1
  • 4.
    4 Data landscape ischanging 1. Amount of digital data is growing at increasing rate (IOT, digitalization, wearables, phones/ tablets) 2. Data types are shifting as well: 1. from text to audio-visual 2. from professional to personal/social (social media) 3. from semi-structured to unstructured
  • 5.
    [Jussi Karlgren, NLPSthlm Meetup 2014]
  • 6.
    6 Data landscape ischanging Triple V’s of Big Data: 1. Volume 2. Velocity 3. Variety
  • 7.
    7 Making sense ofData Typical ML Regression
  • 8.
    8 Making sense ofData Neural NetTypical ML Regression
  • 9.
  • 10.
    Neural Net 10 (figure fromLior Rokach, Ben-Gurion University)
  • 11.
    Neural Net 11 (figure fromLior Rokach, Ben-Gurion University)
  • 12.
    Neural Net 12 (figure fromLior Rokach, Ben-Gurion University)
  • 13.
    Neural Net 13 (figure fromLior Rokach, Ben-Gurion University)
  • 14.
    Neural Net 14 (figure fromLior Rokach, Ben-Gurion University)
  • 15.
  • 16.
  • 17.
    Deep Learning ?? 17 •Learning multiple layers • “Back propagation” • Can “theoretically” learn any function! Prior to 2006: • Very slow and inefficient • SVMs, random forests, etc. SOTA
  • 18.
    18 2006+: the 3Deep Learning Conspirators
  • 19.
  • 20.
  • 21.
    — Andrew Ng “I’veworked all my life in Machine Learning, and I’ve never seen one algorithm knock over benchmarks like Deep Learning” Deep Learning: Why? 21
  • 22.
    Different Levels ofAbstraction 22
  • 23.
    Hierarchical Learning • Naturalprogression from low level to high level structure as seen in natural complexity Different Levels of Abstraction Feature Representation 23
  • 24.
    Hierarchical Learning • Naturalprogression from low level to high level structure as seen in natural complexity • Easier to monitor what is being learnt and to guide the machine to better subspaces Different Levels of Abstraction Feature Representation 24
  • 25.
    Hierarchical Learning • Naturalprogression from low level to high level structure as seen in natural complexity • Easier to monitor what is being learnt and to guide the machine to better subspaces • A good lower level representation can be used for many distinct tasks Different Levels of Abstraction Feature Representation 25
  • 26.
    Hierarchical Learning • Naturalprogression from low level to high level structure as seen in natural complexity Different Levels of Abstraction Feature Representation 2626 • Easier to monitor what is being learnt and to guide the machine to better subspaces • A good lower level representation can be used for many distinct tasks
  • 27.
    Classic Deep Architecture Inputlayer Hidden layers Output layer 27
  • 28.
    Modern Deep Architecture Inputlayer Hidden layers Output layer movie time: http://www.cs.toronto.edu/~hinton/adi/index.htm 28
  • 29.
    [Kudos to RichardSocher, for this eloquent summary :) ] • Manually designed features are often over-specified, incomplete and take a long time to design and validate • Learned Features are easy to adapt, fast to learn • Deep learning provides a very flexible, (almost?) universal, learnable framework for representing world, visual and linguistic information. • Deep learning can learn unsupervised (from raw text/audio/ images/whatever content) and supervised (with specific labels like positive/negative) Why Deep Learning ? 29
  • 30.
  • 31.
    31 What about NLP? 1. Language is ambiguous:
 Every sentence has many possible interpretations. 2. Language is productive:
 We will always encounter new words or new constructions 3. Language is culturally specific Some of the challenges in Language Understanding:
  • 32.
    • NLP treatswords mainly (rule-based/statistical approaches at least) as atomic symbols:
 • or in vector space:
 • also known as “one hot” representation. • Its problem ? Language Representation Love Candy Store [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] Candy [0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 …] AND Store [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 …] = 0 ! 32
  • 33.
    Language Representation 33 - JohanBoye, lecture 2 Term-document matrix = Sparse!
  • 34.
    Distributional representations “You shallknow a word by the company it keeps”
 (J. R. Firth 1957) One of the most successful ideas of modern statistical NLP! these words represent banking • Hard (class based) clustering models • Soft clustering models 34
  • 35.
    Distributional hypothesis He filledthe wampimuk, passed it around and we all drunk some We found a little, hairy wampimuk sleeping behind the tree (McDonald & Ramscar 2001) 35
  • 36.
    Distributional semantics Landauer andDumais (1997), Turney and Pantel (2010), … 36
  • 37.
  • 38.
    Distributional representations • Takingit further: • Continuous word embeddings • Combine vector space semantics with the prediction of probabilistic models • Words are represented as a dense vector: Candy = 38
  • 39.
    Word Embeddings: SocherVectorSpace Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: 39
  • 40.
    Word Embeddings: SocherVectorSpace Model adapted rom Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born 40
  • 41.
    • Can theoretically(given enough units) approximate “any” function • and fit to “any” kind of data • Efficient for NLP: hidden layers can be used as word lookup tables • Dense distributed word vectors + efficient NN training algorithms: • Can scale to billions of words ! Why Neural Networks for NLP? 41
  • 42.
    Word Embeddings: SocherVectorSpace Model Figure (edited) from Bengio, “Representation Learning and Deep Learning”, July, 2012, UCLA In a perfect world: the country of my birth the place where I was born ? … 42
  • 43.
    Compositionality Principle of compositionality: the“meaning (vector) of a complex expression (sentence) is determined by: — Gottlob Frege 
 (1848 - 1925) - the meanings of its constituent expressions (words) and - the rules (grammar) used to combine them” 43
  • 44.
    • How dowe handle the compositionality of language in our models? 44 Compositionality
  • 45.
    • How dowe handle the compositionality of language in our models? • Recursion :
 the same operator (same parameters) is applied repeatedly on different components 45 Compositionality
  • 46.
    • How dowe handle the compositionality of language in our models? • Option 1: Recurrent Neural Networks (RNN) 46 RNN 1: Recurrent Neural Networks (we ignore recurrent NN’s for this talk)
  • 47.
    • How dowe handle the compositionality of language in our models? • Option 2: Recursive Neural Networks (also sometimes called RNN) 47 RNN 2: Recursive Neural Networks
  • 48.
  • 49.
    Recursive Neural TensorNetwork 49 code & info: http://www.socher.org/index.php/Main/ ParsingNaturalScenesAndNaturalLanguageWithRecursiveNeuralNetworks Socher, R., Liu, C.C., NG, A.Y., Manning, C.D. (2011) 
 Parsing Natural Scenes and Natural Language with Recursive Neural Networks
  • 50.
    NP PP/IN NP DT NN PRP$NN Parse Tree Recurrent NN for Vector Space 50
  • 51.
    NP PP/IN NP DT NN PRP$NN Parse Tree INDT NN PRP NN Compositionality 51 Recurrent NN: CompositionalityRecurrent NN for Vector Space
  • 52.
    NP IN NP PRP NN Parse Tree DTNN Compositionality 52 Recurrent NN: CompositionalityRecurrent NN for Vector Space
  • 53.
    NP IN NP DT NN PRPNN PP NP (S / ROOT) “rules” “meanings” Compositionality 53 Recurrent NN: CompositionalityRecurrent NN for Vector Space
  • 54.
    Vector Space +Word Embeddings: Socher 54 Recurrent NN: CompositionalityRecurrent NN for Vector Space
  • 55.
    Vector Space +Word Embeddings: Socher 55 Recurrent NN for Vector Space
  • 56.
    Word Embeddings: Turian(2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/56
  • 57.
    Word Embeddings: Turian(2010) Turian, J., Ratinov, L., Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning code & info: http://metaoptimize.com/projects/wordreprs/ 57
  • 58.
  • 59.
    Word Embeddings: Collobert& Weston (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P. (2011) . Natural Language Processing (almost) from Scratch 59
  • 60.
    Polysemous-embeddings: Stanford (2012) EricH. Huang, Richard Socher, Christopher D. Manning, Andrew Y. Ng (2012)
 Improving Word Representations via Global Context and Multiple Word Prototypes 60
  • 61.
    Linguistic Regularities: Mikolov(2013) code & info: https://code.google.com/p/word2vec/ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations 61
  • 62.
    Word Embeddings forMT: Mikolov (2013) Mikolov, T., Le, V. L., Sutskever, I. (2013) . 
 Exploiting Similarities among Languages for Machine Translation 62
  • 63.
    Word Embeddings forMT: Kiros (2014) Kiros, R., Zemel, R. S., Salakhutdinov, R. (2014) . 
 A Multiplicative Model for Learning Distributed Text-Based Attribute Representations 63
  • 64.
    Recursive Deep Models& Sentiment: Socher (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J.,Manning, C., Ng, A., Potts, C. (2013) 
 Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. code & demo: http://nlp.stanford.edu/sentiment/index.html 64
  • 65.
    Paragraph Vectors: Le& Mikolov (2014) Le, Q., Mikolov,. T. (2014) Distributed Representations of Sentences and Documents 65 • add context (sentence, paragraph, document) to word vectors during training ! Results on Stanford Sentiment 
 Treebank dataset:
  • 66.
    Paragraph Vectors: Daiet al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 66
  • 67.
    Paragraph Vectors: Daiet al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 67
  • 68.
    Paragraph Vectors: Daiet al. (2014) Dai, A., Olah,. C., Le, Q., Corrado, G. (2014) Document Embedding with Paragraph Vectors 68 Nearest neighbours to the machine learning paper “Distributed Representations of Sentences and Documents” in arXiv.
  • 69.
  • 70.
    1. Multimodal representationlearning 2. Generating descriptions of images 3. Ranking images and captions (“image-sentence ranking”) Some Current Approaches 70
  • 71.
    Bags of VisualWords 71 Source credit : K. Grauman, B. Leibe
  • 72.
    Bags of VisualWords (Sivic & Zisserman 2003) standard BoW issues however What we get: But we want: • visual word order/relations • location • scale/viewpoint invariance • … 72
  • 73.
    Zero-shot Learning • skip-gramtext model on wikipedia corpus of 5.7 million documents (5.4 billion words) - approach from (Mikolov et al. ICLR 2013) 73 Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., Ranzato, M.A. (2013) 
 Devise: A deep visual-semantic embedding model DeViSE model
  • 74.
    Encoder: A deepconvolutional network (CNN) and long short- term memory recurrent network (LSTM) for learning a joint image-sentence embedding. Decoder: A new neural language model that combines structure and content vectors for generating words one at a time in sequence. Encoder-Decoder pipeline 74 Kiros, R., Salakhutdinov, R., Zemerl, R. S. (2014) 
 Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (Kiros et al 2014)
  • 75.
    • captures Multimodallinguistic regularities Encoder-Decoder pipeline 75
  • 76.
    • captures Multimodallinguistic regularities Encoder-Decoder pipeline 76 (PCA projection of (300-dimensional) word and image representations)
  • 77.
    77 Vinyals, O., Toshev,A., Bengio, S., Erhan. D. (2015) 
 Show and Tell: A Neural Image Caption Generator Joint Visual-Semantic embedding Karpathy, A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions CNN+LSTM CNN+RNN
  • 78.
    78 Karpathy, A., FeiFei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions Joint Visual-Semantic embedding
  • 79.
    79 Joint Visual-Semantic embedding Karpathy,A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions
  • 80.
    80 Joint Visual-Semantic embedding Karpathy,A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions
  • 81.
    Joint Visual-Semantic embedding 81 Karpathy,A., Fei Fei, L. (2015) 
 Deep Visual-Semantic Alignments for Generating Image Descriptions demo
  • 82.
  • 83.
    Download example codesamples from https://github.com/graphific/DL-Meetup-intro 83 git clone --recursive https://github.com/graphific/ DL-Meetup-intro.git Wanna Play ? Code! (more at http://deeplearning.net/ )
  • 84.
    • Theano -CPU/GPU symbolic expression compiler in python (from LISA lab at University of Montreal). http://deeplearning.net/software/theano/ • Pylearn2 - library designed to make machine learning research easy. http://deeplearning.net/software/ pylearn2/ • Torch - Matlab-like environment for state-of-the-art machine learning algorithms in lua (from Ronan Collobert, Clement Farabet and Koray Kavukcuoglu) http://torch.ch/ • more info: http://deeplearning.net/software links/ Wanna Play ? Wanna Play ? General Deep Learning 84
  • 85.
    • RNNLM (Mikolov)
 http://rnnlm.org •NB-SVM
 https://github.com/mesnilgr/nbsvm • Word2Vec (skipgrams/cbow)
 https://code.google.com/p/word2vec/ (original)
 http://radimrehurek.com/gensim/models/word2vec.html (python) • GloVe
 http://nlp.stanford.edu/projects/glove/ (original)
 https://github.com/maciejkula/glove-python (python) • Socher et al / Stanford RNN Sentiment code:
 http://nlp.stanford.edu/sentiment/code.html • Deep Learning without Magic Tutorial:
 http://nlp.stanford.edu/courses/NAACL2013/ Wanna Play ? NLP 85
  • 86.
    • cuda-convnet2 (AlexKrizhevsky, Toronto) (c++/ CUDA, optimized for GTX 580) 
 https://code.google.com/p/cuda-convnet2/ • Caffe (Berkeley) (Cuda/OpenCL, Theano, Python)
 http://caffe.berkeleyvision.org/ • OverFeat (NYU) 
 http://cilvr.nyu.edu/doku.php?id=code:start Wanna Play ? Computer Vision 86
  • 87.
  • 88.
  • 89.
    Impact on ComputerVision (from Clarifai)89
  • 90.
    Impact on AudioProcessing Speech Recognition 90
  • 91.
    Impact on AudioProcessing TIMIT Speech Recognition (from: Clarifai)91
  • 92.
    C&W 2011 Impact onNatural Language Processing Pos: Toutanova et al.
 2003) Ner: Ando & Zhang 
 2005 C&W 2011 92
  • 93.
    Impact on NaturalLanguage Processing Named Entity Recognition: 93