Neural Information Retrieval (or neural IR) is the application of shallow or deep neural networks to IR tasks. In this lecture, we will cover some of the fundamentals of neural representation learning for text retrieval. We will also discuss some of the recent advances in the applications of deep neural architectures to retrieval tasks.
(These slides were presented at a lecture as part of the Information Retrieval and Data Mining course taught at UCL.)
Deep neural methods have recently demonstrated significant performance improvements in several IR tasks. In this lecture, we will present a brief overview of deep models for ranking and retrieval.
This is a follow-up lecture to "Neural Learning to Rank" (https://www.slideshare.net/BhaskarMitra3/neural-learning-to-rank-231759858)
Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.
- Learn to understand what knowledge graphs are for
- Understand the structure of knowledge graphs (and how it relates to taxonomies and ontologies)
- Understand how knowledge graphs can be created using manual, semi-automatic, and fully automatic methods.
- Understand knowledge graphs as a basis for data integration in companies
- Understand knowledge graphs as tools for data governance and data quality management
- Implement and further develop knowledge graphs in companies
- Query and visualize knowledge graphs (including SPARQL and SHACL crash course)
- Use knowledge graphs and machine learning to enable information retrieval, text mining and document classification with the highest precision
- Develop digital assistants and question and answer systems based on semantic knowledge graphs
- Understand how knowledge graphs can be combined with text mining and machine learning techniques
- Apply knowledge graphs in practice: Case studies and demo applications
Deep neural methods have recently demonstrated significant performance improvements in several IR tasks. In this lecture, we will present a brief overview of deep models for ranking and retrieval.
This is a follow-up lecture to "Neural Learning to Rank" (https://www.slideshare.net/BhaskarMitra3/neural-learning-to-rank-231759858)
Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.
- Learn to understand what knowledge graphs are for
- Understand the structure of knowledge graphs (and how it relates to taxonomies and ontologies)
- Understand how knowledge graphs can be created using manual, semi-automatic, and fully automatic methods.
- Understand knowledge graphs as a basis for data integration in companies
- Understand knowledge graphs as tools for data governance and data quality management
- Implement and further develop knowledge graphs in companies
- Query and visualize knowledge graphs (including SPARQL and SHACL crash course)
- Use knowledge graphs and machine learning to enable information retrieval, text mining and document classification with the highest precision
- Develop digital assistants and question and answer systems based on semantic knowledge graphs
- Understand how knowledge graphs can be combined with text mining and machine learning techniques
- Apply knowledge graphs in practice: Case studies and demo applications
This presentation contains the introduction to NOSQL databases, it's types with examples, differentiation with 40 year old relational database management system, it's usage, why and we should use it.
This talk is about how we applied deep learning techinques to achieve state-of-the-art results in various NLP tasks like sentiment analysis and aspect identification, and how we deployed these models at Flipkart
What really are recommendations engines nowadays?
This presentation introduces the foundations of recommendation algorithms, and covers common approaches as well as some of the most advanced techniques. Although more focused on efficiency than theoretical properties, basics of matrix algebra and optimization-based machine learning are used through the presentation.
Table of Contents:
1. Collaborative Filtering
1.1 User-User
1.2 Item-Item
1.3 User-Item
* Matrix Factorization
* Stochastic Gradient Descent (SGD)
* Truncated Singular Value Decomposition (SVD)
* Alternating Least Square (ALS)
* Deep Learning
2. Content Extraction
* Item-Item Similarities
* Deep Content Extraction: NLP, CNN, LSTM
3. Hybrid Models
4. In Production
4.1 Problematics
4.2 Solutions
4.3 Tools
The Text Classification slides contains the research results about the possible natural language processing algorithms. Specifically, it contains the brief overview of the natural language processing steps, the common algorithms used to transform words into meaningful vectors/data, and the algorithms used to learn and classify the data.
To learn more about RAX Automation Suite, visit: www.raxsuite.com
Presentation of the Semantic Knowledge Graph research paper at the 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (Montreal, Canada - October 18th, 2016)
Abstract—This paper describes a new kind of knowledge representation and mining system which we are calling the Semantic Knowledge Graph. At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes). This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics. As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes. This provides numerous benefits: the knowledge graph can be built automatically from a real-world corpus of data, new nodes - along with their combined edges - can be instantly materialized from any arbitrary combination of preexisting nodes (using set operations), and a full model of the semantic relationships between all entities within a domain can be represented and dynamically traversed using a highly compact representation of the graph. Such a system has widespread applications in areas as diverse as knowledge modeling and reasoning, natural language processing, anomaly detection, data cleansing, semantic search, analytics, data classification, root cause analysis, and recommendations systems. The main contribution of this paper is the introduction of a novel system - the Semantic Knowledge Graph - which is able to dynamically discover and score interesting relationships between any arbitrary combination of entities (words, phrases, or extracted concepts) through dynamically materializing nodes and edges from a compact graphical representation built automatically from a corpus of data representative of a knowledge domain.
Ted Willke - The Brain’s Guide to Dealing with Context in Language UnderstandingMLconf
The Brain’s Guide to Dealing with Context in Language Understanding
Like the visual cortex, the regions of the brain involved in understanding language represent information hierarchically. But whereas the visual cortex organizes things into a spatial hierarchy, the language regions encode information into a hierarchy of timescale. This organization is key to our uniquely human ability to integrate semantic information across narratives. More and more, deep learning-based approaches to natural language understanding embrace models that incorporate contextual information at varying timescales. This has not only led to state-of-the art performance on many difficult natural language tasks, but also to breakthroughs in our understanding of brain activity.
In this talk, we will discuss the important connection between language understanding and context at different timescales. We will explore how different deep learning architectures capture timescales in language and how closely their encodings mimic the brain. Along the way, we will uncover some surprising discoveries about what depth does and doesn’t buy you in deep recurrent neural networks. And we’ll describe a new, more flexible way to think about these architectures and ease design space exploration. Finally, we’ll discuss some of the exciting applications made possible by these breakthroughs.
This presentation contains the introduction to NOSQL databases, it's types with examples, differentiation with 40 year old relational database management system, it's usage, why and we should use it.
This talk is about how we applied deep learning techinques to achieve state-of-the-art results in various NLP tasks like sentiment analysis and aspect identification, and how we deployed these models at Flipkart
What really are recommendations engines nowadays?
This presentation introduces the foundations of recommendation algorithms, and covers common approaches as well as some of the most advanced techniques. Although more focused on efficiency than theoretical properties, basics of matrix algebra and optimization-based machine learning are used through the presentation.
Table of Contents:
1. Collaborative Filtering
1.1 User-User
1.2 Item-Item
1.3 User-Item
* Matrix Factorization
* Stochastic Gradient Descent (SGD)
* Truncated Singular Value Decomposition (SVD)
* Alternating Least Square (ALS)
* Deep Learning
2. Content Extraction
* Item-Item Similarities
* Deep Content Extraction: NLP, CNN, LSTM
3. Hybrid Models
4. In Production
4.1 Problematics
4.2 Solutions
4.3 Tools
The Text Classification slides contains the research results about the possible natural language processing algorithms. Specifically, it contains the brief overview of the natural language processing steps, the common algorithms used to transform words into meaningful vectors/data, and the algorithms used to learn and classify the data.
To learn more about RAX Automation Suite, visit: www.raxsuite.com
Presentation of the Semantic Knowledge Graph research paper at the 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (Montreal, Canada - October 18th, 2016)
Abstract—This paper describes a new kind of knowledge representation and mining system which we are calling the Semantic Knowledge Graph. At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes). This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics. As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes. This provides numerous benefits: the knowledge graph can be built automatically from a real-world corpus of data, new nodes - along with their combined edges - can be instantly materialized from any arbitrary combination of preexisting nodes (using set operations), and a full model of the semantic relationships between all entities within a domain can be represented and dynamically traversed using a highly compact representation of the graph. Such a system has widespread applications in areas as diverse as knowledge modeling and reasoning, natural language processing, anomaly detection, data cleansing, semantic search, analytics, data classification, root cause analysis, and recommendations systems. The main contribution of this paper is the introduction of a novel system - the Semantic Knowledge Graph - which is able to dynamically discover and score interesting relationships between any arbitrary combination of entities (words, phrases, or extracted concepts) through dynamically materializing nodes and edges from a compact graphical representation built automatically from a corpus of data representative of a knowledge domain.
Ted Willke - The Brain’s Guide to Dealing with Context in Language UnderstandingMLconf
The Brain’s Guide to Dealing with Context in Language Understanding
Like the visual cortex, the regions of the brain involved in understanding language represent information hierarchically. But whereas the visual cortex organizes things into a spatial hierarchy, the language regions encode information into a hierarchy of timescale. This organization is key to our uniquely human ability to integrate semantic information across narratives. More and more, deep learning-based approaches to natural language understanding embrace models that incorporate contextual information at varying timescales. This has not only led to state-of-the art performance on many difficult natural language tasks, but also to breakthroughs in our understanding of brain activity.
In this talk, we will discuss the important connection between language understanding and context at different timescales. We will explore how different deep learning architectures capture timescales in language and how closely their encodings mimic the brain. Along the way, we will uncover some surprising discoveries about what depth does and doesn’t buy you in deep recurrent neural networks. And we’ll describe a new, more flexible way to think about these architectures and ease design space exploration. Finally, we’ll discuss some of the exciting applications made possible by these breakthroughs.
In this talk we will summarise some of the detectable trends on AI beyond deep learning. We will focus on the current transition from deep learning to deep semantics, describing the enabling infrastructures, challenges and opportunities in the construction of the next generation AI systems. The talk will focus on Natural Language Processing (NLP) as an AI sub-domain and will link to the research at the AI Systems Lab at the University of Manchester.
An introduction to Deep Learning (DL) concepts, such as neural networks, back propagation, activation functions, CNNs, and GANs, along with a simple yet complete neural network.
An introduction to Deep Learning (DL) concepts, such as neural networks, back propagation, activation functions, CNNs, RNNs (if time permits), and the CLT/AUT/fixed-point theorems, along with code samples in Java and TensorFlow.
https://github.com/telecombcn-dl/lectures-all/
These slides review techniques for interpreting the behavior of deep neural networks. The talk reviews basic techniques such as the display of filters and tensors, as well as more advanced ones that try to interpret which part of the input data is responsible for the predictions, or generate data that maximizes the activation of certain neurons.
An introduction to Deep Learning (DL) concepts, starting with a simple yet complete neural network (no frameworks), followed by aspects of deep neural networks, such as back propagation, activation functions, CNNs, and the AUT theorem. Next, a quick introduction to TensorFlow and Tensorboard, and then some code samples with Scala and TensorFlow.
Multimodal Searching and Semantic Spaces: ...or how to find images of Dalmati...Jonathon Hare
Tutorial at the "Reality of the Semantic Gap in Image Retrieval" tutorial at the first international conference on Semantics And digital Media Technology (SAMT 2006). 6th December 2006.
A fast-paced introduction to Deep Learning that starts with a simple yet complete neural network (no frameworks), followed by an overview of activation functions, cost functions, backpropagation, and then a quick dive into CNNs. Next we'll create a neural network using Keras, followed by an introduction to TensorFlow and TensorBoard. For best results, familiarity with basic vectors and matrices, inner (aka "dot") products of vectors, and rudimentary Python is definitely helpful.
This presentation focuses on Deep Learning (DL) concepts, such as neural neworks, backprop, activation functions, and Convolutional Neural Networks, with a short introduction to D3, and followed by a TypeScript-based code sample that replicates the TensorFlow playground. Basic knowledge of matrices is helpful.
This presentation focuses on Deep Learning (DL) concepts, such as neural neworks, backprop, activation functions, and Convolutional Neural Networks, with a short introduction to D3, and followed by a TypeScript-based code sample that replicates the TensorFlow playground. Basic knowledge of matrices is helpful.
A Visual Exploration of Distance, Documents, and DistributionsRebecca Bilbro
Machine learning often requires us to think spatially and make choices about what it means for two instances to be close or far apart. So which is best - Euclidean? Manhattan? Cosine? It all depends! In this talk, we'll explore open source tools and visual diagnostic strategies for picking good distance metrics when doing machine learning on text.
Machine learning often requires us to think spatially and make choices about what it means for two instances to be close or far apart. So which is best - Euclidean? Manhattan? Cosine? It all depends! In this talk, we'll explore open source tools and visual diagnostic strategies for picking good distance metrics when doing machine learning on text.
An introduction to Deep Learning concepts, with a simple yet complete neural network, CNNs, followed by rudimentary concepts of Keras and TensorFlow, and some simple code fragments.
Lecture slides by Mustafa Jarrar at Birzeit University, Palestine.
See the course webpage at: http://jarrar-courses.blogspot.com/2011/09/knowledgeengineering-fall2011.html
and http://www.jarrar.info
and on Youtube:
http://www.youtube.com/watch?v=3_-HGnI6AZ0&list=PLDEA50C29F3D28257
A fast-paced introduction to Deep Learning (DL) concepts, such as neural networks, back propagation, activation functions, CNNs, RNNs (if time permits), and the CLT/AUT/fixed-point theorems, along with a basic code sample in TensorFlow.
During this session you will learn how to manually create a basic neural network that acts as a classifier, and also the segue from linear regression to a neural network.
You'll also learn about GANs (Generative Adversarial Networks) for static images as well as voice, and the former case, their potential impact on self-driving cars.
Similar to A Simple Introduction to Neural Information Retrieval (20)
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Joint Multisided Exposure Fairness for Search and RecommendationBhaskar Mitra
(Slides from my talk at SEA: Search Engines Amsterdam)
Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval.
What’s next for deep learning for Search?Bhaskar Mitra
In this talk, I will share some of my personal reflections on the progress in the field of neural IR and some of the ongoing and future research directions that I am personally excited about. This talk will be informed by my own research in this area as well as my experience both as a developer/organizer of the MS MARCO benchmark and the TREC Deep Learning Track and as an applied researcher previously working on web scale search systems at Bing. My goal in this talk would be to move the conversation beyond neural reranking models towards a richer and bolder vision of search powered by deep learning.
So, You Want to Release a Dataset? Reflections on Benchmark Development, Comm...Bhaskar Mitra
In this talk, I share some of my personal reflections and learnings on benchmark development and community building for making robust scientific progress. This talk is informed by my experience as a developer of the MS MARCO benchmark and as an organizer of the TREC Deep Learning Track. My goal in this talk is to situate the act of releasing a dataset in the context of broader research visions and to draw due attention to considerations of scientific and social outcomes that are invariably salient in the acts of dataset creation and distribution.
Efficient Machine Learning and Machine Learning for Efficiency in Information...Bhaskar Mitra
Emerging machine learning approaches, including deep learning methods, for information retrieval (IR) have recently demonstrated significant improvements in accuracy of relevance estimation at the cost of increasing model complexity and corresponding rise in computational and environmental costs of training and inference. In web search, these costs are further compounded by the necessity to train on large-scale datasets, consume long documents as inputs, and retrieve relevant documents from web-scale collections within milliseconds in response to high volume query traffic. A typical playbook for developing deep learning models for IR involves largely ignoring efficiency concerns during model development and then later scaling these methods by either finding faster approximations of the same models or employing heuristics to reduce the input space over which these models operate. Domain knowledge about the specific IR task and deeper understanding of system design and data structures in whose context these models are deployed can significantly help with not only model simplification but also to inform data-structure specific machine learning model design. Alternatively, predictive machine learning can also be employed specifically to improve efficiency in large scale IR settings. In this talk, I will cover several case studies for both improving efficiency of machine learning models for IR as well as direct application of machine learning to improve retrieval efficiency, and conclude with a brief discussion on potential future directions for efficiency-sensitive benchmarking of machine learning models for IR.
Multisided Exposure Fairness for Search and RecommendationBhaskar Mitra
Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval. The development of expected exposure based metrics also opens up new opportunities and challenges for model optimization. We demonstrate how stochastic ranking policies can be optimized towards target expected exposure and highlight the trade-offs that may exist in optimizing for different fairness dimensions.
Neural Information Retrieval: In search of meaningful progressBhaskar Mitra
The emergence of deep learning based methods for search poses several challenges and opportunities not just for modeling, but also for benchmarking and measuring progress in the field. Some of these challenges are new, while others have evolved from existing challenges in IR benchmarking exacerbated by the scale at which deep learning models operate. Evaluation efforts such as the TREC Deep Learning track and the MS MARCO public leaderboard are intended to encourage research and track our progress, addressing big questions in our field. The goal is not simply to identify which run is "best" but to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This entails a wider conversation in the IR community about what constitutes meaningful progress, how benchmark design can encourage or discourage certain outcomes, and about the validity of our findings. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track--and reflect on the state of the field and the road ahead.
Conformer-Kernel with Query Term Independence @ TREC 2020 Deep Learning TrackBhaskar Mitra
We benchmark Conformer-Kernel models under the strict blind evaluation setting of the TREC 2020 Deep Learning track. In particular, we study the impact of incorporating: (i) Explicit term matching to complement matching based on learned representations (i.e., the “Duet principle”), (ii) query term independence (i.e., the “QTI assumption”) to scale the model to the full retrieval setting, and (iii) the ORCAS click data as an additional document description field. We find evidence which supports that all three aforementioned strategies can lead to improved retrieval quality.
Lecture slides presented at Northeastern University (December, 2020).
Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.
This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019. For the document retrieval task, we adapt the Duet model to ingest a "multiple field" view of documents—we refer to the new architecture as Duet with Multiple Fields (DuetMF). A second submission combines the DuetMF model with other neural and traditional relevance estimators in a learning-to-rank framework and achieves improved performance over the DuetMF baseline. For the passage retrieval task, we submit a single run based on an ensemble of eight Duet models.
Benchmarking for Neural Information Retrieval: MS MARCO, TREC, and BeyondBhaskar Mitra
The emergence of deep learning-based methods for information retrieval (IR) poses several challenges and opportunities for benchmarking. Some of these are new, while others have evolved from existing challenges in IR exacerbated by the scale at which deep learning models operate. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track, and reflect on the road ahead.
Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as items to be recommended, in response to user's need. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this tutorial will be on the fundamentals of neural networks and their applications to learning to rank.
Tutorial presented at ACM SIGIR/SIGKDD Africa Summer School on Machine Learning for Data Mining and Search (AFIRM 2020) conference in Cape Town, South Africa.
A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs.
We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features.
Adversarial and reinforcement learning-based approaches to information retrievalBhaskar Mitra
Traditionally, machine learning based approaches to information retrieval have taken the form of supervised learning-to-rank models. Recently, other machine learning approaches—such as adversarial learning and reinforcement learning—have started to find interesting applications in retrieval systems. At Bing, we have been exploring some of these methods in the context of web search. In this talk, I will share couple of our recent work in this area that we presented at SIGIR 2018.
5 Lessons Learned from Designing Neural Models for Information RetrievalBhaskar Mitra
Slides from my keynote talk at the Recherche d'Information SEmantique (RISE) workshop at CORIA-TALN 2018 conference in Rennes, France.
(Abstract)
Neural Information Retrieval (or neural IR) is the application of shallow or deep neural networks to IR tasks. Unlike classical IR models, these machine learning (ML) based approaches are data-hungry, requiring large scale training data before they can be deployed. Traditional learning to rank models employ supervised ML techniques—including neural networks—over hand-crafted IR features. By contrast, more recently proposed neural models learn representations of language from raw text that can bridge the gap between the query and the document vocabulary.
Neural IR is an emerging field and research publications in the area has been increasing in recent years. While the community explores new architectures and training regimes, a new set of challenges, opportunities, and design principles are emerging in the context of these new IR models. In this talk, I will share five lessons learned from my personal research in the area of neural IR. I will present a framework for discussing different unsupervised approaches to learning latent representations of text. I will cover several challenges to learning effective text representations for IR and discuss how latent space models should be combined with observed feature spaces for better retrieval performance. Finally, I will conclude with a few case studies that demonstrates the application of neural approaches to IR that go beyond text matching.
Neural Models for Information RetrievalBhaskar Mitra
In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing (NLP) tasks, such as language modelling and machine translation. This suggests that neural models may also yield significant performance improvements on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using semantic rather than lexical matching. IR tasks, however, are fundamentally different from NLP tasks leading to new challenges and opportunities for existing neural representation learning approaches for text.
In this talk, I will present my recent work on neural IR models. We begin with a discussion on learning good representations of text for retrieval. I will present visual intuitions about how different embeddings spaces capture different relationships between items, and their usefulness to different types of IR tasks. The second part of this talk is focused on the applications of deep neural architectures to the document ranking task.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Welocme to ViralQR, your best QR code generator.ViralQR
Welcome to ViralQR, your best QR code generator available on the market!
At ViralQR, we design static and dynamic QR codes. Our mission is to make business operations easier and customer engagement more powerful through the use of QR technology. Be it a small-scale business or a huge enterprise, our easy-to-use platform provides multiple choices that can be tailored according to your company's branding and marketing strategies.
Our Vision
We are here to make the process of creating QR codes easy and smooth, thus enhancing customer interaction and making business more fluid. We very strongly believe in the ability of QR codes to change the world for businesses in their interaction with customers and are set on making that technology accessible and usable far and wide.
Our Achievements
Ever since its inception, we have successfully served many clients by offering QR codes in their marketing, service delivery, and collection of feedback across various industries. Our platform has been recognized for its ease of use and amazing features, which helped a business to make QR codes.
Our Services
At ViralQR, here is a comprehensive suite of services that caters to your very needs:
Static QR Codes: Create free static QR codes. These QR codes are able to store significant information such as URLs, vCards, plain text, emails and SMS, Wi-Fi credentials, and Bitcoin addresses.
Dynamic QR codes: These also have all the advanced features but are subscription-based. They can directly link to PDF files, images, micro-landing pages, social accounts, review forms, business pages, and applications. In addition, they can be branded with CTAs, frames, patterns, colors, and logos to enhance your branding.
Pricing and Packages
Additionally, there is a 14-day free offer to ViralQR, which is an exceptional opportunity for new users to take a feel of this platform. One can easily subscribe from there and experience the full dynamic of using QR codes. The subscription plans are not only meant for business; they are priced very flexibly so that literally every business could afford to benefit from our service.
Why choose us?
ViralQR will provide services for marketing, advertising, catering, retail, and the like. The QR codes can be posted on fliers, packaging, merchandise, and banners, as well as to substitute for cash and cards in a restaurant or coffee shop. With QR codes integrated into your business, improve customer engagement and streamline operations.
Comprehensive Analytics
Subscribers of ViralQR receive detailed analytics and tracking tools in light of having a view of the core values of QR code performance. Our analytics dashboard shows aggregate views and unique views, as well as detailed information about each impression, including time, device, browser, and estimated location by city and country.
So, thank you for choosing ViralQR; we have an offer of nothing but the best in terms of QR code services to meet business diversity!
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
How world-class product teams are winning in the AI era by CEO and Founder, P...
A Simple Introduction to Neural Information Retrieval
1. A Simple Introduction to
NEURAL INFORMATION RETRIEVAL
Guest Lecturer
BHASKAR MITRA
Principal Applied Scientist
Microsoft AI and Research
Research Student
Dept. of Computer Science
University College London
March, 2018
2. “
GROUND
RULES
• Let’s make this interactive
• Please ask lots of questions
• Discussions don’t end in this room
The value of science is not to
make things complex, but to find
the inherent simplicity.
-Frank Seide
@UnderdogGeek bmitra@microsoft.com
7. INFORMATION RETRIEVAL (IR)
User has an information need
There exists a collection of information
resources
IR is the activity of retrieving the information
resources relevant to the information need
8. EXAMPLE OF AN IR TASK
(WEB SEARCH)
User expresses information need as a short
textual query
The search engine retrieves top relevant web
documents as information resources
We will use web search as the main example of
an IR task in the rest of this lecture
query
Information
need
retrieval system indexes a
document corpus
results ranking (document list)
Relevance
(documents satisfy
information need)
9. CHALLENGES IN IR [SLIDE 1/3]
• Vocabulary mismatch
Q: How many people live in Sydney?
Sydney’s population is 4.9 million
[relevant, but missing ‘people’ and ‘live’]
Hundreds of people queueing for live music in Sydney
[irrelevant, and matching ‘people’ and ‘live’]
• Need to interpret words based on context (e.g., temporal)
Today Recent In older (1990s)
TREC data
query:
“uk prime minister”
Vocab mismatch:
• Worse for short texts
• Still an issue for long texts
10. Need to learn Q-D relationship
that generalizes to the tail
• Unseen Q
• Unseen D
• Unseen information needs
• Unseen vocabulary
CHALLENGES IN IR [SLIDE 2/3]
11. Query and document
vary in length
• Models must handle
variable length input
• Relevant docs have
irrelevant sections
CHALLENGES IN IR [SLIDE 3/3]
12. NEURAL
NETWORKS
Chains of parameterized linear transforms (e.g., multiply weight,
add bias) followed by non-linear functions (σ)
Popular choices for σ:
Parameters trained using backpropagation
E2E training over millions of samples in batched mode
Many choices of architecture and hyper-parameters
Non-linearity
Input
Linear transform
Non-linearity
Linear transform
Predicted output
forwardpass
backward
pass
Expected output
loss
Tanh ReLU
13. can’t separate using a linear model!
Input features
Label
surface kerberos book library
1 0 1 0 ✓
1 1 0 0 ✗
0 1 0 1 ✓
0 0 1 1 ✗
library booksurface kerberos
+0.5
+0.5
-1
-1 -1
-1
+1 +1
+0.5
+0.5
H1 H2
But let’s consider a tiny neural
network with one hidden layer…
VISUAL
MOTIVATION FOR
HIDDEN UNITS
Consider the following “toy” challenge for
classifying tech queries:
Vocab: {surface, kerberos, book, library}
Labels:
“surface book”, “kerberos library” ✓
“kerberos surface”, “library book” ✗
14. VISUAL
MOTIVATION FOR
HIDDEN UNITS
Or more succinctly…
Input features Hidden layer
Label
surface kerberos book library H1 H2
1 0 1 0 1 0 ✓
1 1 0 0 0 0 ✗
0 1 0 1 0 1 ✓
0 0 1 1 0 0 ✗
library booksurface kerberos
+0.5
+0.5
-1
-1 -1
-1
+1 +1
+0.5
+0.5
H1 H2
But let’s consider a tiny neural
network with one hidden layer…
can separate using a linear model!
Consider the following “toy” challenge for
classifying tech queries:
Vocab: {surface, kerberos, book, library}
Labels:
“surface book”, “kerberos library” ✓
“kerberos surface”, “library book” ✗
15. WHY ADDING DEPTH HELPS
Deeper networks can split the input space
in many (non-independent) linear regions
than shallow networks
Montúfar, Pascanu, Cho and Bengio. On the number of linear regions of deep neural networks NIPS 2014
18. THE SOFTMAX FUNCTION
In neural classification models, the softmax function is popularly used to
normalize the neural network output scores across all the classes
19. CROSS ENTROPY
The cross entropy between two probability
distributions 𝑝 and 𝑞 over a discrete set of
events is given by,
If 𝑝 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 = 1and 𝑝𝑖 = 0 for all
other values of 𝑖 then,
20. CROSS ENTROPY WITH
SOFTMAX LOSS
Cross entropy with softmax is a popular loss
function for classification
23. TYPES OF VECTOR REPRESENTATIONS
Local (or one-hot) representation
Every term in vocabulary T is represented by a
binary vector of length |T|, where one position
in the vector is set to one and the rest to zero
Distributed representation
Every term in vocabulary T is represented by a
real-valued vector of length k. The vector can
be sparse or dense. The vector dimensions may
be observed (e.g., hand-crafted features) or
latent (e.g., embedding dimensions).
24. Hinton, Geoffrey E. Distributed representations. Technical Report CMU-CS-84-157, 1984
25. OBSERVED (OR EXPLICIT)
DISTRIBUTED
REPRESENTATIONS
The choice of features is a key consideration
The distributional hypothesis states that
terms that are used (or occur) in similar
context tend to be semantically similar
[Harris, 1954]
Firth [1957] famously purported this idea of
distributional semantics by stating “a word
is characterized by the company it keeps”.
Zellig S Harris. Distributional structure. Word, 10(2-3):146–162, 1954.
Firth, J. R. (1957). A synopsis of linguistic theory 1930–1955. In Studies in Linguistic Analysis, p. 11. Blackwell, Oxford.
Turney and Pantel. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research 2010.
26. MINOR NOTE: SPOT THE DIFFERENCE!
DISTRIBUTED REPRESENTATION
Vector representations of items as
combinations of different features
or dimensions (as opposed to
one-hot)
DISTRIBUTIONAL SEMANTICS
Linguistic items with similar
distributions (e.g. context words)
have similar meanings
http://www.marekrei.com/blog/26-things-i-learned-in-the-deep-learning-summer-school/
27. EXAMPLE: TERM-CONTEXT VECTOR SPACE
T: vocabulary, C: set of contexts, S: sparse matrix |T| x |C|
(PPMI: Positive Pointwise Mutual Information)
C0 c1 c2 … cj … c|C|
t0
t1
t2
…
ti Sij
…
t|T|
Turney and Pantel. From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research 2010
t
t
t
t
t
t t
t
t
28. EXAMPLE: SALTON’S VECTOR SPACE
D: collection, T: vocabulary, S: sparse matrix |D| x |T|
t0 t1 t2 … tj … t|T|
d0
d1
d2
…
di Sij
…
d|D|
S
G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, Nov. 1975
idf
29. NOTIONS OF
SIMILARITY
Two terms are similar if their feature
vectors are close
But different feature spaces may capture
different notions of similarity
Is Seattle more similar to…
Sydney (similar type)
or
Seahawks (similar topic)
Depends on your choice of features
30. NOTIONS OF
SIMILARITY
Consider the following toy corpus…
Now consider the different vector
representations of terms you can derive
from this corpus and how the items that
are similar differ in these vector spaces
34. NOTIONS OF
SIMILARITY
Consider the following toy corpus…
Now consider the different vector
representations of terms you can derive
from this corpus and how the items that
are similar differ in these vector spaces
35. RETRIEVAL USING VECTOR REPRESENTATIONS
Map both query and candidate documents
into the same vector space
Retrieve documents closest to the query
e.g., using Salton’s vector space model
Where, 𝑣 𝑞 and 𝑣 𝑑 are vectors of TF-IDF
scores over all terms in the vocabulary
G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, Nov. 1975
𝑠𝑖𝑚 𝑞, 𝑑 =
𝑣 𝑞. 𝑣 𝑑
𝑣 𝑞 . 𝑣 𝑑
36. REGULARITIES IN OBSERVED FEATURE SPACES
Some feature spaces capture
interesting linguistic regularities
e.g., simple vector algebra in the
term-neighboring term space may
be useful for word analogy tasks
Levy, Goldberg and Ramat-Gan. Linguistic Regularities in Sparse and Explicit Word Representations. CoNLL 2014
37. EMBEDDINGS
An embedding is a representation of items
in a new space such that the properties of,
and the relationships between, the items are
preserved from the original representation.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016.
39. EMBEDDINGS
Compared to observed feature spaces:
• Embeddings typically have fewer dimensions
• The space may have more disentangled principle
components
• The dimensions may be less interpretable
• The latent representations may generalize better
41. LET’S TAKE AN IR
EXAMPLE
In Salton’s vector space, both
these passages are equidistant
from the query “Albuquerque”
A latent feature representation
may put the first passage closer
to the query because of terms
like “population” and “area”
Passage about Albuquerque
Passage not about Albuquerque
Query: “Albuquerque”
42. HOW TO LEARN TERM EMBEDDINGS?
Multiple approaches have been
proposed for learning embeddings
from <term, context, count> data
Popular approaches include matrix
factorization or stochastic gradient
descent (SGD)
C0 c1 c2 … cj … c|C|
t0
t1
t2
…
ti Xij
…
t|T|
43. LATENT SEMANTIC ANALYSIS (LSA)
Perform SVD on X to obtain
its low-rank approximation
Involves finding a solution
to X = 𝑈Σ𝑉T
The embedding for the ith
term is given by Σk 𝑡𝑖
Scott C. Deerwester, Susan T Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. Indexing by latent semantic analysis. JASIS, 1990.
44. Scott C. Deerwester, Susan T Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. Indexing by latent semantic analysis. JASIS, 1990.
LATENT SEMANTIC ANALYSIS (LSA)
45. WORD2VEC
Goal: simple (shallow) neural model
learning from billion words scale corpus
Predict middle word from neighbors
within a fixed size context window
Two different architectures:
1. Skip-gram
2. CBOW
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
47. THE SKIP-GRAM LOSS
S is the set of all windows over the training text
c is the number of neighbours we need to predict on either side of the term 𝑡𝑖
Full softmax is computationally impractical - hierarchical softmax or negative sampling is employed instead
49. THE CBOW LOSS
Note: from every window of text skip-gram generates 2 x c training samples
whereas CBOW generates one – that’s why CBOW trains faster than skip-gram
50. WORD ANALOGIES
WITH WORD2VEC
W2v is popular for word analogy tasks
But remember the same relationships also
exist in the observed feature space, as we
saw earlier
51. Let 𝑥𝑖𝑗 be the frequency of the pair 𝑡𝑖, 𝑡𝑗 in
the training data, then
t0 t1 t2 … tj … t|T|
t0
t1
t2
…
ti Xij
…
t|T|
A MATRIX INTERPRETATION OF WORD2VEC
cross-entropy error
actual co-occurrence
probability
predicted co-occurrence
probability
52. Replace the cross-entropy error
with a squared-error and apply a
saturation function f(…) over 𝑥𝑖𝑗
GLOVE
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
ℒ 𝐺𝑙𝑜𝑉𝑒 = −
𝑖=1
|𝑇|
𝑗=1
|𝑇|
𝑓 𝑥𝑖,𝑗 𝑙𝑜𝑔 𝑥𝑖,𝑗 − 𝑤𝑖
⊺
𝑤𝑗
2
squared error
predicted co-occurrence
probability
saturation function
actual co-occurrence
probability`
53. PARAGRAPH2VEC
W2v style model where context is
document, not neighboring term
Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, 2014.
54. RECAP: HOW TO LEARN TERM EMBEDDINGS?
Learn from <term, context, count> data
Choice of context (e.g., neighboring term or container document) defines what relationship you are
modeling
Choice of learning algorithm (e.g., matrix factorization or SGD) defines how well you
model the relationship
Choice of context and learning algorithm are independent – you can use matrix
factorization with neighboring term context, or a w2v-style neural network with
document context (e.g., paragraph2vec)
58. RECAP: RETRIEVAL USING VECTOR REPRESENTATIONS
Generate vector
representation of query
Generate vector
representation of document
Estimate relevance from q-d
vectors
59. Compare query and document
directly in the embedding space
POPULAR APPROACHES TO INCORPORATING
TERM EMBEDDINGS FOR MATCHING
Use embeddings to generate
suitable query expansions
estimate relevance estimate relevance
60. E.g.,
Generalized Language Model [Ganguly et
al., 2015]
Neural Translation Language Model
[Zuccon et al., 2015]
Average term embeddings [Le and Mikolov,
2014, Nalisnick et al., 2016, Zamani and Croft, 2016,
and others]
Word mover’s distance [Kusner et al., 2015,
Guo et al., 2016]
Compare query and document
directly in the embedding space
estimate relevance
61. GENERALIZED LANGUAGE MODEL
Traditional language modeling based IR approach may estimate q-d relevance as follows,
where, 𝑝 𝑡 𝑞|𝑑 is the
probability of generating
term 𝑡 𝑞 from document 𝑑
62. GENERALIZED LANGUAGE MODEL
Traditional language modeling based IR approach may estimate q-d relevance as follows,
𝑝 𝑡 𝑞|𝑑 and 𝑝 𝑡 𝑞|𝐷 are the
probabilities of randomly
sampling term 𝑡 𝑞 from
document 𝑑 and the full
collection 𝐷, respectively
𝑝 𝑡 𝑞|𝐷 has a smoothing effect
on the 𝑝 𝑡 𝑞|𝑑 estimation
63. GENERALIZED LANGUAGE MODEL
GLM includes additional smoothing based on term similarity in the embedding space
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth JF Jones. Word embedding based generalized language model for information retrieval. In SIGIR, 2015.
64. GENERALIZED LANGUAGE MODEL
GLM includes additional smoothing based on term similarity in the embedding space
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth JF Jones. Word embedding based generalized language model for information retrieval. In SIGIR, 2015.
65. GENERALIZED LANGUAGE MODEL
GLM includes additional smoothing based on term similarity in the embedding space
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth JF Jones. Word embedding based generalized language model for information retrieval. In SIGIR, 2015.
66. GENERALIZED LANGUAGE MODEL
GLM includes additional smoothing based on term similarity in the embedding space
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth JF Jones. Word embedding based generalized language model for information retrieval. In SIGIR, 2015.
Probability of generating the
term from the document
based on similarity in the
embedding space
Probability of generating the term
from the full collection based on
similarity in the embedding space
67. NEURAL TRANSLATION LANGUAGE MODEL
Translation Language Model:
Neural Translation Language Model:
TLM estimates 𝑝 𝑡 𝑞|𝑡 𝑑 from q-d paired
data similar to statistical machine translation
NTLM uses term-term similarity in the
embedding space to estimate 𝑝 𝑡 𝑞|𝑡 𝑑
Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In ADCS, 2015.
68. AVERAGE TERM EMBEDDINGS
Q-D relevance
estimated by
computing cosine
similarity between
centroid of q and d
term embeddings
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
69. WORD MOVER’S DISTANCE
Based on the Earth Mover’s Distance (EMD)
[Rubner et al., 1998]
Originally proposed by Wan et al. [2005, 2007],
but used WordNet and topic categories
Kusner et al. [2015] incorporated term
embeddings
Adapted for q-d matching by Guo et al. [2016]
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In CV, 1998.
Xiaojun Wan and Yuxin Peng. The earth mover’s distance as a semantic measure for document similarity. In CIKM, 2005.
Xiaojun Wan. A novel document similarity measure based on earth mover’s distance. Information Sciences, 2007.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In ICML, 2015.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. Semantic matching by non-linear word transportation for information retrieval. In CIKM, 2016.
70.
71. CHOICE OF TERM EMBEDDINGS
FOR DOCUMENT RANKING
RECAP: for the query “Albuquerque” the relevant document
may contain terms like “population” and “area”
Documents about “Santa Fe” not relevant for this query
“Albuquerque” ↔ “population” (Topically similar) ✓
“Albuquerque” ↔ “Santa Fe” (Typically similar) ✗
Standard LSA and para2vec capture topical similarity,
whereas w2v and GloVe capture a mix of both Top/Typ-ical
Passage about Albuquerque
Passage not about Albuquerque
Query: “Albuquerque”
72. DUAL EMBEDDING SPACE MODEL
What if I told you that everyone
using word2vec is throwing half
the model away?
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
73. DUAL EMBEDDING SPACE MODEL
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
IN-OUT captures a more
Topical notion of similarity
than IN-IN and OUT-OUT
Effect is exaggerated when
embeddings are trained on
short text (e.g., queries)
74. DUAL EMBEDDING SPACE MODEL
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
Average term embeddings model, but use IN embeddings for
query terms and OUT embeddings for document terms
75. DUAL EMBEDDING SPACE MODEL
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
76. CHALLENGE
IN+OUT Embeddings for 2.7M words
trained on 600M+ Bing queries
http://bit.ly/DataDESM
Can you come up with
interesting t-SNE visualizations
that demonstrates the
differences between IN-IN and
IN-OUT term similarities? Download
77. A TALE OF TWO QUERIES
“PEKAROVIC LAND COMPANY”
Hard to learn good representation for
the rare term pekarovic
But easy to estimate relevance based
on count of exact term matches of
pekarovic in the document
“WHAT CHANNEL ARE THE
SEAHAWKS ON TODAY”
Target document likely contains ESPN
or sky sports instead of channel
The terms ESPN and channel can be
compared in a term embedding space
Matching in the term space is necessary to handle rare terms. Matching in the
latent embedding space can provide additional evidence of relevance. Best
performance is often achieved by combining matching in both vector spaces.
78. QUERY: CAMBRIDGE (Font size is a function of term-term cosine similarity)
Besides the term “Cambridge”, other related terms (e.g., “university”, “town”,
“population”, and “England”) contribute to the relevance of the passage
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
79. QUERY: CAMBRIDGE (Font size is a function of term-term cosine similarity)
However, the same terms may also make a passage about Oxford look somewhat
relevant to the query “Cambridge”
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
80. QUERY: CAMBRIDGE (Font size is a function of term-term cosine similarity)
A passage about giraffes, however, obviously looks non-relevant in the
embedding space…
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
81. QUERY: CAMBRIDGE (Font size is a function of term-term cosine similarity)
But the embedding based matching model is more robust to the same passage when “giraffe”
is replaced by “Cambridge”—a trick that would fool exact term based IR models. In a sense,
the embedding based model ranks this passage low because Cambridge is not "an African
even-toed ungulate mammal“.
Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137, 2016.
82. E.g.,
Generalized Language Model [Ganguly et
al., 2015]
Neural Translation Language Model
[Zuccon et al., 2015]
Average term embeddings [Le and Mikolov,
2014, Nalisnick et al., 2016, Zamani and Croft, 2016,
and others]
Word mover’s distance []
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth JF Jones. Word embedding based generalized language model for information retrieval. In SIGIR, 2015.
Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In ADCS, 2015.
Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, 2014.
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In WWW, 2016.
Hamed Zamani and W Bruce Croft. Estimating embedding vectors for queries. In ICTIR, 2016.
Compare query and document
directly in the embedding space
estimate relevance
83. Compare query and document
directly in the embedding space
POPULAR APPROACHES TO INCORPORATING
TERM EMBEDDINGS FOR MATCHING
Use embeddings to generate
suitable query expansions
estimate relevance estimate relevance
84. QUERY EXPANSION USING
TERM EMBEDDINGS
Use embeddings to generate
suitable query expansions
estimate relevance
Find good expansion terms based on nearness
in the embedding space
Better retrieval performance when combined
with pseudo-relevance feedback (PRF) [Zamani
and Croft, 2016] and if we learn query specific term
embeddings [Diaz et al., 2016]
Fernando Diaz, Bhaskar Mitra, and Nick Craswell. Query expansion with locally-trained word embeddings. In ACL, 2016.
Dwaipayan Roy, Debjyoti Paul, Mandar Mitra, and Utpal Garain. Using word embeddings for automatic query expansion. arXiv preprint arXiv:1606.07608, 2016.
Hamed Zamani and W Bruce Croft. Embedding-based query language models. In ICTIR, 2016.
87. LEARNING TO
RANK (LTR)
L2R models represent a rankable item—e.g.,
a document—given some context—e.g., a
user-issued query—as a numerical vector
𝑥 ∈ ℝ 𝑛
The ranking model 𝑓: 𝑥 → ℝ is trained to
map the vector to a real-valued score such
that relevant items are scored higher.
”... the task to automatically construct
a ranking model using training data,
such that the model can sort new
objects according to their degrees of
relevance, preference, or importance.”
- Liu [2009]
Tie-Yan Liu. Learning to rank for information retrieval. Foundation and Trends in Information Retrieval, 2009.
88. APPROACHES
Pointwise approach
Relevance label 𝑦 𝑞,𝑑 is a number—derived from binary or graded human
judgments or implicit user feedback (e.g., CTR). Typically, a regression or
classification model is trained to predict 𝑦 𝑞,𝑑 given 𝑥 𝑞,𝑑.
Pairwise approach
Pairwise preference between documents for a query (𝑑𝑖 ≻ 𝑑𝑗 w.r.t. 𝑞) as
label. Reduces to binary classification to predict more relevant document.
Listwise approach
Directly optimize for rank-based metric, such as NDCG—difficult because
these metrics are often not differentiable w.r.t. model parameters.
Liu [2009] categorizes
different LTR approaches
based on training
objectives:
Tie-Yan Liu. Learning to rank for information retrieval. Foundation and Trends in Information Retrieval, 2009.
89. FEATURES
They can often be categorized as:
Query-independent or static features
e.g., incoming link count and document length
Query-dependent or dynamic features
e.g., BM25
Query-level features
e.g., query length
Traditional L2R models
employ hand-crafted features
that encode IR insights
90. POINTWISE
OBJECTIVES
Regression loss
Given 𝑞, 𝑑 predict the value of 𝑦 𝑞,𝑑
e.g., square loss for binary or categorical
labels,
where, 𝑦 𝑞,𝑑 is the one-hot representation
[Fuhr, 1989] or the actual value [Cossock and
Zhang, 2006] of the label
Norbert Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM TOIS, 1989.
David Cossock and Tong Zhang. Subset ranking using regression. In COLT, 2006.
labels
prediction
0 1 1
91. POINTWISE
OBJECTIVES
Classification loss
Given 𝑞, 𝑑 predict the class 𝑦 𝑞,𝑑
e.g., cross-entropy with softmax over
categorical labels 𝑌 [Li et al., 2008],
where, 𝑠 𝑦 𝑞,𝑑
is the model’s score for label 𝑦 𝑞,𝑑
labels
prediction
0 1
Ping Li, Qiang Wu, and Christopher J Burges. Mcrank: Learning to rank using multiple classification and gradient boosting. In NIPS, 2008.
92. PAIRWISE
OBJECTIVES Pairwise loss generally has the following form [Chen et al., 2009],
where, 𝜙 can be,
• Hinge function 𝜙 𝑧 = 𝑚𝑎𝑥 0, 1 − 𝑧 [Herbrich et al., 2000]
• Exponential function 𝜙 𝑧 = 𝑒−𝑧
[Freund et al., 2003]
• Logistic function 𝜙 𝑧 = 𝑙𝑜𝑔 1 + 𝑒−𝑧
[Burges et al., 2005]
• Others…
Pairwise loss minimizes the average number of
inversions in ranking—i.e., 𝑑𝑖 ≻ 𝑑𝑗 w.r.t. 𝑞 but 𝑑𝑗 is
ranked higher than 𝑑𝑖
Given 𝑞, 𝑑𝑖, 𝑑𝑗 , predict the more relevant
document
For 𝑞, 𝑑𝑖 and 𝑞, 𝑑𝑗 ,
Feature vectors: 𝑥𝑖 and 𝑥𝑗
Model scores: 𝑠𝑖 = 𝑓 𝑥𝑖 and 𝑠𝑗 = 𝑓 𝑥𝑗
Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. In NIPS, 2009.
Ralf Herbrich, Thore Graepel, and Klaus Obermayer. Large margin rank boundaries for ordinal regression. 2000.
Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. In JMLR, 2003.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In ICML, 2005.
93. PAIRWISE
OBJECTIVES
RankNet loss
Pairwise loss function proposed by Burges et al. [2005]—an industry favourite
[Burges, 2015]
Predicted probabilities: 𝑝𝑖𝑗 = 𝑝 𝑠𝑖 > 𝑠𝑗 ≡
𝑒 𝛾.𝑠 𝑖
𝑒 𝛾.𝑠 𝑖 +𝑒
𝛾.𝑠 𝑗
=
1
1+𝑒
−𝛾. 𝑠 𝑖−𝑠 𝑗
Desired probabilities: 𝑝𝑖𝑗 = 1 and 𝑝𝑗𝑖 = 0
Computing cross-entropy between 𝑝 and 𝑝
ℒ 𝑅𝑎𝑛𝑘𝑁𝑒𝑡 = − 𝑝𝑖𝑗. 𝑙𝑜𝑔 𝑝𝑖𝑗 − 𝑝𝑗𝑖. 𝑙𝑜𝑔 𝑝𝑗𝑖 = −𝑙𝑜𝑔 𝑝𝑖𝑗 = 𝑙𝑜𝑔 1 + 𝑒−𝛾. 𝑠 𝑖−𝑠 𝑗
pairwise
preference
score
0 1
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In ICML, 2005.
Chris Burges. RankNet: A ranking retrospective. https://www.microsoft.com/en-us/research/blog/ranknet-a-ranking-retrospective/. 2015.
94. A GENERALIZED CROSS-ENTROPY LOSS
An alternative loss function assumes a single relevant document 𝑑+ and compares it
against the full collection 𝐷
Predicted probabilities: p 𝑑+|𝑞 =
𝑒 𝛾.𝑠 𝑞,𝑑+
𝑑∈𝐷 𝑒 𝛾.𝑠 𝑞,𝑑
The cross-entropy loss is then given by,
ℒ 𝐶𝐸 𝑞, 𝑑+, 𝐷 = −𝑙𝑜𝑔 p 𝑑+|𝑞 = −𝑙𝑜𝑔
𝑒 𝛾.𝑠 𝑞,𝑑+
𝑑∈𝐷 𝑒 𝛾.𝑠 𝑞,𝑑
Computing the softmax over the full collection is prohibitively expensive—LTR models
typically consider few negative candidates [Huang et al., 2013, Shen et al., 2014, Mitra et al., 2017]
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In CIKM, 2013.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014.
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
95. Blue: relevant Gray: non-relevant
NDCG and ERR higher for left but pairwise
errors less for right
Due to strong position-based discounting
in IR measures, errors at higher ranks are
much more problematic than at lower ranks
But listwise metrics are non-continuous and
non-differentiable
LISTWISE
OBJECTIVES
Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 2010.
[Burges, 2010]
96. LISTWISE
OBJECTIVES
Burges et al. [2006] make two observations:
1. To train a model we don’t need the costs
themselves, only the gradients (of the
costs w.r.t model scores)
2. It is desired that the gradient be bigger
for pairs of documents that produces a
bigger impact in NDCG by swapping
positions
Christopher JC Burges, Robert Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In NIPS, 2006.
LambdaRank loss
Multiply actual gradients with the change in
NDCG by swapping the rank positions of the
two documents
97. LISTWISE
OBJECTIVES
According to the Luce model [Luce, 2005], given
four items 𝑑1, 𝑑2, 𝑑3, 𝑑4 the probability of
observing a particular rank-order, say
𝑑2, 𝑑1, 𝑑4, 𝑑3 , is given by:
where, 𝜋 is a particular permutation and 𝜙 is a
transformation (e.g., linear, exponential, or
sigmoid) over the score 𝑠𝑖 corresponding to
item 𝑑𝑖
R Duncan Luce. Individual choice behavior. 1959.
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In ICML, 2007.
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In ICML, 2008.
ListNet loss
Cao et al. [2007] propose to compute the
probability distribution over all possible
permutations based on model score and
ground-truth labels. The loss is then given by
the K-L divergence between these two
distributions.
This is computationally very costly, computing
permutations of only the top-K items makes it
slightly less prohibitive.
ListMLE loss
Xia et al. [2008] propose to compute the
probability of the ideal permutation based on
the ground truth. However, with categorical
labels more than one permutation is possible.
100. So far we have discussed:
1. Unsupervised learning of text representations using shallow
neural networks and employing them in traditional IR models
2. Supervised learning of neural models (shallow or deep) for
the ranking task using hand-crafted features
In the last session, we will discuss:
Supervised training of deep neural networks—with richer
structures—for IR tasks based on raw representations of query
and document text
106. SHIFT-INVARIANT
NEURAL OPERATIONS
Detecting a pattern in one part of the input space is similar to
detecting it in another
Leverage redundancy by moving a window over the whole
input space and then aggregate
On each instance of the window a kernel—also known as a
filter or a cell—is applied
Different aggregation strategies lead to different architectures
107. CONVOLUTION
Move the window over the input space each time applying
the same cell over the window
A typical cell operation can be,
ℎ = 𝜎 𝑊𝑋 + 𝑏
Full Input [words x in_channels]
Cell Input [window x in_channels]
Cell Output [1 x out_channels]
Full Output [1 + (words – window) / stride x out_channels]
108. POOLING
Move the window over the input space each time applying an
aggregate function over each dimension in within the window
ℎ𝑗 = 𝑚𝑎𝑥𝑖∈𝑤𝑖𝑛 𝑋𝑖,𝑗 𝑜𝑟 ℎ𝑗 = 𝑎𝑣𝑔𝑖∈𝑤𝑖𝑛 𝑋𝑖,𝑗
Full Input [words x channels]
Cell Input [window x channels]
Cell Output [1 x channels]
Full Output [1 + (words – window) / stride x channels]
max -pooling average -pooling
109. CONVOLUTION W/
GLOBAL POOLING
Stacking a global pooling layer on top of a convolutional layer
is a common strategy for generating a fixed length
embedding for a variable length text
Full Input [words x in_channels]
Full Output [1 x out_channels]
110. RECURRENT NEURAL
NETWORK
Similar to a convolution layer but additional dependency on
previous hidden state
A simple cell operation shown below but others like LSTM and
GRUs are more popular in practice,
ℎ𝑖 = 𝜎 𝑊𝑋𝑖 + 𝑈ℎ𝑖−1 + 𝑏
Full Input [words x in_channels]
Cell Input [window x in_channels] + [1 x out_channels]
Cell Output [1 x out_channels]
Full Output [1 x out_channels]
111. RECURSIVE NN OR
TREE-RNN
Shared weights among all the levels of the tree
Cell can be an LSTM or as simple as
ℎ = 𝜎 𝑊𝑋 + 𝑏
Full Input [words x channels]
Cell Input [window x channels]
Cell Output [1 x channels]
Full Output [1 x channels]
112. AUTOENCODER
Unsupervised models trained to minimize
reconstruction errors
Information Bottleneck method (Tishby et al., 1999)
The bottleneck layer 𝑥 captures “minimal sufficient
statistics” of 𝑣 and is a compressed representation of
the same
113. SIAMESE NETWORK
Supervised model trained on 𝑞, 𝑑1, 𝑑2 where 𝑑1is relevant to
q, but 𝑑2 is non-relevant
Logistic loss is popularly used—think RankNet where
𝑠𝑖𝑚 𝑣 𝑞, 𝑣 𝑑 is the model score
Typically both left and right models share similar architectures,
but may also choose to share the learnable parameters
114. COMPUTATION
NETWORKS
The “Lego” approach to specifying DNN architectures
Library of computation nodes, each node defines logic for:
1. Forward pass: compute output given input
2. Backward pass: compute gradient of loss w.r.t. inputs,
given gradient of loss w.r.t. outputs
3. Parameter gradient: compute gradient of loss w.r.t.
parameters, given gradient of loss w.r.t. outputs
Chain nodes to create bigger and more complex networks
116. TOOLKITS
A diverse set of options
to choose from!
Figure from https://towardsdatascience.com/battle-of-
the-deep-learning-frameworks-part-i-cff0e3841750
119. SEMANTIC
HASHING
Document autoencoder minimizing
reconstruction error
Input: word counts (vocab size = 2K)
Output: binary vector
Stacked RBMs w/ layer-by-layer pre-
training followed by E2E tuning
Ruslan Salakhutdinov and Geoffrey Hinton. Semantic hashing. In IJAR, 2009.
120. DEEP SEMANTIC
SIMILARITY
MODEL (DSSM)
Siamese network trained E2E on query and
document title pairs
Relevance is estimated by cosine similarity
between query and document embeddings
Input: character trigraph counts (bag of words
assumption)
Minimizes cross-entropy loss against randomly
sampled negative documents
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In CIKM, 2013.
121. CONVOLUTIONAL
DSSM (CDSSM)
Replace bag-of-words assumption by concatenating
term vectors in a sequence on the input
Convolution followed by global max-pooling
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014.
123. DSSM TRAINED ON DIFFERENT TYPES OF DATA
Trained on pairs of… Sample training data Useful for? Paper
Query and document titles <“things to do in seattle”, “seattle tourist attractions”> Document ranking (Shen et al., 2014)
https://dl.acm.org/citation...
Query prefix and suffix <“things to do in”, “seattle”> Query auto-completion (Mitra and Craswell, 2015)
https://dl.acm.org/citation...
Consecutive queries in
user sessions
<“things to do in seattle”, “space needle”> Next query suggestion (Mitra, 2015)
https://dl.acm.org/citation...
Each model captures a different notion of similarity
(or regularity) in the learnt embedding space
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014.
Bhaskar Mitra and Nick Craswell. Query auto-completion for rare prefixes. In CIKM, 2015.
Bhaskar Mitra. Exploring session context using distributed representations of queries and reformulations. In SIGIR, 2015.
124. Nearest neighbors for “seattle” and “taylor swift” based on two DSSM
models – one trained on query-document pairs and the other trained on
query prefix-suffix pairs
DIFFERENT REGULARITIES IN DIFFERENT
EMBEDDING SPACES
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014.
Bhaskar Mitra and Nick Craswell. Query auto-completion for rare prefixes. In CIKM, 2015.
125. DIFFERENT REGULARITIES IN DIFFERENT
EMBEDDING SPACES
Groups of similar search intent
transitions from a query log
The DSSM trained on session query pairs
can capture regularities in the query space
(similar to word2vec for terms)
Bhaskar Mitra. Exploring session context using distributed representations of queries and reformulations. In SIGIR, 2015.
126. DSSM TRAINED ON SESSION QUERY PAIRS
ALLOWS FOR ANALOGIES OVER SHORT TEXT!
Bhaskar Mitra. Exploring session context using distributed representations of queries and reformulations. In SIGIR, 2015.
127. INTERACTION-BASED
NETWORKS
Typically a document is relevant if some part of the
document contains information relevant to the query
Interaction matrix 𝑋—where 𝑥𝑖𝑗 is obtained by comparing
the ith window over query terms with the jth window over
the document terms—captures evidence of relevance from
different parts of the document
Additional neural network layers can inspect the
interaction matrix and aggregate the evidence to estimate
overall relevance
Zhengdong Lu and Hang Li. A deep architecture for matching short texts. In NIPS, 2013.
129. LEXICAL AND SEMANTIC
MATCHING NETWORKS
Mitra et al. [2016] argue that both lexical
and semantic matching is important for
document ranking
Duet model is a linear combination of two
DNNs—focusing on lexical and semantic
matching, respectively—jointly trained on
labelled data
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
130. LEXICAL AND SEMANTIC
MATCHING NETWORKS
Lexical sub-model operates over input matrix 𝑋
𝑥𝑖,𝑗 =
1, 𝑖𝑓 𝑡 𝑞,𝑖 = 𝑡 𝑑,𝑗
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
In relevant documents,
1. Many matches, typically in clusters
2. Matches localized early in document
3. Matches for all query terms
4. In-order (phrasal) matches
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
131. LEXICAL AND SEMANTIC
MATCHING NETWORKS
Convolve using window of size 𝑛 𝑑 × 1
Each window instance compares a query term w/
whole document
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
132. LEXICAL AND SEMANTIC
MATCHING NETWORKS
Semantic sub-model matches in the latent
embedding space
Match query with moving windows over document
Learn text embeddings specifically for the task
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
133. BIG VS. SMALL DATA
REGIMES
Big data seems to be more crucial for models that focus on
good representation learning for text
Partial supervision strategies (e.g., unsupervised pre-training
of word embeddings) can be effective but may be leaving the
bigger gains on the table
Learning to train on unlabeled data
may be key to making progress on
neural ad-hoc retrieval
Which IR models are similar?
Clustering based on query level
retrieval performance.
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In WWW, 2017.
134. CHALLENGE
Duet implementation on CNTK (python)
http://bit.ly/CodeDUETCan you evaluate the duet
model on a popular
community question-
answering task? GET THE CODE
135. MANY OTHER NEURAL ARCHITECTURES
(Palangi et al., 2015)
(Kalchbrenner et al., 2014)
(Denil et al., 2014)
(Kim, 2014)
(Severyn and Moschitti, 2015)
(Zhao et al., 2015) (Hu et al., 2014)
(Tai et al., 2015)
(Guo et al., 2016)
(Hui et al., 2017)
(Pang et al., 2017)
(Jaech et al., 2017)
(Dehghani et al., 2017)
136. BUT WEB DOCUMENTS ARE MORE
THAN JUST BODY TEXT…
URL
incoming
anchor text
title
body
clicked query
137. RANKING DOCUMENTS
WITH MULTIPLE FIELDS
Learn different embedding space for each
document field
Different fields may match different aspects of
the query—learn different query embeddings
for matching against different fields
Represent per field match by a vector, not a
score
Field level dropout during training can
regularize against over-dependency on any
individual field
Hamed Zamani, Bhaskar Mitra, Xia Song, Nick Craswell, and Saurabh Tiwary. Neural ranking models with multiple document fields. In WSDM, 2018.
138. NEURAL MODELS FOR
EMERGING IR TASKS
Conversational response retrieval (Zhou et al., 2016, Yan et al., 2016)
Proactive retrieval (Luukkonen et al., 2016)
Multimodal retrieval (Ma et al., 2015)
Knowledge-based IR (Nguyen et al., 2016)
140. AN INTRODUCTION TO NEURAL
INFORMATION RETRIEVAL
Foundations and Trends® in Information Retrieval
(under review)
http://bit.ly/neuralir-intro
THANK YOU
@UnderdogGeek bmitra@microsoft.com
Editor's Notes
Local representation
Distributed representation
One dimension for “banana” “banana” is a pattern
Brittle under noise More robust to noise
Precise Near “mango”, “pineapple”. (Nuanced)
Add vocab Add dimensions Add vocab Generate more vectors
K dimensions K items K dimensions 2k “regions”