Duet @ TREC 2019 Deep Learning Track

•Download as PPTX, PDF•

0 likes•109 views

This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019. For the document retrieval task, we adapt the Duet model to ingest a "multiple field" view of documents—we refer to the new architecture as Duet with Multiple Fields (DuetMF). A second submission combines the DuetMF model with other neural and traditional relevance estimators in a learning-to-rank framework and achieves improved performance over the DuetMF baseline. For the passage retrieval task, we submit a single run based on an ensemble of eight Duet models.

Technology

Duet @ TREC 2019
Deep Learning Track
B haskar Mitra , Microsof t & Univer sity College London, Canada
bmitra@microsof t.com @ UnderdogGeek
N ick C raswell, Microsof t, USA
nickcr@microsof t.com @nick_craswell

Motivation for participation in TREC 2019 Deep
Learning track
Enrich the document pool to improve
reusability of TREC DL dataset
Benchmark Duet on a large public dataset
Try Duet + Neural Ranking model with
Multiple Fields (NRMF) [Zamani et al., 2018]
Source: original Duet paper [Mitra et al., 2017]

Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet

Duet with Multiple Fields (DuetMF)
Match query against each individual
document field using Duet—separate
parameter set corresponding to each field

Unsupervised pretraining
Randomly sample two documents dpos and dneg from the collection
Randomly pick either the title or the URL of dpos, and treat it as a
pseudo-query qpseudo
Mask corresponding field for both dpos and dneg
Compute RankNet loss over <qpseudo, dpos, dneg>

Summary of runs
We submitted three runs:
1. A DuetMF model for the document reranking task
2. A Learning-to-Rank model for the document retrieval task
• Candidate generation using query likelihood (QL)
• Reranking features: DuetMF, Dual Embedding Space Model (DESM), Sequential
Dependence Model (SDM), Pseudo-Relevance Feedback (PRF), Best Match (BM25), and
features based on query length and domain quality
3. An ensemble of eight Duet models for the passage reranking task
• Code: https://github.com/bmitra-msft/NDRM/blob/master/notebooks/Duet.ipynb

Results
ms_ensemble NDCG@10=0.578 vs. best trad run NDCG@10=0.561
ms_duet_passage NDCG@10=0.614 vs. best trad run NDCG@10=0.556

Ideas for TREC 2020
Deep Learning track
• Pretraining Duet on large document
collections (e.g., Wikipedia + books
corpus)
• Duet with BERT/transformer based
distributed sub-model
• Retrieval, not reranking: using query-term
independence assumption [Mitra et al.,
2019] for fullrank setting

A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs. We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features.

Conformer-Kernel with Query Term Independence @ TREC 2020 Deep Learning Track

Bhaskar Mitra

We benchmark Conformer-Kernel models under the strict blind evaluation setting of the TREC 2020 Deep Learning track. In particular, we study the impact of incorporating: (i) Explicit term matching to complement matching based on learned representations (i.e., the “Duet principle”), (ii) query term independence (i.e., the “QTI assumption”) to scale the model to the full retrieval setting, and (iii) the ORCAS click data as an additional document description field. We find evidence which supports that all three aforementioned strategies can lead to improved retrieval quality.

Adversarial and reinforcement learning-based approaches to information retrieval

Bhaskar Mitra

Traditionally, machine learning based approaches to information retrieval have taken the form of supervised learning-to-rank models. Recently, other machine learning approaches—such as adversarial learning and reinforcement learning—have started to find interesting applications in retrieval systems. At Bing, we have been exploring some of these methods in the context of web search. In this talk, I will share couple of our recent work in this area that we presented at SIGIR 2018.

5 Lessons Learned from Designing Neural Models for Information Retrieval

Bhaskar Mitra

Slides from my keynote talk at the Recherche d'Information SEmantique (RISE) workshop at CORIA-TALN 2018 conference in Rennes, France. (Abstract) Neural Information Retrieval (or neural IR) is the application of shallow or deep neural networks to IR tasks. Unlike classical IR models, these machine learning (ML) based approaches are data-hungry, requiring large scale training data before they can be deployed. Traditional learning to rank models employ supervised ML techniques—including neural networks—over hand-crafted IR features. By contrast, more recently proposed neural models learn representations of language from raw text that can bridge the gap between the query and the document vocabulary. Neural IR is an emerging field and research publications in the area has been increasing in recent years. While the community explores new architectures and training regimes, a new set of challenges, opportunities, and design principles are emerging in the context of these new IR models. In this talk, I will share five lessons learned from my personal research in the area of neural IR. I will present a framework for discussing different unsupervised approaches to learning latent representations of text. I will cover several challenges to learning effective text representations for IR and discuss how latent space models should be combined with observed feature spaces for better retrieval performance. Finally, I will conclude with a few case studies that demonstrates the application of neural approaches to IR that go beyond text matching.

Neural Models for Information Retrieval

Bhaskar Mitra

In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing (NLP) tasks, such as language modelling and machine translation. This suggests that neural models will also yield significant performance improvements on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using semantic rather than lexical matching. IR tasks, however, are fundamentally different from NLP tasks leading to new challenges and opportunities for existing neural representation learning approaches for text. We begin this talk with a discussion on text embedding spaces for modelling different types of relationships between items which makes them suitable for different IR tasks. Next, we present how topic-specific representations can be more effective than learning global embeddings. Finally, we conclude with an emphasis on dealing with rare terms and concepts for IR, and how embedding based approaches can be augmented with neural models for lexical matching for better retrieval performance. While our discussions are grounded in IR tasks, the findings and the insights covered during this talk should be generally applicable to other NLP and machine learning tasks.

Neural Models for Information Retrieval

Bhaskar Mitra

In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing (NLP) tasks, such as language modelling and machine translation. This suggests that neural models may also yield significant performance improvements on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using semantic rather than lexical matching. IR tasks, however, are fundamentally different from NLP tasks leading to new challenges and opportunities for existing neural representation learning approaches for text. In this talk, I will present my recent work on neural IR models. We begin with a discussion on learning good representations of text for retrieval. I will present visual intuitions about how different embeddings spaces capture different relationships between items, and their usefulness to different types of IR tasks. The second part of this talk is focused on the applications of deep neural architectures to the document ranking task.

Neural Models for Document Ranking

Bhaskar Mitra

In the last few years, neural representation learning approaches have achieved very good performance on many natural language processing (NLP) tasks, such as language modelling and machine translation. This suggests that neural models may also yield significant performance improvements on information retrieval (IR) tasks, such as relevance ranking, addressing the query-document vocabulary mismatch problem by using semantic rather than lexical matching. IR tasks, however, are fundamentally different from NLP tasks leading to new challenges and opportunities for existing neural representation learning approaches for text. In this talk, I will present my recent work on neural IR models. We begin with a discussion on learning good representations of text for retrieval. I will present visual intuitions about how different embeddings spaces capture different relationships between items, and their usefulness to different types of IR tasks. The second part of this talk is focused on the applications of deep neural architectures to the document ranking task.

Neural Models for Document Ranking

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

The Duet model

Bhaskar Mitra

Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent semantic space. In traditional information retrieval models, on the other hand, terms have discrete or local representations, and the relevance of a document is determined by the exact matches of query terms in the body text. We hypothesize that matching with distributed representations complements matching with traditional local representations, and that a combination of the two is favourable. We propose a novel document ranking model composed of two separate deep neural networks, one that matches the query and the document using a local representation, and another that matches the query and the document using learned distributed representations. The two networks are jointly trained as part of a single neural network. We show that this combination or ‘duet’ performs significantly better than either neural network individually on a Web page ranking task, and significantly outperforms traditional baselines and other recently proposed models based on neural networks.

A Simple Introduction to Neural Information Retrieval

Bhaskar Mitra

Neural Information Retrieval (or neural IR) is the application of shallow or deep neural networks to IR tasks. In this lecture, we will cover some of the fundamentals of neural representation learning for text retrieval. We will also discuss some of the recent advances in the applications of deep neural architectures to retrieval tasks. (These slides were presented at a lecture as part of the Information Retrieval and Data Mining course taught at UCL.)

Neural Information Retrieval: In search of meaningful progress

Bhaskar Mitra

The emergence of deep learning based methods for search poses several challenges and opportunities not just for modeling, but also for benchmarking and measuring progress in the field. Some of these challenges are new, while others have evolved from existing challenges in IR benchmarking exacerbated by the scale at which deep learning models operate. Evaluation efforts such as the TREC Deep Learning track and the MS MARCO public leaderboard are intended to encourage research and track our progress, addressing big questions in our field. The goal is not simply to identify which run is "best" but to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This entails a wider conversation in the IR community about what constitutes meaningful progress, how benchmark design can encourage or discourage certain outcomes, and about the validity of our findings. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track--and reflect on the state of the field and the road ahead.

Topic ModelingKarol Grzegorczyk

Topics ModelingSvitlana volkova

Basic review on topic modeling

Hiroyuki Kuromiya

Transformation Functions for Text Classification: A case study with StackOver...

Sebastian Ruder

Topic Models - LDA and Correlated Topic Models

Claudia Wagner

FaDA: Fast document aligner with word embedding - Pintu Lohar, Debasis Gangul...

Sebastian Ruder

Topic model an introduction

Yueshen Xu

Exploring Session Context using Distributed Representations of Queries and Re...

Bhaskar Mitra

Search logs contain examples of frequently occurring patterns of user reformulations of queries. Intuitively, the reformulation "san francisco" → "san francisco 49ers" is semantically similar to "detroit" →"detroit lions". Likewise, "london"→"things to do in london" and "new york"→"new york tourist attractions" can also be considered similar transitions in intent. The reformulation "movies" → "new movies" and "york" → "new york", however, are clearly different despite the lexical similarities in the two reformulations. In this paper, we study the distributed representation of queries learnt by deep neural network models, such as the Convolutional Latent Semantic Model, and show that they can be used to represent query reformulations as vectors. These reformulation vectors exhibit favourable properties such as mapping semantically and syntactically similar query changes closer in the embedding space. Our work is motivated by the success of continuous space language models in capturing relationships between words and their meanings using offset vectors. We demonstrate a way to extend the same intuition to represent query reformulations. Furthermore, we show that the distributed representations of queries and reformulations are both useful for modelling session context for query prediction tasks, such as for query auto-completion (QAC) ranking. Our empirical study demonstrates that short-term (session) history context features based on these two representations improves the mean reciprocal rank (MRR) for the QAC ranking task by more than 10% over a supervised ranker baseline. Our results also show that by using features based on both these representations together we achieve a better performance, than either of them individually. Paper: http://research.microsoft.com/apps/pubs/default.aspx?id=244728

TopicModels_BleiPaper_Summary.pptxKalpit Desai

Domain-Specific Term Extraction for Concept Identification in Ontology Constr...

Innovation Quotient Pvt Ltd

Topic model, LDA and all that

Zhibo Xiao

Usage of word sense disambiguation in concept identification in ontology cons...

Innovation Quotient Pvt Ltd

Probabilistic models (part 1)KU Leuven

Modeling documents with Generative Adversarial Networks - John Glover

Sebastian Ruder

Topic ModelsClaudia Wagner

Topic modeling using big data analytics

Farheen Nilofer

Big Data Processing using a AWS Dataset

Vishva Abeyrathne

Amazon Product Sentiment review

Lalit Jain

What's hot

Deep Learning for Search

Bhaskar Mitra

The Duet model

Bhaskar Mitra

A Simple Introduction to Neural Information Retrieval

Bhaskar Mitra

Neural Information Retrieval: In search of meaningful progress

Bhaskar Mitra

Topic ModelingKarol Grzegorczyk

Topics ModelingSvitlana volkova

Basic review on topic modeling

Hiroyuki Kuromiya

Transformation Functions for Text Classification: A case study with StackOver...

Sebastian Ruder

Topic Models - LDA and Correlated Topic Models

Claudia Wagner

FaDA: Fast document aligner with word embedding - Pintu Lohar, Debasis Gangul...

Sebastian Ruder

Topic model an introduction

Yueshen Xu

Exploring Session Context using Distributed Representations of Queries and Re...

Bhaskar Mitra

TopicModels_BleiPaper_Summary.pptxKalpit Desai

Domain-Specific Term Extraction for Concept Identification in Ontology Constr...

Innovation Quotient Pvt Ltd

Topic model, LDA and all that

Zhibo Xiao

Usage of word sense disambiguation in concept identification in ontology cons...

Innovation Quotient Pvt Ltd

Probabilistic models (part 1)KU Leuven

Modeling documents with Generative Adversarial Networks - John Glover

Sebastian Ruder

Topic ModelsClaudia Wagner

Topic modeling using big data analytics

Farheen Nilofer

What's hot (20)

Deep Learning for Search

The Duet model

A Simple Introduction to Neural Information Retrieval

Neural Information Retrieval: In search of meaningful progress

Topic Modeling

Topics Modeling

Basic review on topic modeling

Transformation Functions for Text Classification: A case study with StackOver...

Topic Models - LDA and Correlated Topic Models

FaDA: Fast document aligner with word embedding - Pintu Lohar, Debasis Gangul...

Topic model an introduction

Exploring Session Context using Distributed Representations of Queries and Re...

TopicModels_BleiPaper_Summary.pptx

Domain-Specific Term Extraction for Concept Identification in Ontology Constr...

Topic model, LDA and all that

Usage of word sense disambiguation in concept identification in ontology cons...

Probabilistic models (part 1)

Modeling documents with Generative Adversarial Networks - John Glover

Topic Models

Topic modeling using big data analytics

Similar to Duet @ TREC 2019 Deep Learning Track

Big Data Processing using a AWS Dataset

Vishva Abeyrathne

Amazon Product Sentiment review

Lalit Jain

Learning group dssm - 20170605

Shuai Zhang

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...

Dr Arash Najmaei ( Phd., MBA, BSc)

BDS_QA.pdf

NikunjaParida1

Yarn spark next_gen_hadoop_8_jan_2014

Vijay Srinivas Agneeswaran, Ph.D

IRJET- Review of Existing Methods in K-Means Clustering Algorithm

IRJET Journal

ON DISTRIBUTED FUZZY DECISION TREES FOR BIG DATA

Nexgen Technology

Sawmill - Integrating R and Large Data Clouds

Robert Grossman

2005 fall cs523_lecture_4abhineetverma

Ju3517011704

IJERA Editor

International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.

Semantic Annotation of Documents

subash chandra

IA3_presentation.pptx

KtonNguyn2

HDFS-HC: A Data Placement Module for Heterogeneous Hadoop Clusters

Xiao Qin

An increasing number of popular applications become data-intensive in nature. In the past decade, the World Wide Web has been adopted as an ideal platform for developing data-intensive applications, since the communication paradigm of the Web is sufficiently open and powerful. Data-intensive applications like data mining and web indexing need to access ever-expanding data sets ranging from a few gigabytes to several terabytes or even petabytes. Google leverages the MapReduce model to process approximately twenty petabytes of data per day in a parallel fashion. In this talk, we introduce the Google’s MapReduce framework for processing huge datasets on large clusters. We first outline the motivations of the MapReduce framework. Then, we describe the dataflow of MapReduce. Next, we show a couple of example applications of MapReduce. Finally, we present our research project on the Hadoop Distributed File System. The current Hadoop implementation assumes that computing nodes in a cluster are homogeneous in nature. Data locality has not been taken into account for launching speculative map tasks, because it is assumed that most maps are data-local. Unfortunately, both the homogeneity and data locality assumptions are not satisﬁed in virtualized data centers. We show that ignoring the datalocality issue in heterogeneous environments can noticeably reduce the MapReduce performance. In this paper, we address the problem of how to place data across nodes in a way that each node has a balanced data processing load. Given a dataintensive application running on a Hadoop MapReduce cluster, our data placement scheme adaptively balances the amount of data stored in each node to achieve improved data-processing performance. Experimental results on two real data-intensive applications show that our data placement strategy can always improve the MapReduce performance by rebalancing data across nodes before performing a data-intensive application in a heterogeneous Hadoop cluster.

IRJET- Multi Label Document Classification Approach using Machine Learning Te...

IRJET Journal

IRJET- Diverse Approaches for Document Clustering in Product Development Anal...

IRJET Journal

Efficient Machine Learning and Machine Learning for Efficiency in Information...

Bhaskar Mitra

Emerging machine learning approaches, including deep learning methods, for information retrieval (IR) have recently demonstrated significant improvements in accuracy of relevance estimation at the cost of increasing model complexity and corresponding rise in computational and environmental costs of training and inference. In web search, these costs are further compounded by the necessity to train on large-scale datasets, consume long documents as inputs, and retrieve relevant documents from web-scale collections within milliseconds in response to high volume query traffic. A typical playbook for developing deep learning models for IR involves largely ignoring efficiency concerns during model development and then later scaling these methods by either finding faster approximations of the same models or employing heuristics to reduce the input space over which these models operate. Domain knowledge about the specific IR task and deeper understanding of system design and data structures in whose context these models are deployed can significantly help with not only model simplification but also to inform data-structure specific machine learning model design. Alternatively, predictive machine learning can also be employed specifically to improve efficiency in large scale IR settings. In this talk, I will cover several case studies for both improving efficiency of machine learning models for IR as well as direct application of machine learning to improve retrieval efficiency, and conclude with a brief discussion on potential future directions for efficiency-sensitive benchmarking of machine learning models for IR.

Cg33504508

IJERA Editor

Adopting the DSM paradigm: defining federation scenarios through resource br...

Christos Tranoris

pptbutest

Similar to Duet @ TREC 2019 Deep Learning Track (20)

Big Data Processing using a AWS Dataset

Amazon Product Sentiment review

Learning group dssm - 20170605

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...

BDS_QA.pdf

Yarn spark next_gen_hadoop_8_jan_2014

IRJET- Review of Existing Methods in K-Means Clustering Algorithm

ON DISTRIBUTED FUZZY DECISION TREES FOR BIG DATA

Sawmill - Integrating R and Large Data Clouds

2005 fall cs523_lecture_4

Ju3517011704

Semantic Annotation of Documents

IA3_presentation.pptx

HDFS-HC: A Data Placement Module for Heterogeneous Hadoop Clusters

IRJET- Multi Label Document Classification Approach using Machine Learning Te...

IRJET- Diverse Approaches for Document Clustering in Product Development Anal...

Efficient Machine Learning and Machine Learning for Efficiency in Information...

Cg33504508

Adopting the DSM paradigm: defining federation scenarios through resource br...

ppt

More from Bhaskar Mitra

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

Joint Multisided Exposure Fairness for Search and Recommendation

Bhaskar Mitra

(Slides from my talk at SEA: Search Engines Amsterdam) Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval.

What’s next for deep learning for Search?

Bhaskar Mitra

In this talk, I will share some of my personal reflections on the progress in the field of neural IR and some of the ongoing and future research directions that I am personally excited about. This talk will be informed by my own research in this area as well as my experience both as a developer/organizer of the MS MARCO benchmark and the TREC Deep Learning Track and as an applied researcher previously working on web scale search systems at Bing. My goal in this talk would be to move the conversation beyond neural reranking models towards a richer and bolder vision of search powered by deep learning.

So, You Want to Release a Dataset? Reflections on Benchmark Development, Comm...

Bhaskar Mitra

In this talk, I share some of my personal reflections and learnings on benchmark development and community building for making robust scientific progress. This talk is informed by my experience as a developer of the MS MARCO benchmark and as an organizer of the TREC Deep Learning Track. My goal in this talk is to situate the act of releasing a dataset in the context of broader research visions and to draw due attention to considerations of scientific and social outcomes that are invariably salient in the acts of dataset creation and distribution.

Multisided Exposure Fairness for Search and Recommendation

Bhaskar Mitra

Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval. The development of expected exposure based metrics also opens up new opportunities and challenges for model optimization. We demonstrate how stochastic ranking policies can be optimized towards target expected exposure and highlight the trade-offs that may exist in optimizing for different fairness dimensions.

Neural Learning to Rank

Bhaskar Mitra

Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.

Neural Learning to Rank

Bhaskar Mitra

Lecture slides presented at Northeastern University (December, 2020). Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.

Benchmarking for Neural Information Retrieval: MS MARCO, TREC, and Beyond

Bhaskar Mitra

The emergence of deep learning-based methods for information retrieval (IR) poses several challenges and opportunities for benchmarking. Some of these are new, while others have evolved from existing challenges in IR exacerbated by the scale at which deep learning models operate. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track, and reflect on the road ahead.

Neural Learning to Rank

Bhaskar Mitra

Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as items to be recommended, in response to user's need. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this tutorial will be on the fundamentals of neural networks and their applications to learning to rank.

Learning to Rank with Neural Networks

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

Neural Learning to Rank

Bhaskar Mitra

Neu-IR 2017: welcome

Bhaskar Mitra

Neural Text Embeddings for Information Retrieval (WSDM 2017)

Bhaskar Mitra

Query Expansion with Locally-Trained Word Embeddings (ACL 2016)

Bhaskar Mitra

Query Expansion with Locally-Trained Word Embeddings (Neu-IR 2016)

Bhaskar Mitra

More from Bhaskar Mitra (17)

Search and Society: Reimagining Information Access for Radical Futures

Joint Multisided Exposure Fairness for Search and Recommendation

What’s next for deep learning for Search?

So, You Want to Release a Dataset? Reflections on Benchmark Development, Comm...

Multisided Exposure Fairness for Search and Recommendation

Neural Learning to Rank

Benchmarking for Neural Information Retrieval: MS MARCO, TREC, and Beyond

Neural Learning to Rank

Learning to Rank with Neural Networks

Deep Learning for Search

Neural Learning to Rank

Neu-IR 2017: welcome

Neural Text Embeddings for Information Retrieval (WSDM 2017)

Query Expansion with Locally-Trained Word Embeddings (ACL 2016)

Query Expansion with Locally-Trained Word Embeddings (Neu-IR 2016)

Recently uploaded

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

DevOps and Testing slides at DASA Connect

Kari Kakkonen

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Recently uploaded (20)

Connector Corner: Automate dynamic content and events by pushing a button

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

UiPath Test Automation using UiPath Test Suite series, part 3

How world-class product teams are winning in the AI era by CEO and Founder, P...

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

"Impact of front-end architecture on development cost", Viktor Turskyi

Neuro-symbolic is not enough, we need neuro-*semantic*

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

DevOps and Testing slides at DASA Connect

GraphRAG is All You need? LLM & Knowledge Graph

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

The Art of the Pitch: WordPress Relationships and Sales

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Epistemic Interaction - tuning interfaces to provide information for AI support

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Duet @ TREC 2019 Deep Learning Track

1. Duet @ TREC 2019 Deep Learning Track B haskar Mitra , Microsof t & Univer sity College London, Canada bmitra@microsof t.com @ UnderdogGeek N ick C raswell, Microsof t, USA nickcr@microsof t.com @nick_craswell

2. Motivation for participation in TREC 2019 Deep Learning track Enrich the document pool to improve reusability of TREC DL dataset Benchmark Duet on a large public dataset Try Duet + Neural Ranking model with Multiple Fields (NRMF) [Zamani et al., 2018] Source: original Duet paper [Mitra et al., 2017]

3. What is the Duet model?

4. Duet on TREC CAR

5. Adapting Duet for MS MARCO

6. What is NRMF?

7. Duet with Multiple Fields (DuetMF)

8. Duet with Multiple Fields (DuetMF) Match query against each individual document field using Duet

9. Duet with Multiple Fields (DuetMF) Match query against each individual document field using Duet—separate parameter set corresponding to each field

10. Duet with Multiple Fields (DuetMF) Match query against each individual document field using Duet—separate parameter set corresponding to each field Aggregate match vectors from each Duet sub-model to estimate overall relevance of document to query

11. Duet with Multiple Fields (DuetMF) Match query against each individual document field using Duet—separate parameter set corresponding to each field Aggregate match vectors from each Duet sub-model to estimate overall relevance of document to query Structured dropout across fields and Duet sub-models

12. Duet with Multiple Fields (DuetMF) Match query against each individual document field using Duet—separate parameter set corresponding to each field Aggregate match vectors from each Duet sub-model to estimate overall relevance of document to query Structured dropout across fields and Duet sub-models Train using RankNet loss over <q, dpos, dneg>

13. Unsupervised pretraining Randomly sample two documents dpos and dneg from the collection Randomly pick either the title or the URL of dpos, and treat it as a pseudo-query qpseudo Mask corresponding field for both dpos and dneg Compute RankNet loss over <qpseudo, dpos, dneg>

14. Summary of runs We submitted three runs: 1. A DuetMF model for the document reranking task 2. A Learning-to-Rank model for the document retrieval task • Candidate generation using query likelihood (QL) • Reranking features: DuetMF, Dual Embedding Space Model (DESM), Sequential Dependence Model (SDM), Pseudo-Relevance Feedback (PRF), Best Match (BM25), and features based on query length and domain quality 3. An ensemble of eight Duet models for the passage reranking task • Code: https://github.com/bmitra-msft/NDRM/blob/master/notebooks/Duet.ipynb

15. Results ms_ensemble NDCG@10=0.578 vs. best trad run NDCG@10=0.561 ms_duet_passage NDCG@10=0.614 vs. best trad run NDCG@10=0.556

16. Ideas for TREC 2020 Deep Learning track • Pretraining Duet on large document collections (e.g., Wikipedia + books corpus) • Duet with BERT/transformer based distributed sub-model • Retrieval, not reranking: using query-term independence assumption [Mitra et al., 2019] for fullrank setting

17. Questions?

Duet @ TREC 2019 Deep Learning Track

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Duet @ TREC 2019 Deep Learning Track

Similar to Duet @ TREC 2019 Deep Learning Track (20)

More from Bhaskar Mitra

More from Bhaskar Mitra (17)

Recently uploaded

Recently uploaded (20)

Duet @ TREC 2019 Deep Learning Track