Copulas for Information Retrieval (SIGIR'13)

521 views
401 views

Published on

In many domains of information retrieval, system estimates of document relevance are based on multidimensional quality criteria that have to be accommodated in a unidimensional result ranking. Current solutions to this challenge are often inconsistent with the formal probabilistic framework in which constituent scores were estimated, or use sophisticated learning methods that make it difficult for humans to understand the origin of the final ranking. To address these issues, we introduce the use of copulas, a powerful statistical framework for modeling complex multi-dimensional dependencies, to information retrieval tasks. We provide a formal background to copulas and demonstrate their effectiveness on standard IR tasks such as combining multidimensional relevance estimates and fusion of results from multiple search engines. We introduce copula-based versions of standard relevance estimators and fusion methods and show that these lead to significant performance improvements on several tasks, as evaluated on large-scale standard corpora, compared to their non-copula counterparts. We also investigate criteria for understanding the likely effect of using copula models in a given retrieval scenario.

This work together with Arjen P. de Vries and Kevyn Collins-Thompson has been accepted for full oral presentation at the 36th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR) in Dublin, Ireland. The full version of this paper is available at: http://dl.acm.org/citation.cfm?id=2484066

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
521
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Copulas for Information Retrieval (SIGIR'13)

  1. 1. Copulas for Information Retrieval Carsten Eickhoff, Arjen P. de Vries, Kevyn Collins-Thompson
  2. 2. Copulas – What is it all about? • Assume two sufficiently different commodities • Rare elemental metals • Pork bellies • No apparent correlations 0 1 2 3 4 5 6 Rare Earths Pork Bellies
  3. 3. Copulas – What is it all about? • Two seemingly independent variables • Yet, for rare extreme cases, there are co-movements • “Tail dependencies” • Copulas decouple observations and dependencies • IR models are good at estimating marginals • Copulas are good at combining them
  4. 4. Overview 1. Non-linear Dependency Structures in IR 2. Copulas – Intuition & Background 3. Multivariate Relevance Estimation 4. When to use them? 5. Score Fusion 6. Conclusion & Future Directions
  5. 5. 1 Non-Linear Dependency Structures in IR
  6. 6. Multivariate Relevance Modelling • IR Systems index and retrieve a growing variety of document types • Many structured, or at least “complex” • Single-criteria relevance frameworks do not perform well • Multi-criteria models tend to be either: a) Naïve (e.g., independence assumption), or, b) Hard to qualitatively interpret for humans (e.g., L2R)
  7. 7. Non-Linear Dependencies • Non-linear dependency structures are still a challenge • TREC 2010 Faceted Blog Distillation Task, Topic 1171, “mysql” • Relevance Criteria: • Topicality • Subjectivity
  8. 8. Non-Linear Dependencies • Pearson’s ᵨ= 0.18 • So, there is no real dependency • …right?
  9. 9. Non-Linear Dependencies • Pearson’s ᵨ= 0.18 • So, there is no real dependency • …right?
  10. 10. Non-Linear Dependencies • Pearson’s ᵨ= 0.18 • So, there is no real dependency • …right? • In the lower third of the scale, we note ᵨ= 0.37
  11. 11. Non-Linear Dependencies • Pearson’s ᵨ= 0.18 • So, there is no real dependency • …right? • In the lower third of the scale, we note ᵨ= 0.37, • And in the upper third, it turns to ᵨ= -0.4
  12. 12. 2 Copulas – Intuition & Background
  13. 13. Copulas (from copulare, to join) • Copulas model complex non-linear dependencies between variables that simple correlations can't capture • Decouple marginal distributions from dependency structure • Approximate joint multivariate distributions • Applied previously in portfolio and risk management, meteorology, river flooding predictions, …
  14. 14. Formal Basics • Given a k-dimensional rv • Map to unit cube • Describe joint cdf with copula • Isolation of a component • Copula’s zero
  15. 15. Closing the circle • Recall the example TREC topic 1171 • Linear combination: AP = 0.14, below collection average (0.25) • Fit Clayton copula to model joint relevance distribution • AP rises to 0.22
  16. 16. 3 Multivariate Relevance Estimation
  17. 17. Joint Relevance Estimation • Estimate marginal distributions from data • Estimate copula fitting parameters to maximize posterior probability of observing data • Use copula to represent joint probability of relevance
  18. 18. Joint Relevance Estimation • We study three different scenarios: • Opinionated blog posts • Personalized bookmarks • Child-friendly websites • Use original training portion of the corpora where available • A 90/10 split otherwise
  19. 19. Results I – Opinionated Blog Posts • TREC Blogs08 dataset • 1.3 M documents • Relevance dimensions: Topicality & Subjectivity • Significantly higher performance than linear combination model
  20. 20. Results II – Personalized Bookmarks • Dataset by Vallet & Castells • 339k documents • Relevance Dimensions: Topicality & Personal relevance • Significantly performance gains in some metrics
  21. 21. Results III – Child-friendly Websites • Dataset from the PuppyIR project (http://puppyir.eu) • 22k documents • Relevance Dimensions: Topicality & Child-suitability • Worse-than-baseline performance
  22. 22. 4 Copulas – When to use them?
  23. 23. When to use them? • Previously: Strongly varying performance for different settings • Is there a way of predicting the merit? • Recall: copulas model tail dependencies between dimensions
  24. 24. Types of Tail Dependencies
  25. 25. Measuring Tail Dependencies • According to Frees and Valdez 1998: IL and IU measure strength of lower and upper tail dependencies • Anderson-Darling test of goodness-of-fit between copula and observed data Domain Frees Tail index Anderson-Darling Actual Retrieval Performance Opinionated Blogs IL = 0.07 0.67 Copulas > linear Personalized Bookmarks IU = 0.49 0.47 Copulas = linear Child-friendly Websites IL = IU = 0 0.046 Copulas < linear
  26. 26. 5 Copulas for Score Fusion
  27. 27. Score Fusion • A different angle on relevance estimation • Combine individual retrieval system scores instead of modelling relevance from content criteria • In this setting, submissions to historic TRECs serve as criteria • We randomly draw k individual runs and combine them using copulas
  28. 28. Fusion Methods • Established: • Copula-based:
  29. 29. Results – TREC 4 • Results are averaged across 200 randomizations per setting of k • Relative improvements over the best, worst and median fused run in terms of percentages of MAP • Small but consistent improvements over non-copula fusion baselines
  30. 30. Robustness - CombSUM • Fusion approaches are often sensitive to weak contributions • We control the number of weak submissions added to the fusion • Copulas’ explicit modeling of dependency structure is more robust
  31. 31. Robustness - CombMNZ • Fusion approaches are often sensitive to weak contributions • We control the number of weak submissions added to the fusion • Copulas’ explicit modeling of dependency structure is more robust
  32. 32. 6 Conclusion and Future Directions
  33. 33. Conclusion • Copulas decouple observations and dependencies • IR models are good at estimating marginal • Copulas are good at combining them • We use them for multivariate relevance estimation • Strongly scenario-dependent performance • Tail indices & goodness of fit tests as estimators of expected performance • Copulas for score fusion • Robust to outliers
  34. 34. The Road Ahead • Currently, we use single copulas for relevance modelling • Copula mixtures and composite Archimedean copulas for higher accuracy • Here, we use pre-existing copula families and fit them to data • Instead, can we formalize copulas from scratch to include domain knowledge? • So far, we explored two-dimensional relevance spaces • What happens as we move into higher-order systems?
  35. 35. Thank You!

×