Finding significant contextual features is a challenging task in the development of interactive information retrieval (IR) systems. This paper investigated a simple method to facil- itate such a task by looking at aggregated relevance judge- ments of retrieved documents. Our study suggested that the agreement on relevance judgements can indicate the effec- tiveness of retrieved documents as the source of significant features. The effect of highly agreed documents gives us prac- tical implication for the design of adaptive search models in interactive IR systems.

1. Effects of Highly Agreed Documents in Relevancy Prediction
Masegosa, A. R.†, Joho, H.‡, and Jose, J. M.‡
†Department of Computer Science and A.I. ‡Department of Computing Science
University of Granada University of Glasgow
Finding significant contextual (SC) features to model the
relevance of documents (docs) is an important task.
Motivated by the findings of highly relevant docs having a lower
level of disagreement between assessors [1], we exploited
aggregated relevance judgements for relevancy prediction.
Hypothesis: Highly agreed (non-)relevant docs are more
effective to extract the features for relevancy prediction
than less agreed documents.
We focused on object and interaction features to model the
document relevancy in this study.
Data was collected from a laboratory-based user study [2]:
24 users searching in the web for 4 different topics.
Click-through (CT) and perceived bookmarked (BM)
documents were collected. Distribution of CT data in Figure 1.
121 features were extracted as candidate SC features (Table1).
4 well-known probabilistic classifiers jointly a feature selection
scheme were used to measure the performance of each
feature category (more details can be found in [3]).
To analyse the influence of the level of agreement, we
devised 3 conditions (Table 2) to assess documents relevancy.
Data was partitioned and evaluated across each condition
and across each CT cardinality (All CT, CT>1 and CT>2).
1. Introduction
2. Methodology
• Portion of relevant/Non-relevant was similar across the
frequency of click-through made in documents (Figure 1).
• When all documents were considered, the effect of highly
agreed documents was found to be small for relevancy
prediction (All CT in C1-C3 in Figure 2).
• When those docs with more than one click-through were
considered, the prediction accuracy was improved significantly.
• The performance of object features was improved when the
agreement on positive relevance was increased.
• The performance of interaction features was further improved
by increasing the agreement on negative relevance.
• Implications: Exploiting aggregated relevance judgements can
enable us to model document relevancy and find significant
features more effectively. For example, filtering out a single CT
doc might be a simple but effective way to reduce a noise
in relevancy modelling.
3. Results and Implications
Table 1: Categorization of Candidate Features
+3.0% +2.7% +2.7%
+6.6%
+10.2%
+7.8%
+0.8%
+12.0%
+8.0%
+3.5%
+8.3%
+6.2%
C1 C2 C3
All CT CT>1 CT>2 Mean
Figure 2: Performance of Object Features
Note: Increment respect to baseline performance (50%). Average across the seven sub-categories.
Note: The same doc could be found by different users and could have different relevancy assessments.
Figure 1: Click-Through (CT) Distribution
+2.3%+2.6%+2.3%
+10.9%
+9.6%
+2.2%
+19.6%
+14.1%
+3.3%
+10.9%
+8.6%
+2.6%
C1 C2 C3
All CT CT>1 CT>2 Mean
Figure 3: Performance of Interaction Features
References
[1] E. Sormunen. Liberal relevance criteria of TREC -: counting on negligible documents?
In Proceedings of the 25th SIGIR Conference, 324-330, 2002.
[2] H. Joho and J. M. Jose. Slicing and dicing the information space using local contexts.
In Proceedings of the First IIiX Symposium, 111-126, 2006.
[3] A. R. Masegosa, H. Joho, and J. M. Jose, Evaluating Query-Independent Object
Features for Relevancy Prediction. In Proceedings of the ECIR 2007, 283-294, 2007.
Table 2: Relevance Aggregation Methods
NA: Negative Agreement = 1- “BM docs” / “CT docs”, PA: Positive Agreement = “BM docs” / “CT docs”
* This work was supported by ALGRA project (TIN2004-06204-C03-02), FPU scholarship (AP2004-4678) and EPSRC (EP/C004108/1).