Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Treatment of Ties in AP Correlation

35 views

Published on

The Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an observer with respect to a true and objective ranking, and measure the agreement between two observers in the absence of a true ranking. These two variants prove useful in cases where ties are possible in either ranking, and may indeed result in very different scores. AP correlation was devised to incorporate a top-heaviness component into Kendall tau, penalizing more heavily if differences occur between items at the top of the rankings, making it a very compelling coefficient in Information Retrieval settings. However, the treatment of ties in AP correlation remains an open problem. In this paper we fill this gap, providing closed analytical formulations of AP correlation under the two scenarios of ties contemplated in Kendall tau. In addition, we developed an R package that implements these coefficients.

Published in: Science
  • Be the first to comment

  • Be the first to like this

The Treatment of Ties in AP Correlation

  1. 1. Amsterdam October 3rd Future Work • A �e means that items really are indis�nguishable: a �ed pair in X is expected to be �ed in Y too • A �e means that items are within some threshold like ΔnDCG<0.05: �es aren't transi�ve anymore Open source package ircor 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 X Y τ � 0.891 τap � 0.883 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 X Y τa � 0.874 τap,a � 0.873 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 X Y τa � 0.866 τap,a � 0.788 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 X Y τb � 0.885 τap,b � 0.831 Examples Some extreme cases: • If there are no �es, τ=τa=τb and τap=τap,a=τap,�es • If all items are �ed, τa=τap,a=0 and both τb and τap,�es are undefined Yilmaz τap Correla�on With Ties Defini�ons: := number of items �ed with (inclusive), := rank of the first item in 's group (typical sports ranking) Scenario B: agreement between two observersScenario A: accuracy of observer w.r.t. integral truth • Tied pairs don't contribute • Tied pairs are not expected • Ignore all top items (they always contribute 0) contribution of items above 's group contribution of items within 's group (in all permutations in which they're ranked above) • A concordant's contribution depends on 's rank across permutations • Ignore all top items (they always contribute 0) • Each item is concordant in half the permutations • And it's contribution depends on 's rank across permutations • τap assumes is the true ranking, so let's symmetrize: and adapt τap to ignore ties: Kendall τ Correla�on With Ties "this effect [of ties] may arise either because the objects really are indistinguishable, [...] or because the observer is unable to discern such differences as exist" Scenario A (Woodbury, 1940): accuracy of observer w.r.t. integral truth Scenario B (Student, 1921): agreement between two observers • Tied pairs don't contribute anything • All pairs are expected to contribute #ties in X and Y "average value of τ over all the t! possible ways of assigning integral ranks to the tied members" A B C D E F G H C A B D F H G E A B C D E F G H C A B D F H G E • Tied pairs don't contribute anything • Tied pairs are not expected to contribute Rank Correla�on Without Ties Kendall (1938): Yilmaz et al. (2008): • To compare two rankings and over the same set of items • Very common in IR: rank pages by crawling policy, terms for relevance feedback, systems by mean score... • Specially in Evalua�on: performance across topic sets, effec�veness measures, assessors, pooling methods... • Based on the concordance between pairs of items: A B C D E F G H C A B D F H G E T�� T�������� �� T��� �� AP C���������� Julián Urbano and Mónica Marrero

×