1.
Amsterdam
October 3rd
Future Work
• A �e means that items really are indis�nguishable: a �ed pair in X is expected to be �ed in Y too
• A �e means that items are within some threshold like ΔnDCG<0.05: �es aren't transi�ve anymore
Open source package
ircor
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τ � 0.891
τap � 0.883
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τa � 0.874
τap,a � 0.873
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τa � 0.866
τap,a � 0.788
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
X
Y
τb � 0.885
τap,b � 0.831
Examples
Some extreme cases:
• If there are no �es, τ=τa=τb and τap=τap,a=τap,�es
• If all items are �ed, τa=τap,a=0
and both τb and τap,�es are undefined
Yilmaz τap Correla�on With Ties
Defini�ons: := number of items �ed with (inclusive), := rank of the first item in 's group (typical sports ranking)
Scenario B: agreement between two observersScenario A: accuracy of observer w.r.t. integral truth
• Tied pairs don't contribute
• Tied pairs are not expected
• Ignore all top items (they always contribute 0)
contribution of items above 's group
contribution of items within 's group
(in all permutations in which they're ranked above)
• A concordant's contribution depends on 's rank across permutations
• Ignore all top items (they always contribute 0)
• Each item is concordant in half the permutations
• And it's contribution depends on 's rank across permutations
• τap assumes is the true ranking, so let's symmetrize:
and adapt τap to ignore ties:
Kendall τ Correla�on With Ties
"this effect [of ties] may arise either because the objects really are indistinguishable,
[...] or because the observer is unable to discern such differences as exist"
Scenario A (Woodbury, 1940):
accuracy of observer w.r.t. integral truth
Scenario B (Student, 1921):
agreement between two observers
• Tied pairs don't contribute anything
• All pairs are expected to contribute #ties in X and Y
"average value of τ over all the t! possible ways
of assigning integral ranks to the tied members"
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
• Tied pairs don't contribute anything
• Tied pairs are not expected to contribute
Rank Correla�on Without Ties
Kendall (1938): Yilmaz et al. (2008):
• To compare two rankings and over the same set of items
• Very common in IR: rank pages by crawling policy, terms for relevance feedback, systems by mean score...
• Specially in Evalua�on: performance across topic sets, effec�veness measures, assessors, pooling methods...
• Based on the concordance between pairs of items:
A
B
C
D
E
F
G
H
C
A
B
D
F
H
G
E
T�� T�������� �� T��� �� AP C����������
Julián Urbano and Mónica Marrero