Your SlideShare is downloading. ×
Comparison based evaluation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Comparison based evaluation

18
views

Published on

rank aggregation

rank aggregation

Published in: Technology, Sports

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
18
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Comparison-based evaluationJose Antonio Martinez Torres
  • 2. Problem
  • 3. Law of Large number1 spin-$5001000 spin$100,000
  • 4. 254
  • 5. 432
  • 6. Experiments• Movies Dataset– 100,000 ratings– 1000 users– 1700 movies• http://www.grouplens.org/node/73
  • 7. True rating• Based on the law of large numbers, averagerating using this movies dataset can work verywell.• However, by using a small sample, averagerating would significantly differ from the truerating.
  • 8. = { 2, 4, 5,5,4,4,5,3,3,4,5,4,5,4,5,4,5,4,3,2,3,3,4,5,4,4,5,5,3,4}True rating = 4= { 2, 4 }Rating = 3
  • 9. Kendall tau• Statistic used to measure the degree ofsimilarity between two rankings.• Practical use:– Compare how close the top-10 results producedby Google and Bing are.
  • 10. ABCDECDABERank 1 Rank 2T = 6 - 4 / ½ (5) (5-1) = 0.2Dissimilarity goes from 1 to -1 where 1 means the two rankings arethe same and -1 means one ranking is the reverse of the other
  • 11. Movie1 = {3, 4, 5,4,5,3,2,4,4,5,4,3,4,5,4,3,2,3,4,4,4,5,4,3,3,2}Movie2 = {4, 3, 2,3,2,3,4,3,2,1,2,3,4,3,4,3,2,3,2,3,4,3,4,3,4,3}Movie3 = {3, 4, 2,4,5,3,2,4,,4,5,4,3,4,5,4,3,2,3,4,4,4,5,4,3,3,2}Movie4 = {2, 3, 4,5,3,4,4,4,5,4,3,3,2}Movie5 = {3, 4, 5,2,4,5,4,3,3,2}Movie6 = {5, 3, 4,3,3,5,4,3,3,2}Movie7 = {1, 3, 4,3,2,3,43,2,3,4,2,3,4,2,3,4,3,3,2}..Movie1700= {3, 4, 5,4,2,3,3,2,3,2,1,1,2,3}Ranking 1 = 3 , 4 , 3, 2, 3, 5, 1Ranking 2 = 3.5 ,3.5 , 3.5, 2.5, 3.5, 4, 2...Calculate 1 to n ranking n and true rating Kendall tau correlation
  • 12. 0.40.50.60.70.80.911 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50KendalltvalueError rate k-distanceSerie…