Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Evaluation 1: System-Oriented
Tetsuya Sakai
@tetsuyasakai
Waseda University
August 24, 2015@ASSIA 2015, Taipei.
About Tetsuya Sakai
• Professor – Department of Computer Science at Waseda University
• Associate Dean – IT Strategies Div...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
• IR researchers’ goal: build systems
that satisfy the user’s information
needs.
• We cannot ask users all the time, so
we...
Improvements that don’t add up [Armstrong09]
Armstrong et al. analysed 106 papers from SIGIR ’98-’08,
CIKM ’04-’08 that us...
The best IR system in the world
I’ve invented
an IR system
The best IR system in the world
I’ve invented
an IR system
A
I’ve built Test
Collection A
to evaluate it
The best IR system in the world
I’ve invented
an IR system
A
I’ve built Test
Collection A
to evaluate it
A
I’ve evaluated ...
The best IR system in the world
I’ve invented
an IR system
A
I’ve built Test
Collection A
to evaluate it
A
I’ve evaluated ...
A typical test collection
Topic
Relevance assessments
(relevant/nonrelevant documents)
Document collection
Topic
Relevance...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
Recall, Precision and E-measure
[vanRijsbergen79]
• E-measure = (|A∪B|-|A∩B|)/(|A|+|B|)
= 1 – 1/(0.5*(1/Prec) + 0.5*(1/Rec...
F-measure
• F-measure = 1 – E-measure
= 1/(α*(1/Prec) + (1-α)*(1/Rec))
= (β + 1)*Prec*Rec/(β *Prec+Rec)
where α = 1/(β + 1...
Harmonic vs. arithmetic mean
0
0.3
0.6
0.90
0.2
0.4
0.6
0.8
1
00.10.20.30.40.50.60.70.80.91
0.8-1
0.6-0.8
0.4-0.6
0.2-0.4
...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
Interpolated precision
relevant
nonrel
nonrel
relevant
relevant
nonrel
relevant
nonrel
1
2
3
4
5
6
7
8
Rec(r) Prec(r)
0.2 ...
Recall-precision graphs
0 1
0.1 1
0.2 1
0.3 0.6
0.4 0.6
0.5 0.6
0.6 0.6
0.7 0.57
0.8 0.57
0.9 0
1 0
i IPi
0
0.1
0.2
0.3
0....
Average Precision [Buckley05]
• Introduced at TREC-2 (1993), implemented in
trec_eval by Buckley
R: total number of releva...
A user model for AP [Robertson08]
• Different users stop scanning the
ranked list at different ranks. They
only stop at a ...
Normalised Discounted Cumulative Gain
[Jarvelin02]
• Introduced at ACM SIGIR 2000/TOIS 2012, a variant of
the sliding rati...
nDCG: an example
Q-measure [Sakai05AIRS,Sakai07IPM]
• A graded relevance version of AP (see also Graded AP
[Robertson10]).
• Same user mode...
Value of the first relevant document at rank r according to
BR(r) (binary relevance, R=5) [Sakai14PROMISE]
0
0.1
0.2
0.3
0...
Q: An example (with β=1)
Normalised Cumulative Utility [Sakai08EVIA]
• Generalises AP and Q
• NCU =
Σ Pr(r)*NU(r)
r
Normalised
Utility
Prec(r) or B...
Expected Reciprocal Rank [Chapelle09]
• "ERR can be seen as a special case of Normalized Cumulative Utility
(NCU)“ [Chapel...
ERR’s diminishing return property
“Thus, if for example document two merely restates information
already gleaned from docu...
Ranked retrieval measures: summary 1
(not exhaustive)
AP Q-measure ERR nDCG
Handling graded
relevance
Diminishing return
(...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
Time-Biased Gain (TBG) [Smucker12]
Gain at rank r Discounting based on time to reach r
Value of information decays with ti...
U-measure [Sakai13SIGIR]
• U can be used not only for traditional IR, but also for various other
tasks such as session IR,...
Trailtext for U
Just concatenate all the texts that the
user has (probably) read.
For web search, one simple user model
wo...
If the nonrel at rank 2
(snippet) is replaced
with a rel
(snippet + full text),
the value of the rel at
rank 4 is always r...
Ranked retrieval measures: summary 2 (not exhaustive)
AP Q-measure ERR nDCG TBG U-measure
Handling graded relevance
Dimini...
Diversified search – a new IR task
Since 2003 or so
• Given an ambiguous/underspecified query, produce a
single Search Eng...
Diversified search measures summary
α-nDCG
[Clarke08]
ERR-IA
[Chapelle11]
D#-nDCG
[Sakai11SIGIR]
DIN#-nDCG, P+Q#
[Sakai12W...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
So you used a test collection that has n=20 topics to
compute nDCG scores for two systems X and Y.
Which system is more ef...
Population
distribution of X
Population
distribution of Y
Random sampling from normal distributions
Under the above assump...
Under the above assumptions,
obeys where
Population mean
of the difference
Which system is more effective?
Or, which of th...
If you look at the populations, X and Y are
equally effective
If you look at the populations, X and Y are
actually differe...
If H0 is true, this t statistic obeys a t distribution with
φ=(n-1) degrees of freedom.
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-...
If H0 is true, the t statistic obeys a t distribution with
φ=(n-1) degrees of freedom.
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-3...
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-3.5
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
φ=4 φ=99 t...
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-3.5
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
φ=4 φ=99 t...
Example: paired t-test using Excel
Significance level α = 0.05 (95% confidence)
Sample size n = 20
Degrees of freedom φ = ...
Limitations of significance testing (1)
• Normality assumptions: computer-based alternatives (bootstrap
[Savoy97, Sakai06S...
Limitations of significance testing (2)
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-3.5
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1....
Limitations of significance testing (3)
0
0.1
0.2
0.3
0.4
-5.0
-4.5
-4.0
-3.5
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1....
What about the sample size n?
In significance testing, there are four important parameters. If three of
them are set, the ...
Comparing more than two systems
• Conducting a t-test for every system pair is not good (though there
are exceptions [Saka...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
Summary
• Ranked retrieval measures as surrogates of user
satisfaction/performance, with different sets of assumptions. Th...
Tools (by Tetsuya Sakai)
• NTCIREVAL (computes various evaluation measures)
http://research.nii.ac.jp/ntcir/tools/ntcireva...
LECTURE OUTLINE
1. Why evaluate?
2. Set retrieval evaluation measures
3. Ranked retrieval evaluation measures
4. More eval...
References (1)
[Armstrong09] Armstrong, T.G., Moffat, A., Webber, W. and Zobel, J.: Improvements that Don’t Add Up: Ad-hoc...
References (2)
[Jarvelin02] Jarvelin, K. and Kekalainen, J.: Cumulated Gain-based Evaluation of IR Techniques, ACM TOIS, 2...
References (3)
[Sakai05AIRS] Sakai, T.: Ranking the NTCIR Systems based on Multigrade Relevance, AIRS 2004 (LNCS 3411), pp...
References (4)
[Smucker07] Smucker, M.D., Allan, J. and Carterette, B.: A Comparison of Statistical Significance Tests for...
Upcoming SlideShare
Loading in …5
×

assia2015sakai

1,633 views

Published on

Asian Summer School in Information Access 2015 lecture slides - system-oriented evaluation

Published in: Technology
  • Be the first to comment

assia2015sakai

  1. 1. Evaluation 1: System-Oriented Tetsuya Sakai @tetsuyasakai Waseda University August 24, 2015@ASSIA 2015, Taipei.
  2. 2. About Tetsuya Sakai • Professor – Department of Computer Science at Waseda University • Associate Dean – IT Strategies Division of Waseda University • Visiting professor – National Institute of Informatics • Researcher in information retrieval, natural language processing, interaction • Editor-in-chief (Asia/Australasia) – Information Retrieval Journal (Springer) • SIGIR 2013 PC co-chair • SIGIR 2017 general co-chair • NTCIR general co-chair • Toshiba → Cambridge U → Toshiba → NewsWatch → Microsoft Research Asia → Waseda
  3. 3. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  4. 4. • IR researchers’ goal: build systems that satisfy the user’s information needs. • We cannot ask users all the time, so we need measures as surrogates of user satisfaction/performance. • “If you cannot measure it, you cannot improve it.” http://zapatopi.net/kelvin/quotes/ system system system Measure Usersatisfaction Improvements Does it correlate with user satisfaction? Why measure?
  5. 5. Improvements that don’t add up [Armstrong09] Armstrong et al. analysed 106 papers from SIGIR ’98-’08, CIKM ’04-’08 that used TREC data, and reported: • Researchers often use low baselines • Researchers claim statistically significant improvements, but the results are often not competitive with the best TREC systems • IR effectiveness has not really improved over a decade! What we want What we’ve got?
  6. 6. The best IR system in the world I’ve invented an IR system
  7. 7. The best IR system in the world I’ve invented an IR system A I’ve built Test Collection A to evaluate it
  8. 8. The best IR system in the world I’ve invented an IR system A I’ve built Test Collection A to evaluate it A I’ve evaluated my system with A and it’s the best
  9. 9. The best IR system in the world I’ve invented an IR system A I’ve built Test Collection A to evaluate it A I’ve evaluated my system with A and it’s the best I’ve invented an IR system B I’ve built Test collection B to evaluate it B I’ve evaluated my system with B and it’s the best
  10. 10. A typical test collection Topic Relevance assessments (relevant/nonrelevant documents) Document collection Topic Relevance assessments (relevant/nonrelevant documents) Topic Relevance assessments (relevant/nonrelevant documents) : : Topic set : “Qrels”The Sakai Lab home page sakailab.com: relevant www.f.waseda.jp/tetsuya/: relevant http://tanabe-agency.co.jp/talent/sakai_masato/: nonrelevant
  11. 11. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  12. 12. Recall, Precision and E-measure [vanRijsbergen79] • E-measure = (|A∪B|-|A∩B|)/(|A|+|B|) = 1 – 1/(0.5*(1/Prec) + 0.5*(1/Rec)) where Prec=|A∩B|/|B|, Rec=|A∩B|/|A|. A generalised form = 1 – 1/(α*(1/Prec) + (1-α)*(1/Rec)) = 1 – (β + 1)*Prec*Rec/(β *Prec+Rec) where α = 1/(β + 1). A: Relevant docs B: Retrieved docs A ∩ B 2 2 2
  13. 13. F-measure • F-measure = 1 – E-measure = 1/(α*(1/Prec) + (1-α)*(1/Rec)) = (β + 1)*Prec*Rec/(β *Prec+Rec) where α = 1/(β + 1). • F with β=b is often expressed as Fβ Fb. • F1 = 2*Prec*Rec/(Prec+Rec) i.e. harmonic mean of Prec and Rec 2 2 2 User attaches β times as much importance to Rec as Prec (dE/dRec=dE/dPrec when Prec/Rec=β) [vanRijsbergen79]
  14. 14. Harmonic vs. arithmetic mean 0 0.3 0.6 0.90 0.2 0.4 0.6 0.8 1 00.10.20.30.40.50.60.70.80.91 0.8-1 0.6-0.8 0.4-0.6 0.2-0.4 0-0.2 0 0.3 0.6 0.90 0.2 0.4 0.6 0.8 1 00.10.20.30.40.50.60.70.80.91 0.8-1 0.6-0.8 0.4-0.6 0.2-0.4 0-0.2 Prec=0, Rec=1 Prec=0.5, Rec=0.5 F1=0 F1=0.5 Prec=0.1, Rec=0.9 F1=0.18 (Prec+Rec)/2=0.5 (Prec+Rec)/2=0.5 (Prec+Rec)/2=0.5 Balance important Balance NOT important
  15. 15. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  16. 16. Interpolated precision relevant nonrel nonrel relevant relevant nonrel relevant nonrel 1 2 3 4 5 6 7 8 Rec(r) Prec(r) 0.2 1 0.2 0.5 0.2 0.33 0.4 0.5 0.6 0.6 0.6 0.5 0.8 0.57 0.8 0.5 R=5 0 1 0.1 1 0.2 1 0.3 0.6 0.4 0.6 0.5 0.6 0.6 0.6 0.7 0.57 0.8 0.57 0.9 0 1 0 i IPiInterpolated Precision IPi = max Prec(r) r s.t. Rec(r)>=i “The major issue addressed by interpolation is that it rarely happens that any particular recall point is achieved.” [Buckley05, p.56]
  17. 17. Recall-precision graphs 0 1 0.1 1 0.2 1 0.3 0.6 0.4 0.6 0.5 0.6 0.6 0.6 0.7 0.57 0.8 0.57 0.9 0 1 0 i IPi 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 To draw a Rec-Prec curve for a set (T) of topics, plot ΣT IPi / |T| for each i Interpolated Precision IPi = max Prec(r) r s.t. Rec(r)>=i Recall level i Interpolatedprecisionati
  18. 18. Average Precision [Buckley05] • Introduced at TREC-2 (1993), implemented in trec_eval by Buckley R: total number of relevant docs r: document rank I(r): flag indicating a relevant doc C(r): number of relevant docs within ranks [1,r] Highly rel Partially rel Highly rel Partially rel Partially rel Partially rel = Most widely-used binary-relevance IR metric since 1990s, but cannot distinguish between Systems A and B.. System A System B
  19. 19. A user model for AP [Robertson08] • Different users stop scanning the ranked list at different ranks. They only stop at a relevant document. • The user distribution is uniform across all (R) relevant documents. • At each stopping point, compute utility (Prec). • Hence AP is the expected utility for the user population.
  20. 20. Normalised Discounted Cumulative Gain [Jarvelin02] • Introduced at ACM SIGIR 2000/TOIS 2012, a variant of the sliding ratio [Pollack68] • Popular “Microsoft version” [Burges05] : Original definition [Jarvelin02] not recommended: a system that returns a relevant document at rank 1 and one that returns a relevant document at rank b are treated as equally effective, where b is the logarithm base (patience parameter). b’s cancel out in the Burges definition. md: document cutoff (e.g. 10) g(r): gain value at rank r e.g. 1 if doc is partially relevant 3 if doc is highly relevant g*(r) gain value at rank r of an ideal ranked list
  21. 21. nDCG: an example
  22. 22. Q-measure [Sakai05AIRS,Sakai07IPM] • A graded relevance version of AP (see also Graded AP [Robertson10]). • Same user model as AP, but the utility is computed using the blended ratio BR(r) instead of Prec(r). • where β: patience parameter (β=0 ⇒ BR=Prec, hence Q=AP) Combines Precision and normalised cumulative gain (nCG) [Jarvelin02]
  23. 23. Value of the first relevant document at rank r according to BR(r) (binary relevance, R=5) [Sakai14PROMISE] 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 β=0.1 β=1 β=10 r<=R ⇒ BR(r)=(1+β)/(r+βr)=1/r=P(r) r>R ⇒ BR(r)=(1+β)/(r+βR) rank Large β ⇒ more tolerance to relevant docs at low ranks
  24. 24. Q: An example (with β=1)
  25. 25. Normalised Cumulative Utility [Sakai08EVIA] • Generalises AP and Q • NCU = Σ Pr(r)*NU(r) r Normalised Utility Prec(r) or BR(r) Stopping probability Rank-biased Graded-uniform
  26. 26. Expected Reciprocal Rank [Chapelle09] • "ERR can be seen as a special case of Normalized Cumulative Utility (NCU)“ [Chapelle09, p.625] • No recall component where Probability that the user is finally satisfied at r Utility at r
  27. 27. ERR’s diminishing return property “Thus, if for example document two merely restates information already gleaned from document one and hence is of no actual benefit to this user, he may wish to assign it a negative document utility, no matter how ‘relevant’ its content might have been to the original information need.” [Cooper73, p.90] “This is a diminishing return property which seems highly desirable for most IR tasks: if we have already shown a lot of relevant documents, there should be less added value in showing more relevant documents.” [Chapelle11,p.582] Rank-biased NCU [Sakai08EVIA] also has this property
  28. 28. Ranked retrieval measures: summary 1 (not exhaustive) AP Q-measure ERR nDCG Handling graded relevance Diminishing return (navigational intent) Discriminative power [Sakai06SIGIR,07SIGIR] Widely used Used widely at NTCIR NCU = [Sakai08EVIA] f( stopping_probability_over_r, utility_at_r ) How many statistically significant system pairs can be obtained (See Section 5) There are a few graded-relevance versions, but AP almost always means binary-relevance AP
  29. 29. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  30. 30. Time-Biased Gain (TBG) [Smucker12] Gain at rank r Discounting based on time to reach r Value of information decays with time Time to reach r: reads (r-1) snippets, and possibly click some docs and read them Snippet reading time: a constant Doc reading time: linear with doc length
  31. 31. U-measure [Sakai13SIGIR] • U can be used not only for traditional IR, but also for various other tasks such as session IR, aggregated search, summarisation, question answering etc. • While other measures are based on ranks, U abandons the notion of rank. Focusses on the amount of text that the user has read within a search session. Instead of ranks, uses the positions of relevant pieces of information on a trailtext
  32. 32. Trailtext for U Just concatenate all the texts that the user has (probably) read. For web search, one simple user model would be to assume that users read all snippets, plus parts of relevant documents
  33. 33. If the nonrel at rank 2 (snippet) is replaced with a rel (snippet + full text), the value of the rel at rank 4 is always reduced Satisfies diminishing return fixed-length snippets Position-based discounting for U
  34. 34. Ranked retrieval measures: summary 2 (not exhaustive) AP Q-measure ERR nDCG TBG U-measure Handling graded relevance Diminishing return (navigational intent) Discriminative power [Sakai06SIGIR,07SIGIR] Considers document lengths and search engine snippets Handles nonlinear traversal [Sakai14PROMISE] Widely used Users do NOT always scan from top to bottom! TREC Contextual Suggestion
  35. 35. Diversified search – a new IR task Since 2003 or so • Given an ambiguous/underspecified query, produce a single Search Engine Result Page that satisfies different user intents! • Challenge: balancing relevance and diversity SERP(SearchEngineResultPage) Highly relevant near the top Give more space to popular intents? Give more space to informational intents? Cover many intents Diversity test collections have relevance assessments for each intent, rather than for each topic
  36. 36. Diversified search measures summary α-nDCG [Clarke08] ERR-IA [Chapelle11] D#-nDCG [Sakai11SIGIR] DIN#-nDCG, P+Q# [Sakai12WWW] U-IA [Sakai13SIGIR] Handling per-intent graded relevance Handling intent probabilities Handling both informational and navigational intents Per-intent diminishing return Discriminative power [Sakai06SIGIR,07SIGIR] Concordance test [Sakai12WWW,13IRJ] Considers document lengths and search engine snippets Widely used Agree with simple measures? M-measure@NTCIR MobileClick
  37. 37. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  38. 38. So you used a test collection that has n=20 topics to compute nDCG scores for two systems X and Y. Which system is more effective? Scores for X, Y: Per-topic difference: Sample mean of the differences: Sample variance: 0.0750 0.0251
  39. 39. Population distribution of X Population distribution of Y Random sampling from normal distributions Under the above assumptions, obeys where Population mean Population variance Population mean of the difference
  40. 40. Under the above assumptions, obeys where Population mean of the difference Which system is more effective? Or, which of these hypotheses is true? If you look at the populations, X and Y are equally effective If you look at the populations, X and Y are actually different obeys
  41. 41. If you look at the populations, X and Y are equally effective If you look at the populations, X and Y are actually different Which of these hypotheses is true? All we have is the sample data: If H0 is true, this t statistic obeys a t distribution with φ=(n-1) degrees of freedom. Sum of squares Number of independent variables in a sum of squares = accuracy of the sum
  42. 42. If H0 is true, this t statistic obeys a t distribution with φ=(n-1) degrees of freedom. 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 Observed value of t0 computed from sample P-value: area under curve = probability of observing t0 or something more extreme IF H0 is true.
  43. 43. If H0 is true, the t statistic obeys a t distribution with φ=(n-1) degrees of freedom. 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 t0 P-value: area under curve = probability of observing t0 or something more extreme IF H0 is true.(1-α) Significance level α: areas under curve = a pre-determined probability (e.g. 5%) of observing something very rare
  44. 44. 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 t0 α/2 α/2 (1-α) p-value If p-value <= α, then something highly unlikely (e.g. 5% chance) under H0 has happened ⇒ H0 is probably wrong, with (1-α)% (e.g. 95%) confidence! We reject H0, and say that the difference is statistically significant at the significance level of α. The population means are probably different!
  45. 45. 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 t0 α/2 α/2 (1-α) p-value If p-value > α, then what we have observed is something we expect under H0. We accept H0, and say that the difference is NOT statistically significant at the significance level of α. This just means that we cannot tell from data whether H0 is true.
  46. 46. Example: paired t-test using Excel Significance level α = 0.05 (95% confidence) Sample size n = 20 Degrees of freedom φ = 20-1 = 19 Sample mean Sample variance t statistic p-value = T.DIST.2T( 2.116, 19 ) = 0.048 < α X is statistically significantly better than Y at α=0.05. Mean nDCG over 20 topics X 0.3450 Y 0.2700
  47. 47. Limitations of significance testing (1) • Normality assumptions: computer-based alternatives (bootstrap [Savoy97, Sakai06SIGIR], randomisation test [Smucker07]) that do not rely on the assumptions are available. But the results are similar to those obtained by the t-test. • Dichotomous decision: p-value = 0.049 < α ⇒ statistically significant! Publish a paper! p-value = 0.051 > α ⇒ not statistically significant! Put it in the drawer! Saying “p-value=0.049” is much more informative than saying “significant at α=0.05”. Report the p-value! [Sakai14forum]
  48. 48. Limitations of significance testing (2) 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 t0 p-value We get a statistically significant result whenever p-value is small ⇔ t-value is large. t-value is large when (a) sample size n is large; or (b) sample effect size is large. Difference measured in standard deviation units If n is large, you can get a statistically significant result with ANYTHING!
  49. 49. Limitations of significance testing (3) 0 0.1 0.2 0.3 0.4 -5.0 -4.5 -4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 φ=4 φ=99 t0 p-value t-value is large when (a) sample size n is large; or (b) sample effect size is large. Don’t just report the p-value. Report the sample effect size! [Sakai14forum] 0.4734 in the previous example. This reflect how substantial the difference may be.
  50. 50. What about the sample size n? In significance testing, there are four important parameters. If three of them are set, the fourth one is uniquely determined. α: probability of Type I error β: probability of Type II error effect size: magnitude of the difference sample size n: number of topics H0 is true H1 is true H0 accepted 1-α β H0 rejected α 1-β Detecting a nonexistent difference Missing a true difference While IR test collections typically have n=50 topics, it is possible to determine the right n by setting α, β, and the minimum effect size that you want to detect [Sakai15IRJ]. Statistical power: ability to detect a true difference
  51. 51. Comparing more than two systems • Conducting a t-test for every system pair is not good (though there are exceptions [Sakai15IRJ]) - the familywise error rate problem. • Use a proper multiple comparison procedure. • Recommended: randomised Tukey HSD test [Carterette12,Sakai14PROMISE]. • [Sakai14forum] says do an ANOVA (analysis of variance) test first, followed by a Tukey HSD test. But this also causes a problem similar to the familywise error rate. If you are interested in the difference between every system pair, conduct Tukey without conducting ANOVA.
  52. 52. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Summary 7. References
  53. 53. Summary • Ranked retrieval measures as surrogates of user satisfaction/performance, with different sets of assumptions. They maybe compared using discriminative power [Sakai06SIGIR,07SIGIR], concordance test [Sakai12WWW,13IRJ] etc. We want measures that reliably measure what we want to measure! • Principles and limitations of statistical significance testing, esp. paired t-test. Report the p-values and effect sizes [Sakai14IRJ]. Type I errors, Type II errors (1-power), effect sizes and sample sizes. A multiple comparison procedure should be used for more than two systems. Let’s write good IR papers!
  54. 54. Tools (by Tetsuya Sakai) • NTCIREVAL (computes various evaluation measures) http://research.nii.ac.jp/ntcir/tools/ntcireval-en.html • BOOTS (bootstrap hypothesis test as an alternative to the t-test) http://research.nii.ac.jp/ntcir/tools/boots-en.html • Discpower (randomisation test as an alternative to the t-test, and randomised Tukey HSD test for comparing more than two systems) http://research.nii.ac.jp/ntcir/tools/discpower-en.html • Topic set size design Excel tools (how many topics do we need?): http://www.f.waseda.jp/tetsuya/tools.html
  55. 55. LECTURE OUTLINE 1. Why evaluate? 2. Set retrieval evaluation measures 3. Ranked retrieval evaluation measures 4. More evaluation measures 5. Statistical significance, power, effect sizes 6. Reporting your results 7. Summary 8. References
  56. 56. References (1) [Armstrong09] Armstrong, T.G., Moffat, A., Webber, W. and Zobel, J.: Improvements that Don’t Add Up: Ad-hoc Retrieval Results Since 1998, ACM CIKM 2009, pp.601-610, 2009. [Buckley05] Buckley, C. and Voorhees, E.M.: Retrieval System Evaluation, In TREC: Experiment and Evaluation in Information Retrieval (Voorhees, E.M. and Harman, D.K., eds.), Chapter 3, The MIT Press, 2005. [Burges05] Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.: Learning to Rank Using Gradient Descent, ICML 2005, pp.89-96, 2005. [Chapelle09] Chapelle, O., Metzler, D., Zhang, Y., Grispan, P.: Expected Reciprocal Rank for Graded Relevance, ACM CIKM 2009, pp.621-630, 2009. [Chapelle11] Chapelle, O., Ji, S., Liao, C., Velipasaoglu, E., Lai, L., Wu, S.L.: Intent-based Diversification of Web Search Results: Metrics and Algorithms, Information Retrieval, 14(6), pp.572-592, 2011. [Clarke08] Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Buttcher, S. and MacKinnon, I.: Novelty and Diversity in Information Retrieval Evaluation, ACM SIGIR 2008, pp.659-666, 2008. [Cooper73] Cooper, W.S.: On Selecting a Measure of Retrieval Effectiveness, JASIS 24(2), pp.87–100, 1973.
  57. 57. References (2) [Jarvelin02] Jarvelin, K. and Kekalainen, J.: Cumulated Gain-based Evaluation of IR Techniques, ACM TOIS, 20(4), p.422-446, 2002. [Pollack68] Pollack, S.M.: Measures for the Comparison of Information Retrieval Systems, American Documentation, 19(4), pp.387- 397, 1968. [Robertson08] Robertson, S.E.: A New Interpretation of Average Precision, ACM SIGIR 2008, pp.689-690, 2008. [Robertson10] Robertson, S.E., Kanoulas, E., Yilmaz, E.: Extending Average Precision to Graded Relevance Judgments, ACM SIGIR 2010, pp.603-610, 2010. [Savoy97] Savoy, J.: Statistical Inference in Retrieval Effectiveness Evaluation, Information Processing and Management, 33(4), pp.495- 512, 1997.
  58. 58. References (3) [Sakai05AIRS] Sakai, T.: Ranking the NTCIR Systems based on Multigrade Relevance, AIRS 2004 (LNCS 3411), pp.251-262, 2005. [Sakai06SIGIR] Sakai, T.: Evaluating Evaluation Metrics based on the Bootstrap, ACM SIGIR 2006, pp.525-532, 2006. [Sakai07SIGIR] Sakai, T.: Alternatives to Bpref, ACM SIGIR 2007, pp.71-78, 2007. [Sakai07IPM] Sakai, T.: On the Reliability of Information Retrieval Metrics based on Graded Relevance, Information Processing and Management, 43(2), pp.531-548, 2007. [Sakai08EVIA] Sakai, T. and Robertson, S.: Modelling A User Population for Designing Information Retrieval Metrics, EVIA 2008, pp.30- 41, 2008. [Sakai11SIGIR] Sakai, T. and Song, R.: Evaluating Diversified Search Results Using Per-Intent Graded Relevance, ACM SIGIR 2011, pp.1043-1052, 2011. [Sakai12WWW] Sakai, T.: Evaluation with Informational and Navigational Intents, WWW 2012, pp.499-508, 2012. [Sakai13SIGIR-U] Sakai, T., Dou, Z.: Summaries, Ranked Retrieval and Sessions: A Unified Framework for Information Access Evaluation, ACM SIGIR 2013, pp.473-482, 2013. [Sakai13IRJ] Sakai, T. and Song, R.: Diversified Search Evaluation: Lessons from the NTCIR-9 INTENT Task, Information Retrieval, 16(4), pp.504-529, Springer, 2013. [Sakai14PROMISE] Sakai, T.: Metrics, Statistics, Tests, PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173), Springer, pp.116-163, 2014. [Sakai14forum] Sakai, T.: Statistical Reform in Information Retrieval?, SIGIR Forum, 48(1), pp.3-12, 2014. [Sakai15IRJ] Sakai, T.: Topic Set Size Design, Information Retrieval Journal, submitted.
  59. 59. References (4) [Smucker07] Smucker, M.D., Allan, J. and Carterette, B.: A Comparison of Statistical Significance Tests for Information Retrieval Evaluation, ACM CIKM 2007, pp.623-632, 2007. [Smucker12] Smucker, M.D. and Clarke, C.L.A.: Time-based Calibration of Effectiveness Measures, ACM SIGIR 2012, pp. 95–104 , 2012. [vanRijsbergen79] van Rijsbergen, C.J., Information Retrieval, Chapter 7, Butterworths, 1979.

×