Peer Review in the LiquidPub project


Published on

Slides presented at the 2nd Snow Workshop (

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • For instance if t =1, n = 4 p1(0) = probability that the first number is different from the first in the ordered ranking p1(1) 0 probability that the first number is equal to the first in the ordered ranking
  • Check numbers
  • Notice that if we restricted the world to the top N papers and accepted only 30% of them, or 20% of them, we would have essentially a random process. The bottom right suggests that it is not true that for weaker papers we can have a clear decision We were not looking for this results Acceptance rate is over-rated
  • Randomness in review Due to many reasons: subjective reviews limited size of the sample for a given reviewer absolute subjective marking vs. relative ranking
  • Peer Review in the LiquidPub project

    1. 1. Reviewing Peer Review
    2. 2. Metric Dimensions Quality Fairness Efficiency Statistics Kendall Distance Divergence Disagreement Biases Robustness Unbiasing Effort invariant alternatives Effort vs. quality Min. criteria
    3. 3. Quality-related Metrics: real vs. ideal <ul><li>Real peer review ranking vs. ideal ranking </li></ul><ul><ul><li>Ideal ? </li></ul></ul><ul><ul><ul><li>Subjective vs. Objective </li></ul></ul></ul><ul><ul><ul><li>But each process could/should define approximate indicators of quality like: citations, downloads, community voting, success in a second phase, publication, citations, patents… </li></ul></ul></ul><ul><li>IF an approximate ideal ranking is available we can measure the difference in various ways, e.g </li></ul><ul><ul><li>Kendall  distance / Kendall  rank correlation </li></ul></ul><ul><ul><li>Divergence metric </li></ul></ul>
    4. 4. Divergence Metric ©√ Nt = 1/n Normalized t Nt = n/n N-Divergence Ndiv(1,n, C) = n-1/n ©√ When the second ranking is random, we have: indipendent correlated inv. correlated
    5. 5. prior vs. after discussion ca. 74 (36%) contributions have been effected by the discussion phase
    6. 6. Results: peer review ranking vs. citation count Div Normalized t
    7. 8. Fairness <ul><li>Definition : A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it </li></ul><ul><li>The key is in the assignment of a paper to reviewers: a paper assignment is unfair if the specific assignment influences ( makes more predictable ) the fate of the paper. </li></ul>
    8. 9. Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 > + |min bias| 13% 5% 9% 7% < - |min bias| 12% 4% 8% 7%           C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
    9. 10. Disagreement metric <ul><li>Through this metric we compute the similarity between the marks given by the reviewers on the same contribution. </li></ul><ul><li>The rationale behind this metric is that in a review process we expect some kind of agreement between reviewers. </li></ul>/13
    10. 11. Disagreement vs number of reviews
    11. 12. The road ahead Real-Time accuracy estimation Speed Ranking Ranking vs. marking
    12. 13. Thank you