SlideShare a Scribd company logo
Reviewing Peer Review
Metric Dimensions Quality Fairness Efficiency Statistics Kendall Distance Divergence Disagreement Biases Robustness Unbiasing Effort invariant alternatives Effort vs. quality Min. criteria
Quality-related Metrics:  real vs. ideal ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Divergence Metric ©√ Nt = 1/n Normalized t Nt = n/n N-Divergence Ndiv(1,n, C) = n-1/n ©√ When the second ranking is random, we have: indipendent correlated inv. correlated
prior vs. after discussion ca. 74 (36%) contributions have been effected by the discussion phase
Results:  peer review ranking vs. citation count  Div Normalized t
 
Fairness ,[object Object],[object Object]
Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 >  + |min bias| 13% 5% 9% 7% <  - |min bias| 12% 4% 8% 7%           C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
Disagreement metric ,[object Object],[object Object],/13
Disagreement vs number of reviews
The road ahead Real-Time accuracy estimation Speed Ranking Ranking vs. marking
Thank you

More Related Content

Viewers also liked

Research and Planning
Research and PlanningResearch and Planning
Research and Planningguestae5007f
 
სახელმძღვანელოები
სახელმძღვანელოებისახელმძღვანელოები
სახელმძღვანელოებიcira75
 
PCQuest CIO Tech Priorities Survey 2014
PCQuest CIO Tech Priorities Survey 2014PCQuest CIO Tech Priorities Survey 2014
PCQuest CIO Tech Priorities Survey 2014Anil Chopra
 
College chapter 1 3
College chapter 1 3College chapter 1 3
College chapter 1 3gmaidekamido
 
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Aliaksandr Birukou
 
Morigi massimo 15102011_b
Morigi massimo 15102011_bMorigi massimo 15102011_b
Morigi massimo 15102011_b
Renzo Carlucci
 
Going Beyond Your Body Speaks: The Silent Language of Toastmasters
Going Beyond Your Body Speaks: The Silent Language of ToastmastersGoing Beyond Your Body Speaks: The Silent Language of Toastmasters
Going Beyond Your Body Speaks: The Silent Language of ToastmastersBeth MacNeil Stinson
 
Benefits Of Collaborative E Learning
Benefits Of Collaborative E LearningBenefits Of Collaborative E Learning
Benefits Of Collaborative E Learning
Wilson Araromi
 
Siteco Learnlight
Siteco LearnlightSiteco Learnlight
Siteco Learnlight
willemvanhoorn
 
дети войны
дети войныдети войны
дети войныmonolite
 
SWAN, HyQue and Nanopublications
SWAN, HyQue and NanopublicationsSWAN, HyQue and Nanopublications
SWAN, HyQue and Nanopublications
Paolo Ciccarese
 
Being a Club Webmaster
Being a Club WebmasterBeing a Club Webmaster
Being a Club Webmaster
Beth MacNeil Stinson
 
Electrical characteristics
Electrical characteristicsElectrical characteristics
Electrical characteristicsdijahapple
 
How Open Is Open?
How Open Is Open?How Open Is Open?
How Open Is Open?
Ian White
 
My Friend And I Picture Text Dictionary
My Friend And I Picture Text DictionaryMy Friend And I Picture Text Dictionary
My Friend And I Picture Text Dictionaryjim mager
 
Prsa social media brief 4 aug 10
Prsa social media brief 4 aug 10Prsa social media brief 4 aug 10
Prsa social media brief 4 aug 10
Hoosier Chapter of PRSA
 

Viewers also liked (19)

Research and Planning
Research and PlanningResearch and Planning
Research and Planning
 
სახელმძღვანელოები
სახელმძღვანელოებისახელმძღვანელოები
სახელმძღვანელოები
 
Heart Attack1
Heart Attack1Heart Attack1
Heart Attack1
 
PCQuest CIO Tech Priorities Survey 2014
PCQuest CIO Tech Priorities Survey 2014PCQuest CIO Tech Priorities Survey 2014
PCQuest CIO Tech Priorities Survey 2014
 
1 6 Attempt 2
1 6 Attempt 21 6 Attempt 2
1 6 Attempt 2
 
College chapter 1 3
College chapter 1 3College chapter 1 3
College chapter 1 3
 
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...
 
Morigi massimo 15102011_b
Morigi massimo 15102011_bMorigi massimo 15102011_b
Morigi massimo 15102011_b
 
Going Beyond Your Body Speaks: The Silent Language of Toastmasters
Going Beyond Your Body Speaks: The Silent Language of ToastmastersGoing Beyond Your Body Speaks: The Silent Language of Toastmasters
Going Beyond Your Body Speaks: The Silent Language of Toastmasters
 
Benefits Of Collaborative E Learning
Benefits Of Collaborative E LearningBenefits Of Collaborative E Learning
Benefits Of Collaborative E Learning
 
Siteco Learnlight
Siteco LearnlightSiteco Learnlight
Siteco Learnlight
 
Overview
OverviewOverview
Overview
 
дети войны
дети войныдети войны
дети войны
 
SWAN, HyQue and Nanopublications
SWAN, HyQue and NanopublicationsSWAN, HyQue and Nanopublications
SWAN, HyQue and Nanopublications
 
Being a Club Webmaster
Being a Club WebmasterBeing a Club Webmaster
Being a Club Webmaster
 
Electrical characteristics
Electrical characteristicsElectrical characteristics
Electrical characteristics
 
How Open Is Open?
How Open Is Open?How Open Is Open?
How Open Is Open?
 
My Friend And I Picture Text Dictionary
My Friend And I Picture Text DictionaryMy Friend And I Picture Text Dictionary
My Friend And I Picture Text Dictionary
 
Prsa social media brief 4 aug 10
Prsa social media brief 4 aug 10Prsa social media brief 4 aug 10
Prsa social media brief 4 aug 10
 

Similar to Peer Review in the LiquidPub project

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
Aliaksandr Birukou
 
Is peer review any good? A quantitative analysis of peer review
Is peer review any good? A quantitative analysis of peer reviewIs peer review any good? A quantitative analysis of peer review
Is peer review any good? A quantitative analysis of peer review
Aliaksandr Birukou
 
Introduction to Core Assessments
Introduction to Core AssessmentsIntroduction to Core Assessments
Introduction to Core Assessments
Resolver Inc.
 
Quality key users
Quality key usersQuality key users
Quality key users
Antti Jakobsson
 
Impact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with RealityImpact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with Reality
Donna Smith-Moncrieffe
 
evaluation in infomation retrival
evaluation in infomation retrivalevaluation in infomation retrival
evaluation in infomation retrival
jetaime
 
Detecting incorrectly implemented experiments
Detecting incorrectly implemented experimentsDetecting incorrectly implemented experiments
Detecting incorrectly implemented experiments
Optimizely
 
IRJET-Fake Product Review Monitoring
IRJET-Fake Product Review MonitoringIRJET-Fake Product Review Monitoring
IRJET-Fake Product Review Monitoring
IRJET Journal
 
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
ijmvsc
 
Effect of Computer-Based Testing on Candidates
Effect of Computer-Based Testing on CandidatesEffect of Computer-Based Testing on Candidates
Effect of Computer-Based Testing on Candidates
Assessment Systems
 
IM426 3A G5.ppt
IM426 3A G5.pptIM426 3A G5.ppt
IM426 3A G5.ppt
MohamedSalem979344
 
Supplier Classification and Selecetion With Artificial Neural Network
Supplier Classification and Selecetion With Artificial Neural NetworkSupplier Classification and Selecetion With Artificial Neural Network
Supplier Classification and Selecetion With Artificial Neural Network
VOLKAN YILDIRIM
 
The State of Clinical Outsourcing
The State of Clinical OutsourcingThe State of Clinical Outsourcing
The State of Clinical Outsourcing
The Avoca Group
 
Assessment Group Managers Training-SMCC1.pptx
Assessment Group Managers Training-SMCC1.pptxAssessment Group Managers Training-SMCC1.pptx
Assessment Group Managers Training-SMCC1.pptx
HctorMurciaForero
 
Machine Learning Project - Default credit card clients
Machine Learning Project - Default credit card clients Machine Learning Project - Default credit card clients
Machine Learning Project - Default credit card clients
Vatsal N Shah
 
Managing Risk in Outsourced Clinical Trials
Managing Risk in Outsourced Clinical TrialsManaging Risk in Outsourced Clinical Trials
Managing Risk in Outsourced Clinical Trials
The Avoca Group
 
Multidimensional scalling
Multidimensional scallingMultidimensional scalling
Multidimensional scalling
ZakkyZamrudi2
 
Online Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
Online Service Rating Prediction by Removing Paid Users and Jaccard CoefficientOnline Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
Online Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
IRJET Journal
 

Similar to Peer Review in the LiquidPub project (20)

Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the...
 
Is peer review any good? A quantitative analysis of peer review
Is peer review any good? A quantitative analysis of peer reviewIs peer review any good? A quantitative analysis of peer review
Is peer review any good? A quantitative analysis of peer review
 
Introduction to Core Assessments
Introduction to Core AssessmentsIntroduction to Core Assessments
Introduction to Core Assessments
 
Quality key users
Quality key usersQuality key users
Quality key users
 
AERA 2007 Developing benchmark
AERA 2007 Developing benchmarkAERA 2007 Developing benchmark
AERA 2007 Developing benchmark
 
Impact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with RealityImpact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with Reality
 
evaluation in infomation retrival
evaluation in infomation retrivalevaluation in infomation retrival
evaluation in infomation retrival
 
Detecting incorrectly implemented experiments
Detecting incorrectly implemented experimentsDetecting incorrectly implemented experiments
Detecting incorrectly implemented experiments
 
IRJET-Fake Product Review Monitoring
IRJET-Fake Product Review MonitoringIRJET-Fake Product Review Monitoring
IRJET-Fake Product Review Monitoring
 
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
PRIORITIZING THE BANKING SERVICE QUALITY OF DIFFERENT BRANCHES USING FACTOR A...
 
Effect of Computer-Based Testing on Candidates
Effect of Computer-Based Testing on CandidatesEffect of Computer-Based Testing on Candidates
Effect of Computer-Based Testing on Candidates
 
IM426 3A G5.ppt
IM426 3A G5.pptIM426 3A G5.ppt
IM426 3A G5.ppt
 
Chapter 9
Chapter 9Chapter 9
Chapter 9
 
Supplier Classification and Selecetion With Artificial Neural Network
Supplier Classification and Selecetion With Artificial Neural NetworkSupplier Classification and Selecetion With Artificial Neural Network
Supplier Classification and Selecetion With Artificial Neural Network
 
The State of Clinical Outsourcing
The State of Clinical OutsourcingThe State of Clinical Outsourcing
The State of Clinical Outsourcing
 
Assessment Group Managers Training-SMCC1.pptx
Assessment Group Managers Training-SMCC1.pptxAssessment Group Managers Training-SMCC1.pptx
Assessment Group Managers Training-SMCC1.pptx
 
Machine Learning Project - Default credit card clients
Machine Learning Project - Default credit card clients Machine Learning Project - Default credit card clients
Machine Learning Project - Default credit card clients
 
Managing Risk in Outsourced Clinical Trials
Managing Risk in Outsourced Clinical TrialsManaging Risk in Outsourced Clinical Trials
Managing Risk in Outsourced Clinical Trials
 
Multidimensional scalling
Multidimensional scallingMultidimensional scalling
Multidimensional scalling
 
Online Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
Online Service Rating Prediction by Removing Paid Users and Jaccard CoefficientOnline Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
Online Service Rating Prediction by Removing Paid Users and Jaccard Coefficient
 

More from Aliaksandr Birukou

Publishing conference proceedings internationally: how does it work
Publishing conference proceedings internationally: how does it workPublishing conference proceedings internationally: how does it work
Publishing conference proceedings internationally: how does it work
Aliaksandr Birukou
 
Технические аспекты публикации на нескольких языках – как правильно связать DOI
Технические аспекты публикации на нескольких языках – как правильно связать DOIТехнические аспекты публикации на нескольких языках – как правильно связать DOI
Технические аспекты публикации на нескольких языках – как правильно связать DOI
Aliaksandr Birukou
 
Conference Identity: persistent identifiers for conferences
Conference Identity: persistent identifiers for conferencesConference Identity: persistent identifiers for conferences
Conference Identity: persistent identifiers for conferences
Aliaksandr Birukou
 
Springer LOD conference portal. Demo paper - screenshots
Springer LOD conference portal. Demo paper - screenshotsSpringer LOD conference portal. Demo paper - screenshots
Springer LOD conference portal. Demo paper - screenshots
Aliaksandr Birukou
 
PersistentIDs and CrossMark for Conference Proceedings
PersistentIDs and CrossMark for Conference ProceedingsPersistentIDs and CrossMark for Conference Proceedings
PersistentIDs and CrossMark for Conference Proceedings
Aliaksandr Birukou
 
Publishing conference proceedings internationally: Tips and tricks
Publishing conference proceedings internationally: Tips and tricksPublishing conference proceedings internationally: Tips and tricks
Publishing conference proceedings internationally: Tips and tricks
Aliaksandr Birukou
 
Linked Open Data about Springer Nature conferences. The story so far
Linked Open Data about Springer Nature conferences. The story so farLinked Open Data about Springer Nature conferences. The story so far
Linked Open Data about Springer Nature conferences. The story so far
Aliaksandr Birukou
 
Creating a dataset of peer review in computer science conferences published b...
Creating a dataset of peer review in computer science conferences published b...Creating a dataset of peer review in computer science conferences published b...
Creating a dataset of peer review in computer science conferences published b...
Aliaksandr Birukou
 
Linked Data Initiatives at Springer Verlag
Linked Data Initiatives at Springer Verlag Linked Data Initiatives at Springer Verlag
Linked Data Initiatives at Springer Verlag
Aliaksandr Birukou
 
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Aliaksandr Birukou
 
An Integrated Solution for Runtime Compliance Governance in SOA
An Integrated Solution for Runtime Compliance Governance in SOAAn Integrated Solution for Runtime Compliance Governance in SOA
An Integrated Solution for Runtime Compliance Governance in SOA
Aliaksandr Birukou
 
20101112 librinnovando liquidpub
20101112 librinnovando liquidpub20101112 librinnovando liquidpub
20101112 librinnovando liquidpub
Aliaksandr Birukou
 
Liquid Journals - Intro and RoadMap
Liquid Journals - Intro and RoadMapLiquid Journals - Intro and RoadMap
Liquid Journals - Intro and RoadMap
Aliaksandr Birukou
 
Liquid Journals. Overview. How social computing and liquid knowledge will sha...
Liquid Journals. Overview. How social computing and liquid knowledge will sha...Liquid Journals. Overview. How social computing and liquid knowledge will sha...
Liquid Journals. Overview. How social computing and liquid knowledge will sha...
Aliaksandr Birukou
 
Automated Experimentation in Social Informatics
Automated Experimentation in Social InformaticsAutomated Experimentation in Social Informatics
Automated Experimentation in Social Informatics
Aliaksandr Birukou
 
General presentation of the LiquidPub project
General presentation of the LiquidPub projectGeneral presentation of the LiquidPub project
General presentation of the LiquidPub project
Aliaksandr Birukou
 
General presentation of the LiquidPub project
General presentation of the LiquidPub projectGeneral presentation of the LiquidPub project
General presentation of the LiquidPub project
Aliaksandr Birukou
 
LiquidPub: Services at Service of Science
LiquidPub: Services at Service of ScienceLiquidPub: Services at Service of Science
LiquidPub: Services at Service of Science
Aliaksandr Birukou
 
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
Aliaksandr Birukou
 

More from Aliaksandr Birukou (19)

Publishing conference proceedings internationally: how does it work
Publishing conference proceedings internationally: how does it workPublishing conference proceedings internationally: how does it work
Publishing conference proceedings internationally: how does it work
 
Технические аспекты публикации на нескольких языках – как правильно связать DOI
Технические аспекты публикации на нескольких языках – как правильно связать DOIТехнические аспекты публикации на нескольких языках – как правильно связать DOI
Технические аспекты публикации на нескольких языках – как правильно связать DOI
 
Conference Identity: persistent identifiers for conferences
Conference Identity: persistent identifiers for conferencesConference Identity: persistent identifiers for conferences
Conference Identity: persistent identifiers for conferences
 
Springer LOD conference portal. Demo paper - screenshots
Springer LOD conference portal. Demo paper - screenshotsSpringer LOD conference portal. Demo paper - screenshots
Springer LOD conference portal. Demo paper - screenshots
 
PersistentIDs and CrossMark for Conference Proceedings
PersistentIDs and CrossMark for Conference ProceedingsPersistentIDs and CrossMark for Conference Proceedings
PersistentIDs and CrossMark for Conference Proceedings
 
Publishing conference proceedings internationally: Tips and tricks
Publishing conference proceedings internationally: Tips and tricksPublishing conference proceedings internationally: Tips and tricks
Publishing conference proceedings internationally: Tips and tricks
 
Linked Open Data about Springer Nature conferences. The story so far
Linked Open Data about Springer Nature conferences. The story so farLinked Open Data about Springer Nature conferences. The story so far
Linked Open Data about Springer Nature conferences. The story so far
 
Creating a dataset of peer review in computer science conferences published b...
Creating a dataset of peer review in computer science conferences published b...Creating a dataset of peer review in computer science conferences published b...
Creating a dataset of peer review in computer science conferences published b...
 
Linked Data Initiatives at Springer Verlag
Linked Data Initiatives at Springer Verlag Linked Data Initiatives at Springer Verlag
Linked Data Initiatives at Springer Verlag
 
Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...Diversity-aware search for people, content, events AND Diversity-aware hiring...
Diversity-aware search for people, content, events AND Diversity-aware hiring...
 
An Integrated Solution for Runtime Compliance Governance in SOA
An Integrated Solution for Runtime Compliance Governance in SOAAn Integrated Solution for Runtime Compliance Governance in SOA
An Integrated Solution for Runtime Compliance Governance in SOA
 
20101112 librinnovando liquidpub
20101112 librinnovando liquidpub20101112 librinnovando liquidpub
20101112 librinnovando liquidpub
 
Liquid Journals - Intro and RoadMap
Liquid Journals - Intro and RoadMapLiquid Journals - Intro and RoadMap
Liquid Journals - Intro and RoadMap
 
Liquid Journals. Overview. How social computing and liquid knowledge will sha...
Liquid Journals. Overview. How social computing and liquid knowledge will sha...Liquid Journals. Overview. How social computing and liquid knowledge will sha...
Liquid Journals. Overview. How social computing and liquid knowledge will sha...
 
Automated Experimentation in Social Informatics
Automated Experimentation in Social InformaticsAutomated Experimentation in Social Informatics
Automated Experimentation in Social Informatics
 
General presentation of the LiquidPub project
General presentation of the LiquidPub projectGeneral presentation of the LiquidPub project
General presentation of the LiquidPub project
 
General presentation of the LiquidPub project
General presentation of the LiquidPub projectGeneral presentation of the LiquidPub project
General presentation of the LiquidPub project
 
LiquidPub: Services at Service of Science
LiquidPub: Services at Service of ScienceLiquidPub: Services at Service of Science
LiquidPub: Services at Service of Science
 
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
Diversity versus accuracy: solving the apparent dilemma facing recommender sy...
 

Recently uploaded

Unit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdfUnit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdf
Thiyagu K
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
Jheel Barad
 
PART A. Introduction to Costumer Service
PART A. Introduction to Costumer ServicePART A. Introduction to Costumer Service
PART A. Introduction to Costumer Service
PedroFerreira53928
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
Celine George
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
rosedainty
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
Steve Thomason
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
Balvir Singh
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Thiyagu K
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
Tamralipta Mahavidyalaya
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
kaushalkr1407
 
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdfESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
Fundacja Rozwoju Społeczeństwa Przedsiębiorczego
 
2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
Sandy Millin
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
Vivekanand Anglo Vedic Academy
 
Synthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptxSynthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptx
Pavel ( NSTU)
 
How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17
Celine George
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
GeoBlogs
 

Recently uploaded (20)

Unit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdfUnit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdf
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
 
PART A. Introduction to Costumer Service
PART A. Introduction to Costumer ServicePART A. Introduction to Costumer Service
PART A. Introduction to Costumer Service
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
 
Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)Template Jadual Bertugas Kelas (Boleh Edit)
Template Jadual Bertugas Kelas (Boleh Edit)
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
 
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdfESC Beyond Borders _From EU to You_ InfoPack general.pdf
ESC Beyond Borders _From EU to You_ InfoPack general.pdf
 
2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
 
Synthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptxSynthetic Fiber Construction in lab .pptx
Synthetic Fiber Construction in lab .pptx
 
How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
 

Peer Review in the LiquidPub project

  • 2. Metric Dimensions Quality Fairness Efficiency Statistics Kendall Distance Divergence Disagreement Biases Robustness Unbiasing Effort invariant alternatives Effort vs. quality Min. criteria
  • 3.
  • 4. Divergence Metric ©√ Nt = 1/n Normalized t Nt = n/n N-Divergence Ndiv(1,n, C) = n-1/n ©√ When the second ranking is random, we have: indipendent correlated inv. correlated
  • 5. prior vs. after discussion ca. 74 (36%) contributions have been effected by the discussion phase
  • 6. Results: peer review ranking vs. citation count Div Normalized t
  • 7.  
  • 8.
  • 9. Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 > + |min bias| 13% 5% 9% 7% < - |min bias| 12% 4% 8% 7%           C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
  • 10.
  • 12. The road ahead Real-Time accuracy estimation Speed Ranking Ranking vs. marking

Editor's Notes

  1. For instance if t =1, n = 4 p1(0) = probability that the first number is different from the first in the ordered ranking p1(1) 0 probability that the first number is equal to the first in the ordered ranking
  2. Check numbers
  3. Notice that if we restricted the world to the top N papers and accepted only 30% of them, or 20% of them, we would have essentially a random process. The bottom right suggests that it is not true that for weaker papers we can have a clear decision We were not looking for this results Acceptance rate is over-rated
  4. Randomness in review Due to many reasons: subjective reviews limited size of the sample for a given reviewer absolute subjective marking vs. relative ranking