Christopher Weyant, The New Yorker
Coming: 9th July
The Metric Tide
Report of the Independent Review
of the Role of Metrics in Research
Assessment and Management
Re-thinking: what is good research?
• Transformative understanding
• Structure of DNA, RNAi, Higgs boson
• Transformative methods
• X-ray crystallography, PCR, CRISPR
• Transformative technology
• Silicon chip, graphene
• Transformative impact
• Policy, behaviour, legislation
How do we “measure” good research
• Publication in a high impact factor journal
• Citation counts
• Expert evaluation
Culture sustains traditional (subscription model) publishing
• High bar to entry spurs competition
• High bar to entry slows publication, reduces researcher productivity
• Conservative peer review
• Eye-catching research trumps quality?
• IF-based rewards foster cheating
• Cheating undermines public trust
• Restricts access
• Poor fit to public policy in the digital age
Re-imagining scientific publication in the open era
• Universal preprints (like arXiv)
• Rapid dissemination
• Constructive/collaborative criticism
• Commenter credit
• Open Access Journals
• Open, PLOS-style peer review: original & competent research
• Confirmatory studies or negative results also encouraged
• Reviewer credit
• Competition on service and price: value for money
• Universal access: more readers - more scrutiny
• Accessible Data/Software
• Re-use and re-analysis
• Disincentive to fraud
Downsides and challenges
• Quality concerns of pay-to-publish model?
• Bohannon, Beall, and vanity publishing? Mitigated by openness
• Delegation of quality control to the reader:
mistakes only detected after publication?
• Especially risky for medical research?
• BIG question: How do we get the incentives right?
• Post-publication reward mechanisms that are journal-independent
• Evaluate academics on what they do (research+; see DORA)
• Do we need selective journals to foster high-quality research?
• Do we need journals to foster disciplines & sub-disciplines?