Peer review

295 views

Published on

This is a copy of a talk i gave at the Vision Sciences Society meeting in 2012. The talk presentas a review of the history and evolution of peer review and scientific publishing, an evaluation of its current function based on survey data, and a proposal for improving the system.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
295
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Peer review

  1. 1. The state of scientific publishing and a new model Dwight Kravitz Chris Baker
  2. 2. Motivation: Improving Cost and Quality NIH Budget / # Papers Budget per Paper ($1000) Published
  3. 3. Peer Review: Outline• History and evolution• Modern purpose• Improving peer review• Post-publication
  4. 4. History of Peer Review and Publishing• Problems facing authors – Disseminating results – Claiming ownership of results• Publishing – Publishers had distribution channels – Provides a record of when results were
  5. 5. History of Peer Review and Publishing• Problems facing publishers – Establishing veracity of results – Deciding which results to publish given limited• Peer review space – Stamp of scientific approval – Prioritizes papers for Henry Oldenburg, 1665 inclusion by publishers Medical Essays and Observations, 1731
  6. 6. 55% Reject42% Revise3% Accept
  7. 7. 55% Reject42% Revise3% Accept
  8. 8. Given Revision: 98% Accept 2% Reject
  9. 9. 64% of papers are rejected at least once
  10. 10. Direct Costs of the Current System• Total Number of Reviews: 6.3 (2-15)• Total Days under Review: 122 (21-321)• Total Hours for Revisions: 68 (5-300)• Total Days to Publish: 221 (21- 533)• Publication Fees: $2000• Thousands of dollars and hundreds of hours per paper• Long delays impair every form of assessment• Variance hinders planning research programs and careers
  11. 11. Peer Review: Outline• History and evolution – Invention – Current form• Modern purpose• Improving peer review• Post-publication
  12. 12. Original Purpose• Reasons for publishing – Disseminating results – Claiming ownership of results• Reasons for peer review – Establishing veracity of results – Deciding which results to publish given limited space Henry Oldenburg, 1665 Medical Essays and Observations, 1731
  13. 13. Modern Purpose• Reasons for publishing – Disseminating results – Claiming ownership of results• Reasons for peer review – Establishing veracity of results Abstract: 9495 – Deciding which results to Full Text: 8564 publish given limited PDF: 2549 space
  14. 14. Pre-reception• Peer review allows authors to test the paper before sending it to the whole field – Catches fundamental errors – Highlights important overlooked issues
  15. 15. Problems with Pre-reception• Reviews are impure measures of quality
  16. 16. Problems with Pre-reception• Reviews are impure measures of quality• Review process is opaque to the wider field
  17. 17. Prioritization of the Literature• Loop ranks papers based on which journal publishes them – Provides a way of deciding which papers to read – Compensates for long publication lags
  18. 18. Problems with Prioritization• Reviews not focused solely on quality• Inefficient (avg. lag is 221 days)• Creates detrimental short-term incentives
  19. 19. Problems with Prioritization Budget per Paper ($1000) http://pmretract.heroku.com
  20. 20. Problems with Prioritization Budget per Paper ($1000) Retractions per 100k Papers http://pmretract.heroku.com
  21. 21. Problems with Prioritization Fang &Casadevall, 2011
  22. 22. Problems with Prioritization• Reviews not focused on solely on quality• Inefficient (avg. lag is 221 days)• Creates detrimental short-term incentives• Doesn’t work
  23. 23. Problems with Prioritization
  24. 24. Problems with Prioritization
  25. 25. Problems with Prioritization
  26. 26. Problems with Prioritization
  27. 27. Peer Review: Outline• History and evolution – Invention – Current form• Modern purpose – Pre-reception – Prioritization• Improving peer review• Post-publication
  28. 28. Improving the current system• Make publishing more efficient and less variable • Days to Publish: 221 (21- 533) • Years to First Cite: ~1.5 + research• Capture reviews in the prioritization • Total Number of Reviews: 6.3 • Total Hours of Reviewing: ~31.5Decouple peer review rather than• Focus reviews on science from publishing
  29. 29. Proposed System
  30. 30. Decoupling peer review from publishing• Allows reviewers to focus on scientific merit rather than publication• Allows reviews to be captured for prioritization and rewards for reviewers• Makes publishing simpler and more efficient• Eliminates the short-term incentive to produce least publishable units
  31. 31. Peer Review: Outline• History and evolution – Invention – Current form• Modern purpose – Pre-reception – Prioritization• Improving peer review – Decoupling from publishing• Post-publication
  32. 32. Format for papers
  33. 33. The need for post-publication review Proportion of field Why Shru Useful Field ? g Datapoint Altering
  34. 34. The need for post-publication review We need more reviews to Proportion of field approximate these complex distributions. Also enables personalized prioritization of the literature. Why Shru Useful Field ? g Datapoint Altering
  35. 35. Incentivizing post-publication review
  36. 36. Organizing the literature• The utility of modern peer review and publishing is prioritizing and organizing the literature.
  37. 37. Organizing the literature• The utility of modern peer review and publishing is prioritizing and organizing the literature.
  38. 38. Peer Review: Outline• History and evolution – Invention – Current form• Modern purpose – Pre-reception – Prioritization• Improving peer review – Decoupling from publishing• Post-publication review

×