SlideShare a Scribd company logo
Future of Peer Review




                 Image by James Yang http://www.jamesyang.com


       Maria Kowalczuk, PhD
Deputy Biology Editor, BioMed Central
Cartoon by Nick D Kim, strange-matter.net
Role of peer review
• Journal peer review is just a step in the scientific
  process.
• Popular only since 1960s.
• Online publishing and open access have changed
  the publishing landscape.
• Traditional peer review models are single blind
  peer review and double blind peer review.
Criticism of peer review
• Slow
• Expensive
• Inconsistent
• Biased
• Nepotistic
• Open to abuse
Innovative peer review models
• Open peer review
Open (non-anonymous) peer review




Randomised Controlled Trial (BMJ 1999; 318: 23 – 27):
- no effect on report quality, recommendation, or time taken to review
- increased likelihood of reviewers declining to review
Publishing peer review documents




- In all 4 EMBO publications, including EMBO J, EMBO Reports
-‘Peer Review Process File’ shows all referee reports , author responses and editorial
decision letters
- Referees remain anonymous; opt-out is possible
- 95% of take-up rate; willingness of referees to review unchanged
Randomised Controlled Trial (BMJ 2010;341:c5729):
- no effect on report quality or recommendation
- higher refusal rate and reviewers take longer
Open peer review – BMC-series
• Medical BMC-series titles and some independent journals
  operate open peer review and pre-publication history
  visible to all
• Full transparency crucial in medical research
• Full credit to the referees
• Both referees and editors are more accountable for
  decisions
• Biologists are more sceptical, but BMC Cancer publishes a
  lot of basic science with open peer review
Peer review at Biology Direct – key aspects
• The author suggests suitable reviewers from the journal's
  Editorial Board
• An article is rejected if three Editorial Board Members do not
  agree to review it
• If three EBMs formally agree (after initial skim-reading),
  authors can pursue publication (with or without revision),
  Editor-in-Chief has final word on publication
• Reviewers’ comments and authors’ responses are published
Peer review at Biology Direct
Innovative peer review models
• Open peer review
• Minimal re-review
Re-review opt-out – BMC Biology




Authors can opt out of re-review; if the editors judge the revisions sufficient,
the article is published, often accompanied by a critical Commentary.
Discussed in Editorial: http://www.biomedcentral.com/1741-7007/11/18.
Minimising re-review - JCB
Innovative peer review models
• Open peer review
• Minimal re-review
• Journal cascades
Journal cascades




                         Flagships


                      Subject-specific   BMC
BMC series               journals
BMC Neuroscience                         Independent
BMC Public Health                        Journals
                    BMC Research Notes
Neuroscience Peer Review Consortium
Innovative peer review models
•   Open peer review
•   Minimal re-review
•   Journal cascades
•   Separating interest from soundness
Separating interest from soundness
Whatever the model, peer review aims to establish
• whether the research and its description/ interpretation is
  sound
• level of interest

The importance of interest levels and extent of advance is
  different depending on the editorial goals of the journal

Separating soundness from interest level: BMC series, new
  ‘Mega journals’
Journals publishing all sound science

•   BMC Research Notes
•   SpringerPlus
•   PLoS ONE
•   Biology Open (BiO)
•   Scientific Reports (NPG)
•   PeerJ
•   F1000Research
Innovative peer review models
•   Open peer review
•   Minimal re-review
•   Journal cascades
•   Separating interest from soundness
•   Separating peer review from journal
Conclusions
   Traditional peer review is widely used but also widely
    criticised.
   Innovative models have been proposed to:
     - increase transparency of the peer review process,

     - reduce the publication cost and burden on

       reviewers,
     - increase speed of peer review and publication,

     - involve the scientific community in the process.
Conclusions
   Successful innovations:
     - open peer review,

     - journal cascades.

   New initiatives that need to be tested further:
     - peer reviewing manuscripts outside journals,

     - post publication peer review.
Thank you!
     Maria Kowalczuk, PhD
Deputy Biology Editor, BioMed Central
Maria.Kowalczuk@biomedcentral.com

More Related Content

What's hot

Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05
Wouter Gerritsma
 
JBM Flyer
JBM FlyerJBM Flyer
How Often Are Studies Wrong?
How Often Are Studies Wrong?How Often Are Studies Wrong?
How Often Are Studies Wrong?
Ivan Oransky
 
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
plumanalytics
 
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
DeVonne Parks, CEM
 
The Retraction Watch Database
The Retraction Watch DatabaseThe Retraction Watch Database
The Retraction Watch Database
Ivan Oransky
 
Bastian twin-beasts-2015
Bastian twin-beasts-2015Bastian twin-beasts-2015
Bastian twin-beasts-2015
Hilda Bastian
 
Conference on Vaccines
Conference on VaccinesConference on Vaccines
Conference on Vaccines
OMICS International
 
Evaluating e reference
Evaluating e referenceEvaluating e reference
Evaluating e reference
Elaine Lasda
 
Health Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature SearchingHealth Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature Searching
Jamie Halstead
 
Manuscript Management Systems: Understanding the Workflow from Submission to ...
Manuscript Management Systems: Understanding the Workflow from Submission to ...Manuscript Management Systems: Understanding the Workflow from Submission to ...
Manuscript Management Systems: Understanding the Workflow from Submission to ...
Amado III Tandoc, MD, DPSP
 
Retractions, Peer Review, and Transparency
Retractions, Peer Review, and TransparencyRetractions, Peer Review, and Transparency
Retractions, Peer Review, and Transparency
Ivan Oransky
 
Complexities and approaches to predatory publishing
Complexities and approaches to predatory publishingComplexities and approaches to predatory publishing
Complexities and approaches to predatory publishing
C0pe
 
4th World STM Journal Forum / CAST July 28, 2021
4th World STM Journal Forum / CAST July 28, 20214th World STM Journal Forum / CAST July 28, 2021
4th World STM Journal Forum / CAST July 28, 2021
C0pe
 
Literature Searching for BSc Health Promotion 2014
Literature Searching for BSc Health Promotion 2014Literature Searching for BSc Health Promotion 2014
Literature Searching for BSc Health Promotion 2014
JoWilson13
 
COPE General Intro Core Practices
COPE General Intro Core PracticesCOPE General Intro Core Practices
COPE General Intro Core Practices
C0pe
 

What's hot (16)

Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05
 
JBM Flyer
JBM FlyerJBM Flyer
JBM Flyer
 
How Often Are Studies Wrong?
How Often Are Studies Wrong?How Often Are Studies Wrong?
How Often Are Studies Wrong?
 
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
Practical Uses of Altmetrics - Hear librarians talk about how they use new me...
 
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
January 13, 2016 NISO Webinar: Ensuring the Scholarly Record: Scholarly Retra...
 
The Retraction Watch Database
The Retraction Watch DatabaseThe Retraction Watch Database
The Retraction Watch Database
 
Bastian twin-beasts-2015
Bastian twin-beasts-2015Bastian twin-beasts-2015
Bastian twin-beasts-2015
 
Conference on Vaccines
Conference on VaccinesConference on Vaccines
Conference on Vaccines
 
Evaluating e reference
Evaluating e referenceEvaluating e reference
Evaluating e reference
 
Health Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature SearchingHealth Promotion Introduction To Literature Searching
Health Promotion Introduction To Literature Searching
 
Manuscript Management Systems: Understanding the Workflow from Submission to ...
Manuscript Management Systems: Understanding the Workflow from Submission to ...Manuscript Management Systems: Understanding the Workflow from Submission to ...
Manuscript Management Systems: Understanding the Workflow from Submission to ...
 
Retractions, Peer Review, and Transparency
Retractions, Peer Review, and TransparencyRetractions, Peer Review, and Transparency
Retractions, Peer Review, and Transparency
 
Complexities and approaches to predatory publishing
Complexities and approaches to predatory publishingComplexities and approaches to predatory publishing
Complexities and approaches to predatory publishing
 
4th World STM Journal Forum / CAST July 28, 2021
4th World STM Journal Forum / CAST July 28, 20214th World STM Journal Forum / CAST July 28, 2021
4th World STM Journal Forum / CAST July 28, 2021
 
Literature Searching for BSc Health Promotion 2014
Literature Searching for BSc Health Promotion 2014Literature Searching for BSc Health Promotion 2014
Literature Searching for BSc Health Promotion 2014
 
COPE General Intro Core Practices
COPE General Intro Core PracticesCOPE General Intro Core Practices
COPE General Intro Core Practices
 

Similar to Future of peer review

BioMed Central , an editorial perspective
BioMed Central, an editorial perspectiveBioMed Central, an editorial perspective
BioMed Central , an editorial perspective
BioMedCentral
 
BIOMEDICAL .ppt
BIOMEDICAL .pptBIOMEDICAL .ppt
BIOMEDICAL .ppt
AnandKumar459862
 
Open access workshop wits - 24th october 2013 - copy
Open access workshop   wits - 24th october 2013 - copyOpen access workshop   wits - 24th october 2013 - copy
Open access workshop wits - 24th october 2013 - copy
Nkaba Senne
 
Understanding scientific peer review
Understanding scientific peer reviewUnderstanding scientific peer review
Understanding scientific peer review
Samir Haffar
 
An Overview of BioMed Central and the Growth of Open Access Publishing_
An Overview of BioMed Central and the Growth of Open Access Publishing_An Overview of BioMed Central and the Growth of Open Access Publishing_
An Overview of BioMed Central and the Growth of Open Access Publishing_
biomedcentraljp
 
PeerJ - Innovative Publishing Models Panel at OpenCon 2014
PeerJ - Innovative Publishing Models Panel at OpenCon 2014PeerJ - Innovative Publishing Models Panel at OpenCon 2014
PeerJ - Innovative Publishing Models Panel at OpenCon 2014
Right to Research
 
Publishing in ajp lung 4 21
Publishing in ajp lung 4 21Publishing in ajp lung 4 21
Publishing in ajp lung 4 21
amcever
 
Publishing in academic journals medicine and health
Publishing in academic journals medicine and healthPublishing in academic journals medicine and health
Publishing in academic journals medicine and health
uoblibraries
 
Reporting research: writing papers, responding to reviewers, and appealing
Reporting research: writing papers, responding to reviewers, and appealingReporting research: writing papers, responding to reviewers, and appealing
Reporting research: writing papers, responding to reviewers, and appealing
Trish Groves
 
The journal club (education purpose)
The journal club (education purpose)The journal club (education purpose)
The journal club (education purpose)
Harikesh Maurya
 
How to select a journal to publish Article
How to select a journal to publish ArticleHow to select a journal to publish Article
How to select a journal to publish Article
NileshMarkad
 
Journal knowhows
Journal  knowhows Journal  knowhows
Journal knowhows
Prof Viyatprajna Acharya
 
Systematic Reviews in the Health Sciences
Systematic Reviews in the Health SciencesSystematic Reviews in the Health Sciences
Systematic Reviews in the Health Sciences
Becky Morin
 
systematic review : why & How
systematic review : why & Howsystematic review : why & How
systematic review : why & How
Hesham Al-Inany
 
May 17 editors ag_mexico city
May 17 editors ag_mexico cityMay 17 editors ag_mexico city
May 17 editors ag_mexico city
higiniols
 
3-7-systematic-review-abawi-2021.ppt
3-7-systematic-review-abawi-2021.ppt3-7-systematic-review-abawi-2021.ppt
3-7-systematic-review-abawi-2021.ppt
DrSandeepKautish
 
EVIDENCE BASED PUBLIC HEALTH.pptx
EVIDENCE BASED PUBLIC HEALTH.pptxEVIDENCE BASED PUBLIC HEALTH.pptx
EVIDENCE BASED PUBLIC HEALTH.pptx
Dr Qurat-ul-ain ayuob
 
Systematic review & meta analysis
Systematic review & meta analysis Systematic review & meta analysis
Systematic review & meta analysis
Ahmad Abed Ahmadi
 
presentation-1what-do-we-know-about-peer-review.ppt
presentation-1what-do-we-know-about-peer-review.pptpresentation-1what-do-we-know-about-peer-review.ppt
presentation-1what-do-we-know-about-peer-review.ppt
Imran807905
 
Critical appraisal
Critical appraisalCritical appraisal
Critical appraisal
Chai-Eng Tan
 

Similar to Future of peer review (20)

BioMed Central , an editorial perspective
BioMed Central, an editorial perspectiveBioMed Central, an editorial perspective
BioMed Central , an editorial perspective
 
BIOMEDICAL .ppt
BIOMEDICAL .pptBIOMEDICAL .ppt
BIOMEDICAL .ppt
 
Open access workshop wits - 24th october 2013 - copy
Open access workshop   wits - 24th october 2013 - copyOpen access workshop   wits - 24th october 2013 - copy
Open access workshop wits - 24th october 2013 - copy
 
Understanding scientific peer review
Understanding scientific peer reviewUnderstanding scientific peer review
Understanding scientific peer review
 
An Overview of BioMed Central and the Growth of Open Access Publishing_
An Overview of BioMed Central and the Growth of Open Access Publishing_An Overview of BioMed Central and the Growth of Open Access Publishing_
An Overview of BioMed Central and the Growth of Open Access Publishing_
 
PeerJ - Innovative Publishing Models Panel at OpenCon 2014
PeerJ - Innovative Publishing Models Panel at OpenCon 2014PeerJ - Innovative Publishing Models Panel at OpenCon 2014
PeerJ - Innovative Publishing Models Panel at OpenCon 2014
 
Publishing in ajp lung 4 21
Publishing in ajp lung 4 21Publishing in ajp lung 4 21
Publishing in ajp lung 4 21
 
Publishing in academic journals medicine and health
Publishing in academic journals medicine and healthPublishing in academic journals medicine and health
Publishing in academic journals medicine and health
 
Reporting research: writing papers, responding to reviewers, and appealing
Reporting research: writing papers, responding to reviewers, and appealingReporting research: writing papers, responding to reviewers, and appealing
Reporting research: writing papers, responding to reviewers, and appealing
 
The journal club (education purpose)
The journal club (education purpose)The journal club (education purpose)
The journal club (education purpose)
 
How to select a journal to publish Article
How to select a journal to publish ArticleHow to select a journal to publish Article
How to select a journal to publish Article
 
Journal knowhows
Journal  knowhows Journal  knowhows
Journal knowhows
 
Systematic Reviews in the Health Sciences
Systematic Reviews in the Health SciencesSystematic Reviews in the Health Sciences
Systematic Reviews in the Health Sciences
 
systematic review : why & How
systematic review : why & Howsystematic review : why & How
systematic review : why & How
 
May 17 editors ag_mexico city
May 17 editors ag_mexico cityMay 17 editors ag_mexico city
May 17 editors ag_mexico city
 
3-7-systematic-review-abawi-2021.ppt
3-7-systematic-review-abawi-2021.ppt3-7-systematic-review-abawi-2021.ppt
3-7-systematic-review-abawi-2021.ppt
 
EVIDENCE BASED PUBLIC HEALTH.pptx
EVIDENCE BASED PUBLIC HEALTH.pptxEVIDENCE BASED PUBLIC HEALTH.pptx
EVIDENCE BASED PUBLIC HEALTH.pptx
 
Systematic review & meta analysis
Systematic review & meta analysis Systematic review & meta analysis
Systematic review & meta analysis
 
presentation-1what-do-we-know-about-peer-review.ppt
presentation-1what-do-we-know-about-peer-review.pptpresentation-1what-do-we-know-about-peer-review.ppt
presentation-1what-do-we-know-about-peer-review.ppt
 
Critical appraisal
Critical appraisalCritical appraisal
Critical appraisal
 

Future of peer review

  • 1. Future of Peer Review Image by James Yang http://www.jamesyang.com Maria Kowalczuk, PhD Deputy Biology Editor, BioMed Central
  • 2. Cartoon by Nick D Kim, strange-matter.net
  • 3. Role of peer review • Journal peer review is just a step in the scientific process. • Popular only since 1960s. • Online publishing and open access have changed the publishing landscape. • Traditional peer review models are single blind peer review and double blind peer review.
  • 4. Criticism of peer review • Slow • Expensive • Inconsistent • Biased • Nepotistic • Open to abuse
  • 5. Innovative peer review models • Open peer review
  • 6. Open (non-anonymous) peer review Randomised Controlled Trial (BMJ 1999; 318: 23 – 27): - no effect on report quality, recommendation, or time taken to review - increased likelihood of reviewers declining to review
  • 7. Publishing peer review documents - In all 4 EMBO publications, including EMBO J, EMBO Reports -‘Peer Review Process File’ shows all referee reports , author responses and editorial decision letters - Referees remain anonymous; opt-out is possible - 95% of take-up rate; willingness of referees to review unchanged
  • 8.
  • 9. Randomised Controlled Trial (BMJ 2010;341:c5729): - no effect on report quality or recommendation - higher refusal rate and reviewers take longer
  • 10. Open peer review – BMC-series • Medical BMC-series titles and some independent journals operate open peer review and pre-publication history visible to all • Full transparency crucial in medical research • Full credit to the referees • Both referees and editors are more accountable for decisions • Biologists are more sceptical, but BMC Cancer publishes a lot of basic science with open peer review
  • 11.
  • 12. Peer review at Biology Direct – key aspects • The author suggests suitable reviewers from the journal's Editorial Board • An article is rejected if three Editorial Board Members do not agree to review it • If three EBMs formally agree (after initial skim-reading), authors can pursue publication (with or without revision), Editor-in-Chief has final word on publication • Reviewers’ comments and authors’ responses are published
  • 13. Peer review at Biology Direct
  • 14. Innovative peer review models • Open peer review • Minimal re-review
  • 15. Re-review opt-out – BMC Biology Authors can opt out of re-review; if the editors judge the revisions sufficient, the article is published, often accompanied by a critical Commentary. Discussed in Editorial: http://www.biomedcentral.com/1741-7007/11/18.
  • 16.
  • 18. Innovative peer review models • Open peer review • Minimal re-review • Journal cascades
  • 19. Journal cascades Flagships Subject-specific BMC BMC series journals BMC Neuroscience Independent BMC Public Health Journals BMC Research Notes
  • 21.
  • 22. Innovative peer review models • Open peer review • Minimal re-review • Journal cascades • Separating interest from soundness
  • 23. Separating interest from soundness Whatever the model, peer review aims to establish • whether the research and its description/ interpretation is sound • level of interest The importance of interest levels and extent of advance is different depending on the editorial goals of the journal Separating soundness from interest level: BMC series, new ‘Mega journals’
  • 24. Journals publishing all sound science • BMC Research Notes • SpringerPlus • PLoS ONE • Biology Open (BiO) • Scientific Reports (NPG) • PeerJ • F1000Research
  • 25.
  • 26.
  • 27.
  • 28. Innovative peer review models • Open peer review • Minimal re-review • Journal cascades • Separating interest from soundness • Separating peer review from journal
  • 29.
  • 30.
  • 31.
  • 32. Conclusions  Traditional peer review is widely used but also widely criticised.  Innovative models have been proposed to: - increase transparency of the peer review process, - reduce the publication cost and burden on reviewers, - increase speed of peer review and publication, - involve the scientific community in the process.
  • 33. Conclusions  Successful innovations: - open peer review, - journal cascades.  New initiatives that need to be tested further: - peer reviewing manuscripts outside journals, - post publication peer review.
  • 34. Thank you! Maria Kowalczuk, PhD Deputy Biology Editor, BioMed Central Maria.Kowalczuk@biomedcentral.com

Editor's Notes

  1. I am part of the team of five Biology and Medical Editors at BioMed Central. Our role is maintaining best practice in peer review across all BioMed Central journals, as well as consistency in advising on all aspect of research and publication ethics. The Biology and Medical Editors are responsible for defining BioMed Central’s editorial policies and work closely with in-house editors, external academic editors and the Committee on Publication Ethics (COPE) to ensure that the journals’ editorial processes are consistent with recognized best practice in scholarly publishing.
  2. This is the picture that comes to mind when thinking about peer review process.
  3. “Science is the ultimate peer review environment, and journal peer review has a limited and particular place and role within it.” Kent Anderson, Scholarly Kitchen.Initially manuscripts were assessed only by journal editors. Peer review became more widespread in 1960s with the increasing specialization of science and a rise in submissions and publications.Publishing landscape has changed with the advent of online publishing and open access. The technological revolution has affected the way referees are selected.Widely used, traditional peer review models are single blind peer review and double blind peer review.
  4. Expectations of a reasonable time to publication vary hugely by field. In much of maths and physics, much scientific comms happens in advance of peer reviewed publication, and timescales of a year or more can be seen as fairly acceptable. In life sciences, peer reviewed publication is key, and delays due to reviewers asking for additional experiments etc are increasingly causing concern.
  5. Also operated by BMJ OPEN2006: Nature’s open peer review trial:Despite enthusiasm for the concept, open peer review was not widely popular, either among authors or by scientists invited to comment.http://www.nature.com/nature/peerreview/debate/nature05535.html
  6. Biology Direct type model - in theoretical areas, the disagreements over interpretation between reviewers and authors are key to the intellectual contribution of the paper, so publishing the debate and the difference of opinion is key.There are also limits on the number of times a particular EBM reviews mss from particular authors.
  7. Again – non-OA journals experiment with peer review models, too. The re-review opt-out is proving very popular – Journal of Cell Biology has recentlytaken on a similar approach.
  8. A cross-publisher initiativeTransfers of articles can also happen inter-publisher via initiatives such as the neuroscience peer review consortium
  9. BMC series introduced this idea of separating Soundness from level of interest, and having an inclusive policy, publishing sound research but highlighting the best. This is the basis of several new megajournals – e.gPloS One, SpringerPlus.
  10. This seems to be the only journal that actually depends on post-publication peer review. A lot of journal allow comments (all BMC ones do), and altmetrics, but this is only in addition to pre-publication peer review.Some journals publish ‘provisional’ versions of articles and then allow public comments and discussion, in parallel with formal peer review. E.g. Atmospheric Chemistry and Physics, PeerJPrePrints, Arxive and other pre-print servers.
  11. Traditional peer review is widely used but also widely criticised.A number of innovative models have been proposed to increase transparency of the peer review process, reduce the publication cost and burden on reviewers, increase speed of peer review and publication, and involve the scientific community in the process.Journals have been successful in implementing open peer review in a number of biomedical journals, and decreasing the number of times a manuscript is peer reviewed by enabling transfer of manuscripts and reviewer reports between journals.New initiatives of peer reviewing manuscripts outside journals and post publication peer review still need to be tried and tested.
  12. Traditional peer review is widely used but also widely criticised.A number of innovative models have been proposed to increase transparency of the peer review process, reduce the publication cost and burden on reviewers, increase speed of peer review and publication, and involve the scientific community in the process.Journals have been successful in implementing open peer review in a number of biomedical journals, and decreasing the number of times a manuscript is peer reviewed by enabling transfer of manuscripts and reviewer reports between journals.New initiatives of peer reviewing manuscripts outside journals and post publication peer review still need to be tried and tested.