More Related Content
Similar to Understanding scientific peer review
More from Samir Haffar
Understanding scientific peer review
- 1. Understanding scientific peerreview Samir Haffar M.D. Assistant Professor of Gastroenterology
- 2. 1st documented descriptionof a peer review process Ishāq bin Ali Al-Rahawi (854 – 931)
- 3. 1st documented descriptionof a peer review process • This book states that it is the duty of a physician to make duplicate notes of the patient’s condition on each visit. • When the patient had been cured or had died, the notes of the physician were to be examined by a local council of physicians, who would assess whether the physician had practiced according to the standard of that era
- 4. Peer review toadjudicate scientific publications “Royal Society of London” in 1752
- 5. Peer review toadjudicate scientific publications • The “Royal Society of London” established in 1752 a committee to assess publications submitted to their journal “Philosophical Transactions” • The purpose was to help the editor choose appropriately- themed articles and not to strictly assess their quality
- 6. Peer review aswe know it now • Began in the United States in the early 1900s • First journals to implement it Science JAMA American Practitioner
- 7. Introduction of photocopiers(1959) Peer review process was facilitated by introduction of photocopiers, which allowed production and circulation of multiple copies of submitted articles to external reviewers without fear of losing the original documents
- 8. • Assisting editorialdecisions as to whether accept or reject a paper • Helping to improve the quality of submitted papers Peer review is perceived as contributing in 2 ways
- 9.
- 10. Systems of scientificpeer review • Single-blind peer review Reviewers aware of authors’ identity but not vice versa Norm in clinical medicine and biomedical journals • Double-blind peer review Authors & reviewers not aware of each other’s identity Widely used in imaging, nursing and humanities journals • Open peer review Authors & reviewers known to each other BMJ moved to an open review more than a decade ago 70 journals from BMC use variable models of open peer review BMC: BioMed Central
- 11. Single-blind peer review •Advantages Reviewers free from authors’ influence Forthright in critiques without fear of conflicts • Disadvantages Advantageous to well-known authors Detrimental to junior authors Discrimination against authors because of nationality, native language, gender or institution
- 12. Double-bind peer review •Advantages: Offering all advantages of single-blind peer review May protect authors from reviewers’ bias • Disadvantages: Certain abuse persist in single or double-blind review Plagiarism of non published data for personal benefit Deliberate delay of publication to usurp the authors Take longer time: time to uncover author’s identity They were often able to identify authors correctly • Effect on quality: No difference in quality or decision to publish in small trials between blinded/unblinded reviewers
- 13. Reasons for blindingfailure • Reviewers successfully identify the authors as often as 25-50% of the time in both biomedical & social journals • Common reasons for blinding failure: Reviewer familiarity with authors’ work Authors inadvertently revealed their identity in manuscript
- 14. How can authorsavoid blinding failure? Use of the first person Change “we have shown that..” to “Jones et al. have shown that..” Excessive citations from a single author or institution Citations that are in press Use of the author’s initials in the manuscript: to add after acceptance Acknowledgments: can be written after acceptance Failure to remove names of institutions from images & drawings Use of figures from previous publications Krinsky G. AJR 1999;172:1474
- 15. Open peer review •Advantages: Increases transparency Reviewers more constructive in their critiques Could help prevent plagiarism • Disadvantages: Possible animosities between authors & reviewers Higher decline rate from reviewers to participate Increase in time taken to write reviews • Effect on quality: Contradictory and inconclusive Despite the lack of evidence in favor of open reviewing, many journals began to implement or experiment with this system. 70 journals from BMC now use variable models of open peer review
- 16. Peer review asa source of bias Difficult to detect & quantify • No significant results Can be denied publication Go to multiple journals leading to delay Published in low impact factor journals Published in languages other than English • Reviewers ask authors Delete or combine outcomes Modify analysis Perform post hoc subgroup analyses These actions can introduce classical biases
- 17. Possible role ofpeer review to introduce bias
- 18. Which system ofpeer review is better? • Difficult to answer the question with certitude based on present data • Each system has advantages & challenges • Both double blind & open peer reviews more supported at present • Double-blind peer review appears less biased at face value Burdens on editorial staff reduced by a checklist system & software
- 19. Peer review processis far from perfect “Peer review process is expensive, slow, prone to bias, open to abuse, anti-innovatory, and unable to detect fraud” Dr. Richard Smith Former editor of The BMJ
- 20. Fraud in peer-reviewedjournals • John Darsee (US cardiologist) Considered a shining star in his field Published over 100 papers & abstracts at Harvard & Emory Colleagues were suspicious of his works & reported him in 1981 Withdrawn: Emory (8 papers, 32 absts), Harvard (9 papers, 21 absts) • Diederik Stapel (Dutch psychologist) Award from “Society of experimental Social Psychology” Dean of social and behavioral sciences faculty/Tilburg university 2010 The University suspended Staple's data in September 2011 At least 55 publications in peer-reviewed journals were affected
- 21. Despite the continuedcriticism of peer review, few would vote for its abolition, and most would advocate for its improvement, so more research is needed Rennie D, Flanagin A. Three Decades of Peer Review Congresses. JAMA 2018;319:350-53.
- 22. How to improvethe peer review process?
- 23. Groups involves inpeer review • Authors Wish to publish their work for career promotion or secure funding • Editors Under pressure to identify novel & sound research • Reviewers Try to fit a time-consuming process with busy schedules & often without remuneration or academic credit • Publishers Compete in a transforming landscape of fast and abundant science publishing All these groups should be engaged collectively to improve the peer review process https://doi.org/10.1016/j.pan.2017.08.012
- 24. Role of authors •Should adhere to instructions set by journals when they prepare their manuscripts • Penelope* is an automated tool that checks manuscripts and gives immediate feedback to authors before submission • Would be sound to allow authors the right to appeal against decision of editors & reviewers when they find it to be unfair * https://www.penelope.ai
- 25. Role of editors •Few studies of editors’ behaviors have been made, and little is known beyond what they reveal themselves • A judicious editor can play an adjudicating role and prevent many of the potential abuses of reviewers such as hostile comments, unsubstantiated criticisms, and delay of competitors’ manuscripts • Some argue for code of practice for editors which could help1 (1) Newton DP. Accountability in research 2010;17(3):130-45.
- 26. Role of reviewers •Reviewers trained in epidemiology & statistics, of younger age, and who spent up to 3 hours on a review, have been shown to be associated with higher review quality in one study • Using standard checklists by reviewers rather than giving a report in their own words could be helpful • Publons* a platform focusing on peer review, has developed Publons Academy which organizes practical peer review training courses for early career researchers * https://publons.com/home/
- 27. Review Quality Instrument(Version 3.2) van Rooyen S et al. J Clin Epidemiol 1999;52:625–62.
- 28. Review Quality Instrument(Version 3.2) van Rooyen S et al. J Clin Epidemiol 1999;52:625–62.
- 29. Checklist to reviewmanuscripts Only for guidance – reviewers not obliged to answer all questions Title Is the title informative? Is it too long? Does it relate to content of article? Key words Are the keywords appropriate? Do they reflect the content of the article? Abstract Is the abstract structured? Is there an aim and a hypothesis? Introduction Do authors explain background of problem & list recent relevant studies? Materials/methods Is the materials and methods section structured? Are following subheading used: Study design, Subjects, Methods,…. Subjects Do authors follow the recommended reporting guidelines for their study? Are precise inclusion and exclusion criteria provided? Methods Are methods explained in sufficient detail? Statistical analysis Do the authors explain their rationale for using different tests? Are the name, version and manufacturer of statistical programs provided? Results Do the authors explain any missing values? Are the tables informative? Are column & row titles informative? Discussion Do the authors describe the limitations of their study? Do the authors draw clear conclusions based solely on their results? References Are the references up-to-date and formatted according to journal style? Smolčić VŠ, Šimundić AM. Biochemia Medica 2014;24(3):321–8
- 30. Role of publishers •There is too little sound research on peer review • The American Association for the Advancement of Science, publisher of Science, announced the PRE action which makes peer review material available alongside published papers • Lee and Moher, specialists in publication science at University of Washington and Ottawa Hospital Research Institute, call publishers to enable opening of what they name “black box of peer review” It is now time to conduct systematic experimental studies to assess various peer review practices and improve quality & transparency PRE: Peer Review Evaluation
- 32. You can readreviewers’ comments & authors’ reply
- 33. International congress onpeer review and scientific publication • Launched by American Medical Association and British Medical Association in 1986 • The congress is held every 4 years • Activities of this congress are reported in special themed issues of JAMA http://www.peerreviewcongress.org
- 34. 8th International congresson peer review and scientific publication (Chicago - Sep 2017) • Encouraged systematic research on peer review • Introduction of simple but mandatory checklist in editorial process: 4 criteria: randomization, blinding, sample size, and exclusions that if properly reported might reduce the risk of bias • Requirement for registration of randomized clinical trials • Assessments of early positive experiences with data sharing Among trials registered at ClinicalTrials.gov*: 33% completed trials & 57% terminated trials were not published * Rennie D, Flanagin A. Three Decades of Peer Review Congresses. JAMA 2018;319:350-53.
- 35. Role of policymakers • Policy makers are interested and invest in peer review • The EU initiated the project PEERE (www.peere.org) • Multidisciplinary international project running from 2014 to 2018, collects quantitative & qualitative research on how peer review is conducted, & seeks to make evidence- based recommendations to improve the current system PEERE: new frontiers of peer review
- 36. As long ashumans are involved in judging the scientific quality of a manuscript, a completely bias-free system is unlikely
- 37. Conclusion • Peer reviewprocess is far from perfect • All of its systems entwined with advantages & challenges • Double-blind review system is supported on equipoise & fair-play principles • There is little sound research on peer review, it is time to perform large experimental systematic studies on various peer review practices to improve quality & transparency
- 38. • Improve research •Channel appropriately novel findings & observations • Detect fraud when it occurs Despite these limitations and concerns, the peer-review process remain an invaluable tool to
- 39.