Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The peer review process


Published on

Presentació realitzada per Ana Marušić en el marc del Seminari sobre la revisió per experts (peer review) que va tenir lloc a la Facultat de Biblioteconomia i Documentació de la UB el 20 de juny de 2011, dins el marc del programa de doctorat “Informació i Documentació en la Societat del Coneixement”. Aquest seminari va ser organitzat conjuntament amb l'EASE (European Association of Science Editors).

Published in: Technology, Education
  • Be the first to comment

The peer review process

  1. 1. The peer review process (II) Ana Marušić Croatian Medical Journal University of Split School of Medicine
  2. 2. What is the peer review process?
  3. 3. Einstein’s answer to Physical Review in 1936
  4. 4. Reviewer comments• Comments to editor• Comments to author• Reviewer recommendations and comments may not agree, may even be contradictory• Reviewers are consultants, not decision makers• Editors are decision makers, ask them if you have questions about reviewer comments, editorial decision, or the process
  5. 5. Ethical responsibilites of reviewers• Declare competing interests• Ensure that reviewer is qualified (= a peer)• Inform the editor who actually did the review (e.g. if passed onto a colleague)• Treat material in confidence• Take steps to avoid biased review• Deliver courteous and timely reviews
  6. 6. Reviewer misconduct• Cistron submitted DNA sequence for interleukin-1 (IL-1) to Nature• Paper reviewed by Gillis (Immunex): reject• Sequence published in PNAS (corrected)• Cistron and Immunex file patents for IL-1• Immunex patent contains 7 errors from original (rejected) Nature paper• Cistron sues Immunex ($21mn settlement) Rennie 1999
  7. 7. Advice to novice reviewers eview/mistakes/index.htm#
  8. 8. Advice to novice reviewers: Confidentiality• Researchers who agree to participate as reviewers take on the responsibility to provide authors and/or applicants with the best reviews possible. There is a trust recognized between the reviewer and author(s) of any work reviewed.
  9. 9. Advice to novice reviewers: Confidentiality• The reviewer has completed a review within the prescribed time frame and sent the review response form back to the editor. It contained comments intended for the editor, including a recommendation to accept with minor revisions, and helpful suggestions that the authors of the manuscript might incorporate to improve the paper. This reviewer, who shares an office with another researcher, has left the manuscript with the accompanying comments, in a common area that is used by both researchers. The document remains in the area for a two week period during which time his officemate happens to peruse the manuscript.
  10. 10. Advice to novice reviewers: ConfidentialityHas the reviewer acted responsibly?No, this represents a violation of confidentiality.Yes, since the reviewer has already submitted the reviewform with comments.
  11. 11. Advice to novice reviewers: Impartiality• The selection criteria for peer reviewers include relevant research expertise, sound judgment, good communication skills, and objectivity. Funding agencies and journal editors rely on peer reviewers to make recommendations to accept, reject, or revise proposals/manuscripts based primarily on their scientific merit.
  12. 12. Advice to novice reviewers: Impartiality• A chemistry professor specializing in efforts to develop a safe and efficient chemical warfare agent decontamination system is solicited by a journal editor to review a manuscript that is very close to work she is currently engaged in.• The abstract describes experiments that are quite similar to the experiments being conducted by the chemistry professors collaborators. The potential reviewer realizes that there are only a handful of researchers specializing in this area who could competently provide an assessment of the manuscript, and feels that accepting the request to review would be the responsible thing to do.
  13. 13. Advice to novice reviewers: Impartiality• At this point, the chemistry professor has just seen the title and abstract. However, the chemistry professor also realizes that by reviewing the complete paper, she will be privy to protocols that could aid in completing her own work.How should the chemistry professor proceed?• Agree to review the manuscript, then decide whether she can remain impartial after reading the complete document.• Notify the editor about her concerns about impartiality before reading the complete document.
  14. 14. Advice to novice reviewers: Responsiveness• Participation in peer review is considered a valuable service to science in general and to ones field of study in particular. The main mission of peer review is to assess the quality of research to be conducted, in the form of proposal submissions, or research completed in the form of manuscript submissions. Reviewers base their recommendations to funding agencies or journal editors on their research expertise and experience.
  15. 15. Advice to novice reviewers: Responsiveness• An editor for a high profile cellular biology journal contacts a biologist with an expertise in cellular regeneration, and request that he review a manuscript on this topic. The biologist is very intrigued, because the authors of the paper describe an innovative approach in the abstract.• The editor asked that the review be completed and submitted within two weeks. Although the time frame would be reasonable during a different time of the year, the biologist has several research investigations currently demanding his attention as well as a presentation at an international conference to complete. Although the biologist estimates he will be unable to review the manuscript for at least three to four weeks, he accepts the editors request.
  16. 16. Advice to novice reviewers: ResponsivenessHas the reviewer responded in a responsible manner?• Yes, since he has every intention of providing a comprehensive review.• No, despite his intention to provide a comprehensive review, he knows he will not do it within the agreed time frame.
  17. 17. Advice to novice reviewers: Disclosure of competing interests• While conflict is a common human experience, it can become an issue in research integrity when the conflicts (financial, personal, political), are between interest and duties. According to Shamoo (1992, 1993) and Resnik (2001), most of the concerns with COIs arise because personal interest can undermine duties relating to scientific objectivity.
  18. 18. Advice to novice reviewers: Disclosure of competing interests• A researcher from an academic institution receives a request by a program officer to participate in a round of reviews for proposals in an area in which she has considerable background. The review process is a single blind review where she will know the name of the authors of the proposals, but the reviewers will remain anonymous to the authors.• The researcher agrees to participate because she feels the opportunity to review proposals will enhance her professional service record. She is given a number of proposals to read, but is assigned to take the lead on reviewing one proposal in particular. As the researcher proceeds with reviewing this assigned proposal, she realizes that she may have a conflict of interests with one of the authors.
  19. 19. Advice to novice reviewers: Disclosure of competing interests• Earlier in her career, she and the author in question had a mutual affiliation with a research endeavor at a private research lab. There collaborative efforts resulted in several unpublished reports. Since the affiliation was more than five years ago, and the duration of the affiliation was less than nine months, the researcher is uncertain whether she should identify and certify this on the pre-meeting and post-meeting Conflict of Interest Certification forms (COI_Information.pdf, 2005).
  20. 20. Advice to novice reviewers: Disclosure of competing interestsHow should the reviewer proceed with this proposal?• Say nothing to the program officer and submit the review without any mention of a possible COI.• Say nothing to the program officer until submitting the review that includes a disclosure of possible COI.• Decides to say nothing unless it is brought up by the program officer.• Reveal this conflict of interest immediately and let the program officer determine how to proceed.
  21. 21. Advice to novice reviewers: Review quality• Proposal applicants and authors rely on peer reviewers providing objective, comprehensive and fair assessments of their proposals and submitted manuscripts. Reviewer comments and recommendations can significantly enhance the quality of research to be conducted, or make necessary modifications of conclusion drawn. The quality of the review depends on the required level of demonstrated expertise, research objectivity, and sufficient time spent in the activity.
  22. 22. Advice to novice reviewers: Review quality• A researcher at a private research institute maintains a busy schedule with responsibilities conducting and coordinating multiple research projects. She receives a request to participate in a round of proposal reviews for a national research funding agency. Because she has previously received funding from this agency, she feels obliged to participate, even though there is a strong likelihood she will not be able to provide a comprehensive review of the assigned proposals. Despite her reservations, the researchers strong sense of obligation may lead her to accept the request against her better judgment.
  23. 23. Advice to novice reviewers: Review qualityHow should the researcher proceed?• Decide to review proposals despite her reservations.• Immediately decline to review proposals, but provide an explanation to the program officer.• Wait several weeks before deciding not to review proposals, providing an explanation to the review committee for her inability to participate and the delayed response.• Wait several weeks before deciding not to review the proposals, and provides no explanation to the review committee for either herunavailability to participate or her delayed response.
  24. 24. Advice to novice reviewers: Constructive criticism• Journal editors attempt to select peer reviewers based on their competencies in areas relevant to the submitted manuscript. Useful and practical comments can be incorporated in later revisions, enhance the quality of the research, and hopefully, improve the chances for acceptance of a submitted paper.
  25. 25. Advice to novice reviewers: Constructive criticism• A peer reviewer is perusing a submission that details a study with the objective of examining the superiority of the newly developed instrument to measure a general mental health of a person. The author, a graduate student submitting her first manuscript, designed the study with 20 healthy volunteers, 1) each one used the standard Medical Outcome Study (MOS) mental health scale soon after they entered the study and 2) after 3 months they used the newly developed instrument to measure the response.
  26. 26. Advice to novice reviewers: Constructive criticism• The data consisted of pre (MOS scale) and post (new instrument) study results on 20 subjects. The author decided to test whether the mean post-study result is better than the mean pre-study result using a standard pooled t- test.• The peer reviewer, who has a substantial background in statistics, disagrees with the selection of the standard pooled t-test. The reviewer comments that “had the author invested in a basic statistic text and bothered to read it, he would have selected this test instead of wasting the reviewers time”.
  27. 27. Advice to novice reviewers: Constructive criticismHas the reviewer provided useful information to the author?• Yes, the reviewers comment provide useful information on how to address the problem.• No, the reviewers comment did not provide useful information on how to address the problem.
  28. 28. Advice to novice reviewers: Objectivity• According to Rockwell (2006), researchers serving as reviewers are obliged to judge in a fair manner and objectively the quality and significance of the work under review. "He/she is obligated to support and encourage publication of work of high quality while appropriately challenging flawed work."
  29. 29. Advice to novice reviewers: Objectivity• A senior researcher with an established reputation in a specialized field is routinely sought after by funding agencies and editorial boards to review proposals and submitted articles. Her current level of demand contrasts sharply with her early professional experience following completion of her doctorate, when the researchers quest for a research position took almost 1 ½ years. She holds a personal bias against several institutions that chose not to hire her following completion of her doctorate. The researcher has agreed to review a number of proposals for a research funding agency.
  30. 30. Advice to novice reviewers: Objectivity• As she begins reviewing her first proposal, she recognizes that the principal investigator is affiliated with one of the institutions she holds a personal bias against. There is a question as to whether her bias against the institution may also extend to researchers affiliated with this institution.
  31. 31. Advice to novice reviewers: ObjectivityHow should the researcher proceed?• Engage in a self-reflection regarding her ability to impartially review this proposal.• Decide to review the proposal without any self-reflection.• Decide to recuse herself from reviewing this proposal without any self-reflection.• After engaging in self-reflection, decide that her bias impacts objectivity and ask to recuse herself from reviewing the proposal.
  32. 32. Questions?
  33. 33. BMJ Question 1. is the paper important?• Has the research addressed a question that had to be answered, or is it just “another brick in the wall”?• The question matters more than the answer. If the question was important and the answer is valid, then it doesn’t matter if the answer is negative or boring.• Is this something that clinicians or scientists, policy makers, or the public need to know, remembering that there’s more for them to know than they can possibly know?
  34. 34. BMJ Question 2. Is the paper original?• Ideally, you know the literature in the field covered in the paper you have been asked to review; wise to conduct a literature search before you write your review• Almost all of a series of RCTs described as “the first” in major journals were not the first.• Has this never been done before?• If the question has been addressed before does this add importantly (for example, a much bigger or better designed study; first time in this population)?
  35. 35. BMJ Question 2. Is the paper original? (continued)• Remember that some things that are “well known” are not based on any evidence.• If you think the research unoriginal please give us references to previous work. Don’t just say “it’s unoriginal.”• If there are other important studies that the authors don’t reference, please provide references
  36. 36. BMJ Questions 3 and 4. Is the Introduction section appropriate?• Good explanation of study context/background?• Is there a brief, relevant literature review?• Are the aims of the study, and study hypothesis/question stated clearly?
  37. 37. BMJ Question 5: Are the research methods valid?• Identify the strengths as well as the weaknesses of the study methods• Is the design right for answering the research question?• Were the data collected adequately? Was the sampling right?• Are the methods described adequately and completely?• Are the analyses right? Should they be redone?
  38. 38. BMJ Question 5: are the research methods valid? (continued)• If you are not strong in statistics, just say so and focus on you’re the areas you know best• At a minimum, check the data in the abstract, text, and tables for consistency; add up some of the data in the tables and point out obvious errors
  39. 39. BMJ Question 5: Are the methods valid – do the paper and the information reported follow ethical requirements?• For research involving human participants – Was the study reviewed and approved by an ethics review committee? – Was informed consent appropriately obtained?• For research involving animals – Were guidelines for the protection of animals in experiments followed• For case reports – Are patients privacy protected? If patients are identifiable, was appropriate permission obtained
  40. 40. BMJ Question 6. Are the results presented adequately?• Outcomes or observations in logical order?• For quantitative research - are data presented numerically, with measures of statistical significance and variability?• For qualitative research – are observations reported in categories or themes?
  41. 41. BMJ Question 7. Is the Discussion section appropriate?• Does it provide an answer to the research question?• Are the findings discussed in light of other relevant literature?• Are the study limitations clearly and completely described?
  42. 42. BMJ Question 8: are the conclusions reasonable?• Are the conclusions supported by the data or evidence in the paper?• Do the conclusions go beyond the merits of the paper
  43. 43. BMJ Question 9 – are the tables and figures appropriate?• Are they cited in the text?• Are they overly simple or too complicated?• Are the data in the tables or figures redundant with each other or the text?• Are the data clearly presented?• Check data for consistency• Are there too many or too few tables and figures?
  44. 44. BMJ Question 10: are the references appropriate?• Are the references up to date?• Are they relevant?• Are they any important, relevant reference that are missing?
  45. 45. BMJ Question 11: is the paper appropriate for the journal’s readers?• Know the journal and the journal’s readership• If uncertain, look for the journal’s mission statement or description (often available online); may also be in the journal’s instructions for authors• If you’re still uncertain, don’t comment on this point
  46. 46. BMJ Question 12. Are the title and abstract adequate?• Assess title and abstract after you have reviewed the manuscript, tables, and figures• Does the title give a clear message about the study? Is it too long? Too short? Is a subtitle needed?• Is the abstract structured? Consistent with the text, tables, and figures? Complete?
  47. 47. BMJ Question 13: is the writing clear?• Do not point out every typo, spelling error, and grammatical mistake• Provide general comment on the clarity of the paper• Is the manuscript too long or too short?• Remember to be constructive
  48. 48. Manuscript and cover letter• Manuscript presents facts and figures.• Cover letter is your advertisment.
  49. 49. Cover letter1. Manuscript title and authors’ names2. Statement that the manuscript has not been published or is under consideration for publishing elsewhere (abstract up to 400-500 words is not considered previous publication)3. Reasons why you think the journal should publish you article4. (description of individual author’s contributions)
  50. 50. Editor’s Decisions• Acceptance• Revise and Resubmit• Rejection
  51. 51. How to Reply to Request for Revision• Address editor’s concerns• Address reviewers’ concerns• Itemize replies to editor’s and and reviewer’s comments• If you disagree with a comment, explain• Do not ignore comments• Ask for deadline or timing of when revision should be submitted• Request for revision is not a guarantee of acceptance
  52. 52. Sample author response letter – JAMA March 6, 2010 Dear Dr. Editor, We are pleased to submit our revised manuscript “Smoking Status is a Clinical Indicator for Alcohol Misuse in Adults” (JAMA06-4240) for your consideration for publication in JAMA. We thank you and the reviewers for the careful consideration that was given to the original version of the manuscript. We have addressed the issues raised by each reviewer in the revision and describe how we have addressed each issue below. Each of the co-authors has reviewed and approved of the revision. Please let us know if you have any additional questions. My contact information is listed below. Thank you for the opportunity to revise this manuscript. Sincerely, Dr Author
  53. 53. Comments of Reviewer A1. Introduction: This is much too long. It appears to have been lifted straightout of your grant proposal.The introduction has been shortened to 400 words.2. Methods, page 9: Please explain why you included these control variables.In light of this comment, the analyses are no longer adjusted for these controlvariables. As suggested, we conducted additional analysis to examine whetherthe original associations, and we found that our adjusted and unadjustedoutcomes were not substantially different. See manuscript page 8 for thisadditional information.3. Results, page 10: How did you calculate the response rate?The response rate was derived by multiplying the household response rate(89%) by the person response rate (93%) and the sample frame response rate(99%). This method consistent with the JAMA recommended source documentfor calculating response rates for multistage sample designs. See manuscriptpage 9 for this explanation.4. Confidence intervals for all measures of effect would improve thepresentation of the results.We have provided either standard errors or confidence intervals for allreported effects in the text and the tables.
  54. 54. Conclusions: What you should expect from the editorial process• Prompt acknowledgment and timely assessment• If reviewed, a rigorous review• Confidentiality• Fairness• Clear, straightforward, and timely communication
  55. 55. Timeline for publishing research