Analyzing Peer Review
•Download as PPTX, PDF•
2 likes•3,101 views
I present an insider’s view on peer review drawing from my experience at the journals of the American Physical Society (Physical Review B, Physical Review Letters, and Physical Review X) where I have worked since 2003. First, I discuss the basic elements of peer review (editorial screening, rejection without external review, referee selection, consultation with Editorial Board Members, assessment of referee reports, handling of conflicting referee recommendations, selection of a subset of accepted papers for highlighting). In the process, I present some commonly used arguments by authors that can actually backfire, and some anecdotal excerpts of correspondence. Second, I discuss some recent trends in science publishing, from launching new journals to providing new services to authors. I focus on one recent trend, the highlighting of select sets of papers by publishers. Third, I discuss citation impact metrics for journals (Impact Factor, EigenFactor, h5 index) and for subsets of journals (e.g., Editors’ Suggestions, papers highlighted in APS Physics, etc.). This leads naturally to the questions (a) whether editors and referees can pick out, at the time of acceptance, the papers destined to be highly cited or otherwise influential; and (b) whether such papers tend to be controversial at the time of publication and after. I present some data on these questions. Overall, my aim is for the audience to appreciate the imperfect and imprecise nature of editorial decision-making that is sometimes unappreciated by a community trained in the hard sciences. Finally, for the benefit of the younger audience, I present a brief outline of the editorial job and career prospects of editors.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Analyzing Peer Review
Similar to Analyzing Peer Review (20)
Recently uploaded
Recently uploaded (20)
Analyzing Peer Review
- 1. Manolis Antonoyiannakis Associate Editor, Phys. Rev. B Bibliostatistics Analyst, American Physical Society Adj. Assoc. Res. Scientist, Columbia University Analyzing Peer Review Nanoscale Quantum Optics - Early Stage Researchers Workshop COST Action MP1403 Malta, November 2015
- 2. Outline 2 1. Basic elements of peer review 2. Some recent trends in science publishing 3. Citation impact metrics for sets of papers (journals & subsets of journals) 1. Outline of editorial job & career prospects
- 4. Inside the Physical Review Editorial Office • ~ 40 in-house editors (PRA, PRB, PRC, PRD, PRE, PRL, PRX, PR Applied, Physics) • ~ 60 remote editors (active researchers) • ~ 100 support staff • PRB: 16 editors (10 full-time, 5 part-time, 1 Lead Editor)
- 5. RMP PRL PRE PRA PRB PRC PRD Physical Review 1993 1893 1998 PRST-AB 2005 PRST-PER 2008 PHYSICS 2011 PRX 2014 PR Applied {Open Access Electronic Only 1929 RMP 1958 PRL 1970 PR splits into ABCD 1913 APS takes over Physical Review APS Journals
- 6. The American Physical Society is not so American (or is it?) North America 24% Latin America 4% Europe 38%Middle East & Africa 4% Indian Subcontine nt 4% Japan 7% Pacific Rim 19% North America 35% Latin America 3% Europe 49% Middle East & Africa 2% Indian Subcontine nt 1% Japan 5% Pacific Rim 5% 29 countries of origin for Physical Review Editors (2010) Submissions Referees Editors (origin) Editors (origin)
- 7. Comprehensive reports in Condensed Matter and Material Physics, Publishes 4,000—5,000 annually EigenFactor® 0.59 (2nd in all Physics); Impact Factor 3.7 (1st within scope) Short reports in all areas of applied, fundamental, and interdisciplinary physics Publishes 2,500—3,000 annually EigenFactor® 0.94 (1st in all Physics) Impact Factor 7.5 A limited number of papers from all areas of pure, applied, and interdisciplinary physics; Publishes ~200 annually EigenFactor® 0.01 Impact Factor 9.0 Established in 2014. Applied Physics with emphasis on the intersection between Physics and Engineering. Publishes Regular Articles, Letters, and Reviews. Published 116 in 2014, projected to 250 in 2015. Credit: A. Chantis (PRB)
- 8. Review process at Physical Review peer review internal review (by editor) review by Editorial Board Member (EBM) Appeal to Editor in Chief (procedural only) Appeal to Editor 3rd round (if needed) 2nd round 1st round New paper Review process in a nutshell 8
- 9. • Help good papers get published as quickly as possible • Filter out unsuitable papers by editorial rejection & peer review • Add value to papers: • Improve papers via editorial & peer review • Select the best papers to highlight: in Physics; as Editors’ Suggestions; or recommend in APS Tip Sheet to popular press • Help researchers become skilled referees “advance and diffuse the knowledge of physics” 9 Editor’s Role: Select & promote quality research through rigorous peer review
- 10. Challenges for Editors • Influential papers are often controversial • Experts’ judgment not always faultless • Editors’ own knowledge of field & people is limited • Editors’ time constraints (3-4 NEW papers daily/editor) • Selective journals are subjective by definition: 41st chair effect • Social, cultural factors affect behavior of authors & referees and can thereby affect the fate of papers • Responsive, conscientious, knowledgeable referees are hard to find
- 11. What is it? An editorial rejection letter, upon initial receipt, with editors’ judgment of impact / innovation / interest / significance / importance Why? To preserve time & effort of referees (our most precious resource)… … and help authors find a better-suited journal with minimal delay How do editors decide? Red flags that may warrant editorial rejection - Obvious marginal extension or incremental advance; too specialized - Subject matter or readership does not fit - Sloppy presentation, opaque writing / too much jargon & acronyms -introduction: lacks clarity, no context, poorly describes prior work, no broad picture, too focused on technical details, no compelling motivation -referencing: too many old / specialized / self- / ‘confined’ references - no punch-line in conclusions: what is the main message of the paper? why is it important? how does it advance the field? Rejection Without External Review (RWER) 11
- 15. “This is fine as far as it goes. From here on, it’s who you know.” How do the editors find referees for a paper?
- 16. We look for referees in: • references (authors of, referees of) • related papers in Web of Science, Google Scholar, SPIN, NASA, APS database (authors, citing papers) • suggested referees • referee expertise in APS database (>60,000 referees) • mental database We generally avoid: • Undesirable referees • Coauthors (current or previous) • Referees at same institution as authors • Acknowledged persons • Direct competitors (if known) • Busy referees (currently reviewing for PR/PRL) • Overburdened referees (> 15 mss/past year) • Consistently slow referees (>8 weeks to review) • Referees who consistently provide poor reports How do the editors find referees for a paper? 16
- 17. While on the topic of refereeing… Physics Today, Sept. 2005, p. 43 On 30 July, Tate replied that he regretted Einstein’s decision to withdraw the paper, but stated that he would not set aside the journal’s review procedure. In particular, he wrote, “I could not accept for publication in THE PHYSICAL REVIEW a paper which the author was unwilling I should show to our Editorial Board before publication.” In 1936, Einstein submits a paper to the Phys. Review. It gets reviewed and returned to the authors with a critical report. Einstein is upset & withdraws the paper. Einstein, in conversations with colleague Robertson, realizes there was an error. He publishes the correct results in the J of the Franklin Institute. Who was the referee whose report upset Einstein so much? Why, Robertson!
- 18. How not to argue for your paper: An insider’s view Credit: K. Dusling (PRL)
- 19. Typical misunderstandings & faulty arguments when corresponding with editors This subject is very important, so you should publish my paper. Not every paper on an important topic warrants publication in a high- profile journal The broader subject may have broad interest, but what about this paper? The referee found no mistake, (s)he only said it is not interesting. Two referees recommend publication, only one does not. Many papers on this topic have been published in PRL, see .... Correctness is necessary but not sufficient for publication. So what? Look at what the referee said. It is the content of a report that matters, not the vote. So, enough already. This is an argument against publication, not for publication... 19 Credit: D van Heijnsbergen (PRL)
- 20. I am entitled to two rounds of review and expect the editor to have another two referees look at my paper Although two rounds of review are common, they are not guaranteed. I have published 234 papers and have an h-index of 42. How can the editor reject my paper? The editor has no research experience in this field. How can they reject my paper without external review? You published that prior paper which is clearly less sophisticated than ours We are mindful of the authors’ prior record, especially in borderline cases. But we focus on the paper at hand. The editor approaches the paper as a general reader, and over time, builds considerable experience. Also, she may have discussed the paper with (a) other editorial colleagues, or (b) with an Editorial Board Member. Peer review is a complex & imperfect process. Journals are ‘distributions’: some papers clearly deserved publication, others barely made it. Maybe the prior paper was in a field that was hot at the time, and the bar was lower. Etc. Typical misunderstandings & faulty arguments
- 21. As seen from the authors’ perspective - Referee comments wrong / unjustified? RRR - Referee does not understand my paper? RRR - Referee biased / unfair / has competing interest? RRR - Editor wrongly sides with the critical referee? RRR - Referee asks me to cite irrelevant papers? RRR - Editor does not provide clear yes/no decision? RRR - Editor does not firmly reject my paper? RRR Revise, Respond & Resubmit (RRR): A common[*] 1st-round remedy 21 [*] But not universal. See next slide.
- 22. However, please keep in mind that the Editors need a clear reason to publish Try to be a stricter judge for your paper than the referees / editors would be Ask yourself (honestly): Would it be a mistake for the editors NOT to publish your paper? 22
- 23. Useful resources for authors (1) “Whitesides’ Group: Writing a Paper”, George M. Whitesides, Advanced Materials 16, 1375 (2004) A classic paper on how to write scientific papers that every researcher should read. (2) “What Editors Want”, Lynn Worsham, The Chronicle of Higher Education, September 8, 2008 http://chronicle.com/jobs/news/2008/09/2008090801c.htm A journal editor reveals the most common mistakes academics make when they submit manuscripts. (3) Strunk and White, The Elements of Style (MacMillan: New York 1979, 3rd ed. So successful that it is known not by its title but as “The Little Book”. Check out APS tutorials on authoring & refereeing Some editorial talks are found on internet (Google search)
- 24. George Whitesides on writing a paper http://pubs.acs.org/userimages/ContentEditor/1305035664639/Whitesides-ACS-Writing-a-Scientific-Paper.pdf https://www.youtube.com/watch?v=q3mrRH2aS98 ______________________________________________________________________
- 25. Outline 25 1. Basic elements of peer review 2. Recent trends in science publishing 3. Citation impact metrics for sets of papers (journals & subsets of journals) 1. Outline of editorial job & career prospects
- 26. “O.K., let’s slowly lower in the grant money.” Recent trends in science publishing: Worldwide R&D investment tops $1.5 trillion[*] [*] 2014 Global R&D Funding Forecast, http://www.battelle.org/docs/tpp/2014_global_rd_funding_forecast.pdf
- 27. 27 Growth of research papers
- 28. Growth of research papers A century of physics Roberta Sinatra, Pierre Deville, Michael Szell, Dashun Wang & Albert-László Barabási Nature Physics 11, 791–796 (2015) doi:10.1038/nphys3494 Exponential!
- 29. 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 Vietnam war begins WWII Published Papers per year by all APS journals Credit: A. Chantis (PRB) Exponential!
- 30. Publishers respond to growth of research output & competition in various ways 30 0. Allow old journals to grow 1. Launch new journals E.g.: From 2004-2014, Thomson Reuters adds 102 new “physics” journals (35% increase) New journals are: - Broad & interdisciplinary - small & exclusive - niche & low-impact But also… 2. Provide select sets of papers (highlights) of higher quality than ‘average’ paper in source journal + more visibility
- 31. Proliferation of journals Journal Publisher Launch Year 2014 size Optics Express OSA 1997 3306 Nano Letters ACS 2001 1100 Nature Materials NPG 2002 146 Nature Physics NPG 2005 124 Nature Nanotechnology NPG 2006 135 PLoS One PLoS 2006 29778 Nature Photonics NPG 2007 113 ACS Nano ACS 2007 1313 Applied Physics Express JPS 2008 423 Nature Communications NPG 2010 2784 Biomedical Optics Express OSA 2010 346 PRX APS 2011 216 Scientific Reports NPG 2011 3929 Optical Materials Express OSA 2011 279 AIP Advances AIP 2011 561 Photonics Research OSA 2013 42 APL Materials AIP 2013 198 PR Applied APS 2014 115 ACS Photonics ACS 2014 181 Optica OSA 2014 [180] Science Advances AAAS 2015 APL Photonics AIP 2016 N/A
- 32. Proliferation of highlighting services Journal Publisher Highlighting mechanism Launch year Nature NPG News & Views 1926 [*] Science AAAS This Week in Science 1985 Science AAAS Perspective 1989 Science AAAS Editors' Choice 2000 Class Quant Grav IOP IOPselect 2001 EPL IOP IOPselect 2001 Environm Res Lett IOP IOPselect 2001 J Microm Microeng IOP IOPselect 2001 J of Optics IOP IOPselect 2001 J of Physics A: Math and Gen IOP IOPselect 2001 J of Physics AB: AMO IOP IOPselect 2001 J of Physics: Cond Matt IOP IOPselect 2001 Laser Phys IOP IOPselect 2001 Laser Phys Lett IOP IOPselect 2001 Meas Sci and Tech IOP IOPselect 2001 Nanotechnology IOP IOPselect 2001 New J of Physics IOP IOPselect 2001 Physica Scripta IOP IOPselect 2001 Plasma Sources Science & Technology IOP IOPselect 2001 European J of Physics IOP IOPselect 2001 Nature NPG Research Highlight 2003 J Phys Soc Japan JPS Editors' Choice 2003 Nature NPG Editor's Summary 2005 PRL APS Editors' Suggestion 2007 PRB APS Editors' Suggestion 2008 Any APS journal APS Synopsis 2008 Any APS journal APS Viewpoint 2008 Any OSA journal OSA Spotlight on Optics 2009 PRC APS Editors' Suggestion 2012 PRA APS Editors' Suggestion 2013 PRD APS Editors' Suggestion 2014 PRE APS Editors' Suggestion 2014 Any ACS journal ACS ACS Editors' Choice 2014 Science AAAS Research News Numerous IOP journals IOP featured articles 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 Proliferation after 2000 Select papers in APS journals: • Physics (Viewpoint or Synopsis) • Editors’ Suggestion
- 34. Outline 34 1. Basic elements of peer review 2. Recent trends in science publishing 3. Citation impact metrics for sets of papers (journals & subsets of journals) 4. Outline of editorial job & career prospects
- 35. Impact statistics “My question is: Are we making an impact?”
- 36. Citation Impact Metrics Metric Measures Remarks Caution Impact Factor (2-1 years) citations/paper Average metric; large journals cannot have high IF Small journals, highly skewed distributions with outliers EigenFactor (5 years) eigenvector centrality in network of journals market share of reader’s time; scales with total citations h5 (5 years) highest no. papers cited ≥ h5 times High-end metric: no. ‘significant’ papers 0 5 10 15 0 200 400 600 No.papers CY=2013, PY=2011-2012 R² = 0.90655 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 0 200,000 400,000 600,000 800,000 EigenFactor(2013) Total Cites Eigenfactor Score
- 38. h5 ranking, Physics & Mathematics
- 39. 0 5 10 15 20 25 30 35 40 45 50 0 200 400 600 800 1000 2011 Impact Factor Papers published annually Physics Viewpoints Nat Phys PRL Suggestions Nano L Adv Mat Nat Mat RMP Nat Phot Small Adv Fun Mat Impact Factors for journals… …and for highlighted sets of papers PRB Suggestions 1. Large journals cannot have ‘high’ impact factors 2. Highlighted papers are cited considerably above other papers, on average: Viewpointed papers ≈ 3 × PRL PRL Suggestions ≈ 2 × PRL PRB Suggestions ≈ 2 × PRB
- 40. PRL: Metrics for highlighted papers • Manuscript PDF Downloads (relative to average PRL): – Any Highlight: 2.5× – Viewpoint: 3× • Press coverage: – Any Highlight: 40% – Viewpoint: 60% • Citations (relative to average PRL): – Any Highlight: 2.5× – Viewpoint: 3× Credit: K. Dusling (PRL)
- 41. PRB – Citation Impact for Rapids & Suggestions 41 Impact Factors PRB (official IF): 3.7 Rapid Communications “IF”: ~ 5 Editors’ Suggestions “IF”: ~ 6 Thomson Reuters Essential Science Indicators ESI Top Papers Editors’ Suggestions 1 in 20 vs PRB 1 in 100 A new mark of prestige: Editors’ Suggestions cited considerably more than other papers
- 42. • It helps to start with tackling an important problem! • Quality writing cannot be overemphasized (main text, logical cohesion, presentation of the problem and its context; but also title, abstract, introduction, conclusions, references, figures) • Look at previously highlighted papers & their descriptions (Viewpoints, Synopses, Suggestions) to get an idea of which papers are selected • @ selection process, editors scrutinize paper, its potential impact, advance and applicability, the referee comments, etc. What can I do for my paper to be highlighted?
- 43. Leaving the average behind 43 “Much of the world is controlled as much by the “tails” of distributions as by means or averages: by the exceptional, not the mean; by the catastrophe, not the steady drip; by the very rich, not the “middle class.” We need to free ourselves from “average” thinking.” Philip W. Anderson “Some Thoughts About Distribution in Economics,” in W. B. Arthur et al., eds., The Economy as an Evolving Complex System II (Reading, MA: Addison-Wesley, 1997), 566
- 44. 1 1 12 2 23 3 4 How hard is consensus? Top-10 cited PRL’s in 2001-2006 Rounds of Review Publish as is P w/minor edits P w/major edits Review after major edits Reject No recommend. Ed. Board Member recommends acceptance
- 45. 1 1 12 2 23 3 4 How hard is consensus? Top-10 cited PRL’s in 2001-2006 Rounds of Review Publish as is P w/minor edits P w/major edits Review after major edits Reject No recommend. Influential papers are often controversial: Top-10 cited Letters are 10 times more likely to attract a Comment In 10 out of the top-20 cited papers in PRL (published 1991-2000 in plasmonics, photonic crystals & negative refraction) at least one (& sometimes both) reports were negative in the 1st round of review
- 46. Outline 46 1. Basic elements of peer review 2. Recent trends in science publishing 3. Citation impact metrics for sets of papers (journals & subsets of journals) 1. Outline of editorial job & career prospects
- 47. An unconventional career – why? Starting to feel like this… “Sometimes I wonder if there’s more to life than unlocking the mysteries of the universe.” “I’m looking for a position where I can slowly lose sight of what I originally set out to do with my life, with benefits.” …I decided to try working on scholarly publications in a non-profit environment. It has worked for me, so far… And fearing I might end up like this…
- 48. Editorial job in APS Society publisher (non-profit) Leading professional institution Semi-academic environment Job security & stability Opportunities to learn & grow within the job: • learn more physics • writing • design own projects and make them happen (e.g., bibliostatistics, coding, data science) • some exceptionally talented colleagues to learn from Meet new faces, see new places (travel & remote work) Modest salary (Long Island & NY areas are quite expensive) Excellent benefits
- 49. Desired traits of an editor Integrity Service-oriented Critical thinking Emotional intelligence (maturity, humility,…) Communication skills Sense of humor Common sense! Self-motivated & able to work independently Research background in at least one field (typically: PhD + post-doc) Willingness to learn (about physics + people)
- 50. Editorial jobs Websites of interest: Physics Today APS website (APS hires 2-3 editors/year) http://www.aps.org/about/jobs/index.cfm Nature website http://www.nature.com/npg_/work/index.html
- 51. Thank you, and good luck! Feedback? Questions? manolis@aps.org
- 52. Authors’ justification for Rapid Communication The highlight of this work only has two points. Therefore, this work is suitable as a rapid communication. On conflict of interest In view of the content of the paper, some scientist (including those cited as authors of previous investigations) may have some conflict of interest, as they failed to see the new effects. On suggested referees The Referees should be specialists in semiconductor science and not my close colleagues. I seldom suggest them from the reference list, however this time they are either my close colleagues or already passed away. So please find the Referees for this manuscript and in case they agree to be not anonymous, I can try to sort them as suggested or not suggested. On undesirable referees Don't send our MS to those Referees, who pretend to understand our work. Prior submission to other journal I had submitted my paper to another journal where people also did not really like it. Language editing: We have endeavored to seek and destroy any typos that escaped our latest search (we caught two). Editor accusations: Changes in our submission of 28 March, are small in order to avoid accusation of a substantial improvement of my original submission. And on a lighter note: Excerpts from authors’ letters
- 53. Referee will not be terrorized You sent me a reminder yesterday as well - I will not be terrorized by you or anyone else. Final verdict: My recommendation is: Do not resuscitate! Let this paper die. Suggesting alternative referee I suggest you try [Referee X] at U. of Minnesota. Tell him I said he would do it! It is too damn cold there for him to be doing much at the moment. Referee burnout Must pass - referee burnout - overwhelmed and underpaid! For mortals The paper as written is unacceptable for publication in PRL but I sincerely hope that the authors can rewrite this manuscript so that mortals can read and understand. On other referees I would like to point out that your referees (excluding myself) seem to be nincompoops. Clarified manuscript The authors have clarified their manuscript significantly and now it is clear that I do not understand it. And on a lighter note: Excerpts from referee reports
- 54. APS & Columbia colleagues for stimulating discussions & data (A Chantis, A Begley, A Klironomos, A Melikyan, S Kancharla, R Garisto, J Dadap, and others) Columbia University for access to Web of Science and bibliography And to all those authors, and especially the anonymous referees, who make our job possible THANK YOU! Acknowledgments