A referee's plea reviewed


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A referee's plea reviewed

  1. 1. Based on Mark Allman’s A Referees Plea presentation - May 21, 2001A REFEREES PLEAREVIEWEDStenio FernandesCIn/UFPENov 2012
  3. 3. • The importance of understanding the peerreview (PR) process In general Avoid silly mistakes Adjust the level of expectancy of your submission To not be upset with some reviews Targeted submissions Know how your paper will be treatedUNDERSTANDING THE PEER REVIEWPROCESS
  4. 4.  Peer review is a very old process Ishaq al-Rahwi (854-931) of Syria first described the PR process The Philosophical Transactions of the Royal Society is the first journalto formalize the peer review process (300 years ago) Validation by peers and publication in a scientific journal the method through which authors register, validate, disseminate, andarchive their discoveries and results. process and the speed at which articles are peer reviewed andpublished are key elements in the appropriate accreditation ofscientific findings. Recent Survey 85% of scientists agree that PR greatly helps scientificcommunication 93% of them believe peer review is necessary.
  5. 5.  Single Blind Review The names of the reviewers are hidden from the author. Traditional method of reviewing (the most common type) Advantage Reviewer anonymity allows for impartial decisions free from influence by theauthors Disadvantages: Authors fear the risk that reviewers working in the same field maywithhold submission of the review in order to delay publication,thereby giving the reviewer the opportunity to publish first Reviewers may use their anonymity as justification for beingunnecessarily critical or harsh when commenting on the author’swork. Prejudice. Yes, it happens all the time!TYPES OF PEER REVIEW (1/3)
  6. 6. TYPES OF PEER REVIEW (1/3)
  7. 7.  Double Blind Review Both the reviewer and the author remain anonymous Advantages: Author anonymity prevents any reviewer bias based on, forexample, an author’s country of origin or previous controversialwork. Articles written by ‘prestigious’ or renowned authors areconsidered on the basis of the content of their papers, rather thanon the author’s reputation. Disadvantage: It is uncertain whether a paper can ever truly be ‘blind’ –especially in specialty ‘niche’ areas. Reviewers can often identifythe author through the paper’s style, subject matter or self-citation.TYPES OF PEER REVIEW (2/3)
  8. 8. TYPES OF PEER REVIEW (2/3)
  9. 9. Open ReviewReviewer and author are known to each otherAdvantage best way to prevent malicious comments, stop plagiarism,prevent reviewers from drawing upon their own ‘agenda’ andencourage open, honest reviewing.Disadvantage the opposite view: a less honest process in which politeness or fear of retribution maycause a reviewer to withhold or tone down criticism. Junior reviewers may hesitate to criticize more esteemed authors forfear of damaging their prospectsTYPES OF PEER REVIEW (3/3)
  10. 10. TYPES OF PEER REVIEW (3/3)
  11. 11.  Again: Understand the Purpose of Peer Review Peer review is a critical element of scholarly publication, and one of themajor cornerstones of the scientific process. Peer Review serves two key functions Acts as a filter: Ensures research is properly verified before being published Improves the quality of the research: rigorous review by other experts helps toimprove key points and correct inadvertent errors On Being Asked To Review Does the article you are being asked to review truly match yourexpertise? Only accept an invitation if you are competent to review the article Do you have time to review the paper? an article will take about 5 hours to review properly Are there any potential conflicts of interest? A conflict of interest will not necessarily eliminate you from reviewing anarticleREVIEWER GUIDELINES – FIRST STEPS
  13. 13.  Referee 2007 – 2007 - IEEE Transactions on Multimedia 2007 – 2007 - Journal of Internet Engineering 2009 – Atual - Computer Communications 2009 – Atual - Peer-to-Peer Networking andApplications Journal 2009 – Atual - Journal of Network and SystemsManagement 2009 – Atual - Journal of Network and ComputerApplications 2008 – Atual - Ad Hoc Networks 2010 – Atual - Journal of The Brazilian ComputerSociety 2011 – Atual - Wireless Communications andMobile Computing 2012 – Atual - Wireless Networks 2009 – Atual - Computer Networks 2012 – Atual - Journal of Communications andNetworks Editorial Board 2012 – Atual - International Journal on Advancesin Intelligent Systems 2012 - Atual - Journal of Mobile and UbiquitousComputing TechnologyReferee IEEE ICC, GLOBECOM,INFOCOM, WCNC, GHTCE,ICNS, CCNC, HPSR, BCNTPC Member IEEE WCNC, CCNC,INFOCOM TMA WORKSHOPCHAIR IEEE TRICANS, ACM HPN CONNEPI 2010 (interestingstuff, long history)GENERAL GUIDELINES FROMEDITORS AND TPC CHAIRS
  14. 14.  the article should not be disclosed to a third party you would be expected to evaluate the article according to thefollowing Originality Is the article sufficiently novel and interesting to warrant publication? Is the research question an important one? Structure Are all the key elements present: abstract, introduction, methodology, results,related work, discussion (lessons learned), conclusions? Title: Does it clearly describe the article? Abstract: Does it reflect the content of the article? Introduction: Does it describe what the author hoped to achieve accurately,and clearly state the problem being investigated? Method: Is there sufficient information present for you to replicate theresearch? Results: has appropriate analysis been conducted? Are the statistics correct? Conclusion/Discussion: Are the claims in this section supported by the results,do they seem reasonable?CONDUCTING THE REVIEW
  15. 15.  Commentary should be courteous and constructive You should indicate whether your comments are your ownopinion or are reflected by the data If the review is not blinded (yes, it’s true!) Do the authors have a "track record" of working in this area? Are they from a reputable institution? Think about the consequences of this? (my question) Subjectively: do you believe what the authors are telling you? do you suspect some consistent error in the hypothesis, methods,analysis of data, etc.? Is there some chance that there is scientific fraud or plagiarisminvolved in this manuscript?CONDUCTING THE REVIEW
  16. 16.  Finally, on balance, when considering the whole article, do thefigures and tables inform the reader, are they an importantpart of the story? Previous Research: If the article builds upon previous research does it reference thatwork appropriately? Are there any important works that have been omitted? Are the references accurate?CONDUCTING THE REVIEW
  17. 17.  RELEVANCE Is the topic of the paper relevant to the journal? Is the paper aimed at the readership? Is the technical depth of the paper relevant to the journal? CONTENT Is the paper needed (does it fill a gap)? Are the references up to date? Is the content (theories, concepts, methodologies) sound? Does the originality of the material warrant publication? PRESENTATION Is the paper well organised? Do any parts require further explanation of clarification?REVIEWER CONFIDENTIAL COMMENTS TOEDITOR - ELSEVIER
  18. 18.  Section I. Overview A. Reader Interest 1. How relevant is this manuscript to the readers of this journal? B. Content Is the manuscript technically sound? What do you see as this manuscripts contribution to the literature inthis field? What do you see as the strongest aspect of this manuscript? What do you see as the weakest aspect of this manuscript? C. Presentation Are title, abstract, and/or keywords satisfactory?REVIEWER CONFIDENTIAL COMMENTS TOEDITOR - SPRINGER
  19. 19.  Please identify and discuss the contribution of this manuscript.Please include in your discussion: Does the paper have significant tutorial content? Is there enough background provided so that the generalist incommunications can understand its main contributions? Elaborate. Does the paper contain original contributions? What is the natureof the contributions? Is there a description of lessons learned that are given to thereader to help the reader avoid pitfalls in his own work? Is there a need for a paper such as this in the communicationscommunity? For example, are there articles that are already available which covermore or less the same topic at about the same depth? Please discuss the quality of the citations in this manuscriptREVIEWER CONFIDENTIAL COMMENTS TOEDITOR - IEEE
  20. 20.  Please comment on the organization of the paper, and offer anysuggestions that you think will improve the paper and itsreadability Please comment on the technical correctness of the manuscriptin general identify any specific technical inaccuracies that you find, and makesuggestions for correcting those If the manuscript does not require major revision, please providea list of minor changes Please provide a summary comment on the overall suitability ofthe paper for publication in IEEE *REVIEWER CONFIDENTIAL COMMENTS TOEDITOR - IEEE
  22. 22.  I now realize that the community generates a large amount of“junk” I dont mean bad ideas Each paper I reviewed for SIGCOMM had at least a nugget of an interestingidea. Definition of Junk: "it took me twice as long to read this as it shouldhave" In other words, the paper was sloppy I had a very difficult time trying to figure out what the paper wasproposing What the experiments were really showing. The following is a short list of *easy* ways to improve yourresearch papers Most comments are general Last comment is directed at the internetworking research community0 - LARGE AMOUNT OF “JUNK”
  23. 23.  It seems to me that everyone has access to a good spellchecker these days. Not using the spell checker just indicates that either you aresloppy or you do not care. If youre sloppy with your writing why should I believe you arenot sloppy with your experiments/analysis as well? If you dont care about your paper, why should I?1 - FIRE UP THE SPELL CHECKER,PLEASE!
  24. 24.  Proofread your paper I see lots of mistakes that indicate the paper was not looked overbefore it was submitted (e.g., "The foo algorithm works betterthan the the bar algorithm."). Again, if you do not take enough care in presenting your work,why should I be compelled to recommend accepting it? Sloppy papers give the indication of sloppy research As well as reflecting poorly on your work, such papers are often verydifficult to read and understand The point of writing a paper is to convey new information to theworld Therefore, there is major value in clear writing. If a referee cant get the point of the paper, or has a very difficult timegetting the main idea, there is a high likelihood that the paper will berejected2 - SLOPPY PAPERS ARE LONG SHOTS
  25. 25.  I print out all papers I review. I read them in my lazy boy. Iscribble notes on them. Do not assume that I am looking at color plots! Color plots come out of my black and white printer very badly in mostcases. Besides, most conferences/journals publish black and white papers Make it easy to tell the difference between the lines on your plots. Do not plot on a grey background. Make the font on the plots readable without a magnifying glass Help me out! Please! Often the plots are key to understanding thepaper If I cannot understand the paper I will recommend rejecting it3 - I READ SUBMISSIONS IN MY LAZYBOY
  26. 26.  You do not need to give an in-depth tutorial of allbackground material. How much background material to include? It depends on the conference to which you are submitting,the maturity of the background work, etc. There are some topics (e.g., TCP congestion control)that are well understood by people attending anynetworking conference. You do not need to describe the algorithms in painfuldetail. A short paragraph re-capping the main ideas andproviding references to the original works shouldsuffice.4 - DONT WASTE MY TIME (I)
  27. 27.  You do not need to provide me with an example of everykind of plotting strategy you use When you first show a plot with notation that I mightnot understand, explain the notation But, using half a page to show me a plot that adds nothing tomy understanding of the topic is a less than compelling useof real estate5 - DONT WASTE MY TIME (II)
  28. 28.  Roughly stick to the page limit and the requested format. Dont make me try to guess how long your paper really isbecause you 1.75-spaced the paper rather than double-spacing it as requested in the call for papers. Editors usually check this Authors have been relying on technical reports to “extend” the paper6 - DONT WASTE MY TIME III
  29. 29.  If you need a constant for the foo algorithm and you definec=17, I want to know why you chose 17. Also, some items in papers jump out at reviewers as odd. For instance, I was recently reviewing a paper that simulateda network with a bottleneck bandwidth of "about T1 rate". Theactual rate of the link turned out to be something like 1.3Mbps. why the authors used 1.3 Mbps rather than using T1 rate (~1.5Mbps)? Even if the explanation you end up with is rough, it is better than noexplanation at all in the vast majority of the cases.7 - ANSWER ALL THE EASY QUESTIONS
  30. 30.  8. Make My Life Easy It is not a super-big deal, but it is always nice to get a paper that is easy to read This includes things like using a very clear font that is of readablesize and numbering the pages8 - MAKE MY LIFE EASY
  31. 31.  Your paper does not solve the entire worlds networkingproblems. Really. Many papers rub reviewers the wrong way by making boldclaims about solving all sorts of problems without any sort ofevidence (or only weak evidence) It is much better to show some restraint and perspective whendiscussing your results. It is alright to solve small problems, or to only showpreliminary results that *may indicate* a solution. Just dont over claim what you have shown.9 - A LITTLE RESTRAINT
  32. 32.  10. Use Units "The length of our data transfers is 1" means nothing. 1 what? 1packet? 1 KB? 1 minute? Sometimes it is obvious from the context -- sometimes it isnt. If youuse units you dont have to worry about it.10 - USE UNITS
  33. 33.  Theorems are quite a necessary part of many papers. However, it is not necessary to give *only* a theorem with nosummary or context. Summaries aid me as a reviewer to gain an intuitiveunderstanding before trying to dig into the math to make surewhat you have it correct thus making my task easier and less error prone, I believe. As a reader of a paper, I often just want the high-order bitwithout slogging into the theorems and proofs to get said bit.11 - WERE ALL NOT MATHEMATICIANS
  34. 34.  One common mistake that authors make is failing to includerelevant references. It is better to err on the side of citing too much related work Next, when previous work must be criticized, do so in aconstructive manner. In other words, do not say something like "Zevons [1] work is easilyshown to be less effective than the work presented in this paper" --Mr. Zevon might be reviewing your paper! Something more along the lines of: "Zevon [1] first introduced the fooalgorithm … We show a more robust algorithm” Finally, do not overstate what a paper you are citing says It is often better to stick with the authors conclusions12 - ON REFERENCES
  35. 35. S. KeshavACM CCR EditorJuly 2011WISE WORDS FROM AWISEMAN
  36. 36. THE ACM SIGCOMM’S CCR - JULY 2011
  37. 37.  13 technical submissions All of them were rejected by the Area Editors on the advice of thereviewers One could ask: were all the papers so terrible? Some submissions were too broad, others too narrow, Some too incremental, some too radical, Some were just not interesting enough I feel that this attitude has permeated our community at large A similar spirit of harsh criticism is used to judge papers at SIGCOMM,MOBICOM, CoNEXT, and probably every other top-tier computer scienceconference Reviewers seem only to want to find fault with papers rather than appreciate insights Despite inevitable errors lack of technical completenessTHE ACM SIGCOMM’S CCR - JULY 2011
  38. 38. 1. Subconscious desire to get one’s back1. if my paper has been rejected from a venue due to sharp criticism…2. Why not pay this back with sharp criticism of my own?2. A desire to prove one’s expertise1. if I can show that a paper is not perfect, that shows how clever I am3. A biased view of what papers in a particular area should looklike1. I’m the expert in my field so I think I know what every paper in myfield should look like4. unrealistic expectations1. I may not write perfect papers but I expect to read only perfect ones5. Subconscious attitudes are exacerbated by1. a ballooning of reviewer workloads2. journals in computer science languishing in their roles, conferencepapers being held to archival standard
  39. 39.  One negative consequence is the stifling of innovation they scale the walls by adding epsilon to delta until theincrementality threshold is breached well-studied areas are further overstudied to the detriment of others A second negative consequence is that it turns someresearchers off do not want to take part in a game where they cannot respect thewinners or the system PC chairs and Area Editors have to change our mental attitude from finding reasons to reject a paper to finding reasons to accept a paper We will certainly be trying to do this from now on at CCR
  40. 40.  As a reviewer of a paper, it is your duty to critique a paper andpoint out its flaws. But can you overlook minor flaws ? Can you find the greater good? With this small change, the system will also change. One review at a time
  41. 41. Based ontrue factsEXAMPLES: FROM GOODTO BAD (AND VICE-VERSA)
  42. 42. Dont judge peer reviewby its occasional failingshttp://www.guardian.co.uk/science/blog/2008/apr/15/peerreviewersdoamagnificen
  44. 44. • Title: be creative• Abstract: act as a marketing person• Introduction: be clear. State scenarios, need(motivation), topic importance, a quick view onprevious work/your solution/results, paperorganization• Methodology: make it reproducible (at least, clear)• Results: be creative on presentation (not much),interpret data, infer things (not much), discuss thelessons learned• Related work: show both broad and close researchpapers; state the difference to yours (during andat the end of the section)• Conclusion: do not add new material. Summarizemain findings and make it coherent withtitle/abstract/intro“Convincethe refereethat it isworthwhilereadingthe nextsection”(me)
  45. 45. Author makes adraftColleaguemakes a revisionProfessionalproofreading
  47. 47. Open AccessExplainedPUBLISHERSVS. OPENACCESS