Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tutorial on reviewing your own paper (and those of others)


Published on

Published in: Business, Technology
  • Be the first to comment

Tutorial on reviewing your own paper (and those of others)

  1. 1. Reviewing Your Own Paper(and those of others)Tim GrantRetired But Active Researcher (R-BAR)Chair, ISCRAM Publications & AcademicStandards Committee (PASC)r̅
  2. 2. Overview• Goal:– To help listeners to produce better submissions toconferences, journals, & books by gainingknowledge of reviewing own & other papers• Outline:– Reviewing: history, process, products, roles– Reviewing criteria & how to review– Shortcomings, criticisms & failure of reviewing– Online possibilities– Key resources– An exercise for the attendeeLinköping, 5 Jun 13Tutorial on reviewing2
  3. 3. Introduction: about me• Career:– 1966-87: Royal Air Force officer (UK & SG)– 1987-2004: Consultant, ICT services industry (NL)– 2001-09: Visiting Professor, University of Pretoria (ZA)– 2004-12: Professor, Netherlands Defence Academy (NL)• Qualifications:– 1969: Bachelor of Science, Aero Engineering, Bristol (UK)– 1984: Defence Fellowship (Masters), Brunel (UK)– 1996: PhD, Artificial Intelligence, Maastricht (NL)• ISCRAM:– 2009-date: Board member & PASC chairLinköping, 5 Jun 13Tutorial on reviewing3
  4. 4. Introduction: reviewing (1)• Reviewing (aka refereeing):– Subjecting draft proposals, papers, posters, demonstrations,theses, courses, subjects, etc to critical evaluation• Why?– Quality control; help authors to learn– Admission to “Body of Knowledge” (BoK)– Assignment of credit & priority to authors• By whom?– Independent experts working in same field– Can be board / panel / tribunal of experts– More usually peers (other researchers / authors)Linköping, 5 Jun 13Tutorial on reviewing4
  5. 5. Introduction: reviewing (2)• Where?– Evaluation of applications for funding– Review of project reports by researchers to assesssuccessful progress / completion– Review of draft conference presentations, journal articles,and monographs to check they meet quality standards– Evaluation of set of papers after publication for review article– Evaluation of quality of work produced by individuals, teams,departments, and institutions to help determine:• Appointments• Promotions• Levels of fundingLinköping, 5 Jun 13Tutorial on reviewing5RIN, 2010
  6. 6. Brief history (1)• 854-931: Ishap bin Ali Al Rahwi, “Ethics of Physician”:– 1stdescription of (medical) peer review• 1453: printing press• 1543-1564: Copernicus, Servetus, Galileo, Versalius:– Review by non-peers• 1620: Francis Bacon’s New Philosophy:– Scientific method• 1662: Royal Society, London, chartered:– 1stlearned society• 1665: Philosophical Transactions:– 1stscientific journal; editor selects publicationsLinköping, 5 Jun 13Tutorial on reviewing6Spier, 2002
  7. 7. Brief history (2)• 1752: Royal Society adopts review procedure:– Pioneered in Edinburgh since 1731– Need for editorial committee (initially to fill excess space)• 1890: Typewriter & carbon copies (1959 Xerox):– Easier to circulate submissions to reviewers• Approx 1940: Diversity & specialization of material:– Excess journal space vanishes; need to discriminate– Need for external reviewers• 1989-98: International Congresses on Peer Review• Since approx 1990: PC, Internet, email, WWW:– Online journals; Open Access movementLinköping, 5 Jun 13Tutorial on reviewing7Spier, 2002
  8. 8. Reviewing process (1)Linköping, 5 Jun 13 8Problem Research Write up Review PublishResearchFundingTeach EvaluateTeachingDesignDevelopProduct ServiceValorizationEvaluatecitations / impactpatentsuse in coursesIdeally should predictcitations / impact, patents,& use in coursesBoK
  9. 9. Reviewing process (2)• Steps in review process:1. Authors submit draft paper to editors/publishers2. Publisher’s staff log submission & acknowledge receipt3. Editors assign draft paper to reviewers4. Reviewers assess draft paper, recommending acceptance,rejection, or modification• Typically 2 or 3 reviewers1. Editors weigh recommendations, make decision, & informauthors2. Authors modify paper & resubmit revised versionLinköping, 5 Jun 13Tutorial on reviewing9RIN, 2010
  10. 10. Reviewing process (3)• Steps in ISCRAM review process:1. Authors upload draft paper to ConfTool2. ConfTool logs, acknowledges receipt, & informs Track chair3. Track chair assigns draft paper to reviewers in ConfTool4. Reviewers assess draft paper, recommending acceptance,rejection, or modification in ConfTool5. Meta-reviewers weigh recommendations in ConfTool6. Track chair makes decision & informs authors via ConfTool7. Authors modify paper & upload revised version, details ofchanges made & copyright form to ConfTool8. ConfTool logs, acknowledges receipt, & informs Track chairLinköping, 5 Jun 13Tutorial on reviewing10(Adapted from) RIN, 2010
  11. 11. Reviewing process (4)• Variations on reviewing process:– Submission:• (Extended) abstract + paper; paper only– Submission types:• Full paper; short/work-in-progress paper; practitionerpaper; poster; demonstration; exhibit– Reviewing:• By committee; blind; double-blind; restricted; crowd (open)– Medium:• Paper; CD-ROM; USB; online• Restricted (to attendees or to members) vs open accessLinköping, 5 Jun 13Tutorial on reviewing11
  12. 12. Reviewing process (5)• Steps in review process (with abstract, eg for book):1. Authors submit abstract to editors/publishers2. Editors assess abstract, make decision, & inform authors3. Authors submit draft paper to editors/publishers4. Publisher’s staff log & acknowledge receipt5. Editors assign draft paper to reviewers6. Reviewers assess draft paper, recommending acceptance,rejection, or modification7. Editors weigh recommendations, make decision, & informauthors8. Authors modify paper & resubmit revised versionLinköping, 5 Jun 13Tutorial on reviewing12(Adapted from) RIN, 2010
  13. 13. Reviewing process (6)• Steps in review process (for journal):1. Authors submit abstract to editors/publishers2. Editors assess abstract, make decision, & inform authors3. Authors submit draft paper to editors/publishers4. Publisher’s staff log & acknowledge receipt5. Editors assign draft paper to reviewers6. Reviewers assess draft paper, recommending acceptance,rejection, or modification7. Editors weigh recommendations, make decision, & inform authors8. Authors modify draft paper & resubmit revised version9. Reviewers assess revised paper & make new recommendation10. Editors weigh recommendations, make decision, & inform authors11. Authors modify paper & resubmit revised versionLinköping, 5 Jun 13Tutorial on reviewing13(Adapted from) RIN, 2010
  14. 14. Reviewing process (7)• Conferences:– Usually 1 iteration; paper only; 8-10 pages– (Less often) 2 iterations: first abstract, then paper• Journals:– 2 or more iterations; paper only; 20-30 pages– At least 50% new material for conference paper• Books:– 2 iterations:• Extended abstract or chapter proposal (typically 2 pages)• Chapter / contribution; 20-30 pagesLinköping, 5 Jun 13Tutorial on reviewing14
  15. 15. Reviewing process (8)• Differences between disciplines:– Humanities & social sciences:• Double-blind common• Authors may be allowed to nominate reviewer(s)– Sciences:• Single-blind common; increasingly double-blind– Medical field:• Increasingly open-reviewLinköping, 5 Jun 13Tutorial on reviewing15
  16. 16. Products from reviewing• Paper template• Copyright form• Review template• Email templates for:– Acknowledging receipt– Informing Track chair of upload– Assigning papers to reviewers– Rejection, acceptance, & conditional acceptance– Final acceptance of camera-ready copyLinköping, 5 Jun 13Tutorial on reviewing16
  17. 17. Roles in reviewing• Author: writes, submits & modifies paper• Track chair: assigns submissions to reviewers• Reviewer: reviews set (2..6) of submissions• Meta-reviewer: moderates (2..4) reviews• (Editor: combines Track chair & meta-reviewer roles)• Programme Committee = {Editors}– ISCRAM: Scientific Committee = {Track chairs}– Moderates across tracks• Programme Chair (& co-chairs):– Decides conference scheduleLinköping, 5 Jun 13Tutorial on reviewing17
  18. 18. Reviewing procedures• Review procedure documents (eg ISCRAM):– Review process:• Overview for conference organizers– Review timeline:• Helps conference organizers schedule activities– Review guidelines:• For reviewers of full, short / work-in-progress, & practitionerpapers– Meta-review & final decision:• For meta-reviewers, track chairs, & Scientific Committee– Best (Student) Paper procedure:• For Best Paper sub-committees & conference organizersLinköping, 5 Jun 13Tutorial on reviewing18
  19. 19. Criteria for reviewing (1)Criterion ISCRAM BNAIC PlanSIG ICIW(Discipline) IS + CRAM AI Planning Cyber/info warRelevance x x x xSignificance/contribution x - x xOriginality x (full only) x x -Validity x (full only) x x xClarity/readability x x x x(Others) Best Paper? Award? Method, drawings,abstract, keywords,referencesOverall score x x x xReviewer’s confidence x x x -Remarks for author(s) x x x xRemarks for PC x x x -Linköping, 5 Jun 13 19Tutorial on reviewing
  20. 20. Criteria for reviewing (2)• Relevance:– Papers should be clearly relevant to conference’s subjectarea, not generalist– ISCRAM (= IS + CRAM):• Technical papers (information systems, computer science, IT)should describe implications for crisis response and/oremergency management• Vice versa for crisis response / emergency management papers– Tips:• Read Call For Papers!• Ensure paper fits into topic or track• Does this year’s conference have special theme?– If so, try to make link to theme in paper (N.B. nice to have)Linköping, 5 Jun 13Tutorial on reviewing20
  21. 21. Criteria for reviewing (3)• Significance / contribution:– Does paper make a significant contribution to literature?• Significance means contribution opens up (or closes off) previouslyunexplored lines of research or of development (valorization)– Does paper state the contribution it makes?– Does paper link this contribution to pre-existing literature?– Does paper identify the limitations of this contribution?• From method & from what is outside scope of paper• Limitations can always be topics for further research!– Tips:• Know the relevant literature!• Don’t overclaim, i.e. overstate contributionLinköping, 5 Jun 13Tutorial on reviewing21
  22. 22. Criteria for reviewing (4)• Originality / novelty:– Ideas must be non-trivial, new & timely, not “more of same”• Case study can be new• Literature survey paper can be new if it “adds value”, egstructuring literature in a new way• Application of established ideas to another field can be new:– Idea may be known in IS literature, but not yet applied to CRAM.– And vice versa - which is why practitioner papers are so important– Tips:• Know the relevant literature!• Where can, exploit multi-disciplinary nature (e.g. IS + CRAM)Linköping, 5 Jun 13Tutorial on reviewing22
  23. 23. Criteria for reviewing (5)• Validity / soundness / thoroughness:– Is purpose of paper clearly stated? And paper’s scope?– Is research question / hypothesis clearly stated?– Are appropriate methods used?• Is best practice for method followed? (eg case study, literature survey)– Is argument logically sound? Statistics & equations correct?– Is treatment /discussion thorough?• In enough detail for another researcher to reproduce what you did?– Tips:• Ensure paper tells story:– Not necessarily the order in which you did your research!• Reflect on your contribution in context of related literatureLinköping, 5 Jun 13Tutorial on reviewing23
  24. 24. Criteria for reviewing (6)• Clarity / readability / structure:– Well structured? Grammatically correct? Easy to read?– Abbreviations in full on 1stuse? Jargon known to readers?– Citations given in full & easy to find?– Can oldies (60+) read graphics without magnifying glass?– Tips:• Apply “magic number of 5 +/- 2” to number of sections• First section should be “Introduction” or “Motivation”• Last section should be “Conclusions & Further Research”• Review relevant literature in 2ndor 2nd-to-last section• Do not exceed page or word limit!• Use spelling & grammar checker!Linköping, 5 Jun 13Tutorial on reviewing24
  25. 25. How to review (1)• Reviewing is extension of reading paper• Keshav’s three-pass method for reading:– Pass (1): Quick-scan (5-10 mins):• Read title, abstract, & introduction• Read section & sub-section headings (structure)• Glance at equations (if any)• Read conclusions• Glance over references to see which ones you know– Should be able to answer “five Cs”:• Category; context; correctness; contribution, clarityLinköping, 5 Jun 13Tutorial on reviewing25Keshav, 2012
  26. 26. How to review (2)– Pass (2): Read paper with more care (1 hour):• Ignore details such as proofs• Jot down key points, or make notes in margin• Look carefully at figures, diagrams, & other illustrations• Mark relevant unread references for further reading• Answer:– What does paper do? (Rationale/motivation, aims, hypothesis)– How does it do it? (Participants, method)– What did it find? (Findings, implications, falsifiable, limitations)Linköping, 5 Jun 13Tutorial on reviewing26Keshav, 2012
  27. 27. How to review (3)– Pass (3): Virtually re-create paper (4-5 hours):• Making same assumptions, re-create paper in own way• Compare re-creation with actual paper:– Identify & challenge every statement– Jot ideas down as you go• Self-reflect: watch out for community groupthink / bias• Answer:– Weaknesses:» Implicit assumptions; experimental / analytical errors; missingcitations– Strengths:» Presentation technique; support for conclusions; internal efficacyLinköping, 5 Jun 13 27Keshav, 2012Tutorial on reviewing
  28. 28. How to review (4)• Now add reviewing-specific steps:– Scoring according to reviewing template– Writing “Remarks to authors”:• Briefly summarize paper in own words• State what you think contribution(s) are:– Not those stated by author(s)• Give specific comments based on jottings / margin notes:– Separate major points from minor ones (eg typos)– By relevance; significance; originality; validity; clarity– Justifying your scores• Conclude comments (eg summarize good & bad points)• Recommend modificationsLinköping, 5 Jun 13Tutorial on reviewing28Roscoe, 2007
  29. 29. How to review (5)• Approaches to reviewing own paper:1. Write paper & then review it, modifying as needed2. Write paper, already building in information reviewer needs• I prefer 2ndapproach (next 6 slides)• Mindset: Reviewer is always right!– True: modify as reviewer recommends (Relevance,Significance, Originality, & Validity)– False: reviewer misunderstood because draft paper is poorlyexpressed; express more clearly (Clarity)• You’ll still get Remarks to Authors:– But they should be fewer and less majorLinköping, 5 Jun 13 29Tutorial on reviewing
  30. 30. How to review (6)• How I plan a paper (1/6):– Look at Call For Papers– Have I done research that is worth presenting?• If none, STOP– Which topic / track does it fit into?• If none, STOP– What message do I want to give?• Sketch story (5 plus/minus 2 bullets)– Write single sentence describing purpose of paper:• “The purpose of this paper is to present / survey / show ...”Linköping, 5 Jun 13Tutorial on reviewing30
  31. 31. How to review (7)• How I plan a paper (2/6):– Extract title from purpose of paper:• This must “sell” your paper!Eg “Tweak the tweet: leveraging proliferation with a prescriptive syntaxto support citizen reporting” (Starbird & Stamberger, ISCRAM 2010)– Extract top-level headings from story:• Don’t forget 5 plus/minus 2:1.Introduction2.Relevant theory3.Experiment4.Discussion5.Conclusions & Further ResearchLinköping, 5 Jun 13 31Tutorial on reviewing
  32. 32. How to review (8)• How I plan a paper (3/6):– Expand 1. Introduction:1.1 Background (or Motivation)– Identifies “gap” in current Body of Knowledge– Motivates why research is worth doing– Shows relevance to conference & topic / track– Outlines what paper covers, eg“This paper focuses on technical capabilities for cyber warfare.”1.2 Purpose & scope– Insert one-sentence purpose here– Scope is what your research & this paper does not include, eg“Legal issues are outside the scope of this paper.”1.3 Paper structure (or Layout)– So reader knows what to expect (sections, appendices)Linköping, 5 Jun 13 32Tutorial on reviewing
  33. 33. How to review (9)• How I plan a paper (4/6):– Expand 3. Experiment:Eg 3.1 Design; 3.2 Execution; 3.3 Results & analysis– Expand 5. Conclusions & Further Research:(5.1) Summary– Not same as Abstract(5.2) Contribution(s)– Just most important ones– Don’t overstate(5.3) Limitations– From method used– From what is outside scope(5.4) Further researchLinköping, 5 Jun 13 33Tutorial on reviewing
  34. 34. How to review (10)• How I plan a paper (5/6):– Allocate page budget (see CFP or template for limit):– Write Abstract:• Note word limit (in CFP, author instructions, or template):– Two-thirds background & motivation:» Conclude with one-sentence purpose of paper– One-third (intended) structure of paperLinköping, 5 Jun 13 34Title, abstract & keywords ½ page or 5%Introduction 5%Conclusions & Further Research 5%References 10%(other sections) divide up 75%Tutorial on reviewing
  35. 35. How to review (11)• How I plan a paper (6/6):– Write one-sentence objective for each section:1. Introduction“The objective of this section is to describe the background,motivate the research, state the purpose and scope of thepaper, and outline its structure.”2. Relevant theory“The objective of this section is to summarize the theoryrelevant to the content of this paper.”• N.B. To guide writing only; may be deleted once written– Allocate (sub-) sections to co-authors (if any):• In consultation, taking into account roles & abilitiesLinköping, 5 Jun 13 35Tutorial on reviewing
  36. 36. Key resources (1)• Overviews (recommended):– RIN, 2010. Peer Review: A guide for researchers. ResearchInformation Network (March), Accessed 2 May 13. *– Wikipedia. 2013. 2 May 2013– Rowland, F. 2002. The Peer Review Process: A report to theJISC Scholarly Communications Group. 2 May 2013. *Linköping, 5 Jun 13Tutorial on reviewing40
  37. 37. Key resources (2)• Papers/articles (1/3):– Calcagno, V., Demoinet, E., Gollner, K., Guidi, L., Ruths, D. & deMazancourt, C. 2012. Flows of Research Manuscripts AmongScientific Journals Reveal Hidden Submission Patterns. ScienceMagazine, 338, 1065-9 (November)– Cormode, G. 2008. How NOT to Review a Paper: the tools andtechniques of the adversarial reviewer. SIGMOD Record, 37, 4. *– Griswold, W.G. (not dated). How to Read an Engineering ResearchPaper. *• See also link to paperform.pdf. *– Hill, S. & Provost, F. 2003. The Myth of the Double Blind Review?Author identification using only citations. SIGKDD Explorations, 5,3, 179-184– Jefferson, T., Alderson, P., Wager, E. & Davidoff, F. 2002. Effects ofEditorial Peer Review: A systematic review. JAMA, 287, 2784-6Linköping, 5 Jun 13 41Tutorial on reviewing
  38. 38. Key resources (3)• Papers/articles (2/3):– Kershav, S. 2012. How to Read a Paper. (June 26). *• See Ian McLean’s 2012 Literature Review Matrix, based on Kershav, at *– Mahoney, M.J. 1977. Publication Prejudices: An experimental studyof confirmatory bias in the peer review system. Cognitive Therapyand Research, 1, 2, 161-175– McCook, A. 2006. Is Peer Review Broken? The Scientist Magazine(February 1)– Roberts. 1999. Scholarly Publishing, Peer Review and the Internet.First Monday, 4, 4 (5 April)– Roscoe, T. 2007. Writing Reviews for System Conferences. *Linköping, 5 Jun 13 42Tutorial on reviewing
  39. 39. Key resources (4)• Papers/articles (3/3):– Smith, R. 2006. Peer Review: A flawed process at the heart ofscience and journals. J R Soc Med, 99, 178-182– Spier, R. 2002. The History of the Peer Review Process. Trends inBiotechnology, 20, 8, 357-8 (August)– Van Rooyen, S., Godlee, F., Evans, S., Smith, R. & Black, N. 1998.Effect of Blinding and Unmasking on the Quality of Peer Review: Arandomized trial. JAMA, 280, 3, 234-7– Walt, S.M. 2013. On Writing Well. Foreign Policy (February 15). *• Itself an example of good writing.– Weiss, R. 2005. Many Scientists Admit to Misconduct: Degrees ofdeteption vary in poll. Washington Post (June 9)Linköping, 5 Jun 13 43Tutorial on reviewing
  40. 40. Key resources (5)• Journals:– Learned Publishing– Journal of Scholarly Publishing– Journal of American Society for Information Science– Journal of Documentation• Bibliography:– Bailey, C.W. 2011. Scholarly Electronic PublishingBibliography. Version 80(30 November), accessed 2 May 13• Learned society:– Association of Learned and Professional Society Publishers(ALPSP): Accessed 2 May 13Linköping, 5 Jun 13Tutorial on reviewing44
  41. 41. Key resources (6)• Books:– Peek, R.P. & Newby, G.B. (eds). 1996. Scholarly Publishing: Theelectronic frontier. MIT Press, Cambridge, MA– Page, G., Campbell, R. & Meadows, A.J. 1997. Journal Publishing.Cambridge University Press, Cambridge, UK– Meadows, A.J. 1998. Communicating Research. Academic Press,San Diego, CA– Tenopir, C. & King, D.W. 2000. Towards Electronic Journals:Realities for scientists, librarians and publishers. SLA Publishing,Washington DC– Fredrikson, E.H. (ed). 2001. A Century of Scientific Publishing. IOSPublishing, Amsterdam, NL– Abel, R.E. & Newlin, L.W. (eds). 2002. Scholarly Publishing: Books,journals, publishers, and libraries in the twentieth century. JohnWiley & Sons, New YorkLinköping, 5 Jun 13 45Tutorial on reviewing
  42. 42. Key resources (7)• Wikipedia (English, all accessed 2 May 13):–––––––• See also Scholarly paper and Peer review sections–––öping, 5 Jun 13Tutorial on reviewing46
  43. 43. Exercise for listener• “Beer versus science” as example of open reviewing:– Grim, T. 2008. A Possible Role of Social Activity to ExplainDifferences in Publication Output Among Ecologists. Oikos, 177,484-7. *• Inverse linear relation between beer input and publications output– Mack, C.A. 2008. In Defense of Beer-Drinking Scientists. (March 21). *– Mack, C.A. 2008. In Defense of my Defense of Beer-DrinkingScientists. (March 27). *– Mack, C.A. 2008. More on Beer-Drinking Scientists: A response to DrGrim. (April 9). *– Van Noorden, R. 2010. Make Mine a Double. (Sep 15).• Let’s do field research!Linköping, 5 Jun 13 47Tutorial on reviewing
  44. 44. Any questions?Tim GrantRetired But Active Researcher (R-BAR) (0)638 193 749With thanks to:Julie Dugdale, Simon French, Mark S Pfaff, Murray Turoff,Arien vd Wal