Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Peer review - Why does it matter for your academic career?


Published on

Presentation provided in the context of the Young Researchers Special Issue for the International Journal of Technology-Enhanced Learning (IJTEL). Download available via

  • Be the first to comment

  • Be the first to like this

Peer review - Why does it matter for your academic career?

  1. 1. Peer review – Why does it matter for your academic career? Dr. Marco Kalz Centre for Learning Sciences and Technologies (CELSTEC) Open University of the Netherlands
  2. 2. Structure• History of Peer review• Peer review from 2 perspectives• How to conduct a peer review?• Issues with peer review
  3. 3. History of peer review• Spier (2002) mentions the bible as early peer review example• Royal Society of London created peer review in 1665• First peer-reviewed journal: Philosophical Transactions• Target: Improve papers
  4. 4. Peer-review as quality assurance
  5. 5. Peer-review as quality assurance • Submit primarily to peer-reviewed (and SSCI/SCI indexed) journals • Impact counts (however calculated) • Grow your citation networks
  6. 6. Acting as a peer-reviewer• Part of an academic job• Service to the academic community• Staying up-to-date & learning from others• Establishing relations to journal editors
  7. 7. How to conduct a peer-review?
  8. 8. Strive for a quality standards Do the quality of review that you wish others would do for you. Bieber (1995)
  9. 9. Structure of a review• Review (sent to authors)• Remarks for editors (not sent to authors)• Overall evaluation• Confidence level• Relevance• Novelty• Significance• Soundness/Technical quality
  10. 10. The review process• Read a paper 3 times: – 1. Getting an impression – 2. In depth analysis – 3. Judge the paper • 3 types of comments: • Review form • Comments • Comments in the paper itself
  11. 11. How to handle paper load?Faltings (2004) recommends – Apply filtering – Don’t spend time on papers which are not acceptable anyway
  12. 12. Paper checklistEvery paper must state (Faltings 2004):•The problem addressed•Solution or insight proposed•An example showing that it works•An evaluation, ideally in comparisonwith existing techniques
  13. 13. Common problems• Assumptions not explicitly discussed• Potential bias not identified• Unrealistic examples• Treatment & Effect exchanged• Fuzzy method section• Paper not well structured
  14. 14. Comment guidelinesBe constructive, honest and neutral•Find at least one positive comment•Back up your decision by an explanation•Keep always a professional style•Recognize hard work•Limit harm by lowering your confidencelevel
  15. 15. Example comments I“The paper is building on an outdated state-of-the art and its findings are therefor not solid”. “The state-of-the art covers only partially important related work.Especially the work by X (2004) and Y (2009) should be included in the analysis”.
  16. 16. Example comments II“The presented solution is only applicablein the presented context but I don’t think it can work in a different one”. “The external validity has not been controlled in the method section of thepapers. Therefor the transferability of the presented approach is unsure”.
  17. 17. Peer-reviewers as pimps? Frey, B. S. (2003). Publishing as prostitution? Choosing between one’s own ideas and academic success. Public Choice, 116(1), 205-223. Springer. Photo by technokitten
  18. 18. Peer-reviewers as pimps?
  19. 19. Issues with peer reviewPhoto by Grevel Blind review (Reviewer stays anonymous) +++ ??? --- Potentially open for bias of a reviewer due to – a competetive relation to the authors – personal preferences/problems – Hiding in anonymity
  20. 20. Issues with peer reviewDouble-blind review (reviewer & authoranonymous)+++ No direct bias problems+++ “Objective”--- No chance for communication--- Problem of dishonest reviews Photo by Jess and Colin
  21. 21. Issues with peer review• Speed of communication• A new paradigm is ignored (Kuhn, 1970)• Reward of peer-reviewers
  22. 22. New approaches for peer reviewOpen Peer Review•Introduced by BMJ in 1999, speeding upreview process, no differences in quality(van Rooyen et al., 1999)•Other, hybrid models•Post-publication reviews•Social Media
  23. 23. New approaches for peer review
  24. 24. Thank you for your attention!
  25. 25. References• Bieber, M. (1995). How to review. Available at• Faltings, B. (2004). How to write a review. Invited presentation 2nd European Starting AI researcher Symposium. In conjunction with the 16th European Conference on Artificial Intelligence. Valencia, Spain. August 22 – 27, 2004. Available at• Frey, B. S. (2003). Publishing as prostitution? Choosing between one’s own ideas and academic success. Public Choice, 116(1), 205-223. Springer.• van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. British Medical Journal, 318(7175), 23-7.• Smith, R. (1999). Opening up BMJ peer review. British Medical Journal, 318(7175), 4-5.• Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20(8), 357-358.