Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reproducible Research and Best Practices for Paper Review

622 views

Published on

Jacek Zurada, IEEE Life Fellow 2014 TAB Chair (IEEE V-P Technical Activities)

  • Be the first to comment

  • Be the first to like this

Reproducible Research and Best Practices for Paper Review

  1. 1. Reproducible Research: Best Practices for Reviewing (and Writing) a Paper Jacek Zurada, IEEE Life Fellow Candidate for 2019 IEEE President-Elect (2020 President) May 20, 2018
  2. 2. These Select (non-comprehensive) Slides are Opinions of the Author and not Necessarily Opinions of IEEE Summarize the Authors’ editorial experience and related workshops in his multiple roles in IEEE that include: Candidate for 2019 IEEE President-Elect (2020 President) 2014 TAB Chair (IEEE V-P Technical Activities) Editor-in-Chief of IEEE Transactions on Neural Networks 1998-2003 Chair of IEEE TAB Periodicals Committee 2010-11 Chair of IEEE TAB Periodicals Review and Advisory Committee 2012-13 2
  3. 3. Reproducible Research -Reproducible Research embraces the idea that the ultimate research product (i.e., paper, report, communications, etc.) contains full disclosure of the environment used to produce the results such as the code, data, experimental and laboratory conditions etc. that are needed to reproduce the results and possibly expand it through new work
  4. 4. Why Reproducible Research? -Motivations and pressures come from •The digitized web-of-science world and the publication business that has exploded in volume since 2000 •Authors/researchers/readers •Reviewers, editors, evaluators •Sponsors, funding agencies •Economy •.... and common sense 4
  5. 5. Reproducible Research: report
  6. 6. Reproducible Research Across Disciplines - Theoretical sciences • Easy - Experimental sciences • Physics, chemistry: moderate • Life sciences: moderate to difficult - Engineering and computational sciences • Theoretical: easy • Practical/computational/experimental: moderate, but definitely should not be difficult!
  7. 7. Who Holds the Keys to Reproducible Research? ▸Authors (all) ▸Sponsors ▸Reviewers ▸Associate/Action Editors ▸Editors-in-Chief ▸AND READERS DEMAND IT
  8. 8. How to Review a Paper for Originality? ▸What novel aspects are revealed in the paper? ▸It should be one or more - new experimental approach - new phenomenon or material, or properties thereof - new method, relevant application or model - new theorem(s) - new data generated - new interpretations or insights These selected best practices are contributed by an experienced editor and are not official IEEE review guidelines
  9. 9. How to Review a Paper for RR? (underlined passages connect to RR) ▸Experimental paper: needs a hypothesis, provides experimental validation (equipment, materials, approach, method need to be discussed) ▸Modeling paper: assumptions, mathematical approach, tools used, method, illustrations ▸Computational paper: inputs, computational approach, tools used, method, initial conditions, possibly computing statistics (if results not directly tractable) These selected best practices are contributed by an experienced editor and are not official IEEE review guidelines
  10. 10. How to Review a Paper for RR? ▸Work should be very well-referenced and credit given to previous contributors ▸Results in the paper should be compared with either benchmark results or other authors’ results ▸Physical and computational experiments must be repeatable, if deterministic (initial conditions, assumptions) ▸When computational experiments are non-tractable, repetitive experiments and their statistics needed, instead These selected best practices are contributed by an experienced editor and are not official IEEE review guidelines
  11. 11. How to Review a Paper for RR? ▸Research should be replicable by others in the public domain (data open publicly) ▸Formulas must be verifiable ▸If a theorem is proven, the proof must be completely tractable: assumptions, theorems, derivations, proofs ▸All quantities must be clearly defined (i.e. ‘MSEs’ are often vague and left to interpretations) ▸Tests on benchmark data or problems essential ▸Discussion of alternative solutions These selected best practices are contributed by an experienced editor and are not official IEEE review guidelines
  12. 12.  IEEE DataPort is a web-based, cloud-services platform/data repository supporting the RR needs of authors and readers, https://ieee-dataport.org/  Incubated through the IEEE Big Data Initiative  Like IEEE Xplore, but Data-focused  Full service platform • Referenceable datasets through Data Object Identifiers (DOI) • Linkable to/from published and unpublished documents through URLs How IEEE DataPort Helps with RR
  13. 13. How IEEE CodeOcean Helps with RR? IEEE publishes a large volume of articles that reference code and algorithms - By using Code Ocean’s services, algorithms in IEEE articles can now be executed in real time - IEEE authors can upload, share, and run their code on the Code Ocean platform - IEEE Xplore users can discover, view, and execute code in real- time - Users will discover and access the bidirectional link between IEEE Xplore and Code Ocean - The service is free and accessible through IEEE Xplore - https://codeocean.com/signup/ieeeData as of 31December2016

×