Berlin 6 Open Access Conference: Jelena Kovacevic

741 views

Published on

www.berlin6.org

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
741
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Jelena Kovačević: MR Tools for Next-Generation Bioinformatics Systems 06/06/09 This is a brief version of the presentation I gave at Icassp 2007 in the special session on reproducible research. My goal was to give some input based on my experience during the 5 years I served as the Editor-in-Chief of the IEEE Transactions on Image Processing.
  • Berlin 6 Open Access Conference: Jelena Kovacevic

    1. 1. How to Encourage and Publish Reproducible Research Jelena Kovačević Center for Bioimage Informatics Department of Biomedical Engineering Department of Electrical & Computer Engineering Carnegie Mellon University
    2. 2. Theory Vs Experimentation <ul><li>Theoretical disciplines </li></ul><ul><ul><li>Mathematics </li></ul></ul><ul><li>Experimental disciplines </li></ul><ul><ul><li>Biology </li></ul></ul><ul><li>A hybrid is born: Computational sciences </li></ul><ul><ul><li>Should follow good practices from both </li></ul></ul><ul><ul><li>SP falls in there: How are we doing? </li></ul></ul>Axioms Lemmas Proof MIT CMU Stanford Harvard hypothesis
    3. 3. Issues <ul><li>Cultural </li></ul><ul><ul><li>Innovation above all else </li></ul></ul><ul><ul><li>TIP Transactions reviewing questions </li></ul></ul><ul><ul><ul><li>1. Is the paper technically sound? </li></ul></ul></ul><ul><ul><ul><li>2. Is the coverage of the topic sufficiently comprehensive and balanced? </li></ul></ul></ul><ul><ul><ul><li>3. How would you describe the technical depth of the paper? </li></ul></ul></ul><ul><ul><ul><li>4. How would you rate the technical novelty of the paper? </li></ul></ul></ul><ul><ul><li>Can lead to paradox </li></ul></ul><ul><li>Educational </li></ul><ul><ul><li>Our students undertrained in statistics </li></ul></ul><ul><ul><li>Typically reimplement everything </li></ul></ul><ul><li>Data </li></ul><ul><ul><li>We collaborate and data might not be ours </li></ul></ul><ul><li>IP </li></ul><ul><ul><li>Data issues </li></ul></ul><ul><ul><li>Companies and agencies protecting their IP </li></ul></ul><ul><li>Collaborative </li></ul><ul><ul><li>With colleagues within the university/company, outside </li></ul></ul>
    4. 4. How Do We Publish RR? <ul><li>Not likely to happen overnight </li></ul><ul><ul><li>Encourage and reward “good behavior” (Child psychology 101) </li></ul></ul><ul><li>Ideas </li></ul><ul><ul><li>Special section in Transactions for RR? </li></ul></ul><ul><ul><li>Establish a paper award for an RR paper? </li></ul></ul><ul><ul><li>Form a rough guideline of what each paper should contain for an RR designation? </li></ul></ul><ul><ul><li>Everything we read is partly “on faith” </li></ul></ul>
    5. 5. How to Make Papers RR? <ul><ul><li>Used in my group </li></ul></ul><ul><ul><li>Compilation of ideas from Barni and EPFL groups (Vetterli, Vandewalle et al.) </li></ul></ul><ul><ul><li>Compendium (Gentleman & Lang) </li></ul></ul><ul><ul><li>Freeze the code upon </li></ul></ul><ul><ul><ul><li>Submission </li></ul></ul></ul><ul><ul><ul><li>Acceptance </li></ul></ul></ul><ul><ul><li>“ Good intentions” enforced </li></ul></ul><ul><ul><li>Students do projects and reproduce </li></ul></ul>
    6. 6. An Entirely NonRR Case Study <ul><li>Data set </li></ul><ul><ul><li>15 papers published in the TIP </li></ul></ul><ul><ul><li>EDICS category using both theory and experimentation </li></ul></ul><ul><ul><li>Stayed away from standards as well as biomed </li></ul></ul><ul><ul><li>For all algorithms, competing ones exist </li></ul></ul><ul><ul><li>RR </li></ul></ul><ul><ul><ul><li>block-diagram? </li></ul></ul></ul><ul><ul><ul><li>pseudo code? </li></ul></ul></ul><ul><ul><ul><li>data available? </li></ul></ul></ul><ul><ul><ul><li>code available? </li></ul></ul></ul><ul><ul><ul><li>proof available? </li></ul></ul></ul><ul><li>Ratings (0, 0.5, 1) </li></ul><ul><ul><li>Algorithm and experimental setup </li></ul></ul><ul><ul><ul><li>algorithm explained? </li></ul></ul></ul><ul><ul><ul><li>data explained? </li></ul></ul></ul><ul><ul><ul><li>data size? </li></ul></ul></ul><ul><ul><ul><li>details on parameters used? </li></ul></ul></ul><ul><ul><ul><li>comparison to competing algorithms? </li></ul></ul></ul>
    7. 7. Results of the Entirely NonRR Case Study <ul><ul><li>All papers had proofs, none had code available </li></ul></ul><ul><ul><li>Sufficient detail on algorithms, none had a block-diagram </li></ul></ul><ul><ul><li>Data used, data size and availability all below average </li></ul></ul><ul><ul><li>Half of the cases were the parameters specified </li></ul></ul><ul><ul><li>Comparisons to competing algorithms: quarter </li></ul></ul><ul><ul><li>Pleasant surprise: 60%, pseudo-code was available </li></ul></ul>

    ×