Some anecdotes Wanted to build on a very notorious bioinformatics paper;  asked for pipeline script; main author sent us ...
Focus on replication issue Most journals already require data, methods  (software) to be available to reviewers and reade...
So: Should there be review standards for bioinformatics? YES. What should they focus on? REPLICATION DURING  REVIEW. Fo...
A specific proposal A position statement and/or white paper from OBF  and others. A JS/static Web site (no DB/tracking!)...
An example paragraph.Bioinformatics review criterion: The software is  available, +1; the software license allows readers ...
Upcoming SlideShare
Loading in...5
×

BOSC 2012 panel discussion

4,760

Published on

Should we have standard bioinformatics review criteria?

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
4,760
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BOSC 2012 panel discussion

  1. 1. Some anecdotes Wanted to build on a very notorious bioinformatics paper; asked for pipeline script; main author sent us a Perl script that had a syntax error in the third line. (<- joke goes here) Was talking with collaborators about digital normalization, they said “oh, we’ve been doing that for ages. Here are refs.” Refs were to high profile papers that said “we filtered k-mers.” No software, no parameters, no validation. High profile paper on metagenome assembly that used computational spike-in data (concluding that existing approaches had low sensitivity). Asked for data; no record of it.A very real problem is that there is no recourse in these situations. There is no way to check if these people are doing good science or not. WTH???
  2. 2. Focus on replication issue Most journals already require data, methods (software) to be available to reviewers and readers.  http://ivory.idyll.org/blog/journal-data-policies.html (#bosc12)  Although for software, this is most often honored in the breach… It is easy to justify this for replication and good science! Note, no explicit requirement to allow modification & distribution. Bigger goals:  A gateway path to open source, reusability?  Developers are nervous about releasing crappy code, so code quality will almost certainly improve.
  3. 3. So: Should there be review standards for bioinformatics? YES. What should they focus on? REPLICATION DURING REVIEW. For what kinds of bioinformatics? ANY ARTICLE WHERE UNPUBLISHED SOFTWARE IS USED. How should they work? REVIEWER OPT-IN.  No journal infrastructure required.  Allows for judgement of reviewer to play a role.  Most editors are timid and will not overrule reviewer, especially when their own journal guidelines support.  If you can get 1 reviewer /paper to do it, you have changed the field.
  4. 4. A specific proposal A position statement and/or white paper from OBF and others. A JS/static Web site (no DB/tracking!) with  Checklist for reviewers that generates score and back- link;  Links to position statement & journal policies;  Dynamically generate summary paragraph with score, that links back to Web page. Encourage reviewers (us!!) to use as opt-in component of review.  Can be one component, minor or major, of review, at reviewer discretion;  Serves as polite education rather than merely
  5. 5. An example paragraph.Bioinformatics review criterion: The software is available, +1; the software license allows readers to run it, +1; the software is available, + 1; I didnt download it. The data is available, +1; the data format is straightforward, +1; I didnt download the data.Score: 5/5. See http://short/url/to/brc for more information.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×