Some anecdotes Wanted to build on a very notorious bioinformatics paper; asked for pipeline script; main author sent us a Perl script that had a syntax error in the third line. (<- joke goes here) Was talking with collaborators about digital normalization, they said “oh, we’ve been doing that for ages. Here are refs.” Refs were to high profile papers that said “we filtered k-mers.” No software, no parameters, no validation. High profile paper on metagenome assembly that used computational spike-in data (concluding that existing approaches had low sensitivity). Asked for data; no record of it.A very real problem is that there is no recourse in these situations. There is no way to check if these people are doing good science or not. WTH???
Focus on replication issue Most journals already require data, methods (software) to be available to reviewers and readers. http://ivory.idyll.org/blog/journal-data-policies.html (#bosc12) Although for software, this is most often honored in the breach… It is easy to justify this for replication and good science! Note, no explicit requirement to allow modification & distribution. Bigger goals: A gateway path to open source, reusability? Developers are nervous about releasing crappy code, so code quality will almost certainly improve.
So: Should there be review standards for bioinformatics? YES. What should they focus on? REPLICATION DURING REVIEW. For what kinds of bioinformatics? ANY ARTICLE WHERE UNPUBLISHED SOFTWARE IS USED. How should they work? REVIEWER OPT-IN. No journal infrastructure required. Allows for judgement of reviewer to play a role. Most editors are timid and will not overrule reviewer, especially when their own journal guidelines support. If you can get 1 reviewer /paper to do it, you have changed the field.
A specific proposal A position statement and/or white paper from OBF and others. A JS/static Web site (no DB/tracking!) with Checklist for reviewers that generates score and back- link; Links to position statement & journal policies; Dynamically generate summary paragraph with score, that links back to Web page. Encourage reviewers (us!!) to use as opt-in component of review. Can be one component, minor or major, of review, at reviewer discretion; Serves as polite education rather than merely
An example paragraph.Bioinformatics review criterion: The software is available, +1; the software license allows readers to run it, +1; the software is available, + 1; I didnt download it. The data is available, +1; the data format is straightforward, +1; I didnt download the data.Score: 5/5. See http://short/url/to/brc for more information.