Should we have standard bioinformatics review criteria?

1. Some anecdotes
 Wanted to build on a very notorious bioinformatics paper;
asked for pipeline script; main author sent us a Perl script
that had a syntax error in the third line. (<- joke goes here)
 Was talking with collaborators about digital normalization,
they said “oh, we’ve been doing that for ages. Here are
refs.” Refs were to high profile papers that said “we
filtered k-mers.” No software, no parameters, no
validation.
 High profile paper on metagenome assembly that used
computational spike-in data (concluding that existing
approaches had low sensitivity). Asked for data; no record
of it.
A very real problem is that there is no recourse in these
situations. There is no way to check if these people
are doing good science or not. WTH???

2. Focus on replication issue
 Most journals already require data, methods
(software) to be available to reviewers and readers.
 http://ivory.idyll.org/blog/journal-data-policies.html
(#bosc12)
 Although for software, this is most often honored in the
breach…
 It is easy to justify this for replication and good
science!
 Note, no explicit requirement to allow modification &
distribution.
 Bigger goals:
 A gateway path to open source, reusability?
 Developers are nervous about releasing crappy code, so
code quality will almost certainly improve.

3. So:
 Should there be review standards for bioinformatics? YES.
 What should they focus on? REPLICATION DURING
REVIEW.
 For what kinds of bioinformatics? ANY ARTICLE WHERE
UNPUBLISHED SOFTWARE IS USED.
 How should they work? REVIEWER OPT-IN.
 No journal infrastructure required.
 Allows for judgement of reviewer to play a role.
 Most editors are timid and will not overrule reviewer, especially
when their own journal guidelines support.
 If you can get 1 reviewer /paper to do it, you have changed the
field.

4. A specific proposal
 A position statement and/or white paper from OBF
and others.
 A JS/static Web site (no DB/tracking!) with
 Checklist for reviewers that generates score and back-
link;
 Links to position statement & journal policies;
 Dynamically generate summary paragraph with score,
that links back to Web page.
 Encourage reviewers (us!!) to use as opt-in
component of review.
 Can be one component, minor or major, of review, at
reviewer discretion;
 Serves as polite education rather than merely

5. An example paragraph.
Bioinformatics review criterion: The software is
available, +1; the software license allows readers
to run it, +1; the software is available, + 1; I didn't
download it. The data is available, +1; the data
format is straightforward, +1; I didn't download
the data.
Score: 5/5. See http://short/url/to/brc for more
information.