2017 11-15-reproducible research

Yannick Wurm
http://wurmlab.github.io
Reproducible Research &
Sustainable Software

Baby steps towards improved
efﬁciency, robustness &
reproducibility

Geoffrey Chang: Crystallographer
• Beckman FoundationYoung Investigator
Award
• Presidential Early Career Award
Journal of Molecular Biology (2003) Chang. Structure
of MsbA from Vibrio cholera: a multidrug resistance ABC
transporter homolog in a closed conformation.
PNAS (2004) Ma & Chang. Structure of the multidrug
resistance efﬂux transporter EmrE from Escherichia coli.
Science (2005) Reyes & Chang. Structure of the ABC
transporter MsbA in complex with ADP vanadate and
lipopolysaccharide.
Science (2005) Pornillos et al. X-ray structure of the
EmrE multidrug transporter in complex with a substrate.
Science (2001) Chang & Roth. Structure of MsbA from
E. coli: a homolog of the multidrug resistance ATP binding
cassette (ABC) transporters.
Science (2001) Chang & Roth.

earch Institute in
next year, in a cer-
Chang received a
Award
rs, the
young
ated a
apers
ctures
ded in
into a
Swiss
per in
bt on a
group
cience
gated,
scover
ispro-
mns of
density
m had
ucture.
d used
energy from adenosine triphosphate to trans-
port molecules across cell membranes. These
so-called ABC transporters perform many
determination was at the root o
cess: “He has an incredible d
ethic. He really pushed the fie
of getting things to
no one else had be
Chang’s data are go
but the faulty so
everything off.
Ironically, anoth
doc in Rees’s lab, K
exposed the mistake
tember issue of Na
now at the Swiss F
ofTechnology in Zu
the structure of anA
calledSav1866from
aureus. The structur
cally—and unexpe
ent from that of
pulling up Sav186
MsbA from S. typh
computer screen, L
realized in minutes
structurewasinvert
the “hand” of a mol
Flipping fiasco. The structures of MsbA (purple) and Sav1866 (green) overlap
little (left) until MsbA is inverted (right).
California.The next year, in a cer-
e White House, Chang received a
l Early Career Award
ts and Engineers, the
ghest honor for young
. His lab generated a
high-profile papers
e molecular structures
proteins embedded in
nes.
e dream turned into a
In September, Swiss
published a paper in
cast serious doubt on a
cture Chang’s group
ed in a 2001 Science
en he investigated,
horrified to discover
madedata-analysispro-
ipped two columns of
ng the electron-density
which his team had
final protein structure.
ly, his group had used
m to analyze data for
cess: “He has an
ethic. He really p
of get
no on
Chan
but t
every
Iro
doc in
expos
temb
now a
ofTec
the str
called
aureu
cally—
ent f
pullin
MsbA
comp
realiz
struct
the “h
a cha
Flipping fiasco. The structures of MsbA (purple) and Sav1866 (green) overlap
little (left) until MsbA is inverted (right).
Sav1866 Dawson & Locher (2006) Nature
Science(2001)Chang&Roth.Science (2001) Chang & Roth.
Comparison with 3D structure of ortholog
Science (2001) Chang & Roth.

LETTERS I BOOKS I POLICY FORUM I EDUCATION FORUM I PERSPECTIVES
1878 1880 1882
LETTERS
edited by Etta Kavanagh
Retraction
WE WISH TO RETRACT OUR RESEARCH ARTICLE “STRUCTURE OF
MsbA from E. coli:A homolog of the multidrug resistanceATP bind-
ing cassette (ABC) transporters” and both of our Reports “Structure of
the ABC transporter MsbA in complex with ADP•vanadate and
lipopolysaccharide”and“X-raystructureoftheEmrEmultidrugtrans-
porter in complex with a substrate” (1–3).
The recently reported structure of Sav1866 (4) indicated that our
MsbA structures (1, 2, 5) were incorrect in both the hand of the struc-
ture and the topology. Thus, our biological interpretations based on
these inverted models for MsbA are invalid.
Anin-housedatareductionprogramintroducedachangeinsignfor
anomalous differences.This program, which was not part of a conven-
tional data processing package, converted the anomalous pairs (I+ and
I-) to (F- and F+), thereby introducing a sign change. As the diffrac-
tion data collected for each set of MsbA crystals and for the EmrE
crystals were processed with the same program, the structures reported
in (1–3, 5, 6) had the wrong hand.
The error in the topology of the original MsbA structure was a con-
sequence of the low resolution of the data as well as breaks in the elec-
tron density for the connecting loop regions. Unfortunately, the use of
the multicopy refinement procedure still allowed us to obtain reason-
able refinement values for the wrong structures.
The Protein Data Bank (PDB) files 1JSQ, 1PF4, and 1Z2R for
MsbA and 1S7B and 2F2M for EmrE have been moved to the archive
of obsolete PDB entries. The MsbA and EmrE structures will be
recalculated from the original data using the proper sign for the anom-
alous differences, and the new Ca coordinates and structure factors
will be deposited.
We very sincerely regret the confusion that these papers have
caused and, in particular, subsequent research efforts that were unpro-
ductive as a result of our original findings.
GEOFFREY CHANG, CHRISTOPHER B. ROTH,
CHRISTOPHER L. REYES, OWEN PORNILLOS,
YEN-JU CHEN, ANDY P. CHEN
Department of Molecular Biology, The Scripps Research Institute, La Jolla, CA 92037, USA.
References
1. G. Chang, C. B. Roth, Science 293, 1793 (2001).
2. C. L. Reyes, G. Chang, Science 308, 1028 (2005).
3. O. Pornillos, Y.-J. Chen, A. P. Chen, G. Chang, Science 310, 1950 (2005).
4. R. J. Dawson, K. P. Locher, Nature 443, 180 (2006).
5. G. Chang, J. Mol. Biol. 330, 419 (2003).
6. C. Ma, G. Chang, Proc. Natl. Acad. Sci. U.S.A. 101, 2852 (2004).
MsbA from E. coli:A homolog of the multidrug resistanceATP bind-
ing cassette (ABC) transporters” and both of our Reports “Structure of
the ABC transporter MsbA in complex with ADP•vanadate and
lipopolysaccharide”and“X-raystructureoftheEmrEmultidrugtrans-
porter in complex with a substrate” (1–3).
The recently reported structure of Sav1866 (4) indicated that our
MsbA structures (1, 2, 5) were incorrect in both the hand of the struc-
ture and the topology. Thus, our biological interpretations based on
these inverted models for MsbA are invalid.
Anin-housedatareductionprogramintroducedachangeinsignfor
anomalous differences.This program, which was not part of a conven-
tional data processing package, converted the anomalous pairs (I+ and
I-) to (F- and F+), thereby introducing a sign change. As the diffrac-
tion data collected for each set of MsbA crystals and for the EmrE
crystals were processed with the same program, the structures reported
in (1–3, 5, 6) had the wrong hand.
The error in the topology of the original MsbA structure was a con-
sequence of the low resolution of the data as well as breaks in the elec-
1860
Untilrecently,GeoffreyChang’scareerwason
a trajectory most young scientists only dream
about. In 1999, at the age of 28, the protein
crystallographer landed a faculty position at
the prestigious Scripps Research Institute in
San Diego, California.The next year, in a cer-
emony at the White House, Chang received a
Presidential Early Career Award
for Scientists and Engineers, the
country’s highest honor for young
researchers. His lab generated a
stream of high-profile papers
detailing the molecular structures
of important proteins embedded in
cell membranes.
Then the dream turned into a
nightmare. In September, Swiss
researchers published a paper in
Nature that cast serious doubt on a
protein structure Chang’s group
had described in a 2001 Science
paper. When he investigated,
Chang was horrified to discover
thatahomemadedata-analysispro-
2001 Science paper, which described the struc-
tureofaproteincalledMsbA,isolatedfromthe
bacterium Escherichia coli. MsbA belongs to a
huge and ancient family of molecules that use
Sciences and
EmrE, a differ
Crystalliz
five membra
was an incred
postdoc advis
nia Institute o
proteins are a
because they
ously diffic
needed for x-
determination
cess: “He has
ethic. He real
of
no
Ch
bu
ev
do
ex
tem
no
of
the
cal
au
ca
en
pu
A Scientist’s Nightmare: Software
Problem Leads to Five Retractions
SCIENTIFIC PUBLISHING

😥
Geoffrey Chang
• Beckman FoundationYoung Investigator
Award
• Presidential Early Career Award
Science (2001) Chang & Roth. Structure of MsbA from
E. coli: a homolog of the multidrug resistance ATP binding
cassette (ABC) transporters.
Journal of Molecular Biology (2003) Chang. Structure
of MsbA from Vibrio cholera: a multidrug resistance ABC
transporter homolog in a closed conformation.
PNAS (2004) Ma & Chang. Structure of the multidrug
resistance efﬂux transporter EmrE from Escherichia coli.
Science (2005) Reyes & Chang. Structure of the ABC
transporter MsbA in complex with ADP vanadate and
lipopolysaccharide.
Science (2005) Pornillos et al. X-ray structure of the
EmrE multidrug transporter in complex with a substrate.
1860
Untilrecently,GeoffreyChang’scareerwason
a trajectory most young scientists only dream
about. In 1999, at the age of 28, the protein
crystallographer landed a faculty position at
the prestigious Scripps Research Institute in
San Diego, California.The next year, in a cer-
emony at the White House, Chang received a
Presidential Early Career Award
for Scientists and Engineers, the
country’s highest honor for young
researchers. His lab generated a
stream of high-profile papers
detailing the molecular structures
of important proteins embedded in
cell membranes.
Then the dream turned into a
nightmare. In September, Swiss
researchers published a paper in
Nature that cast serious doubt on a
protein structure Chang’s group
had described in a 2001 Science
paper. When he investigated,
Chang was horrified to discover
thatahomemadedata-analysispro-
2001 Science paper, which described the struc-
tureofaproteincalledMsbA,isolatedfromthe
bacterium Escherichia coli. MsbA belongs to a
huge and ancient family of molecules that use
Sciences and
EmrE, a differ
Crystalliz
five membra
was an incred
postdoc advis
nia Institute o
proteins are a
because they
ously diffic
needed for x-
determination
cess: “He has
ethic. He real
of
no
Ch
bu
ev
do
ex
tem
no
of
the
cal
au
ca
en
pu
A Scientist’s Nightmare: Software
Problem Leads to Five Retractions
SCIENTIFIC PUBLISHING

This is costly
For:
•the individual
•collaborators
•the institution
•1000s of researchers performing follow-up
work
•science
•society

This changes
everything.
Cost to sequence 1,000,000 nucleotides
~50,000-fold drop in sequencing costs in 10 years.
Any lab can
sequence
anything!
We generate 50,000x more data per $
than 10 years ago

In fact, while some cancer types show
positive age acceleration, others exhibit
negative age acceleration.
20 types of cancer, 6,000 samples total
Methylation proﬁles analysed
—> cancer cells are 36 years older than normal cells
>500 citations

• Understanding/visualising/analysing/massaging big data is hard.
• Biology/life is complex.
• Biologists lack computational training.
• Field is young.
• Analysis tools (generally) suck:
• badly written
• badly tested
• hard to install
• output quality… often questionable.
• Data sizes keep growing!
• Data formats keep changing :(
Genome bioinformatics is hardBiology is harder than (many) other
data sciences

• Avoid costly mistakes
• Be faster:“stand on the shoulders of giants”
• Increase impact / visibility
Reproducible Research &
Sustainable Software

Community Page
Best Practices for Scientific Computing
Greg Wilson1
*, D. A. Aruliah2
, C. Titus Brown3
, Neil P. Chue Hong4
, Matt Davis5
, Richard T. Guy6¤
,
Steven H. D. Haddock7
, Kathryn D. Huff8
, Ian M. Mitchell9
, Mark D. Plumbley10
, Ben Waugh11
,
Ethan P. White12
, Paul Wilson13
1 Mozilla Foundation, Toronto, Ontario, Canada, 2 University of Ontario Institute of Technology, Oshawa, Ontario, Canada, 3 Michigan State University, East Lansing,
Michigan, United States of America, 4 Software Sustainability Institute, Edinburgh, United Kingdom, 5 Space Telescope Science Institute, Baltimore, Maryland, United
States of America, 6 University of Toronto, Toronto, Ontario, Canada, 7 Monterey Bay Aquarium Research Institute, Moss Landing, California, United States of America,
8 University of California Berkeley, Berkeley, California, United States of America, 9 University of British Columbia, Vancouver, British Columbia, Canada, 10 Queen Mary
University of London, London, United Kingdom, 11 University College London, London, United Kingdom, 12 Utah State University, Logan, Utah, United States of America,
13 University of Wisconsin, Madison, Wisconsin, United States of America
Introduction
Scientists spend an increasing amount of time building and
using software. However, most scientists are never taught how to
do this efficiently. As a result, many are unaware of tools and
practices that would allow them to write more reliable and
maintainable code with less effort. We describe a set of best
practices for scientific software development that have solid
foundations in research and experience, and that improve
scientists’ productivity and the reliability of their software.
Software is as important to modern scientific research as
telescopes and test tubes. From groups that work exclusively on
computational problems, to traditional laboratory and field
scientists, more and more of the daily operation of science revolves
around developing new algorithms, managing and analyzing the
large amounts of data that are generated in single research
projects, combining disparate datasets to assess synthetic problems,
and other computational tasks.
Scientists typically develop their own software for these purposes
because doing so requires substantial domain-specific knowledge.
As a result, recent studies have found that scientists typically spend
30% or more of their time developing software [1,2]. However,
90% or more of them are primarily self-taught [1,2], and therefore
lack exposure to basic software development practices such as
writing maintainable code, using version control and issue
error from another group’s code was not discovered until after
publication [6]. As with bench experiments, not everything must be
done to the most exacting standards; however, scientists need to be
aware of best practices both to improve their own approaches and
for reviewing computational work by others.
This paper describes a set of practices that are easy to adopt and
have proven effective in many research settings. Our recommenda-
tions are based on several decades of collective experience both
building scientific software and teaching computing to scientists
[17,18], reports from many other groups [19–25], guidelines for
commercial and open source software development [26,27], and on
empirical studies of scientific computing [28–31] and software
development in general (summarized in [32]). None of these practices
will guarantee efficient, error-free software development, but used in
concert they will reduce the number of errors in scientific software,
make it easier to reuse, and save the authors of the software time and
effort that can used for focusing on the underlying scientific questions.
Our practices are summarized in Box 1; labels in the main text
such as ‘‘(1a)’’ refer to items in that summary. For reasons of space,
we do not discuss the equally important (but independent) issues of
reproducible research, publication and citation of code and data,
and open science. We do believe, however, that all of these will be
much easier to implement if scientists have the skills we describe.
Education
A Quick Guide to Organizing Computational Biology
Projects
William Stafford Noble1,2
*
1 Department of Genome Sciences, School of Medicine, University of Washington, Seattle, Washington, United States of America, 2 Department of Computer Science and
Engineering, University of Washington, Seattle, Washington, United States of America
Introduction
Most bioinformatics coursework focus-
es on algorithms, with perhaps some
components devoted to learning pro-
gramming skills and learning how to
use existing bioinformatics software. Un-
fortunately, for students who are prepar-
ing for a research career, this type of
curriculum fails to address many of the
day-to-day organizational challenges as-
sociated with performing computational
experiments. In practice, the principles
behind organizing and documenting
computational experiments are often
learned on the fly, and this learning is
strongly influenced by personal predilec-
tions as well as by chance interactions
with collaborators or colleagues.
The purpose of this article is to describe
one good strategy for carrying out com-
putational experiments. I will not describe
profound issues such as how to formulate
hypotheses, design experiments, or draw
conclusions. Rather, I will focus on
relatively mundane issues such as organiz-
ing files and directories and documenting
understanding your work or who may be
evaluating your research skills. Most com-
monly, however, that ‘‘someone’’ is you. A
few months from now, you may not
remember what you were up to when you
created a particular set of files, or you may
not remember what conclusions you drew.
You will either have to then spend time
reconstructing your previous experiments
or lose whatever insights you gained from
those experiments.
This leads to the second principle,
which is actually more like a version of
Murphy’s Law: Everything you do, you
will probably have to do over again.
Inevitably, you will discover some flaw in
your initial preparation of the data being
analyzed, or you will get access to new
data, or you will decide that your param-
eterization of a particular model was not
broad enough. This means that the
experiment you did last week, or even
the set of experiments you’ve been work-
ing on over the past month, will probably
need to be redone. If you have organized
and documented your work clearly, then
repeating the experiment with the new
under a common root directory. The
exception to this rule is source code or
scripts that are used in multiple projects
Each such program might have a projec
directory of its own.
Within a given project, I use a top-leve
organization that is logical, with chrono
logical organization at the next level, and
logical organization below that. A sample
project, called msms, is shown in Figure 1
At the root of most of my projects, I have a
data directory for storing fixed data sets, a
results directory for tracking computa
tional experiments peformed on that data
a doc directory with one subdirectory per
manuscript, and directories such as src
for source code and bin for compiled
binaries or scripts.
Within the data and results directo
ries, it is often tempting to apply a similar
logical organization. For example, you
may have two or three data sets agains
which you plan to benchmark your
algorithms, so you could create one
directory for each of them under data
In my experience, this approach is risky
because the logical structure of your finahttp://software.ac.uk

Speciﬁc Approaches/Tools
1. Write code for humans

Write code for humans (not computers!)
• For
• yourself today, in 6 months & in 3 years
• colleagues / collaborators
• reviewers
• other random people who may reuse/improve your code
• Respect conventions (e.g., a style guide)

te Damian ConwayUse whitespace/indentation!
e Damian Conway
Same information

Line length
Strive to limit your code to 80 characters per line. This ﬁts comfortably on a printed page with a
reasonably sized font. If you ﬁnd yourself running out of room, this is a good indication that you
should encapsulate some of the work in a separate function.

ant_measurements <- read.table(file = '~/Downloads/Web/ant_measurements.txt', header=TRUE, se
ant_measurements <- read.table(file = '~/Downloads/Web/ant_measurements.txt',
header = TRUE,
sep = 't',
col.names = c('colony', 'individual', 'headwidth', 'mass')
)
ant_measurements <- read.table(file = '~/Downloads/Web/ant_measurements.txt', header=TRUE,  
sep='t', col.names = c('colony', 'individual', 'headwidth', 'mass'))
Subset of R style guide
http://r-pkgs.had.co.nz/style.html

http://r-pkgs.had.co.nz/style.html
Subset of R style guide

Write code for humans (not computers!)
• For
• yourself today, in 6 months & in 3 years
• colleagues / collaborators
• reviewers
• other random people who may want to reuse your code
• Respect conventions (e.g., a style guide)
• If it runs "fast enough", no need to optimise (generally…)

Code reviews: ask a peer to
(critically) read your analysis code.
Or do peer-programming sessions

2. Organise mindfully

Eliminate redundancy
DRY: Don’t RepeatYourself
& don't reinvent the wheel.
Organise mindfully

Organise mindfully http://bit.ly/projectstruct
Choose a standard/
template and stick to it!

Organise mindfully http://bit.ly/projectstruct

3. Plan for mistakes

Create code tests that are easy to run
• Unit tests == checking edge cases to see if the function works
# do your stuff
# e.g. define speed() function
library(testthat)
expect_that(speed(km = 0, minutes = 60), equals(0))
expect_that(speed(km = 60, minutes = 60), equals(1))
expect_that(my_model, is_a("lm"))
• Integration tests
• == "full analysis" but on small/fake data with known results
• e.g. on fakeVCF genotype ﬁle of 2 loci (one true positive,
one true negative)
• Add "sanity checks". Nonsensical commands should fail!
speed(km= "twenty", minutes=20) # should fail
speed(km = -4, minutes = 60) # should fail
expect_that(speed(km = -4, minutes = 60), throws_error())
expect_that(nrow(significant_SNPs), 42)

Automatically check consistency with style guide
install.packages("lintr") # once
library(lintr) # everytime
lint("file_to_check.R")

"Continuous integration":
Tests should run automagically.
So you don't have to remember (or ﬁnd time) to do it.
💾http://github.org
Tests run
automatically
http://travis-ci.org
If unexpected
result:
📬

4. Use tools that reduce risks

Use tools that reduce risks
• Ensure computers are set up for productivity. E.g.,:
• use GNU parallel on an 40-core machine is in some cases
more more appropriate than submitting to queue
• If you need to make a "pipeline", use software designed for this.
E.g.:
• Nextﬂow
• Snakemake
• (etc)
• too many examples to discuss here

knitr/rmarkdown/
jupyter
Analysis & report in one.
analysis.Rmd
A minimal R Markdown example
I know the value of pi is 3.1416, and 2 times pi is 6.2832. To c
library(knitr); knit( minimal.Rmd )
A paragraph here. A code chunk below:
1+1
## [1] 2
.4-.7+.3 # what? it is not zero!
## [1] 5.551e-17
Graphics work too
library(ggplot2)
qplot(speed, dist, data = cars) + geom_smooth()
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
● ●
0
40
80
120
5 10 15 20
speed
dist
Figure 1: A scatterplot of cars

Github: Facebook for code
• Easy versioning
• Random people use your stuff
• And ﬁnd problems and ﬁx and improve it!
• Greater impact / better world
• Easily update
• Easily collaborate
• Identify new tools
• Build online reputation Demo
Learn at: https://try.github.io/

Getting help.
• In real life: Make friends with people.Talk to them.
• Qmplus forum?
• Online:
• Speciﬁc discussion mailing lists (e.g.: R, bioruby, MAKER...)
• Programming: http://stackoverﬂow.com
• Bioinformatics: http://www.biostars.org
• Sequencing-related: http://seqanswers.com
• Stats: http://stats.stackexchange.com , R-help mailing list.

Summary: Speciﬁc Approaches/Tools
4. Use tools that reduce risks
Bruno
Vieira
Anurag
Priyam
Ismail
Moghul
Roddy
Pracana
Joe
Colgan
https://wurmlab.github.io
+Emeline Favreau
+Karina Zile
+Eckart Stolle
+Leandro Santiago
+Carlos Martinez-Ruiz

“Can you BLAST this for me?”

Anurag Priyam,  
Mechanical engineering student, IIT Kharagpur
Sure, I can
help you…

“Can you BLAST this for me?”
Antgenomes.org SequenceServer
BLAST made easy
(well, we’re trying...)
Aim: An open source idiot-proof web-
interfacefor custom BLAST

Today: SequenceServer Used in >200 labs
17 contributors.

Choosing a programming language
Good: Bad:
Excel quick & dirty easy to make mistakes
doesn’t scale
R
numbers, stats,
genomics
programming
Unix command-line
== shell == bash
Can’t escape it.
Quick & Dirty. HPC.
programming,
complicated things
Java 1990s user interfaces overcomplicated.
Perl 1980s. Everything.
Python scripting, text ugly
Ruby scripting, text
Javascript/Node
scripting, ﬂexibility(web
& client), community
only little bio-stuff

Ruby.“Friends don’t let friends do Perl” - reddit user
### in PERL:
open INFILE, "my_file.txt";
while (defined ($line = <INFILE>)) {
chomp($line);
@letters = split(//, $line);
@reverse_letters = reverse(@letters);
$reverse_string = join("", @reverse_letters);
print $reverse_string, "n";
}
### in Ruby:
File.open("a").each { |line|
puts line.chomp.reverse
}
• example:“reverse each line in ﬁle”
• read ﬁle; with each line
• remove the invisible “end of line” character
• reverse the contents
• print the reversed line

More ruby examples.
5.times {
puts "Hello world"
}
# Sorting people
people_sorted_by_age = people.sort_by { |person| person.age}
+many tools for bio-data - e.g. check http://biogems.info

2017 11-15-reproducible research

Recommended

Recommended

More Related Content

Similar to 2017 11-15-reproducible research

Similar to 2017 11-15-reproducible research (20)

More from Yannick Wurm

More from Yannick Wurm (20)

Recently uploaded

Recently uploaded (20)

2017 11-15-reproducible research