Challenges with data quality,
sharing, and versioning
David Dooling <ddooling@wustl.edu>
GIA 2009

Production Centers
• Tony Cox, Sanger • David Dooling, WUStL
Sequencing Scale
Scale Quality
Infrastructure Sharing
Data flow Versioning
• Toby Bloom, Broad
Quality
Integration
Standards
Sharing
<ddooling@wustl.edu>

sub scale {
<ddooling@wustl.edu>

Moore’s Law
,-./011-2#
300.-4/#567#
8,9#
:;0.6<-#
:-=>-1?-#
!\"\"\"# !\"\"$# !\"\"!# !\"\"%# !\"\"&# !\"\"'# !\"\"(# !\"\")# !\"\"*# !\"\"+# !\"$\"#
<ddooling@wustl.edu>

Images
200 TB/week
<ddooling@wustl.edu>

Images
10 PB/year
<ddooling@wustl.edu>

Perspective
20 PB/day
<ddooling@wustl.edu>

Perspective
2 PB/s
<ddooling@wustl.edu>

FASTQ
@HWI-EAS404:5:1:6:180#0/1
GCTGGTTTAACTCGAGTATTTGTCCATTCTACTAATTTGAGTGTCTGCTGTGGAAAGGTGTTTGTCATGTATTTT
+HWI-EAS404:5:1:6:180#0/1
aaaa`]aaaa`aa^aa]aaaa^\\`_\\``____`W]a_`T\\[[b__`\\YXUW][MSTNZX^[[`_Z[^``\\X`^a\\
@HWI-EAS404:5:1:6:396#0/1
TATTTACTCTATCCCATTATATACATATTATGATTTCAAAATAACAATGCCAATATAAAAACTAACAATATGATA
+HWI-EAS404:5:1:6:396#0/1
Yaaa_baa`^a]Wa___aaa^I^V]^]NQ_`\\^ZPP[__^_\\a`^a`JYQWVNFFMRQSX_X^a_Y[`^a^\\\\NZ
@HWI-EAS404:5:1:6:1344#0/1
GAGGACTTGCATGCTAGGTTTGGTTCTTGGCTGAATTGCTGAAACTGTCCAAGTATCAGTAGCAAAACATGGGTG
+HWI-EAS404:5:1:6:1344#0/1
aabaaa__]^a`[\\^`]]Y``[ST_]`]WW]]WZ]`^ZT[_X``\\`_WVNYWKDNLTW[Y\\XSVZ^ZTZZVRUX[
@HWI-EAS404:5:1:6:1814#0/1
AAAGCTTACTGCTGTTTAGAATTCTTGCTACAGTCAGGAGAAAGCCGAAAGCTGAACGGGTACTGAATCTTCTAC
+HWI-EAS404:5:1:6:1814#0/1
aa````aa^a`_^`\\`a`XY`^ZX^YW\\^[X\\UWUYOMVZZ\\\\_W^^\\XXTSMHMLLNTTDWU__[WVVY]Y_]X
7 TB/week
<ddooling@wustl.edu>

FASTQ
@HWI-EAS404:5:1:6:180#0/1
GCTGGTTTAACTCGAGTATTTGTCCATTCTACTAATTTGAGTGTCTGCTGTGGAAAGGTGTTTGTCATGTATTTT
+HWI-EAS404:5:1:6:180#0/1
aaaa`]aaaa`aa^aa]aaaa^\\`_\\``____`W]a_`T\\[[b__`\\YXUW][MSTNZX^[[`_Z[^``\\X`^a\\
@HWI-EAS404:5:1:6:396#0/1
TATTTACTCTATCCCATTATATACATATTATGATTTCAAAATAACAATGCCAATATAAAAACTAACAATATGATA
+HWI-EAS404:5:1:6:396#0/1
Yaaa_baa`^a]Wa___aaa^I^V]^]NQ_`\\^ZPP[__^_\\a`^a`JYQWVNFFMRQSX_X^a_Y[`^a^\\\\NZ
@HWI-EAS404:5:1:6:1344#0/1
GAGGACTTGCATGCTAGGTTTGGTTCTTGGCTGAATTGCTGAAACTGTCCAAGTATCAGTAGCAAAACATGGGTG
+HWI-EAS404:5:1:6:1344#0/1
aabaaa__]^a`[\\^`]]Y``[ST_]`]WW]]WZ]`^ZT[_X``\\`_WVNYWKDNLTW[Y\\XSVZ^ZTZZVRUX[
@HWI-EAS404:5:1:6:1814#0/1
AAAGCTTACTGCTGTTTAGAATTCTTGCTACAGTCAGGAGAAAGCCGAAAGCTGAACGGGTACTGAATCTTCTAC
+HWI-EAS404:5:1:6:1814#0/1
aa````aa^a`_^`\\`a`XY`^ZX^YW\\^[X\\UWUYOMVZZ\\\\_W^^\\XXTSMHMLLNTTDWU__[WVVY]Y_]X
350 TB/year
<ddooling@wustl.edu>

Mapping
2 TB/week
<ddooling@wustl.edu>

Mapping
100 TB/year
<ddooling@wustl.edu>

Mapping
42,000 core-hr/week
<ddooling@wustl.edu>

Mapping
5 core-yr/week
<ddooling@wustl.edu>

Mapping
250 core cluster
<ddooling@wustl.edu>

The Weakest Link
<ddooling@wustl.edu>

The Balanced PC
• Clock speed
• AGP
• Front-side bus
• Hypertransport
• 1 Gbps
• PCI-X
• SATA
• PCI-Express
• Infiniband
• Multi-core
• Front-side bus
• GPU
• 10 Gbps
<ddooling@wustl.edu>

The balanced PS 1
10 gosub get(sequencers)
20 gosub get(disk)
30 gosub get(backup_capacity)
40 gosub get(network_capacity)
50 gosub get(cluster_nodes)
1 - Pipeline for Sequencing
<ddooling@wustl.edu>

The unbalanced PS
10 gosub get(sequencers)
20 gosub get(disk)
30 gosub get(backup_capacity)
40 gosub get(network_capacity)
50 gosub get(cluster_nodes)
60 goto 10
<ddooling@wustl.edu>

The GHz race
<ddooling@wustl.edu>

} # scale
<ddooling@wustl.edu>

sub quality {
<ddooling@wustl.edu>

Honda
<ddooling@wustl.edu>

Honda
<ddooling@wustl.edu>

Honda
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Ford
<ddooling@wustl.edu>

Quality is Job 1
<ddooling@wustl.edu>

...must be more than
just a slogan
<ddooling@wustl.edu>

Quality missteps
Initial low fidelity between base
quality values and quality
Tsonev, S. SEP 2007
<ddooling@wustl.edu>

An aside
“basecall calibration predicted vs. observed”
<ddooling@wustl.edu>

Cult of traces
<ddooling@wustl.edu>

Quality is the key
Need high fidelity between prediction and observed
50 bytes per base
20 bytes per base
2 bytes per base
3 bits per base
<ddooling@wustl.edu>

The down side
http://www3.appliedbiosystems.com/cms/
groups/mcb_marketing/documents/
generaldocuments/cms_057559.pdf
http://mammoth.psu.edu/labPhotos/imageOfFlowgram.jpg
<ddooling@wustl.edu>

} # quality
<ddooling@wustl.edu>

sub sharing {
<ddooling@wustl.edu>

1000 Genomes
<ddooling@wustl.edu>

3.8 Tb
<ddooling@wustl.edu>

~50 B/b
<ddooling@wustl.edu>

190 TB
<ddooling@wustl.edu>

Submitted to central
repositories
<ddooling@wustl.edu>

... and replicated
across the pond
<ddooling@wustl.edu>

The goal of this project is to provide a system
for storing and retrieving huge amounts of
data, distributed among a large number of
heterogenous server nodes, under a single
virtual filesystem tree with a variety of
standard access methods.
<ddooling@wustl.edu>

Write-only databases
Search limited to sequence and
values of specific XML entities
submitted as metadata
<ddooling@wustl.edu>

Write-only databases
x
Search limited to sequence and
values of specific XML entities
submitted as metadata
<ddooling@wustl.edu>

Speaking of XML
<?xml version=\"1.0\" encoding=\"UTF-8\"?> <?xml version=\"1.0\" encoding=\"UTF-8\"?> <LS454>
<STUDY_SET xmlns:xsi=\"http://www.w3.org/2001/ <EXPERIMENT_SET xmlns:xsi=\"http://www.w3.org/ <INSTRUMENT_MODEL>GS 20</ <VALUE>TACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
XMLSchema-instance\"> 2001/XMLSchema-instance\"> INSTRUMENT_MODEL> ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
<STUDY alias=\"LowSalternSDbayVir111005\" <EXPERIMENT ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
accession=\"SRP000145\"> alias=\"LowSalternSDbayVir111005_experiment\" <FLOW_SEQUENCE>TACGTACGTACGTACGTACGTACGTACGTACGT ACGTACGTACGTACGTACGTACGTACGTACG</VALUE>
<DESCRIPTOR> expected_number_runs=\"2\" accession=\"SRX000217\"> ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT </RUN_ATTRIBUTE>
<STUDY_TITLE>Solar Salterns, viral <TITLE>454 sequencing of saltern metagenome ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT <RUN_ATTRIBUTE>
fraction from low salinity saltern in San Diego, fragment library</TITLE> ACGTACGTACGTACGTACGTACGTACGTACGTACGTACG</ <TAG>key_sequence</TAG>
CA </STUDY_TITLE> <STUDY_REF accession=\"SRP000145\" FLOW_SEQUENCE> <VALUE>TCAG</VALUE>
<STUDY_TYPE refname=\"LowSalternSDbayVir111005\"/> <FLOW_COUNT>168</FLOW_COUNT> </RUN_ATTRIBUTE>
existing_study_type=\"Metagenomics\"/> <DESIGN> </LS454> </RUN_ATTRIBUTES>
<STUDY_ABSTRACT>Viral community from a <DESIGN_DESCRIPTION>454 Sequencing of </PLATFORM> </RUN>
\"low\" salinity saltern and sequenced at 454 Life viral fraction from low salinity saltern in San <PROCESSING> <RUN alias=\"D1LDSHL\" instrument_model=\"454 GS
Sciences. </STUDY_ABSTRACT> Diego, CA</DESIGN_DESCRIPTION> <BASE_CALLS> 20\" run_date=\"2006-04-06T09:25:19Z\"
<CENTER_NAME>SDSU</CENTER_NAME> <SAMPLE_DESCRIPTOR accession=\"SRS000373\" <SEQUENCE_SPACE>Base Space</ run_file=\"D1LDSHL\" run_center=\"454MSC\"
refname=\"28373\"/> SEQUENCE_SPACE> total_data_blocks=\"1\" accession=\"SRR001054\">
<CENTER_PROJECT_NAME>LowSalternSDbayVir111005</ <LIBRARY_DESCRIPTOR> <BASE_CALLER>454BaseCaller</BASE_CALLER> <EXPERIMENT_REF accession=\"SRX000217\"
CENTER_PROJECT_NAME> <LIBRARY_NAME>lowSalternSDbayVir111005</ </BASE_CALLS> refname=\"LowSalternSDbayVir111005_experiment\"/>
<PROJECT_ID>28373</PROJECT_ID> LIBRARY_NAME> <QUALITY_SCORES qtype=\"phred\"> <DATA_BLOCK name=\"D1LDSHL\" region=\"1\"
</DESCRIPTOR> <LIBRARY_STRATEGY>OTHER</ <QUALITY_SCORER>454BaseCaller</ total_spots=\"70935\" total_reads=\"70935\"
<STUDY_ATTRIBUTES> LIBRARY_STRATEGY> QUALITY_SCORER> number_channels=\"1\" format_code=\"1\" sector=\"0\">
<STUDY_ATTRIBUTE> <LIBRARY_SOURCE>OTHER</LIBRARY_SOURCE> <NUMBER_OF_LEVELS>64</NUMBER_OF_LEVELS> <FILES>
<TAG>NCBI parent project ID</TAG> <LIBRARY_SELECTION>RANDOM</ <MULTIPLIER>1</MULTIPLIER> <FILE filename=\"D1LDSHL01.sff\"
<VALUE>28725</VALUE> LIBRARY_SELECTION> </QUALITY_SCORES> filetype=\"sff\"/>
</STUDY_ATTRIBUTE> <LIBRARY_LAYOUT> </PROCESSING> </FILES>
</STUDY_ATTRIBUTES> <SINGLE/> </EXPERIMENT> </DATA_BLOCK>
</STUDY> </LIBRARY_LAYOUT> </EXPERIMENT_SET> <RUN_ATTRIBUTES>
</STUDY_SET> <LIBRARY_CONSTRUCTION_PROTOCOL> <RUN_ATTRIBUTE>
none provided <TAG>flow_count</TAG>
<?xml version=\"1.0\" encoding=\"UTF-8\"?> </LIBRARY_CONSTRUCTION_PROTOCOL> <?xml version=\"1.0\" encoding=\"UTF-8\"?> <VALUE>168</VALUE>
<SAMPLE_SET xmlns:xsi=\"http://www.w3.org/2001/ </LIBRARY_DESCRIPTOR> <RUN_SET xmlns:xsi=\"http://www.w3.org/2001/ </RUN_ATTRIBUTE>
XMLSchema-instance\"> <SPOT_DESCRIPTOR> XMLSchema-instance\"> <RUN_ATTRIBUTE>
<SAMPLE alias=\"28373\" accession=\"SRS000373\"> <SPOT_DECODE_SPEC> <RUN alias=\"D0IIGP3\" instrument_model=\"454 GS <TAG>flow_sequence</TAG>
<SAMPLE_NAME> <NUMBER_OF_READS_PER_SPOT>2</ 20\" run_date=\"2006-03-17T09:39:51Z\"
<TAXON_ID>496920</TAXON_ID> NUMBER_OF_READS_PER_SPOT> run_file=\"D0IIGP3\" run_center=\"454MSC\" <VALUE>TACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
<COMMON_NAME>saltern metagenome</ <READ_SPEC> total_data_blocks=\"1\" accession=\"SRR001053\"> ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
COMMON_NAME> <READ_INDEX>0</READ_INDEX> <EXPERIMENT_REF accession=\"SRX000217\" ACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGTACGT
</SAMPLE_NAME> <READ_CLASS>Technical Read</ refname=\"LowSalternSDbayVir111005_experiment\"/> ACGTACGTACGTACGTACGTACGTACGTACG</VALUE>
<DESCRIPTION>viral fraction from low READ_CLASS> <DATA_BLOCK name=\"D0IIGP3\" region=\"1\" </RUN_ATTRIBUTE>
salinity saltern in San Diego, CA </ <READ_TYPE>Adapter</READ_TYPE> total_spots=\"51121\" total_reads=\"51121\" <RUN_ATTRIBUTE>
DESCRIPTION> <BASE_COORD>1</BASE_COORD> number_channels=\"1\" format_code=\"1\" sector=\"0\"> <TAG>key_sequence</TAG>
<SAMPLE_ATTRIBUTES> </READ_SPEC> <FILES> <VALUE>TCAG</VALUE>
<SAMPLE_ATTRIBUTE> <READ_SPEC> <FILE filename=\"D0IIGP301.sff\" </RUN_ATTRIBUTE>
<TAG>collection_date</TAG> <READ_INDEX>1</READ_INDEX> filetype=\"sff\"/> </RUN_ATTRIBUTES>
<VALUE>11/10/05</VALUE> <READ_CLASS>Application Read</ </FILES> </RUN>
</SAMPLE_ATTRIBUTE> READ_CLASS> </DATA_BLOCK> </RUN_SET>
<SAMPLE_ATTRIBUTE> <READ_TYPE>Forward</READ_TYPE> <RUN_ATTRIBUTES>
<TAG>lat_lon</TAG> <BASE_COORD>5</BASE_COORD> <RUN_ATTRIBUTE>
<VALUE>32.599040, -117.107356</VALUE> </READ_SPEC> <TAG>flow_count</TAG>
</SAMPLE_ATTRIBUTE> </SPOT_DECODE_SPEC> <VALUE>168</VALUE>
</SAMPLE_ATTRIBUTES> </SPOT_DESCRIPTOR> </RUN_ATTRIBUTE>
</SAMPLE> </DESIGN> <RUN_ATTRIBUTE>
</SAMPLE_SET> <PLATFORM> <TAG>flow_sequence</TAG>
<ddooling@wustl.edu>

} # sharing
<ddooling@wustl.edu>

sub versioning {
<ddooling@wustl.edu>

The Cathedral and the Bazaar
Linux overturned much of what I thought I
knew. I had been preaching the Unix gospel of
small tools, rapid prototyping and evolutionary
programming for years. But I also believed
there was a certain critical complexity above
which a more centralized, a priori approach was
required. I believed that the most important
software (operating systems and really large
tools like the Emacs programming editor)
needed to be built like cathedrals, carefully
crafted by individual wizards or small bands of
mages working in splendid isolation, with no
beta to be released before its time.
<ddooling@wustl.edu>

The Vatican and the Reformation
<ddooling@wustl.edu>

The popes
Will this scale?
<ddooling@wustl.edu>

GenBank genome
http://betterexplained.com/articles/intro-to-distributed-version-control-illustrated/
<ddooling@wustl.edu>

git genome
http://betterexplained.com/articles/intro-to-distributed-version-control-illustrated/
<ddooling@wustl.edu>

The Human Reference
>7 dna:chromosome chromosome:NCBI36:7:1:158821424:1
...AATAACTATATAAGTAAATAAGCAAGCTGTATGAATATACAAAGCTCTCTGGTAAAG
GTAAATACATAAACAAACATAAAAACAGTCCTATTGTAATTTTGGTTTGTAACTCTGCTT
TTTATTTTCTACATAATTTAAAAGGCAAATGCATAAAATGTAATTGTAAATCTGTTAGCT
GGTATACAATGAATAAAGATATAATTTGTCACATCAATAACATAAAAAGAGTAGAGCTAT
ATATATAGCAGTAGAATTTTGGTATGTGATTGAACTTAAGTTGAAATAAATTCAAATTAA
AATGTTATAACTCTAGGATGTTATATGTAATTCTCATAGTAACCAAAAATGAAATATACA
TAGAATATAAACAAAAGGAAATGAGACTAGAAACAAAATGTGTCACTACAAAAAAATCAA
CTAAAGATAAAAAAGAAATAATTGAGAAAATGATTGGCAAAAATCAGTAACTCTGACGTA
TTAAAACTTTCCATGCTACATAAATCTGAAAACTCTATTTCACATAAAACTGGAGCTGAA
AGAAACAAATATTTACCTATAAAGTTAAAAGTTATATAGGGAACAAACACTAATTTTTTT
TAGAAAAAATTATAAAAAGAGTAAAAATATGCCTTATACTACCGTAATTTCATGTTTTAC
AGCTCTGGGAAAATAGAAAATAAAATGTTCTGTTAGCATGAATCCCTCTGTGCCCCC...
<ddooling@wustl.edu>

The Human Reference
<ddooling@wustl.edu>

The Human Reference
(a) 2
A
4(24) B
82
3(2)
5 7 16(2)
3(3) 2
3 3(2) 2
5 58(2)
3(2)
2(2) 8 2(3)
6(2)
2(219) 2 2
23(2) 3
2
2 3 81
3(21)
4(22) 4(3)
13 3(24)
3
A 2(2)
2(2) 2(202)
19(8)
2(19) 2(15) 2 2(34)
2(13)
158
C
5(7) 2(42) 4(9)
2(15)
2(4)
7(8)
3(3) 71
B
18
2
C
2
D
37
F
139 6
E E
13(2) 13(2) 55(3)
2(6) 2(7) 6(3)
4(7)
4 5 2
F 3 D
38(6) 3(5)
160 3(50) 2
G G
2 2(61)
4(51) 2(49)
3(50) 8
2(7)
H
4
2(4)
142 2(50) 5
5(5) 8(6) 5(7)
158
3
3(41)
173
H
(b) (c) 142
G 160
81
13(7) 158
117 93 29
D H
A 184
9(6)
H
48(10)
140 8
8(5)
38(6)
114 G
F
13(2)
13(2) 55(3)
132
207 D
139
A 82
127(2) B E
62
E
37 71
B
F 37
139 D
F
13(2) 55(3)
E D 21 158
32(3) 45(3)
A
13(2) C
s5766
13(2)
38(6)
20(2)
18
F G
B
8
8(5) A
81
18(6) 58(7) E
171 C
G 123(2) 82
B
D Zhi, BJ Raphael, AL Price, H Tang and PA Pevzner. Identifying repeat domains in
large genomes. Genome Biology 2006, 7:R7
<ddooling@wustl.edu>

} # versioning
<ddooling@wustl.edu>

sub thank {\"you\"}
<ddooling@wustl.edu>