The document discusses scaling metagenome assembly to very large datasets. It proposes two approaches: 1) Partitioning the de Bruijn graph to divide the data into smaller, independent components that can be assembled in parallel. However, partitioning faces the same challenges of scaling to large datasets as other assemblers. 2) Digital normalization, which "squashes" redundant high coverage reads in a single pass to reduce the total data while preserving sequence content and gene coverage. Initial assemblies of field samples demonstrate the approaches can assemble petabase-scale datasets on a single compute node. Future work involves optimizing partitions, scaling implementations, and integrating other data types.

1. Scaling metagenome assembly –to infinity and beeeeeeeeeeyond! C. Titus Brown et al. Computer Science / Microbiology Depts Michigan State University In collaboration with Great Prairie Grand Challenge (Tiedje, Jansson, Tringe)

2. SAMPLING LOCATIONS

3. Sampling strategy per site 1 M 1 cM 10 M 1 cM Reference soil 1 M Soil cores: 1 inch diameter, 4 inches deep Total: 8 Reference metagenomes + 64 spatially separated cores (pyrotag sequencing) 10 M

4. Great Prairie sequencing summary 200x human genome…! > 10x more challenging (total diversity)

5. Our perspective Great Prairie project: there is no end to the data! Immense biological depth: estimate ~1-2 TB (10**12) of raw sequence needed to assemble top ~20-40% of microbes. Improvements in sequencing tech Existing methods for scaling assembly simply will not suffice: this is a losing battle. Abundance filtering XXX Better data structures XXX Parallelization is not going to be sufficient; neither are advances in data structures. I think: bad scaling is holding back assembly progress.

6. Our perspective, #2 Deep sampling is needed for these samples Illumina is it, for now. The last thing in the world we want to do is write yet another assembler…pre-assembly filtering, instead. All of our techniques can be used together with any assembler. We’ve mostly stuck with Velvet, for reasons of historical contingency.

7. Two enabling technologies Very efficient k-mer counting Bloom counting hash/MinCount Sketch data structure; constant memory Scales ~10x over traditional data structures k-independent. Probabilistic properties well suited to next-gen data sets. Very efficient de Bruijn graph representation We traverse k-mers stored in constant-memory Bloom filters. Compressible probabilistic data structure; very accurate. Scales ~20x over traditional data structures. K-independent. …cannot directly be used for assembly because of FP.

8. Approach 1: Partitioning Use compressible graph representation to explore natural structure of data: many disconnected components.

9. Partitioning for scaling Can be done in ~10x less memory than assembly. Partition at low k and assemble exactly at any higher k (DBG). Partitions can then be assembled independently Multiple processors -> scaling Multiple k, coverage -> improved assembly Multiple assembly packages (tailored to high variation, etc.) Can eliminate small partitions/contigs in the partitioning phase. In theory, an exact approach to divide and conquer/data reduction.

10. Adina Howe

11. Partitioning challenges Technical challenge: existence of “knots” in the graph that artificially connect everything. Unfortunately, partitioning is not the solution. Runs afoul of same k-mer/error scaling problem that all k-mer assemblers have… 20x scaling isn’t nearly enough, anyway 

12. Digression: sequencing artifacts Adina Howe

13. Partitioning challenges Unfortunately, partitioning is not the solution. Runs afoul of same k-mer/error scaling problem that all k-mer assemblers have… 20x scaling isn’t nearly enough, anyway 

14. Approach 2: Digital normalization “Squash” high coverage reads Eliminate reads we’ve seen before (e.g. “> 5 times”) Digital version of experimental “mRNA normalization”. Nice algorithm! Single-pass Constant memory Trivial to implement Easy to parallelize / scale (memory AND throughput) “Perfect” solution? (Works fine for MDA, mRNAseq…)

15. Digital normalization Two benefits: Decrease amount of data (real, but redundant sequence) Eliminate errors associated that redundant sequence. Single-pass algorithm (c.f. streaming sketch algorithms)

16. Digital normalization validation? Two independent methods for comparing assemblies… by both of them, we get very similar results for raw and treated.

17. Comparing assemblies quantitatively Build a “vector basis” for assemblies out of orthogonal M-base windows of DNA. This allows us to disassemble assemblies into vectors, compare them, and even “subtract” them from one another.

18. Running HMMs over de Bruijn graphs(=> cross validation) hmmgs: Assemble based on good-scoring HMM paths through the graph. Independent of other assemblers; very sensitive, specific. 95% of hmmgsrplB domains are present in our partitioned assemblies. CTC ACT TTC GTA GAC ATA ACC CTA Jordan Fish, Qiong Wang, and Jim Cole (RDP) GTT

19. Digital normalization validation Two independent methods for comparing assemblies… by both of them, we get very similar results for raw and treated. Hmmgs results tell us that Velvet multi-k assembly is also very sensitive. Our primary concern at this point is about long-range artifacts (chimeric assembly).

20. Techniques Developed suite of techniques that work for scaling, without loss of information (?) While we have no good way to assess chimeras and misassemblies, basic sequence content and gene content stay the same across treatments. And… what, are we just sitting here writing code? No! We have data to assemble!

21. Assembling Great Prairie data, v0.8 Iowa corn GAII, ~500m reads / 50 Gb => largest partition ~200k reads 84 Mb in 53,501 contigs > 1kb. Iowa prairie GAII, ~500m reads / 50 Gb => biggest ~100k read partition 102 MB in 70,895 contigs > 1kb. Both done on a single 8-core Amazon EC2 bigmem node, 68 GB of RAM, ~$100. (Yay, we can do it! Boo, we’re only using 2% of reads.) No systematic optimization of partitions yet; 2-4x improvement expected. Normalization of HiSeq is also yet to be done. Have applied to other metagenomes, note; longer story.

22. Future directions? khmer software reasonably stable & well-tested; needs documentation, software engineering love. github.com/ctb/khmer/ (see ‘refactor’ branch…) Massively scalable implementation (HPC & cloud). Scalable digital normalization (~10 TB / 1 day? ;) Iterative partitioning Integrating other types of sequencing data (454, PacBio, …)? Polymorphism rates / error rates seem to be quite a bit higher. Validation and standard data sets? Someone? Please?

23. Lossless assembly; boosting.

24. Acknowledgements: Thek-mer gang: Adina Howe, Jason Pell, ArendHintze, Qingpeng Zhang, Rose Canino-Koning, Tim Brom. mRNAseq: LikitPreeyanon, Alexis Pyrkosz, Hans Cheng, Billie Swalla, and Weiming Li. HMM graph search: Jordan Fish, Qiong Wang, Jim Cole. Great Prairie consortium: Jim Tiedje, Rachel Mackelprang, Susannah Tringe, Janet Jansson Funding: USDA NIFA; MSU, startup and iCER; DOE; BEACON/NSF STC; Amazon Education.

25. Acknowledgements: Thek-mer gang: Adina Howe, Jason Pell, ArendHintze, Qingpeng Zhang, Rose Canino-Koning, Tim Brom. mRNAseq: LikitPreeyanon, Alexis Pyrkosz, Hans Cheng, Billie Swalla, and Weiming Li. HMM graph search: Jordan Fish, Qiong Wang, Jim Cole. Great Prairie consortium: Jim Tiedje, Rachel Mackelprang, Susannah Tringe, Janet Jansson Funding: USDA NIFA; MSU, startup and iCER; DOE; BEACON/NSF STC; Amazon Education.

27. Lumps! Adina Howe

28. Lumps! Adina Howe

29. Knots in the graph are caused by sequencing artifacts.

30. Identifying the source of knots Use a systematic traversal algorithm to identify highly-connected k-mers. Removal of these k-mers (trimming) breaks up the knots. Many, but not all, of these highly-connected k-mers are associated with high-abundance k-mers.

31. Highly connected k-mers are position-dependent Adina Howe

32. HCKs under-represented in assembly Adina Howe

33. HCKs tend to end contigs Adina Howe

34. Our current model Contigs are extended or joined around artifacts, with an observation bias towards such extensions (because of length cutoff). Tendency is for a long contig to be extended by 1-2 reads, so artifacts trend towards location at end of contig. Adina Howe

35. Conclusions (artifacts) They connect lots of stuff (preferential attachment) They result from something in the sequencing (3’ bias in reads) Assemblers don’t like using them The major effect of removing them is to shorten many contigs by a read.

36. Digital normalization algorithm for read in dataset: if median_kmer_count(read) < CUTOFF: update_kmer_counts(read) save(read) else: # discard read

37. Supplemental: abundance filtering is very lossy.

38. Per-partition assembly optimization Strategy: Vary k from 21 to 51, assemble with velvet. Choose k that maximizes sum(contigs > 1kb) Ran top partitions in Iowa corn (4.2m reads, 303 partitions) For k=33, 3.5 mb in 1876 contigs > 1kb, max 15.7 kb For best k for each partition(varied between 31 and 47), 5.7 mb in 2511 contigs > 1kb, max 51.7 kb

39. Comparing assemblies quantitatively Build a “vector basis” for assemblies out of orthogonal M-base windows of DNA. This allows us to disassemble assemblies into vectors, compare them, and even “subtract” them from one another.

40. Comparing assemblies / dendrogram