Downloaded 13 times
More Related Content
Similar to 2015 vancouver-vanbug
More from c.titus.brown
2015 vancouver-vanbug
- 1. Building a platformfor bioinformatics: some exciting new directions for khmer. C. Titus Brown ctbrown@ucdavis.edu March 12, 2015
- 2. Hello! Associate Professor (#tenure!); Schoolof Veterinary Medicine University of California, Davis. More information at: • ged.msu.edu/ ( URL needs to be updated :) • github.com/ged-lab/ • ivory.idyll.org/blog/ • @ctitusbrown
- 3. Warnings This talk containsinformation that may constitute “forward-looking statements.” Generally, the words “believe,” “expect,” “intend,” “estimate,” “anticipate,” “project,” “will” and similar expressions identify forward-looking statements, which generally are not historical in nature. I have been advised to put this disclaimer in as well: Dr. Brown is not currently under treatment for any disorders related to megalomania.
- 4. Introducing k-mers CCGATTGCACTGGACCGA (<-read) CCGATTGCAC CGATTGCACT GATTGCACTG ATTGCACTGG TTGCACTGGA TGCACTGGAC GCACTGGACC ACTGGACCGA
- 5. De Bruijn graphs– assemble on overlaps J.R. Miller et al. / Genomics (2010)
- 6. K-mers give youan implicit alignment CCGATTGCACTGGACCGATGCACGGTACCGTATAGCC CATGGACCGATTGCACTGGACCGATGCACGGTACCG
- 7. K-mers give youan implicit alignment CCGATTGCACTGGACCGATGCACGGTACCGTATAGCC CATGGACCGATTGCACTGGACCGATGCACGGTACCG CATGGACCGATTGCACTGGACCGATGCACGGACCG (with no accounting for mismatches or indels)
- 8. The problem withk-mers CCGATTGCACTGGACCGATGCACGGTACCGTATAGCC CATGGACCGATTGCACTCGACCGATGCACGGTACCG Each sequencing error results in k novel k-mers!
- 9. The opportunity: CCGATTGCACTGGACCGATGCACGGTACCGTATAGCC CATGGACCGATTGCACTCGACCGATGCACGGTACCG The graphcontains information about errors (can be used for error trimming in reads). The graph also contains information about variants (can be used for variant calling).
- 10. Conway T C, Bromage A J Bioinformatics 2011;27:479-486 © The Author 2011. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com One big challenge: scalability! De Bruijn graph size scales with # errors.
- 11. One big challenge:scalability! De Bruijn graph size scales with # errors. Memory usage ~ “real” variation + number of errors Number of errors ~ size of data set
- 12. Conway T C, Bromage A J Bioinformatics 2011;27:479-486 © The Author 2011. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com One big challenge: scalability! De Bruijn graph size scales with # errors.
- 13. Goals • Initial goal:can we assemble large data sets?? • Longer-term goal: can we find efficient (De Bruijn?) graph-based approaches to sequence analysis?
- 14. First attempt: compressible DeBruijn graphs 1% 5% 15%10% Pell et al., 2012 Can use Bloom filters to store De Bruijn graph structures. => Overall structure remains as you squish graphs down.
- 15. Technical challenges met(and defeated) • Exhaustive in-memory traversal of graphs containing 5-15 billion nodes. • Sequencing technology introduces false connections in graph. • Implementation lets us scale ~20x over other approaches. Pell et al., 2012
- 16. Technical challenges met(and defeated) • Exhaustive in-memory traversal of graphs containing 5-15 billion nodes. • Sequencing technology introduces false connections in graph. • Implementation lets us scale ~20x over other approaches, but this is not enough. • Although, see Minia assembler (Chikhi et al.) Pell et al., 2012
- 17. Second attempt: diginorm ConwayT C , Bromage A J Bioinformatics 2011;27:479-486 © The Author 2011. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com
- 18. Random sampling =>deep sampling needed Typically 10-100x needed for robust recovery (30-300 Gbp for human)
- 19. Actual coverage varieswidely from the average. Low coverage introduces unavoidable breaks.
- 20. But! Shotgun sequencingis very redundant! Lots of the high coverage simply isn’t needed. (unnecessary data)
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. Contig assembly nowscales with underlying genome size • Transcriptomes, microbial genomes incl MDA, and most metagenomes can be assembled in under 50 GB of RAM, with identical or improved results. • Memory efficient is improved by use of CountMin Sketch. Brown et al., 2012, arXiv.
- 28.
- 29. Diginorm is onlya good start: • Diginorm alters the coverage of the data set. • Diginorm also discards lots of data! • Various other infelicities… o Repeats go away! o Coverage estimation approach ~poor.
- 30. Diginorm is agood start: • Diginorm works on genomes, metagenomes, and transcriptomes; • Diginorm is streaming and uses sublinear space.
- 31. Third attempt: asemi-streaming framework for sequence analysis https://github.com/ged-lab/2014-streaming/
- 32. Diginorm can detect graphsaturation Zhang et al., submitted.
- 33. This generically permitssemi- streaming approaches. Zhang et al., submitted.
- 34. e.g. E. colianalysis => ~1.2 pass, sublinear memory Zhang et al., submitted.
- 35. => Efficient k-mererror trimming. Zhang et al., submitted. (This all works on metagenomes & transcriptomes, too.)
- 36. Moving some sequence analysisto streaming. ~1.2 pass, sublinear memory Zhang et al., submitted. First pass: digital normalization - reduced set of k-mers. Second pass: spectral analysis of data with reduced k-mer set. First pass: collection of low-abundance reads + analysis of saturated reads. Second pass: analysis of collected low-abundance reads. First pass: collection of low-abundance reads + analysis of saturated reads. (a) (b) (c) two-pass; reduced memory few-pass; reduced memory online; streaming.
- 37. Sublinear time/space read erroranalysis -- Zhang et al., submitted. Read error profile from mouse mRNAseq (c.f. Grabherr et al., 2011).
- 38. Another simple algorithm. Zhanget al., submitted.
- 39. So, that’s prettycool, right? • We provide simple time- and memory-efficient approaches for k-mer spectral analysis of large data sets. • These semi-streaming approaches provide a general framework for applying k-mer spectral approaches to all (deep) sequencing data, including genomes, metagenomes, and RNAseq. • The khmer software provides a functional and reasonably efficient reference implementation, freely available under the BSD license and actively developed at github.com/ged-lab/.
- 40. Stream all thethings! (1/2)
- 41. Stream all thethings! (2/2)
- 42. But that’s notall! Buy now, and you can also get sequence-to-graph alignment for the low, low price of free!* graph = khmer.new_counting_hash(…) aligner = khmer.ReadAligner(graph, trusted=5) score, graph_align, read_align, is_truncated = aligner.align(seq) * Terms and conditions may apply. Not all source code fully works :)
- 43. Pair-HMM-based graph alignment Jordan Fishand Michael Crusoe
- 44. (Full model) Jordan Fishand Michael Crusoe
- 45. This is ageneral API… Many potential uses: • Error correction; • Variant calling; • Counting (to replace mapping) & allelic counts; • Align to multiple references; • Tackle strain variation and polyploidy; • Building consensus graphs from shallow population sequencing; • Consensus graph building from multiple read types; • Protein-guided graph search (BlastGraph & Xander)
- 47.
- 48. Graphalign is stillalpha. • We don’t understand parameters well. • Unoptimized. • Not yet competitive with existing approaches. • Broadly applicable! • Hope to engage w/broader community, soon.
- 49. Concluding thoughts #1 •None of our theory is particularly limited to De Bruijn graphs, although our implementation is deeply tied to them at the moment. • We view these ideas (streaming; graphs) as a potentially substantial improvement over current mainstream approaches. • We are not alone – there is a larger community exploring these approaches! (GA4GH, esp.)
- 50. Concluding thoughts #2 •Our implementations are usable but not yet terribly optimized. • We are moving khmer towards a platform for providing reference implementations of these approaches, as well as for research and development. • We are interested in providing components with decent performance & statistical guarantees, for fun and profit. • Python and C++ FTW!
- 51. Thanks! Please contact meat ctbrown@ucdavis.edu!
Editor's Notes
- #11 A sketch showing the relationship between the number of sequence reads and the number of edges in the graph. Because the underlying genome is fixed in size, as the number of sequence reads increases the number of edges in the graph due to the underlying genome that will plateau when every part of the genome is covered. Conversely, since errors tend to be random and more or less unique, their number scales linearly with the number of sequence reads. Once enough sequence reads are present to have enough coverage to clearly distinguish true edges (which come from the underlying genome), they will usually be outnumbered by spurious edges (which arise from errors) by a substantial factor.
- #13 A sketch showing the relationship between the number of sequence reads and the number of edges in the graph. Because the underlying genome is fixed in size, as the number of sequence reads increases the number of edges in the graph due to the underlying genome that will plateau when every part of the genome is covered. Conversely, since errors tend to be random and more or less unique, their number scales linearly with the number of sequence reads. Once enough sequence reads are present to have enough coverage to clearly distinguish true edges (which come from the underlying genome), they will usually be outnumbered by spurious edges (which arise from errors) by a substantial factor.
- #20 High coverage is essential.
- #21 High coverage is essential.
- #28 Goal is to do first stage data reduction/analysis in less time than it takes to generate the data. Compression => OLC assembly.