The document discusses three main problems with de novo assembly of next generation sequencing data and proposes solutions. The three problems are 1) large memory and compute requirements for assembly, 2) complexity of the assembly process and lack of standardized protocols, and 3) limited training opportunities that are difficult for students. The proposed solutions are standardized assembly protocols called khmer-protocols that provide copy-paste workflows for mRNAseq and metagenome assembly using techniques like digital normalization to reduce memory usage and make assembly scalable. The khmer-protocols are designed to be open, versioned, and reproducible to generate initial assembly results cheaply and easily in the cloud.

1. Making de novo assembly cheap & easy:
standardized protocols for mRNAseq and
metagenome assembly and analysis
C. Titus Brown
Assistant Professor
CSE, MMG, BEACON
Michigan State University
Jan 2014
ctb@msu.edu

2. My lab’s focus
 De novo assembly and efficient/effective use of
NGS, especially for non-model organism.
 Open source software engineering.
 Training and education in NGS.

3. There is quite a bit of life left to sequence & assem
http://pacelab.colorado.edu/

4. Three problems:
1.
Assembly memory & compute requirements?
2.
It’s a complex process; what are good defaults?
3.
Training is limited in opportunity, difficult for
students, not always effective.

5. First problem: lots of data!

6. So, we want to go from raw data:
Name
@SRR606249.17/1
GAGTATGTTCTCATAGAGGTTGGTANNNNT
+
B@BDDFFFHHHHHJIJJJJGHIJHJ####1 Quality score
@SRR606249.17/2
CGAANNNNNNNNNNNNNNNNNCCTGGCTCA
+
CCCF#################22@GHIJJJ

7. …to “assembled” original sequence.
UMD assembly primer (cbcb.umd.edu)

8. Practical memory measurements
Velvet measurements (Adina Howe)

9. Shotgun sequencing & de novo
assembly:
It was the Gest of times, it was the wor
, it was the worst of timZs, it was the
isdom, it was the age of foolisXness
, it was the worVt of times, it was the
mes, it was Ahe age of wisdom, it was th
It was the best of times, it Gas the wor
mes, it was the age of witdom, it was th
isdom, it was tIe age of foolishness
It was the best of times, it was the worst of times, it was the
age of wisdom, it was the age of foolishness

10. Why are big data sets difficult?
Need to resolve errors: the more coverage there is, the
more errors there are.
Memory usage ~ “real” variation + number of errors
Number of errors ~ size of data set

11. The scaling problem
 We can cheaply gather DNA data in quantities
sufficient to swamp straightforward assembly
algorithms running on commodity hardware.
 Since ~2008:
 The field has engaged in lots of engineering
optimization…
 …but the data generation rate has consistently
outstripped Moore’s Law.

12. Our solution: Digital
normalization

13. Digital normalization

14. Digital normalization

15. Digital normalization

16. Digital normalization

17. Digital normalization

18. Contig assembly now scales a lot better.
Most samples can be assembled in < 50 GB of
memory.

19. Diginorm is widely useful, becoming
widely used:
1. Assembly of the H. contortus parasitic nematode
genome, a “high polymorphism/variable coverage”
problem.
(Schwarz et al., 2013; pmid 23985341)
2. Reference-free assembly of the lamprey (P.
marinus) transcriptome, a “big assembly” problem.
(in prep)
3. Osedax symbiont metagenome, a “contaminated
metagenome” problem (Goffredi et al, 2013; pmid

20. Second problem: too many choices!
Read trimming
and filtering
(x100)
What
programs and
options do
you use??
Assembly
(x10)
Quantification
(x20)
Science!
(x 10,000)
Annotation
(x20)

21. Third problem: training
 I teach:
 Summer NGS course (two weeks, KBS); heavily
oversubscribed.
 Many ad hoc workshops
 Fall BEACON course (intro computational science)
 Others teach:
 Summer/fall workshops (Robin Buell)
 Various genomics/bioinformatics courses (Shin-han
Shiu, Rob Britton, ???)

22. Overall training results:
 We can fairly easily get people over the initial
“technical” hump (here are some programs,
here’s how to use them).
 We can begin to teach people the way to think
about the problem.
 People have a really tough time connecting
generic instruction to their own research,
however!
(And people need to learn how to analyze their own

23. Three problems:
1.
Assembly memory & compute requirements?
2.
It’s a complex process; what are good defaults?
3.
Training is limited in opportunity, difficult for
students, not always effective.

24. Solution? khmer-protocols
Read cleaning
 Effort to provide standard “cheap”
assembly protocols for Illumina
mRNAseq & metagenomes in the
cloud.
Diginorm
Assembly
 Entirely copy/paste; ~2-6 days from
raw reads to assembly,
annotations, and differential
expression analysis. ~$150 on
Amazon per data set.
Annotation
RSEM differential
expression
 Open, versioned, forkable, citable.

26. “Eel Pond” mRNAseq protocol
Adapter trim &
quality filter
Group transcripts
EBSeq
(Differential
expression
analysis)
Diginorm to C=20
Annotate x
database
Trim highcoverage reads at
low-abundance
k-mers
RSEM (Map QC
reads to count)
Assemble with
Trinity
Extracting
differentially
expressed genes
& graphing

27. “Kalamazoo” metagenome protocol
Adapter trim &
quality filter
Partition
graph
Map reads to
assembly
Diginorm to C=10
Too big to
assemble?
Split into "groups"
Annotate contigs
with abundances
Trim highcoverage reads at
low-abundance
k-mers
Reinflate groups
(optional
Diginorm to C=5
Small enough to assemble?
Assemble!!!
Prokka

28. Show: Web site
http://khmer-protocols.readthedocs.org/

29. Show: mRNAseq output
Differential expression graph

30. Show: mRNAseq spreadsheet

31. Show: BLAST server

32. Soon: Galaxy integration

33. What khmer-protocols is:
 Starting point.
 Defensible initial solution to get initial results.
Works on ~80% or more of samples, guesstimated.
 Great (?) way to learn
 100% reproducible; methods section on
computational analysis is more or less written for you.
 Fairly fast and inexpensive (comparatively)
(~$100/data set)

34. What khmer-protocols is not:
 The One True Solution.
 The Best Solution.
 Proprietary.
 Closed.
 Slow and expensive (comparatively).

35. Speed up/efficiency?
Walltime to complete assemblies
RAM needed to complete assemblies
occ oases occ trinity ocu oases ocu trinity
occ oases occ trinity ocu oases ocu trinity
500
400
Total memory used (GB)
Total walltime (hrs)
75
50
25
300
200
100
0
0
DN RAW
DN RAW
DN RAW
Sample
DN RAW
DN RAW
DN RAW
DN RAW
DN RAW
Sample
Elijah Lowe

36. Diginorm increases sensitivity (very
slightly :)
Evaluation by homology against a reference gene
37 extra from diginorm, vs 17 lost;
64 extra from diginorm, vs 15 lost;
Elijah Lowe

37. Please use!
 Would love feedback: what worked? What didn’t
work?
 Cannot support khmer protocols on HPC, but can
support it in the cloud; iCER may (?) support it on
HPC -- all of the software is installed.
(We are working on better default support for HPC.)

38. Links & more references
 ged.msu.edu/angus/ - NGS course materials
 khmer-protocols.readthedocs.org – khmer
protocols
 Cloud computing discussion next Wed, 1/22,
2pm, iCER. Don’t e-mail me at: ctb@msu.edu