ADAM
Upcoming SlideShare
Loading in...5
×
 

ADAM

on

  • 891 views

Introductory talk on ADAM -- a system of storing and analyzing genomic data using Avro, Parquet and Spark.

Introductory talk on ADAM -- a system of storing and analyzing genomic data using Avro, Parquet and Spark.

Statistics

Views

Total Views
891
Views on SlideShare
888
Embed Views
3

Actions

Likes
1
Downloads
5
Comments
0

1 Embed 3

https://twitter.com 3

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

ADAM ADAM Presentation Transcript

  • ADAM https://github.com/massie/adam Matt Massie University of California, Berkeley massie@berkeley.edu Saturday, November 2, 13
  • SAM BAM ADAM Sequence Alignment Map (SAM) Binary Alignment Map (BAM) Avro Data Alignment Map (ADAM) Saturday, November 2, 13
  • Pipeline Issues Today: Time and Scale • The time to go from reads to answers is too long • Processing thousands of BAM files for statistical analysis doesn’t scale Saturday, November 2, 13
  • ADAM: Speed and Scale • Read BAM once, perform transformations (e.g. sort, mark duplicates, BQSR) in distributed memory, write the analysisready ADAM file once • Use a distribute filesystem (HDFS), a fast execution system (Spark) and columnar data formats (Parquet) to scale Saturday, November 2, 13
  • Unlocking Genomic Data Shark (SQL) Hadoop M/R Spark Impala (SQL) ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM ADAM Hadoop Distributed File System (HDFS) Local Filesystem ADAM ADAM ADAM ADAM Saturday, November 2, 13 BAM
  • record ADAMRecord { union union union union union union union union union union union { { { { { { { { { { { null, null, null, null, null, null, null, null, null, null, null, string } referenceName = null; int } referenceId = null; long } start = null; int } mapq = null; string } readName = null; string } sequence = null; string } mateReference = null; long } mateAlignmentStart = null; string } cigar = null; string } qual = null; string } recordGroupId = null; union union union union union union union union union union union { { { { { { { { { { { boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, boolean, null null null null null null null null null null null } } } } } } } } } } } http://avro.apache.org/ readPaired = false; properPair = false; readMapped = false; mateMapped = false; readNegativeStrand = false; mateNegativeStrand = false; firstOfPair = false; secondOfPair = false; primaryAlignment = false; failedVendorQualityChecks = false; duplicateRead = false; union { null, string } mismatchingPositions = null; union { null, string } attributes = null; union union union union union union union union union union } { { { { { { { { { { null, null, null, null, null, null, null, null, null, null, string } recordGroupSequencingCenter = null; string } recordGroupDescription = null; long } recordGroupRunDateEpoch = null; string } recordGroupFlowOrder = null; string } recordGroupKeySequence = null; string } recordGroupLibrary = null; int } recordGroupPredictedMedianInsertSize = null; string } recordGroupPlatform = null; string } recordGroupPlatformUnit = null; string } recordGroupSample = null; union { null, int } mateReferenceId = null; Saturday, November 2, 13
  • Parquet http://parquet.io Column-oriented layout Row-oriented layout https://blog.twitter.com/2013/dremel-made-simple-with-parquet Saturday, November 2, 13
  • Genomic Data Example chrom20 TCGA 4M chrom20 GAAT 4M1D chrom20 CCGAT 5M Column Oriented chrom20 chrom20 chrom20 TCGA GAAT CCGAT 4M 4M1D 5M Row Oriented chrom20 Saturday, November 2, 13 TCGA 4M chrom20 GAAT 4M1D chrom20 CCGAT 5M
  • http://spark.incubator.apache.org/ Saturday, November 2, 13
  • Low-Coverage BAM Experiment • 14GB Low-coverage BAM with 145M reads • 10-node ec2 cluster m2.4xlarge • Reduced to 13GB with ADAM • Conversion/upload to HDFS 22mins • Sorted in 7minutes Saturday, November 2, 13
  • High-Coverage BAM Experiment • Input: 237GB NA12878- high coverage, PCR free, whole-genome BAM • Conversion took 4hrs on ec2 m2.4xlarge (8cpu, 68.4gb mem) • Output size: 237GB BAM reduced to 212GB ADAM Saturday, November 2, 13
  • Current Features • • • • • Saturday, November 2, 13 Convert BAM to ADAM (read-oriented) Sort an ADAM file by reference Generate ADAMPileups Print mpileup output Very soon ADAM will be able to mark duplicates (initial benchmarks look good)
  • In progress... • Frank is working on a distributed variant caller (https:// github.com/fnothaft/avocado), local realignment, adam2bam • Chris Hartl is integrating ADAM with GATK (https:// github.com/chartl/GAParquet) DiagnoseTargets, adding new VCF formats to ADAM, BQSR • Christos Kozanitis has been working on Shark and Impala integration for ad-hoc SQL read queries • Collaborations with Mt. Sinai, GenomeBridge and the Broad Institute who are interested in using ADAM Saturday, November 2, 13