A Survey of NGS Data Analysis on Hadoop

•Download as PPTX, PDF•

1 like•3,598 views

This document discusses approaches to analyzing next-generation sequencing (NGS) data on Hadoop. It summarizes three tools - CloudBurst, CloudAligner, and DistMap - that use distributed computing for mapping DNA reads to a reference genome. It also discusses challenges related to processing large NGS datasets and the need for scalable solutions. Hardware approaches like the bina processor are presented as ways to accelerate NGS analysis. Overall, the document outlines the rise of NGS as generating big genomic data and the computational challenges in analyzing this data at scale.

Introduction of NGS Data Analysis on Hadoop
Chung-Tsai Su
SPN Architect, Core Tech
Trend Micro
2014/10/31 @CSIE.NTU
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 1

Q＆A
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 2 http://setmoney.blob.core.windows.net/newsimages/2014/09/04/136352-XXL.jpg

http://www.genome.gov/sequencingcosts/
NGS Era

NGS Pipeline
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 4

High-Level Workflow of NGS
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 5
Read
Mapping
Raw
Reads
(.fq)
Variant
Calling
Sequence
Alignment/
Mapping
(.sam/.bam)
Variant
Calling file
(.vcf)

NGS Data Analysis Pipeline
• GATK best practice
h1t0t/3p1/s20:1/4/wwwCo.bnfidreontaiald | Cinopsyritgihtt u20t1e2 .Torenrdg M/igcroa Itnkc. /guide/best6-practices?bpm=DNAseq

illumina solution
7
http://systems.illumina.com/content/dam/illumina-marketing/
documents/products/brochures/brochure_sequencing_systems_portfolio.pdf

The First $1,000 Genome – illumina HiSeq X Ten
h1t0t/3p1:/2//0s14ystemCso.niflidleunmtiali |n Caop.ycrioghmt 2/0s12y Tsretnedm Miscr/oh Inics.eq-x-sequen8cing-system.html

Expectation of Data Processing
Power for illumina HiSeq X Ten
• A cluster of 10 HiSeq X instruments
• Capable of sequencing up to 18,000 whole human
genomes each year
– Has a run cycle of ~3 days and produces ~150 genomes each
run cycle
– Running the industry standard BWA+GATK analysis pipeline to
perform this analysis on a reasonably high-end (Dual Intel Xeon
E5-2697v2 CPU – 12 core, 2.7 GHz with 96 GB DRAM)
compute server takes ~24 hours per genome.
– To achieve the required throughput of 150 genomes every three
days, at least 50 of these servers are required.
• Should meet a target of ~28 minutes for the completion
of the mapping, aligning, sorting, de-duplication and
variant calling of each genome.
h1t0t/3p1/:2/0/1w4 ww.Ceodnfidicenotiagl | eConpyorigmht 2e0.1c2 Toremnd /Mdicrroa Ingc.en/ 9

Literature Survey
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 10

Literature
• CloudBurst, 2009
• CloudAligner, 2011
• DistMap, 2013
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 11

10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 12

Algorithm of CloudBurst
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 13
Seed-and-Extend
Algorithm

Experiments$
Performance of CloudBurst
Scalability+
16000
14000
12000
10000
8000
6000
4000
2000
0
Running Time vs Number of Reads on Chr 1
0 1 2 3 4 5 6 7 8
Runtime (s)
Millions of Reads
0 1
2 3
4
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 14

Speedup over Serial RMAP
EECS$584$–$Fall$2013$
Speedup+over+serial+RMAP+
40
35
30
25
20
15
10
5
0
Speedup over serial RMAP
0 1 2 3 4
Speedup
Number of Mismatches
chr1 chr22
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 15

Experiments$
Speedup on EC2
Speedup+on+EC2+
1800
1600
1400
1200
1000
800
600
400
200
0
Running Time on EC2
High-CPU Medium Instance Cluster
24 48 72 96
Running time (s)
Number of Cores
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 16

10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 17

Overhead of Disk I/O
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 18

Architecture of CloudAligner
Seed-and-Extend
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 19
Algorithm

Performance on Small Data
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 20

Performance on Large Data
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 21

Performance on Amazon EMR
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 22

Comparison with CloudBurst and CloudAligner
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 23

10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 24

Workflow of DistMap
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 25

Evaluation of Read Mapping tools
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 26

Comparison of DistMap and other tools for
distributed mapping
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 27

Market Movement
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 28

Hardware Solution -
The World’s First NGS Bioinformatics Processor
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 29

h1t0t/3p1/:2/0/1w4 ww.Cbonifnidean.tical o| Cmopy/rpighrto 20d12u Tcretn.dh Mticmro Ilnc. 30

Architecture of bina Technology
h1t0t/3p1/:2/0/1w4 ww.Cbonifnidean.tical o| Cmopy/rtigehtc 2h01n2 Torelnod gMiycr.oh Intcm. l 31

h1t0t/3p1s/2:0/1/4www.dConnafidnenetixal u| Cso.pcyorigmht 2/i0m12 aTrgeneds M/iucrso Iencc.ases/dnanex3u2s_CHARGE_prod1.png

Summary
• NGS is a new page for Big Data Era
• Need more CS experts to solve scalability and
performance issues
• Also, need more Data Scientist to discover the
secrets/insights of Human Genome
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 33

http://technews.tw/2014/08/02/gene-big-data/
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 34 http://technews.tw/2014/08/02/gene-big-data/

Q＆A
10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 35

This document discusses the advantages of using Make and Snakemake for bioinformatics workflows compared to other options like shell scripts and Perl. It outlines key features of Make like its focus on files, ability to restart safely, and support for parallelization. The document then introduces Snakemake as building upon Make's strengths while addressing weaknesses like lack of control flow and cluster support. It provides examples of using Snakemake with cluster resources and utilities like logs, dry-runs and workflow diagrams. Finally, it discusses best practices like versioning, caching, integration with R, leaving paper trails and ensuring reproducibility.

Reproducible bioinformatics pipelines with Docker and Anduril

Christian Frech

Christian Frech presented on using Docker and Anduril to create reproducible bioinformatics pipelines. Anduril is a pipeline framework that aims to make pipelines modular, bundled with their execution environment, and able to be run on clusters. It uses a proprietary scripting language but can embed other languages. Frech demonstrated an RNA-seq analysis pipeline built in Anduril, which generated QC plots, differential expression results, network and enrichment analyses. While adoption of Anduril has been limited by its scripting language, Docker can be used to containerize components and make pipelines fully reproducible and portable.

Computational infrastructure for NGS data analysis

cursoNGS

Journal Seminar: Is Singularity-based Container Technology Ready for Running ...

Kento Aoyama

1. The study evaluates the performance of running MPI applications in Singularity containers on HPC clouds and local clusters. 2. Experiments were conducted on Chameleon Cloud using Intel Xeon Haswell nodes and a local cluster using Intel Knights Landing nodes. 3. Results show that Singularity incurs less than 8% overhead for MPI point-to-point and collective communications and for HPC applications, demonstrating its potential for efficiently running MPI workloads on HPC clouds and systems.

LUGM-Update of the Illumina Analysis Pipeline

Hai-Wei Yen

Computational workflows for omics analyses at the IARC

Matthieu Foll

This document discusses the use of computational workflows and nextflow at the International Agency for Research on Cancer (IARC). IARC uses high throughput sequencing and omics data to study cancer causes and prevention. While research groups have different scientific questions, nextflow helps avoid duplicating bioinformatics efforts and promotes best practices. Key benefits of nextflow include easy installation, reproducibility, and ability to run pipelines on any machine or cluster. Challenges include debugging, handling multiple inputs, and deleting intermediate files. Overall nextflow has changed how IARC conducts bioinformatics research for the better.

Workshop NGS data analysis - 1

Maté Ongenaert

This document discusses formal verification of functional code. It begins with motivations for formal verification like improving software dependability for mission-critical and safety-critical systems. It then provides an overview of formal verification, describing it as mathematically proving properties of a system's mathematical model. The document discusses examples of formal verification that programmers may already be familiar with, like proving properties of algorithms. It also covers formal verification methods in more depth, like model checking, theorem proving, and temporal logic. Finally, it discusses examples of projects that used formal verification like miTLS, F*, and seL4 microkernel.

Progressive Provenance Capture Through Re-computation

Paul Groth

Provenance capture relies upon instrumentation of processes (e.g. probes or extensive logging). The more instrumentation we can add to processes the richer our provenance traces can be, for example, through the addition of comprehensive descriptions of steps performed, mapping to higher levels of abstraction through ontologies, or distinguishing between automated or user actions. However, this instrumentation has costs in terms of capture time/overhead and it can be difficult to ascertain what should be instrumented upfront. In this talk, I'll discuss our research on using record-replay technology within virtual machines to incrementally add additional provenance instrumentation by replaying computations after the fact.

NGS: Mapping and de novo assembly

Bioinformatics and Computational Biosciences Branch

This document provides an introduction and overview of common methods for processing and analyzing next generation sequencing (NGS) data, including mapping NGS reads and de novo assembly of NGS reads. It discusses various NGS applications such as RNA-Seq, epigenetics, structural variation detection, and metagenomics. Key steps in read alignment such as choosing an alignment program and viewing alignments are outlined. Considerations for choosing an alignment program based on library type, read type, and platform are also reviewed. Popular alignment programs including Bowtie, BWA, TopHat, and Novoalign are mentioned.

Sciunits: Resuable Research Object

Tanu Malik

This document introduces sciunits, which are reusable research objects that capture application executions, repeat executions, and reproduce executions with different input arguments. Sciunits are versioned, self-contained, and use provenance for self-documentation. They address the lack of an easily creatable, readily reusable, efficiently versioned discrete unit of computation. The document describes the sciunit architecture, packaging process, versioning solution, storage and retrieval, provenance visualization, and summarization techniques. It also provides examples of sciunit applications and performance results for packaging, repeating, and versioning sciunits.

Software Dev

Adrian Wisernig

The document discusses using the pHash library for audio fingerprinting. pHash is a C/C++ library that supports fingerprinting of audio, video, and images. It generates hashes based on the bark scale frequencies that humans can hear and compares the bit error rate between hashes to determine a match. The document outlines a project to use pHash for audio fingerprinting, including a bash script to run the process and an 87 line C++ file to calculate hashes and determine matches. Areas for potential improvement are also discussed.

Big data solution for ngs data analysis

Yun Lung Li

This document outlines a presentation on big data solutions for NGS data analysis using software containerization and distributed analytics. It discusses Docker containerization and its use at Atgenomix to simplify cluster environments. It also covers NGS genome analysis techniques like read mapping, variant calling, and using Spark and Hadoop for data parallelization. Elasticsearch is introduced for distributed, RESTful search and analytics of variant data.

Interactive Data Analysis for End Users on HN Science Cloud

Helix Nebula The Science Cloud

The document provides a status report on testing the Helix Nebula Science Cloud for interactive data analysis by end users of the TOTEM experiment. It summarizes the deployment of a "Science Box" platform on the Helix Nebula Cloud using technologies like EOS, CERNBox, SWAN and SPARK. Initial tests of the platform were successful in 2017 using a single VM. Current tests involve a scalable deployment with Kubernetes and using SPARK as the computing engine. Synthetic benchmarks and a TOTEM data analysis example show the platform is functioning well with room to scale out storage and computing resources for larger datasets and analyses.

Galaxy RNA-Seq Analysis: Tuxedo Protocol

Hong ChangBum

This document provides an overview of analyzing RNA-Seq data using the Tuxedo protocol in Galaxy. It describes experimental design considerations, quality control of sequencing data using FastQC, mapping reads to a reference genome using Tophat, determining differential expression with Cuffdiff, and visualizing results using IGV and CummeRbund. The tutorial walks through an example analysis on Drosophila melanogaster RNA-Seq data, covering topics such as setting file formats, running alignment and expression tools, extracting workflows, and useful Galaxy resources.

Manycores for the Masses

Intel® Software

NERSC is the production high-performance computing (HPC) center for the United States Department of Energy (DOE) Office of Science. The center supports over 6,000 users in 600 projects, using a variety of applications in materials science, chemistry, biology, astrophysics, high energy physics, climate science, fusion science, and more. NERSC deployed the Cori system on over 9,000 Intel® Xeon Phi™ processors. This session describes the optimization strategy for porting codes that target traditional manycore architectures to the processors. We also discuss highlights and lessons learned from the optimization process on 20 applications associated with the NERSC Exascale Science Application Program (NESAP).

Attackboard slides dac12-0605

Yoshi Shih-Chieh Huang

The document introduces Attackboard, a novel dependency-aware traffic generator for exploring network-on-chip (NoC) design space. Attackboard rebuilds state machines from packet dependency traces to generate traffic, greatly reducing storage overhead while maintaining accuracy. It extracts packet arrival patterns to reconstruct state machines, called attackboards, for each processing element. These attackboards are then used to drive traffic simulation at runtime instead of storing large traces. Attackboard achieves better tradeoffs between accuracy, size, and simulation time compared to previous trace-based approaches.

A Library for Emerging High-Performance Computing Clusters

Intel® Software

This document discusses the challenges of developing communication libraries for exascale systems using hybrid MPI+X programming models. It describes how current MPI+PGAS approaches use separate runtimes, which can lead to issues like deadlock. The document advocates for a unified runtime that can support multiple programming models simultaneously to avoid such issues and enable better performance. It also outlines MVAPICH2's work on designs like multi-endpoint that integrate MPI and OpenMP to efficiently support emerging highly threaded systems.

NANO266 - Lecture 9 - Tools of the Modeling Trade

University of California, San Diego

UCSD NANO 266 Quantum Mechanical Modelling of Materials and Nanostructures is a graduate class that provides students with a highly practical introduction to the application of first principles quantum mechanical simulations to model, understand and predict the properties of materials and nano-structures. The syllabus includes: a brief introduction to quantum mechanics and the Hartree-Fock and density functional theory (DFT) formulations; practical simulation considerations such as convergence, selection of the appropriate functional and parameters; interpretation of the results from simulations, including the limits of accuracy of each method. Several lab sessions provide students with hands-on experience in the conduct of simulations. A key aspect of the course is in the use of programming to facilitate calculations and analysis.

RESTful Triple Spaces of Things

Open University, KMi

This document proposes adapting Triple Space Computing (TSC) to the Internet of Things (IoT) using a RESTful HTTP API. It presents Otsopack, an open source middleware that exposes TSC querying primitives over HTTP. The evaluation covers 4 case studies of running Otsopack on different devices: an XBee gateway with Python, a FoxG20 board with a Python implementation, a Samsung Galaxy Tab using Android, and a regular computer. The studies show semantic reasoning is feasible but performance varies greatly between devices. A RESTful interface helps build a distributed system but semantic capabilities are still challenging for constrained IoT devices.

Use C++ and Intel® Threading Building Blocks (Intel® TBB) for Hardware Progra...

Intel® Software

In this presentation, we focus on an alternative approach that uses nodes that contain Intel® Xeon® processors and Intel® Xeon Phi™ coprocessors. Programming models and the development tools are identical for these resources, greatly simplifying development. We discuss how the same models for vectorization and threading can be used across these compute resources to create software that performs well on them. We further propose an extension to the Intel® Threading Building Blocks (Intel® TBB) flow graph interface that enables intra-node distributed memory programming, simplifying communication, and load balancing between the processors and coprocessors. Finally, we validate this approach by presenting a benchmark of a risk analysis implementation that achieves record-setting performance.

LO-PHI: Low-Observable Physical Host Instrumentation for Malware Analysis

Pietro De Nicolao

Lichtenberg bosc2010 wordseeker

BOSC 2010

This document summarizes a concurrent bioinformatics software tool called WordSeeker that uses parallel and distributed computing approaches to efficiently discover genome-wide patterns and word-based genomic signatures. The software uses techniques like radix trees, suffix trees, and clustering to analyze input sequences in parallel across multiple cores and nodes. It was tested on the Arabidopsis thaliana genome and able to analyze full genomes in under 5 hours using distributed memory across 5 nodes. Future work will focus on additional parallelization and post-processing capabilities.

OVH AntiDDoS : Threat Detection

Steven Le Roux

This document discusses OVH's Anti-DDoS infrastructure which uses Apache Kafka and Apache Storm for threat detection. It handles 3 Tbps of traffic across 17 datacenters and 32 points of presence. Data is ingested into Kafka topics and processed through Storm topologies to perform stream processing and scoring to detect attacks like scans and router grouping. The document also discusses configuration, hardware, benchmarks and thanks various open source contributors.

Scientific Computing @ Fred Hutch

Dirk Petersen

Variability, Bugs, and Cognition

Andrzej Wasowski

The document discusses variability, bugs, and cognition. It summarizes findings from analyzing known variability bugs. Key findings include: bugs often appear in unanticipated configurations that were not considered by programmers; many bugs involve features defined across subsystems rather than locally; and managing variability and large numbers of configurations can negatively impact understandability and maintainability. Controlled experiments are proposed to study how variability affects the time and accuracy of bug finding.

Introduction to next generation sequencing

VHIR Vall d’Hebron Institut de Recerca

This document provides an introduction to next generation sequencing (NGS) technologies. It begins with an outline of topics to be covered, including the evolution of NGS technologies, their descriptions and comparisons, bioinformatics challenges of NGS data analysis, and some aspects of NGS data analysis workflows and tools. The document then delves into explanations of specific NGS platforms, their performance characteristics, and the sequencing processes. It discusses the large computational infrastructure and data management needs of NGS, as well as quality control, preprocessing of NGS data, and popular analysis tools and workflows.

NGS Targeted Enrichment Technology in Cancer Research: NGS Tech Overview Webi...

QIAGEN

This slidedeck discusses the most biologically efficient, cost-effective method for successful NGS. The GeneRead DNA QuantiMIZE Kits enable determination of the optimum conditions for targeted enrichment of DNA isolated from biological samples, while the GeneRead DNAseq Panels V2 allow you to quickly and reliably deep sequence your genes of interest. Applications in translational and clinical research are highlighted.

What's hot

Whole Genome Sequencing - Data Processing and QC at SciLifeLab NGI

Phil Ewels

Formal Verification of Functional Code

Martin Děcký

Progressive Provenance Capture Through Re-computation

Paul Groth

NGS: Mapping and de novo assembly

Bioinformatics and Computational Biosciences Branch

Sciunits: Resuable Research Object

Tanu Malik

Software Dev

Adrian Wisernig

Big data solution for ngs data analysis

Yun Lung Li

Interactive Data Analysis for End Users on HN Science Cloud

Helix Nebula The Science Cloud

Galaxy RNA-Seq Analysis: Tuxedo Protocol

Hong ChangBum

Manycores for the Masses

Intel® Software

Attackboard slides dac12-0605

Yoshi Shih-Chieh Huang

A Library for Emerging High-Performance Computing Clusters

Intel® Software

NANO266 - Lecture 9 - Tools of the Modeling Trade

University of California, San Diego

RESTful Triple Spaces of Things

Open University, KMi

Use C++ and Intel® Threading Building Blocks (Intel® TBB) for Hardware Progra...

Intel® Software

LO-PHI: Low-Observable Physical Host Instrumentation for Malware Analysis

Pietro De Nicolao

Lichtenberg bosc2010 wordseeker

BOSC 2010

OVH AntiDDoS : Threat Detection

Steven Le Roux

Scientific Computing @ Fred Hutch

Dirk Petersen

Variability, Bugs, and Cognition

Andrzej Wasowski

What's hot (20)

Whole Genome Sequencing - Data Processing and QC at SciLifeLab NGI

Formal Verification of Functional Code

Progressive Provenance Capture Through Re-computation

NGS: Mapping and de novo assembly

Sciunits: Resuable Research Object

Software Dev

Big data solution for ngs data analysis

Interactive Data Analysis for End Users on HN Science Cloud

Galaxy RNA-Seq Analysis: Tuxedo Protocol

Manycores for the Masses

Attackboard slides dac12-0605

A Library for Emerging High-Performance Computing Clusters

NANO266 - Lecture 9 - Tools of the Modeling Trade

RESTful Triple Spaces of Things

Use C++ and Intel® Threading Building Blocks (Intel® TBB) for Hardware Progra...

LO-PHI: Low-Observable Physical Host Instrumentation for Malware Analysis

Lichtenberg bosc2010 wordseeker

OVH AntiDDoS : Threat Detection

Scientific Computing @ Fred Hutch

Variability, Bugs, and Cognition

Viewers also liked

Introduction to next generation sequencing

VHIR Vall d’Hebron Institut de Recerca

NGS Targeted Enrichment Technology in Cancer Research: NGS Tech Overview Webi...

QIAGEN

NGS - Basic principles and sequencing platforms

Annelies Haegeman

Aug2014 abrf interlaboratory study plans

GenomeInABottle

The document describes the ABRF Next Generation Sequencing Study which aims to produce reference data sets to establish baseline performance of sequencing platforms and methods. Phase I focused on RNA-Seq and produced major conclusions about intraplatform and interplatform concordance. Phase II will focus on DNA sequencing including performance using different platforms/protocols, damaged DNA, small genomes, and oncogenic mutations. Samples and sequencing plans are described for three projects. The study is a collaboration between ABRF and Genome in a Bottle to generate standardized reference data.

Molecular QC: Interpreting your Bioinformatics Pipeline

Candy Smellie

What is the impact of assay failure in your laboratory and how do you monitor for it? The most heavily degraded samples are not suitable for standard exome coverage: sometimes it’s not even a matter of getting bad sequencing, you might get nothing at all! FFPE artifacts increase with storage time Artifacts go against the statistical power of your variant calling analysis Molecular reference standards help filter out bad mappings and spurious variants Bioinformatics pipelines allow adding Molecular Reference Standards in your joint variant calling pipeline Genome In A Bottle Reference Standards are invaluable for validating variant calling analysis NIST and its collaborators shared datasets created with most NGS technologies Horizon Diagnostics shared annotated, merged variant calls from NIST for the Ashkenazim Trio ~35K variants are predicted having high or moderate impact within the Trio GM24385 (Ashkenazim Son) includes 352 small variants with high/moderate impact which are absent in Father and Mother Routinely monitor the performance of your workflows and assays with independent external controls

Dr. Douglas Marthaler - Use of Next Generation Sequencing for Whole Genome An...

John Blue

A different kettle of fish entirely: bioinformatic challenges and solutions f...

Lex Nederbragt

The document discusses the bioinformatic challenges of whole genome assembly for Atlantic cod and Atlantic salmon using high-throughput sequencing techniques. It provides context on the developments in high-throughput sequencing technologies that have enabled increasing throughput and read lengths over time. The hierarchical structure of genome assembly is explained, from initial reads to longer contigs and scaffolds that are pieced together to reconstruct the original genome fragments.

I Jornada Actualización en Genética Reproductiva y Fertilidad

TECNALIA Research & Innovation

Illumina is a global leader in genomic technologies that provides sequencing and microarray solutions. Their portfolio includes the MiSeq, NextSeq 500, HiSeq 2500, and HiSeq X Ten sequencing platforms. Illumina serves customers in human health, research, reproductive health, forensics, cancer, agriculture and more. Their reproductive and genetic health portfolio addresses the continuum from preconception to pregnancy to genetic conditions. Products highlighted include VeriSeq PGS for embryo screening, Karyomapping for inherited disorder screening, and their non-invasive prenatal testing solution in development. The NextSeq 550 was also introduced as a platform that enables both sequencing and array scanning applications on a single system.

Next generation sequencing in preimplantation genetic screening (NGS in PGS)

Mahidol University, Thailand

This document discusses the use of next generation sequencing (NGS) technology for preimplantation genetic screening (PGS). It provides background on PGS and describes the timeline of technologies used, including fluorescence in situ hybridization (FISH), array comparative genomic hybridization (aCGH), SNP microarrays, and NGS. VeriSeq PGS is highlighted as an NGS assay developed by Illumina for PGS using embryos biopsied on day 3 or 5. Studies validating VeriSeq showed 100% consistency with aCGH and karyotyping in detecting chromosome abnormalities. NGS provides advantages over other methods like higher resolution, batch analysis to reduce costs, and ability to detect mosaicism.

Global Next Generation Sequencing (NGS) Industry By Market Size & Forecast to...

DavidClark206

This research report covers end-to-end market for NGS in terms of the workflow; presequencing, NGS platforms, consumables and services, sequencing services and bioinformatics market. From an insight perspective, this research report focuses on the qualitative data, market size, share, and growth of various segments and sub-segments, competitive landscape, and company profiles. Inquire For Discount (Single User Report Price US $4650) @ http://www.reportsnreports.com/contacts/Discount.aspx?name=257153 .

NGx Sequencing 101-platforms

AllSeq

The field of next-generation sequencing (NGS) has been experiencing explosive growth over the past several years and shows little sign of slowing down. The increasing capabilities and dramatically lowered costs have expanded NGS's reach beyond that of the human genome into nearly every corner of biological research. An overview of the platforms on the market today, including an assessment of their relative strengths and weaknesses, will be presented. The presentation will conclude with a peek into where the technology is going and what will be available in the future.

Next-generation sequencing from 2005 to 2020

Christian Frech

Next-generation sequencing technologies have rapidly advanced since 2005. Key developments include massively parallel sequencing reactions that enabled sequencing of entire human genomes for less than $1,000 by 2015. While Illumina dominates the market, other platforms like Ion Torrent and PacBio are increasing capabilities. Routine human whole genome sequencing is now used in research and medicine, enabling new opportunities like liquid biopsies and single-cell analysis. However, data storage and analysis remain challenges due to the large volumes of sequencing data.

Next-Generation Sequencing an Intro to Tech and Applications: NGS Tech Overvi...

QIAGEN

A Comparison of NGS Platforms.

mkim8

This document provides an overview and comparison of popular next-generation sequencing platforms. It discusses the common sequencing pipeline including library preparation, massively parallel sequencing, and bioinformatics analysis. Popular platforms like Roche 454, Illumina, and SOLiD are described in detail focusing on their specific sequencing chemistries and performance characteristics. Newer third-generation platforms such as Ion Torrent, PacBio, and Oxford Nanopore are also introduced. A wide range of NGS applications from whole genome sequencing to RNA-seq are outlined.

Clinical Applications of Next Generation Sequencing

Bell Symposium & MSP Seminar

The Molecular Diagnostics Laboratory processes around 20,000 specimens annually for various testing categories including infectious diseases, hematological malignancies, solid tumor malignancies, and inherited disorders. Next generation sequencing is proposed as a solution to provide more comprehensive genetic testing by simultaneously evaluating variations in multiple genes from a single sample. While next generation sequencing offers advantages over traditional testing methods, there are also technical and clinical challenges to address in implementation, including optimizing detection of structural variations and interpretation of variants with unclear clinical significance.

Ngs part i 2013

Elsa von Licy

This document provides an overview of next generation sequencing technologies and applications. It summarizes an upcoming webinar series on next generation sequencing and its role in cancer biology. The first webinar will provide an introduction to next generation sequencing technologies and applications and be presented by Quan Peng on April 4, 2013. The following two webinars will focus on next generation sequencing for cancer research and data analysis and be presented on April 11 and 18, 2013 respectively.

Next Generation Sequencing and its Applications in Medical Research - Frances...

Sri Ambati

The so-called “next-generation” sequencing (NGS) technologies allows us, in a short time and in parallel, to sequence massive amounts of DNA, overcoming the limitations of the original Sanger sequencing methods used to sequence the first human genome. NGS technologies have had an enormous impact on biomedical research within a short time frame. This talk will give an overview of these applications with specific examples from Mendelian genomics and cancer research. #h2ony

140127 abrf interlaboratory study proposal

GenomeInABottle

The document describes Phase II of the ABRF Next Generation Sequencing Study which aims to establish reference data sets for evaluating DNA sequencing performance across multiple platforms and laboratories. Phase II will sequence various human and bacterial genomic samples to assess accuracy, coverage, and limits of detection using different platforms and library preparation methods. A collaboration with NIST Genome in a Bottle will provide standardized samples to the participating laboratories. The study aims to provide a resource for ongoing method development and evaluation of sequencing performance.

Galaxy dna-seq-variant calling-presentationandpractical_gent_april-2016

Prof. Wim Van Criekinge

This document provides an overview of variant analysis from next-generation sequencing data. It begins with introductions to the CCA-Drylab@VUmc, TraIT, and Galaxy projects. The focus of the lecture is explained to be variant analysis from NGS data using interactive demos in Galaxy. Background is provided on Illumina sequencing technology and properties of sequencing reads. Key steps in variant analysis are outlined, including quality control and read mapping, variant calling and annotation using tools like FastQC, BWA, FreeBayes, and SnpEff. Formats for storing sequencing data and variants are also introduced, such as FASTQ, SAM/BAM, and VCF.

2016 iHT2 San Diego Health IT Summit

Health IT Conference – iHT2

The document discusses Illumina's role in advancing precision medicine through next-generation sequencing and data analytics. It notes that while sequencing costs have decreased dramatically, challenges remain in interpreting, integrating, and analyzing the large volumes of genomic and other healthcare data. Illumina aims to develop comprehensive, patient-centric analytics platforms and knowledgebases to help address these challenges and enable more effective prevention, diagnosis, and treatment based on a patient's genetics, environment, and lifestyle. The success of these efforts will be measured by improvements in patient outcomes, healthcare costs and efficiencies, and changes in clinical practice guided by integrated genomic and clinical data analysis.

Viewers also liked (20)

Introduction to next generation sequencing

NGS Targeted Enrichment Technology in Cancer Research: NGS Tech Overview Webi...

NGS - Basic principles and sequencing platforms

Aug2014 abrf interlaboratory study plans

Molecular QC: Interpreting your Bioinformatics Pipeline

Dr. Douglas Marthaler - Use of Next Generation Sequencing for Whole Genome An...

A different kettle of fish entirely: bioinformatic challenges and solutions f...

I Jornada Actualización en Genética Reproductiva y Fertilidad

Next generation sequencing in preimplantation genetic screening (NGS in PGS)

Global Next Generation Sequencing (NGS) Industry By Market Size & Forecast to...

NGx Sequencing 101-platforms

Next-generation sequencing from 2005 to 2020

Next-Generation Sequencing an Intro to Tech and Applications: NGS Tech Overvi...

A Comparison of NGS Platforms.

Clinical Applications of Next Generation Sequencing

Ngs part i 2013

Next Generation Sequencing and its Applications in Medical Research - Frances...

140127 abrf interlaboratory study proposal

Galaxy dna-seq-variant calling-presentationandpractical_gent_april-2016

2016 iHT2 San Diego Health IT Summit

Similar to A Survey of NGS Data Analysis on Hadoop

Introduction to OpenDaylight and Hydrogen, Learnings from the Year, What's Ne...

David Meyer

OpenDaylight is an open source SDN platform developed under the Linux Foundation. The presentation discusses OpenDaylight and its first release (Hydrogen). Key points: - OpenDaylight aims to create a common, extensible SDN platform to further adoption and innovation. - The Hydrogen release included 14 projects and delivered core SDN functions. It faced various issues during development. - Looking ahead, the speaker discusses continuing to build the community, improving code quality, and releasing new projects to address areas like security and distributed systems. The overall goal is to advance the OpenDaylight platform.

Building the iRODS Consortium

All Things Open

Extending twitter's data platform to google cloud

Vrushali Channapattan

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Extending Twitter's Data Platform to Google Cloud

lohitvijayarenu

This document summarizes Twitter's project to extend its data platform from an on-premises only model to a hybrid on-premises and cloud model called Partly Cloudy. It discusses design considerations around user experience, scalability, and onboarding. It outlines the workstreams to replicate datasets to Google Cloud Storage while maintaining authentication, authorization, and audit controls. The project has enabled Twitter data processing users to access replicated datasets, use managed Hadoop and Presto clusters in Google Cloud, and explore other Google offerings.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

Big data processing using - Hadoop Technology

Shital Kat

This document summarizes a report on Hadoop technology as a solution to big data processing. It discusses the big data problem, including defining big data, its characteristics and challenges. It then introduces Hadoop as a solution, describing its components HDFS for storage and MapReduce for parallel processing. Examples of common friend lists and word counting are provided. Finally, it briefly mentions some Hadoop projects and companies that use Hadoop.

Accumulo Summit 2014: Addressing big data challenges through innovative archi...

Accumulo Summit

The ability to collect and analyze large amounts of data is a growing problem within the scientific community. The growing gap between data and users calls for innovative tools that address the challenges faced by big data volume, velocity and variety. The Massachusetts Institute of Technology, Lincoln Laboratory (MIT LL) is not immune to these challenges and has developed a set of tools that address many of these challenges. Big data volume stresses the storage, memory, and compute capacity of a computing system and requires access to a computing cloud. Choosing the right cloud is problem specific. Currently, there are four multi-billion dollar ecosystems that dominate the cloud computing environment: enterprise clouds, big data clouds, SQL database clouds, and supercomputing clouds. Each cloud ecosystem has its own hardware, software, conferences, and business markets. The broad nature of business big data challenges make it unlikely that one cloud ecosystem can meet its needs and solutions are likely to require the tools and techniques from more than one cloud ecosystem. The MIT Supercloud was developed to address this challenge. To our knowledge, the MIT SuperCloud is the only deployed cloud system that allows all four ecosystems to co-exist without sacrificing performance or functionality. The velocity of big data velocity stresses the rate at which data can be absorbed and meaningful answers produced. Led by the NSA, a Common Big Data Architecture (CBDA) was developed for the U.S. government based on the Google Big Table NoSQL approach and is now in wide use. MIT/LL played a leading role in developing the CBDA and is a leader in adapting the CBDA to a variety of big data challenges. Big data variety may present the largest challenge and greatest opportunities. The promise of big data is the ability to correlate diverse and heterogeneous data to form new insights. The centerpiece of the CBDA is the NSA developed Apache Accumulo database (capable of millions of entries/second) and the MIT/LL developed D4M schema. These technologies allow vast quantities of highly diverse data (text, computer logs, and social media data, etc.) to be automatically ingested into a common schema that enables rapid query and correlation of every element. The talk will concentrate on how we utilize the aforementioned technologies in our mission to apply advanced technology to problems of national security.

Dagster @ R&S MNT

Simon Späti

This document summarizes a Dagster community meeting at Rohde & Schwarz in May 2021. It introduces Simon as the presenter and discusses how Rohde & Schwarz uses Dagster for their SmartAnalytics product. Key points include: - Rohde & Schwarz uses Dagster to move their ETL pipelines to the cloud for improved scalability. - Their Dagster implementation includes event-driven pipelines using sensors to monitor file uploads and listen to an S3 folder. - Advantages of Dagster include replacing their custom ETL engine, improved troubleshooting with its error reporting, and easier code reuse through resources and solids. - Next steps include adding tests, improving documentation, establishing best practices, and

MapR Edge : Act Locally Learn Globally

ridhav

[DataCon.TW 2017] Data Lake: centralize in on-prem vs. decentralize on cloud

Jeff Hung

Trend Micro has been running big-data in on-premises data center for many years. With Hadoop and its mature ecosystem, we are able to build the centralized Data Lake to serve and fulfill massive data processing loads while manage and encourage new use of data. In recent years, we are shifting our focus to AWS. Due to the decentralized nature of the cloud, the design and thinking for building Data Lake are different. We must identify what are still important no matter in on-prem or on the cloud, and what could be done differently to embrace the cloud model. In this talk, we will elaborate Trend Micro considerations and best practices on building Data Lake in on-prem and on cloud. And share our experience on managing peta-byte scale data with many years of evolution.

Approaching real-time-hadoop

Chris Huang

The document discusses approaches for building real-time applications on Hadoop systems before using Impala. It recommends using HBase to store and query data in real-time, SolrCloud for secondary indexing, and streaming tools like Storm on YARN for continuously processing data. The document provides examples of querying log data and malware information in real-time. It emphasizes clarifying use cases, computing data batches efficiently, and minimizing the gap between batches to approach real-time capabilities. The document advises that Impala is not always needed and that the same problems can occur, so the three-arrow approach of HBase, SolrCloud, and streaming often provides good real-time functionality without overengineering the solution.

Gluecon miller horizon

Mike Miller

The document discusses new technologies that are emerging beyond Hadoop to process big data more effectively. It summarizes Google's Percolator, Dremel, and Pregel systems, which provide incremental processing, ad-hoc querying, and graph processing respectively. These new approaches help address some of Hadoop's limitations and represent the next generation of big data technologies on the horizon.

Take your Data Management Practice to the Next Level with Denodo 7

Denodo

Presentation for slideshare

bolu804

The document describes the development of a Hydrologic Community Modeling System (HCMS) using a workflow engine called TRIDENT. The HCMS will allow for seamlessly integrated hydrologic models with interchangeable and portable modules. It will include libraries for data access, data processing, hydrologic models, and post-analysis tools. TRIDENT facilitates composing, executing, archiving, and sharing scientific workflows. Its use in hydrologic modeling provides benefits like flexible model setup, interactive or automated execution, high-performance computing, and provenance capture. The document introduces several libraries being developed as part of the HCMS.

Are you ready to be edgy? Bringing applications to the edge of the network

Megan O'Keefe

This document discusses edge computing and provides an overview of key concepts: 1. Edge computing extends cloud computing to the edge of networks to address bandwidth and latency issues and enable real-time applications like VR/AR, machine learning, and IoT. 2. Edge computing deployments are shaped differently than cloud but still leverage containers and orchestration tools like Kubernetes. 3. The document demonstrates an edge application management platform called Optikon that uses Kubernetes to deploy and manage applications across edge clusters.

Eclipse IoT Day, March 2017 - LightweightM2M Protocol & Ecosystem

Open Mobile Alliance

DDDP 2019 - Brown to Green

John Archer

Gc vit sttp cc december 2013

Seema Shah

The document discusses grid computing and the speaker's background in the topic. It provides key takeaways about understanding the evolution of technologies like grid computing and envisioning upcoming trends. It then discusses what a grid is, including early definitions, and elements of grid computing like resource sharing, coordinated problem solving, and dynamic virtual organizations. The document also outlines attributes of grid computing related to virtualization, dynamic provisioning, resource pooling, and self-adaptive software. It provides examples of how grids are used and lists common grid components.

DataOps: An Agile Method for Data-Driven Organizations

Ellen Friedman

MobiCloud: Towards Cloud Mobile Hybrid Application Generation using Semantica...

Amit Sheth

Ajith Ranabahu, Amit Sheth, Ashwin Manjunatha, and Krishnaprasad Thirunarayan, 'Towards Cloud Mobile Hybrid Application Generation using Semantically Enriched Domain Specific Languages', International Workshop on Mobile Computing and Clouds (MobiCloud 2010), Santa Clara, CA,October 28, 2010. Paper: http://knoesis.org/library/resource.php?id=865 Project: http://knoesis.wright.edu/research/srl/projects/mobi-cloud/

Similar to A Survey of NGS Data Analysis on Hadoop (20)

Introduction to OpenDaylight and Hydrogen, Learnings from the Year, What's Ne...

Building the iRODS Consortium

Extending twitter's data platform to google cloud

Extending Twitter's Data Platform to Google Cloud

Big data processing using - Hadoop Technology

Accumulo Summit 2014: Addressing big data challenges through innovative archi...

Dagster @ R&S MNT

MapR Edge : Act Locally Learn Globally

[DataCon.TW 2017] Data Lake: centralize in on-prem vs. decentralize on cloud

Approaching real-time-hadoop

Gluecon miller horizon

Take your Data Management Practice to the Next Level with Denodo 7

Presentation for slideshare

Are you ready to be edgy? Bringing applications to the edge of the network

Eclipse IoT Day, March 2017 - LightweightM2M Protocol & Ecosystem

DDDP 2019 - Brown to Green

Gc vit sttp cc december 2013

DataOps: An Agile Method for Data-Driven Organizations

MobiCloud: Towards Cloud Mobile Hybrid Application Generation using Semantica...

Recently uploaded

Gregory Harris - Cycle 2 - Civics Presentation

gharris9

ASONAM2023_presection_slide_track-recommendation.pdf

ToshihiroIto4

Artificial Intelligence, Data and Competition – ČORBA – June 2024 OECD discus...

OECD Directorate for Financial and Enterprise Affairs

This presentation by Juraj Čorba, Chair of OECD Working Party on Artificial Intelligence Governance (AIGO), was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp. This presentation was uploaded with the author’s consent.

Competition and Regulation in Professions and Occupations – OECD – June 2024 ...

OECD Directorate for Financial and Enterprise Affairs

Suzanne Lagerweij - Influence Without Power - Why Empathy is Your Best Friend...

Suzanne Lagerweij

This is a workshop about communication and collaboration. We will experience how we can analyze the reasons for resistance to change (exercise 1) and practice how to improve our conversation style and be more in control and effective in the way we communicate (exercise 2). This session will use Dave Gray’s Empathy Mapping, Argyris’ Ladder of Inference and The Four Rs from Agile Conversations (Squirrel and Fredrick). Abstract: Let’s talk about powerful conversations! We all know how to lead a constructive conversation, right? Then why is it so difficult to have those conversations with people at work, especially those in powerful positions that show resistance to change? Learning to control and direct conversations takes understanding and practice. We can combine our innate empathy with our analytical skills to gain a deeper understanding of complex situations at work. Join this session to learn how to prepare for difficult conversations and how to improve our agile conversations in order to be more influential without power. We will use Dave Gray’s Empathy Mapping, Argyris’ Ladder of Inference and The Four Rs from Agile Conversations (Squirrel and Fredrick). In the session you will experience how preparing and reflecting on your conversation can help you be more influential at work. You will learn how to communicate more effectively with the people needed to achieve positive change. You will leave with a self-revised version of a difficult conversation and a practical model to use when you get back to work. Come learn more on how to become a real influencer!

原版制作贝德福特大学毕业证（bedfordhire毕业证）硕士文凭原版一模一样

gpww3sf4

原版一模一样【微信：741003700 】【贝德福特大学毕业证（bedfordhire毕业证）硕士文凭】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Burning Issue Presentation By Kenmaryon.pdf

kkirkland2

Gregory Harris' Civics Presentation.pptx

gharris9

XP 2024 presentation: A New Look to Leadership

samililja

Pro-competitive Industrial Policy – LANE – June 2024 OECD discussion

OECD Directorate for Financial and Enterprise Affairs

Updated diagnosis. Cause and treatment of hypothyroidism

Faculty of Medicine And Health Sciences

Mẫu PPT kế hoạch làm việc sáng tạo cho nửa cuối năm PowerPoint

1990 Media

Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...

OECD Directorate for Financial and Enterprise Affairs

This presentation by Thibault Schrepel, Associate Professor of Law at Vrije Universiteit Amsterdam University, was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp. This presentation was uploaded with the author’s consent.

Presentatie 4. Jochen Cremer - TU Delft 28 mei 2024

Dutch Power

Artificial Intelligence, Data and Competition – LIM – June 2024 OECD discussion

OECD Directorate for Financial and Enterprise Affairs

2024-05-30_meetup_devops_aix-marseille.pdf

Frederic Leger

Pro-competitive Industrial Policy – OECD – June 2024 OECD discussion

OECD Directorate for Financial and Enterprise Affairs

Presentatie 8. Joost van der Linde & Daniel Anderton - Eliq 28 mei 2024

Dutch Power

Tom tresser burning issue.pptx My Burning issue

amekonnen

Mastering the Concepts Tested in the Databricks Certified Data Engineer Assoc...

SkillCertProExams

• For a full set of 760+ questions. Go to https://skillcertpro.com/product/databricks-certified-data-engineer-associate-exam-questions/ • SkillCertPro offers detailed explanations to each question which helps to understand the concepts better. • It is recommended to score above 85% in SkillCertPro exams before attempting a real exam. • SkillCertPro updates exam questions every 2 weeks. • You will get life time access and life time free updates • SkillCertPro assures 100% pass guarantee in first attempt.

Recently uploaded (20)

Gregory Harris - Cycle 2 - Civics Presentation

ASONAM2023_presection_slide_track-recommendation.pdf

Artificial Intelligence, Data and Competition – ČORBA – June 2024 OECD discus...

Competition and Regulation in Professions and Occupations – OECD – June 2024 ...

Suzanne Lagerweij - Influence Without Power - Why Empathy is Your Best Friend...

原版制作贝德福特大学毕业证（bedfordhire毕业证）硕士文凭原版一模一样

Burning Issue Presentation By Kenmaryon.pdf

Gregory Harris' Civics Presentation.pptx

XP 2024 presentation: A New Look to Leadership

Pro-competitive Industrial Policy – LANE – June 2024 OECD discussion

Updated diagnosis. Cause and treatment of hypothyroidism

Mẫu PPT kế hoạch làm việc sáng tạo cho nửa cuối năm PowerPoint

Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...

Presentatie 4. Jochen Cremer - TU Delft 28 mei 2024

Artificial Intelligence, Data and Competition – LIM – June 2024 OECD discussion

2024-05-30_meetup_devops_aix-marseille.pdf

Pro-competitive Industrial Policy – OECD – June 2024 OECD discussion

Presentatie 8. Joost van der Linde & Daniel Anderton - Eliq 28 mei 2024

Tom tresser burning issue.pptx My Burning issue

Mastering the Concepts Tested in the Databricks Certified Data Engineer Assoc...

A Survey of NGS Data Analysis on Hadoop

3. http://www.genome.gov/sequencingcosts/ NGS Era

5. High-Level Workflow of NGS 10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 5 Read Mapping Raw Reads (.fq) Variant Calling Sequence Alignment/ Mapping (.sam/.bam) Variant Calling file (.vcf)

6. NGS Data Analysis Pipeline • GATK best practice h1t0t/3p1/s20:1/4/wwwCo.bnfidreontaiald | Cinopsyritgihtt u20t1e2 .Torenrdg M/igcroa Itnkc. /guide/best6-practices?bpm=DNAseq

7. illumina solution 7 http://systems.illumina.com/content/dam/illumina-marketing/ documents/products/brochures/brochure_sequencing_systems_portfolio.pdf

8. The First $1,000 Genome – illumina HiSeq X Ten h1t0t/3p1:/2//0s14ystemCso.niflidleunmtiali |n Caop.ycrioghmt 2/0s12y Tsretnedm Miscr/oh Inics.eq-x-sequen8cing-system.html

9. Expectation of Data Processing Power for illumina HiSeq X Ten • A cluster of 10 HiSeq X instruments • Capable of sequencing up to 18,000 whole human genomes each year – Has a run cycle of ~3 days and produces ~150 genomes each run cycle – Running the industry standard BWA+GATK analysis pipeline to perform this analysis on a reasonably high-end (Dual Intel Xeon E5-2697v2 CPU – 12 core, 2.7 GHz with 96 GB DRAM) compute server takes ~24 hours per genome. – To achieve the required throughput of 150 genomes every three days, at least 50 of these servers are required. • Should meet a target of ~28 minutes for the completion of the mapping, aligning, sorting, de-duplication and variant calling of each genome. h1t0t/3p1/:2/0/1w4 ww.Ceodnfidicenotiagl | eConpyorigmht 2e0.1c2 Toremnd /Mdicrroa Ingc.en/ 9

14. Experiments$ Performance of CloudBurst Scalability+ 16000 14000 12000 10000 8000 6000 4000 2000 0 Running Time vs Number of Reads on Chr 1 0 1 2 3 4 5 6 7 8 Runtime (s) Millions of Reads 0 1 2 3 4 10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 14

15. Speedup over Serial RMAP EECS$584$–$Fall$2013$ Speedup+over+serial+RMAP+ 40 35 30 25 20 15 10 5 0 Speedup over serial RMAP 0 1 2 3 4 Speedup Number of Mismatches chr1 chr22 10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 15

16. Experiments$ Speedup on EC2 Speedup+on+EC2+ 1800 1600 1400 1200 1000 800 600 400 200 0 Running Time on EC2 High-CPU Medium Instance Cluster 24 48 72 96 Running time (s) Number of Cores 10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 16

30. h1t0t/3p1/:2/0/1w4 ww.Cbonifnidean.tical o| Cmopy/rpighrto 20d12u Tcretn.dh Mticmro Ilnc. 30

31. Architecture of bina Technology h1t0t/3p1/:2/0/1w4 ww.Cbonifnidean.tical o| Cmopy/rtigehtc 2h01n2 Torelnod gMiycr.oh Intcm. l 31

32. h1t0t/3p1s/2:0/1/4www.dConnafidnenetixal u| Cso.pcyorigmht 2/i0m12 aTrgeneds M/iucrso Iencc.ases/dnanex3u2s_CHARGE_prod1.png

33. Summary • NGS is a new page for Big Data Era • Need more CS experts to solve scalability and performance issues • Also, need more Data Scientist to discover the secrets/insights of Human Genome 10/31/2014 Confidential | Copyright 2012 Trend Micro Inc. 33

Editor's Notes

From the figure, we can see that CloudAligner is 60 to 80% faster than CloudBurst.
We mapped different subsets of the accession SRR035459 to the human chromosome 22 (50 Mbp) allowing up to 3 mismatches. From the figure, we can see that the execution time of both CloudBurst and CloudAligner is proportional to the number of reads, and CloudAligner outperforms Cloud- Burst from 35 to 67%.
With CloudBurst, the limitation of ts approach is the network bandwidth. With CloudAligner, its limitation is in the computation power of the workers in Hadoop. Consequently, if we run CloudAligner on cluster of legacy machines with high speed network, we probably lose the performance advantage over CloudBurst.

A Survey of NGS Data Analysis on Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to A Survey of NGS Data Analysis on Hadoop

Similar to A Survey of NGS Data Analysis on Hadoop (20)

Recently uploaded

Recently uploaded (20)

A Survey of NGS Data Analysis on Hadoop

Editor's Notes