CNS_poster12
•
0 likes•119 views
1) The document discusses optimizing large genome assembly by using stateful continuous bulk processing (CBP) on the Azure cloud platform. CBP allows efficient stateful graph processing and avoids reprocessing unchanged data. 2) The approach involves porting an existing genome assembly pipeline called Contrail to use CBP. Contrail currently uses Hadoop and MapReduce for genome assembly but is inefficient and slow. 3) Using CBP on Azure and a provenance manager called Newt, the ported pipeline can trace data and provenance through multi-stage processing, replay actors on selected inputs, and handle errors like crashes transparently through state management without full reprocessing.
Report
Share
Report
Share
Download to read offline
Recommended
Pycon 2016-open-space
This document discusses big data architecture and cluster optimization using Python. It covers the data analytics cycle, capacity planning for clustering, predictive modeling, tuning operating system parameters, virtual memory configuration, tuning Kafka and Spark clusters, and using PySpark for MapReduce word counting. It also discusses a data science education initiative at the University of Kachchh to establish a data science lab and teach machine learning and data science with Python.
Partitioning SKA Dataflows for Optimal Graph Execution
Presentation at the 9th Workshop on Scientific Cloud Computing and Infrastructure for Workflows and Application Composition
H2O World - GLM - Tomas Nykodym
Generalized linear models (GLMs) are a well-known statistical method for fitting linear models to different types of prediction problems. GLMs allow modeling of non-normal distributions and non-identity link functions. Regularization techniques like lasso, ridge, and elastic net are commonly used with GLMs to prevent overfitting. H2O's GLM implementation handles large datasets with many predictors efficiently using distributed and parallel processing. It supports automatic handling of categorical variables, multiple solvers, and regularization including elastic net. GLMs scaled to "big data" problems can be fit in seconds on H2O using all CPU cores.
Tutorial5
MapReduce is a programming model for processing large datasets in a distributed system. It allows for automatic parallelization and distribution of work. The MapReduce model consists of a map step that processes key-value pairs to generate intermediate key-value pairs, and a reduce step that merges all intermediate values associated with the same intermediate key. As an example, a word count problem can be solved by mapping words to counts, then reducing by word to get the total count for each word. Hadoop is an open-source implementation of MapReduce that provides fault tolerance and locality optimizations for distributed processing of large datasets across clusters.
Cascalog internal dsl_preso
Cascalog is an internal DSL for Clojure that allows defining MapReduce workflows for Hadoop. It provides helper functions, a way to define custom functions analogous to UDFs, and functions to programmatically generate all possible data aggregations from an input based on business requirements. The workflows can be unit tested and executed on Hadoop. Cascalog abstracts away lower-level MapReduce details and allows defining the entire workflow within a single language.
Dynamically Optimizing Queries over Large Scale Data Platforms
Enterprises are adapting large-scale data processing platforms, such as Hadoop, to gain actionable insights from their "big data". Query optimization is still an open challenge in this environment due to the volume and heterogeneity of data, comprising both structured and un/semi-structured datasets. Moreover, it has become common practice to push business logic close to the data via user-defined functions (UDFs), which are usually opaque to the optimizer, further complicating cost-based optimization. As a result, classical relational query optimization techniques do not fit well in this setting, while at the same time, suboptimal query plans can be disastrous with large datasets. In this talk, I will present new techniques that take into account UDFs and correlations between relations for optimizing queries running on large scale clusters. We introduce "pilot runs", which execute part of the query over a sample of the data to estimate selectivities, and employ a cost-based optimizer that uses these selectivities to choose an initial query plan. Then, we follow a dynamic optimization approach, in which plans evolve as parts of the queries get executed. Our experimental results show that our techniques produce plans that are at least as good as, and up to 2x (4x) better for Jaql (Hive) than, the best hand-written left-deep query plans.
Beyond Lists - Functional Kats Conf Dublin 2015
This document summarizes Phillip Trelford's talk at the Functional Kats Conference in 2015. It discusses performance comparisons between different programming languages and data structures, including reactive extensions in C# and F# observables. It also covers topics like immutable vs mutable data structures and choosing pragmatic solutions that optimize for speed. Source code examples are provided for concepts like quicksort algorithms and higher-order functions.
Loffeld_SIAMCSE15
Higher-order finite-volume methods for solving conservation laws can achieve high arithmetic intensity (AI) and improved performance. Theoretical analysis showed that 6th and 8th order methods reach the target AI for modern machines with infinite cache. Measurements of AI using hardware counters on a IBM Blue Gene/Q supercomputer matched the theoretical predictions when using multi-dimensional cache blocking. However, 3D blocking requires too much cache space due to wide halos from higher-order stencils. Iterating rectangular blocks in columns reduces cache usage and allows 6th and 8th order methods to achieve high AI with realistic cache sizes.
Recommended
Pycon 2016-open-space
This document discusses big data architecture and cluster optimization using Python. It covers the data analytics cycle, capacity planning for clustering, predictive modeling, tuning operating system parameters, virtual memory configuration, tuning Kafka and Spark clusters, and using PySpark for MapReduce word counting. It also discusses a data science education initiative at the University of Kachchh to establish a data science lab and teach machine learning and data science with Python.
Partitioning SKA Dataflows for Optimal Graph Execution
Presentation at the 9th Workshop on Scientific Cloud Computing and Infrastructure for Workflows and Application Composition
H2O World - GLM - Tomas Nykodym
Generalized linear models (GLMs) are a well-known statistical method for fitting linear models to different types of prediction problems. GLMs allow modeling of non-normal distributions and non-identity link functions. Regularization techniques like lasso, ridge, and elastic net are commonly used with GLMs to prevent overfitting. H2O's GLM implementation handles large datasets with many predictors efficiently using distributed and parallel processing. It supports automatic handling of categorical variables, multiple solvers, and regularization including elastic net. GLMs scaled to "big data" problems can be fit in seconds on H2O using all CPU cores.
Tutorial5
MapReduce is a programming model for processing large datasets in a distributed system. It allows for automatic parallelization and distribution of work. The MapReduce model consists of a map step that processes key-value pairs to generate intermediate key-value pairs, and a reduce step that merges all intermediate values associated with the same intermediate key. As an example, a word count problem can be solved by mapping words to counts, then reducing by word to get the total count for each word. Hadoop is an open-source implementation of MapReduce that provides fault tolerance and locality optimizations for distributed processing of large datasets across clusters.
Cascalog internal dsl_preso
Cascalog is an internal DSL for Clojure that allows defining MapReduce workflows for Hadoop. It provides helper functions, a way to define custom functions analogous to UDFs, and functions to programmatically generate all possible data aggregations from an input based on business requirements. The workflows can be unit tested and executed on Hadoop. Cascalog abstracts away lower-level MapReduce details and allows defining the entire workflow within a single language.
Dynamically Optimizing Queries over Large Scale Data Platforms
Enterprises are adapting large-scale data processing platforms, such as Hadoop, to gain actionable insights from their "big data". Query optimization is still an open challenge in this environment due to the volume and heterogeneity of data, comprising both structured and un/semi-structured datasets. Moreover, it has become common practice to push business logic close to the data via user-defined functions (UDFs), which are usually opaque to the optimizer, further complicating cost-based optimization. As a result, classical relational query optimization techniques do not fit well in this setting, while at the same time, suboptimal query plans can be disastrous with large datasets. In this talk, I will present new techniques that take into account UDFs and correlations between relations for optimizing queries running on large scale clusters. We introduce "pilot runs", which execute part of the query over a sample of the data to estimate selectivities, and employ a cost-based optimizer that uses these selectivities to choose an initial query plan. Then, we follow a dynamic optimization approach, in which plans evolve as parts of the queries get executed. Our experimental results show that our techniques produce plans that are at least as good as, and up to 2x (4x) better for Jaql (Hive) than, the best hand-written left-deep query plans.
Beyond Lists - Functional Kats Conf Dublin 2015
This document summarizes Phillip Trelford's talk at the Functional Kats Conference in 2015. It discusses performance comparisons between different programming languages and data structures, including reactive extensions in C# and F# observables. It also covers topics like immutable vs mutable data structures and choosing pragmatic solutions that optimize for speed. Source code examples are provided for concepts like quicksort algorithms and higher-order functions.
Loffeld_SIAMCSE15
Higher-order finite-volume methods for solving conservation laws can achieve high arithmetic intensity (AI) and improved performance. Theoretical analysis showed that 6th and 8th order methods reach the target AI for modern machines with infinite cache. Measurements of AI using hardware counters on a IBM Blue Gene/Q supercomputer matched the theoretical predictions when using multi-dimensional cache blocking. However, 3D blocking requires too much cache space due to wide halos from higher-order stencils. Iterating rectangular blocks in columns reduces cache usage and allows 6th and 8th order methods to achieve high AI with realistic cache sizes.
H2O World - Welcome to H2O World with Arno Candel
H2O World 2015
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Comparing pregel related systems
The document summarizes preliminary results from a project comparing the performance of open source implementations of Pregel and related graph processing systems (Hama, Giraph, GPS) on single-source shortest path (SSSP) and PageRank algorithms. Initial results show that Hama does not scale well to larger graphs, while Giraph and GPS scale better. Further analysis of memory usage, network traffic, additional systems like GraphLab and Signal/Collect, and using Green-Marl to generate code for Giraph and GPS is still in progress.
Big data solution capacity planning
Big data solution capacity planning to process 10 million events per day from 10k different servers. Considering 50 GB of data processing per day.
Influxdb and time series data
InfluxDB is an open source time series database designed to handle high write and query speeds for real-time metrics, events, and sensor data. It uses a schemaless data model and stores data as time-stamped points in measurements, which can be queried using a SQL-like language. InfluxDB excels at aggregating and analyzing time series data for use cases like monitoring, analytics, and alerting.
Latency SLOs done right
This document discusses best practices for defining and measuring latency service level objectives (SLOs). It recommends computing SLOs directly from raw log data using histograms, which allow arbitrary percentiles to be derived and are better than averaging sample percentiles. Histograms can also be aggregated over time and used to count the number of requests above a latency threshold regardless of what the threshold was set to initially. Common histogram implementations like HDR-Histogram and t-digest are suggested.
Toronto meetup 20190917
This document discusses elastic distributed deep learning training at scale on-premises and in the cloud. It introduces the architecture of elastic distributed training, which combines high performance synchronization techniques like distributed data parallel with session scheduling and elastic scaling to provide flexibility. This allows training jobs to automatically scale up and down resources based on policies while maintaining high performance. It aims to make distributed training transparent to frameworks like TensorFlow and PyTorch.
The productivity brought by Clojure
The document discusses the productivity benefits of Clojure, including REPL-driven development, immutable data structures, functional programming, and the immutable Datomic database. REPL-driven development allows writing and testing code incrementally. Immutable data structures and functional programming eliminate the need for deep cloning and enable better function composability. The immutable Datomic database allows "time travel" queries on historical data without additional tables. In summary, Clojure offers improvements to test-driven development, data handling, and database queries through these features.
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.
Case study- Real-time OLAP Cubes
This case study describes an approach to building quasi real-time OLAP cubes in Microsoft SQL Server Analysis Services to enable daily comparisons of production forecasts and outcomes. The cubes are partitioned by time to allow independent and frequent updates. Initial attempts failed due to deadlocks from simultaneous partition updates. The working solution takes advantage of a 6 day work week by switching partition dates on Saturdays only and reprocessing partitions then. This allows real-time and historical partition updates without gaps or overlaps in the data.
TAGC2016 schneider
Presentation by Valerie Schneider discussing Genome Reference Consortium (GRC) plans for the mouse and zebrafish reference genome assemblies, presented at the 2016 meeting of the The Allied Genetic Conference (TAGC). Includes description of resources at the National Center for Biotechnology Information (NCBI) for working with reference genome assemblies.
Understanding the reference assembly: CSHL Hackathon
The document discusses the human reference assembly and some key points:
1. The current reference assembly (GRCh38) represents two haplotypes and includes alternate loci to represent structural variation.
2. The assembly is improved from previous versions through the inclusion of 178 regions with alternate loci representing 261 alternate loci and 96 patches of novel sequence totaling over 5 megabytes.
3. Relevant assembly data can be accessed from the Genome Reference Consortium website including sequence files, annotations, and reports on assembly regions like alternate loci and centromeres.
Rna splicing
RNA splicing is the process by which introns, or non-coding sequences, are removed from pre-messenger RNA (pre-mRNA) to produce mature mRNA that can be translated into protein. Most genes contain introns that are removed by a spliceosome, a complex of RNA and proteins, leaving just the coding exons to form mRNA. Alternative splicing allows one gene to encode multiple proteins by selecting different combinations of exons. Errors in splicing can cause diseases if they result in truncated or abnormal proteins.
Rna splicing
The document discusses RNA splicing and the spliceosome. It notes that Phillip Allen Sharp co-discovered RNA splicing and won the 1993 Nobel Prize for discovering that eukaryotic genes contain introns that are removed from messenger RNA through splicing. It then provides details about the spliceosome, a large molecular machine that helps remove introns. The spliceosome is made up of small nuclear ribonucleoproteins (snRNPs) including U1, U2, U4, U5 and U6, which act in complexes E, A, B and C during the splicing process.
Fluorescence in situ Hybridization FISH #glok92
Fluorescence in situ hybridization (FISH) is a process that uses fluorescent probes that bind to specific parts of chromosomes to identify chromosomal abnormalities. FISH can identify the presence and location of specific DNA or RNA sequences on chromosomes within cells and tissues. It allows researchers to view specific chromosome segments or entire chromosomes with fluorescence microscopy. FISH is useful for applications like gene mapping, toxicology studies, detecting chromosomal structural abnormalities, and determining ploidy.
Fluorescent in-situ Hybridization (FISH)
Fluorescent in-situ hybridization (FISH) is a technique that uses fluorescent probes that bind to only parts of the genome to detect and localize specific DNA sequences on chromosomes. It involves denaturing and hybridizing a fluorescent probe to a complementary DNA or RNA target sequence on preserved tissue. The bound probe can then be visualized with a fluorescent microscope to identify specific gene mutations, abnormalities, or other chromosome changes to aid in cancer diagnosis, prenatal testing, and other clinical applications.
Dna sequencing powerpoint
DNA sequencing is the process of determining the order of nucleotides in a DNA fragment. The first DNA sequencing method, called Sanger sequencing, was developed in 1975 and involves copying DNA fragments and labeling the fragments to determine the sequence. DNA sequencing is useful for scientists as it allows study of what DNA codes for through comparative and functional genomics. While it improves healthcare and agriculture, a disadvantage is the risk of providing an incorrect DNA sequence.
Dna Sequencing
The document summarizes DNA sequencing methods. It discusses the DNA double helix structure and how the four nitrogenous bases form complementary pairs between strands. It then describes the two main historical DNA sequencing methods: the Maxam-Gilbert method which uses chemical degradation, and the Sanger method which is based on chain termination using dideoxynucleotides. The Sanger method is now the most common approach and involves sequencing in four separate reactions with one of the four ddNTPs added to each.
DNA SEQUENCING METHOD
DNA sequencing determines the order of nucleotide bases in a DNA molecule. The first methods were developed in the 1970s by Maxam and Gilbert (chemical method) and Sanger (chain termination/dideoxy method). Sanger's method is now most commonly used and involves DNA synthesis with chain termination by dideoxynucleotides to generate fragments of different lengths that can be separated and read to determine the DNA sequence. DNA sequencing has revolutionized biological sciences by enabling diagnosis of genetic diseases, identification of disease-causing mutations, and mapping of genomes. It provides benefits for medicine, forensics, and agriculture.
Fish(flourescent in-situ hybridization)
about the FISH technique and application
http://biotechliveinfo.blogspot.com/2017/02/chromosome-analysis-fluorescent-in-situ.html
Stream Processing Overview
This document provides an overview of stream processing. It discusses how stream processing systems are used to process large volumes of real-time data continuously and produce actionable information. Examples of applications discussed include traffic monitoring, network monitoring, smart grids, and sensor networks. Key concepts of stream processing covered include data streams, operators, windows, programming models, fault tolerance, and platforms like Storm and Spark Streaming.
XSEDE15_PhastaGateway
This document describes a science gateway developed using the Airavata middleware to enable complex industrial flow simulations through a high-performance computing workflow. The gateway lowers barriers to running workflows by automating tasks like file management, scheduling, scripting, and compiling across different HPC systems. It allows users to specify simulation parameters through a web interface and executes workflow steps on HPC resources, returning output upon completion. The document outlines the key components of the gateway, including the PHASTA application integration and deployment on TACC Stampede and CCI IBM Blue Gene resources.
Sessionization with Spark streaming
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
More Related Content
What's hot
H2O World - Welcome to H2O World with Arno Candel
H2O World 2015
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Comparing pregel related systems
The document summarizes preliminary results from a project comparing the performance of open source implementations of Pregel and related graph processing systems (Hama, Giraph, GPS) on single-source shortest path (SSSP) and PageRank algorithms. Initial results show that Hama does not scale well to larger graphs, while Giraph and GPS scale better. Further analysis of memory usage, network traffic, additional systems like GraphLab and Signal/Collect, and using Green-Marl to generate code for Giraph and GPS is still in progress.
Big data solution capacity planning
Big data solution capacity planning to process 10 million events per day from 10k different servers. Considering 50 GB of data processing per day.
Influxdb and time series data
InfluxDB is an open source time series database designed to handle high write and query speeds for real-time metrics, events, and sensor data. It uses a schemaless data model and stores data as time-stamped points in measurements, which can be queried using a SQL-like language. InfluxDB excels at aggregating and analyzing time series data for use cases like monitoring, analytics, and alerting.
Latency SLOs done right
This document discusses best practices for defining and measuring latency service level objectives (SLOs). It recommends computing SLOs directly from raw log data using histograms, which allow arbitrary percentiles to be derived and are better than averaging sample percentiles. Histograms can also be aggregated over time and used to count the number of requests above a latency threshold regardless of what the threshold was set to initially. Common histogram implementations like HDR-Histogram and t-digest are suggested.
Toronto meetup 20190917
This document discusses elastic distributed deep learning training at scale on-premises and in the cloud. It introduces the architecture of elastic distributed training, which combines high performance synchronization techniques like distributed data parallel with session scheduling and elastic scaling to provide flexibility. This allows training jobs to automatically scale up and down resources based on policies while maintaining high performance. It aims to make distributed training transparent to frameworks like TensorFlow and PyTorch.
The productivity brought by Clojure
The document discusses the productivity benefits of Clojure, including REPL-driven development, immutable data structures, functional programming, and the immutable Datomic database. REPL-driven development allows writing and testing code incrementally. Immutable data structures and functional programming eliminate the need for deep cloning and enable better function composability. The immutable Datomic database allows "time travel" queries on historical data without additional tables. In summary, Clojure offers improvements to test-driven development, data handling, and database queries through these features.
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.
Case study- Real-time OLAP Cubes
This case study describes an approach to building quasi real-time OLAP cubes in Microsoft SQL Server Analysis Services to enable daily comparisons of production forecasts and outcomes. The cubes are partitioned by time to allow independent and frequent updates. Initial attempts failed due to deadlocks from simultaneous partition updates. The working solution takes advantage of a 6 day work week by switching partition dates on Saturdays only and reprocessing partitions then. This allows real-time and historical partition updates without gaps or overlaps in the data.
What's hot (9)
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Viewers also liked
TAGC2016 schneider
Presentation by Valerie Schneider discussing Genome Reference Consortium (GRC) plans for the mouse and zebrafish reference genome assemblies, presented at the 2016 meeting of the The Allied Genetic Conference (TAGC). Includes description of resources at the National Center for Biotechnology Information (NCBI) for working with reference genome assemblies.
Understanding the reference assembly: CSHL Hackathon
The document discusses the human reference assembly and some key points:
1. The current reference assembly (GRCh38) represents two haplotypes and includes alternate loci to represent structural variation.
2. The assembly is improved from previous versions through the inclusion of 178 regions with alternate loci representing 261 alternate loci and 96 patches of novel sequence totaling over 5 megabytes.
3. Relevant assembly data can be accessed from the Genome Reference Consortium website including sequence files, annotations, and reports on assembly regions like alternate loci and centromeres.
Rna splicing
RNA splicing is the process by which introns, or non-coding sequences, are removed from pre-messenger RNA (pre-mRNA) to produce mature mRNA that can be translated into protein. Most genes contain introns that are removed by a spliceosome, a complex of RNA and proteins, leaving just the coding exons to form mRNA. Alternative splicing allows one gene to encode multiple proteins by selecting different combinations of exons. Errors in splicing can cause diseases if they result in truncated or abnormal proteins.
Rna splicing
The document discusses RNA splicing and the spliceosome. It notes that Phillip Allen Sharp co-discovered RNA splicing and won the 1993 Nobel Prize for discovering that eukaryotic genes contain introns that are removed from messenger RNA through splicing. It then provides details about the spliceosome, a large molecular machine that helps remove introns. The spliceosome is made up of small nuclear ribonucleoproteins (snRNPs) including U1, U2, U4, U5 and U6, which act in complexes E, A, B and C during the splicing process.
Fluorescence in situ Hybridization FISH #glok92
Fluorescence in situ hybridization (FISH) is a process that uses fluorescent probes that bind to specific parts of chromosomes to identify chromosomal abnormalities. FISH can identify the presence and location of specific DNA or RNA sequences on chromosomes within cells and tissues. It allows researchers to view specific chromosome segments or entire chromosomes with fluorescence microscopy. FISH is useful for applications like gene mapping, toxicology studies, detecting chromosomal structural abnormalities, and determining ploidy.
Fluorescent in-situ Hybridization (FISH)
Fluorescent in-situ hybridization (FISH) is a technique that uses fluorescent probes that bind to only parts of the genome to detect and localize specific DNA sequences on chromosomes. It involves denaturing and hybridizing a fluorescent probe to a complementary DNA or RNA target sequence on preserved tissue. The bound probe can then be visualized with a fluorescent microscope to identify specific gene mutations, abnormalities, or other chromosome changes to aid in cancer diagnosis, prenatal testing, and other clinical applications.
Dna sequencing powerpoint
DNA sequencing is the process of determining the order of nucleotides in a DNA fragment. The first DNA sequencing method, called Sanger sequencing, was developed in 1975 and involves copying DNA fragments and labeling the fragments to determine the sequence. DNA sequencing is useful for scientists as it allows study of what DNA codes for through comparative and functional genomics. While it improves healthcare and agriculture, a disadvantage is the risk of providing an incorrect DNA sequence.
Dna Sequencing
The document summarizes DNA sequencing methods. It discusses the DNA double helix structure and how the four nitrogenous bases form complementary pairs between strands. It then describes the two main historical DNA sequencing methods: the Maxam-Gilbert method which uses chemical degradation, and the Sanger method which is based on chain termination using dideoxynucleotides. The Sanger method is now the most common approach and involves sequencing in four separate reactions with one of the four ddNTPs added to each.
DNA SEQUENCING METHOD
DNA sequencing determines the order of nucleotide bases in a DNA molecule. The first methods were developed in the 1970s by Maxam and Gilbert (chemical method) and Sanger (chain termination/dideoxy method). Sanger's method is now most commonly used and involves DNA synthesis with chain termination by dideoxynucleotides to generate fragments of different lengths that can be separated and read to determine the DNA sequence. DNA sequencing has revolutionized biological sciences by enabling diagnosis of genetic diseases, identification of disease-causing mutations, and mapping of genomes. It provides benefits for medicine, forensics, and agriculture.
Fish(flourescent in-situ hybridization)
about the FISH technique and application
http://biotechliveinfo.blogspot.com/2017/02/chromosome-analysis-fluorescent-in-situ.html
Viewers also liked (10)
Understanding the reference assembly: CSHL Hackathon
Understanding the reference assembly: CSHL Hackathon
Similar to CNS_poster12
Stream Processing Overview
This document provides an overview of stream processing. It discusses how stream processing systems are used to process large volumes of real-time data continuously and produce actionable information. Examples of applications discussed include traffic monitoring, network monitoring, smart grids, and sensor networks. Key concepts of stream processing covered include data streams, operators, windows, programming models, fault tolerance, and platforms like Storm and Spark Streaming.
XSEDE15_PhastaGateway
This document describes a science gateway developed using the Airavata middleware to enable complex industrial flow simulations through a high-performance computing workflow. The gateway lowers barriers to running workflows by automating tasks like file management, scheduling, scripting, and compiling across different HPC systems. It allows users to specify simulation parameters through a web interface and executes workflow steps on HPC resources, returning output upon completion. The document outlines the key components of the gateway, including the PHASTA application integration and deployment on TACC Stampede and CCI IBM Blue Gene resources.
Sessionization with Spark streaming
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
20211119 ntuh azure hpc workshop final
This document discusses using Microsoft Azure cloud computing resources to conduct genome-wide association studies (GWAS) and polygenic risk scoring (PRS) to predict COVID-19 mortality. Key steps include acquiring genotype and phenotype data, performing quality control, running GWAS and PRS analyses using HPC clusters on Azure, and downloading results. Azure provides scalable computing and storage for the large genomic datasets. Its HPC capabilities allow accelerating the analyses, which could otherwise take months to complete on-premises.
Hp Connect 10 06 08 V5
PerfCap offers an integrated performance and capacity planning software solution called PAWZ. PAWZ collects data from nodes, analyzes performance trends, and uses modeling to predict capacity needs and identify systems at risk of saturation. It helps answer questions like how much workload growth an existing configuration can support and what configuration changes would enable more growth. A case study showed PAWZ accurately modeled an Itanium system and identified hardware options to support 200% workload growth. PAWZ automates the capacity planning process.
BioPig for scalable analysis of big sequencing data
This document introduces BioPig, a Hadoop-based analytic toolkit for large-scale genomic sequence analysis. BioPig aims to provide a flexible, high-level, and scalable platform to enable domain experts to build custom analysis pipelines. It leverages Hadoop's data parallelism to speed up bioinformatics tasks like k-mer counting and assembly. The document demonstrates how BioPig can analyze over 1 terabase of metagenomic data using just 7 lines of code, much more simply than alternative MPI-based solutions. While challenges remain around optimization and integration, BioPig shows promise for scalable genomic analytics on very large datasets.
Taking Spark Streaming to the Next Level with Datasets and DataFrames
Structured Streaming provides a simple way to perform streaming analytics by treating unbounded, continuous data streams similarly to static DataFrames and Datasets. It allows for event-time processing, windowing, joins, and other SQL operations on streaming data. Under the hood, it uses micro-batch processing to incrementally and continuously execute queries on streaming data using Spark's SQL engine and Catalyst optimizer. This allows for high-level APIs as well as end-to-end guarantees like exactly-once processing and fault tolerance through mechanisms like offset tracking and a fault-tolerant state store.
Performance
Lightning talk showing various aspectos of software system performance. It goes through: latency, data structures, garbage collection, troubleshooting method like workload saturation method, quick diagnostic tools, famegraph and perfview
Full Stack Load Testing
A talk I gave at the Boston Web Performance Meetup in August 2014.
Performance is one of the most challenging issues in modern web app design, in large part because modeling, testing, and validating performance before deploying to production is so challenging. While many ops teams have nailed down the problem of re-creating pre-production environments that closely mimic production, those environments frequently rely on known-good components beyond the application code itself: AWS ELB, F5 load balancers, CDNs, Varnish, and more.
Testing plug-in components like that can be challenging, because their performance characteristics don't directly align with application metrics.
- How many simultaneous users can my load balancer support? - What sort of network load will I put on my CDN (i.e., how much will it cost?) - How do different user behavior patterns affect performance?
In this meetup, we'll introduce a novel tool in this toolbox: tcpreplay, an open-source tool for replaying packet capture files back at an application. By replaying user traffic to a staging environment, you can test the effects of
- Network saturation to the load balancer - High numbers of users / IPs - Lots of traffic to your other monitoring tools!
Database Camp 2016 @ United Nations, NYC - Brad Bebee, CEO, Blazegraph
The document discusses how graph databases and GPU acceleration can help analyze large datasets with billions of edges and nodes. It provides examples of precision medicine and cyber defense applications that involve very large graph problems. The document also summarizes Blazegraph's GPU acceleration technology, which can provide 200-300x faster performance for graph analytics compared to Apache Spark. Blazegraph uses its DASL language to simplify programming graph algorithms for multi-GPU execution.
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
Apache Spark 2.0 offers many enhancements that make continuous analytics quite simple. In this talk, we will discuss many other things that you can do with your Apache Spark cluster. We explain how a deep integration of Apache Spark 2.0 and in-memory databases can bring you the best of both worlds! In particular, we discuss how to manage mutable data in Apache Spark, run consistent transactions at the same speed as state-the-art in-memory grids, build and use indexes for point lookups, and run 100x more analytics queries at in-memory speeds. No need to bridge multiple products or manage, tune multiple clusters. We explain how one can take regulation Apache Spark SQL OLAP workloads and speed them up by up to 20x using optimizations in SnappyData.
We then walk through several use-case examples, including IoT scenarios, where one has to ingest streams from many sources, cleanse it, manage the deluge by pre-aggregating and tracking metrics per minute, store all recent data in a in-memory store along with history in a data lake and permit interactive analytic queries at this constantly growing data. Rather than stitching together multiple clusters as proposed in Lambda, we walk through a design where everything is achieved in a single, horizontally scalable Apache Spark 2.0 cluster. A design that is simpler, a lot more efficient, and let’s you do everything from Machine Learning and Data Science to Transactions and Visual Analytics all in one single cluster.
SnappyData at Spark Summit 2017
Why does big data always have to go through a pipeline? multiple data copies, slow, complex and stale analytics? We present a unified analytics platform that brings streaming, transactions and adhoc OLAP style interactive analytics in a single in-memory cluster based on Spark.
Taming Big Data!
My talk at the Winter School on Big Data in Tarragona, Spain.
Abstract: We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to leverage the “cloud” (whether private or public) to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers.
Presentation southernstork 2009-nov-southernworkshop
This document discusses data placement scheduling between distributed repositories. It introduces Stork, a batch scheduler for data placement activities that supports plug-in data transfer modules and scheduling of data movement jobs. The document discusses techniques used by Stork such as throttling concurrent transfers, fault tolerance, job aggregation, and adaptive tuning of data transfer protocols. It also covers topics like network reservation, failure awareness, and directions for future work including priority-based scheduling and advance resource reservation.
Network-aware Data Management for Large Scale Distributed Applications, IBM R...
The document discusses network-aware data management for large-scale distributed applications. It provides an outline for a presentation on this topic, including discussing the performance of VSAN and VVOL storage in virtualized environments, the PetaShare distributed storage system and Stork data scheduler, data streaming in high-bandwidth networks, and several other related topics like network reservations and scheduling. The presenter's background and experience working on data transfer scheduling, distributed storage, and high-performance computing networks is also briefly summarized.
Lessons learned from designing QA automation event streaming platform(IoT big...
Event steaming platform - need to test it?
a couple things to understand before your start.
[still not finished - work in progress]
Explore big data at speed of thought with Spark 2.0 and Snappydata
Abstract:
Data exploration often requires running aggregation/slice-dice queries on data sourced from disparate sources. You may want to identify distribution patterns, outliers, etc and aid the feature selection process as you train your predictive models. As you begin to understand your data, you want to ask ad-hoc questions expressed through your visualization tool (which typically translates to SQL queries), study the results and iteratively explore the data set through more queries. Unfortunately, even when data sets can be in-memory, large data set computations take time breaking the train of thought and increasing time to insight . We know Spark can be fast through its in-memory parallel processing. But, Spark 1.x isn’t quite there. Spark 2.0 promises to offer 10X better speed than its predecessor. Spark 2.0 ushers some impressive improvements to interactive query performance. We first explore these advances - compiling the query plan eliminating virtual function calls, and other improvements in the Catalyst engine. We compare the performance to other popular popular query processing engines by studying the spark query plans. We then go through SnappyData (an open source project that integrates Spark with a database that offers OLTP, OLAP and stream processing in a single cluster) where we use smarter data colocation and Synopses data (.e.g. Stratified sampling) to dramatically cut down on the memory requirements as well as the query latency. We explain the key concepts in summarizing data using structures like stratified sampling by walking through some examples in Apache Zeppelin notebooks (a open source visualization tool for spark) and demonstrate how we can explore massive data sets with just your laptop resources while achieving remarkable speeds.
Bio:
Jags is a founder and the CTO of SnappyData. Previously, Jags was the Chief Architect for “fast data” products at Pivotal and served in the extended leadership team of the company. At Pivotal and previously at VMWare, he led the technology direction for GemFire and other distributed in-memory Bio:
Jags Ramnarayan is a founder and the CTO of SnappyData. Previously, Jags was the Chief Architect for “fast data” products at Pivotal and served in the extended leadership team of the company. At Pivotal and previously at VMWare, he led the technology direction for GemFire and other distributed in-memory products.
Intro to Apache Apex - Next Gen Platform for Ingest and Transform
Introduction to Apache Apex - The next generation native Hadoop platform. This talk will cover details about how Apache Apex can be used as a powerful and versatile platform for big data processing. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch alerts, real-time actions, threat detection, etc.
Bio:
Pramod Immaneni is Apache Apex PMC member and senior architect at DataTorrent, where he works on Apache Apex and specializes in big data platform and applications. Prior to DataTorrent, he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs.
Cloud Experiences
The document discusses the Sanger Institute's experiences with moving genomic research workloads and data to the cloud. Key points include:
- Moving an existing web application to AWS mirrors improved performance but required significant code changes.
- Running genomic analysis pipelines and databases on AWS was more cost effective than traditional colocation, though software configuration took effort.
- Large data transfer speeds over the public internet pose challenges for moving terabytes of sequencing data to cloud resources.
- Security, data governance, and funding models require careful consideration when using cloud services for sensitive genomic and medical data. Private clouds may help address some issues.
20141219 workshop methylation sequencing analysis
The document describes the agenda for a hands-on training workshop on methylation sequencing analysis. The training will cover topics such as WGBS dataset quality control, mapping, and basic analysis using tools like methpipe, BEDtools, and the UCSC Genome Browser. The first session in the morning focuses on WGBS dataset QC, mapping with methpipe, and calculating metrics like bisulfite conversion rate and methylation levels in different genomic regions. The second session in the afternoon involves more advanced analysis using BEDtools to analyze methylation differences between genomic regions and between male and female samples, as well as using the UCSC Genome Browser.
Similar to CNS_poster12 (20)
BioPig for scalable analysis of big sequencing data
BioPig for scalable analysis of big sequencing data
Taking Spark Streaming to the Next Level with Datasets and DataFrames
Taking Spark Streaming to the Next Level with Datasets and DataFrames
Database Camp 2016 @ United Nations, NYC - Brad Bebee, CEO, Blazegraph
Database Camp 2016 @ United Nations, NYC - Brad Bebee, CEO, Blazegraph
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
SnappyData, the Spark Database. A unified cluster for streaming, transactions...
Presentation southernstork 2009-nov-southernworkshop
Presentation southernstork 2009-nov-southernworkshop
Network-aware Data Management for Large Scale Distributed Applications, IBM R...
Network-aware Data Management for Large Scale Distributed Applications, IBM R...
Lessons learned from designing QA automation event streaming platform(IoT big...
Lessons learned from designing QA automation event streaming platform(IoT big...
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and Snappydata
Intro to Apache Apex - Next Gen Platform for Ingest and Transform
Intro to Apache Apex - Next Gen Platform for Ingest and Transform
CNS_poster12
- 1. Optimizing Large Genome Assembly in the Cloud Apurva Kumar, Soumyarupa De and Kenneth Yocum University of California, San Diego Genomic Analytics Data • High Throughput Sequencing for human (African male NA18507) produce : 3.5 billions 36 base pairs (bp) reads = 756 MB. (Cost : 1000 USD) Current Problem : To assemble genome in lesser cost and minimal time . Why this problem ? • Only 60 complete human genomes publicly exist(including Steve Jobs who got it sequenced for 100K USD) • Commercialization is still in early stage and growing rapidly Question How do we scale this 3.5 B bp input data to compute the final genome sequence ? Optimizing using Stateful Bulk Processing Newt Lineage Current Assembly Techniques Make use of Stateful Continuous Bulk Processing (aka CBP) : Efficient stateful graph processing model (supports even Google’s “Pregel”) and run it on top of Azure. Output data Contrail implemented as CBP StagesInput flows Output flows Fin edges Fout edges T( , ΔFstate, ΔF1) Fout state Fin state key Δin Δout Translate function for a stage with state as loopback flow Our Approach Aim : To assemble a human genome (3.5 B 36 bp) Genomic Analysis Graph Processing CBP on Azure (+ Newt (Debug)) Genome assembly using graphs Bulk Procesing with State Errors in Contrail Pipeline Debugging via NewtContrail with Newt • Multi-staged pipeline with several MapReduce stages • Types of errors: • De Bruijn graph for a sample short input reads • Eulerian walk across the graph gives the genomic sequence Fail-stop Corrupted Outputs Suspicious Actors • Newt “fail” API triggered on crash - Reports crash culprits to Newt • Find inputs that lead to corruption or fail-stops - Prune selected inputs and replay entire pipeline • Crash avoidance: remove crash culprits immediately and continue - No replay overhead, transparent fault handling - How to handle removed inputs used in other dataflow paths? • Online identification of suspicious actor behavior based on actor’s history - Processing rate (too slow, too fast), selectivity (n-to-1 instead of 1-to-1) - How much history? Backward Tracing Build GraphShort read files Graph refinement Genome sequence Porting completed Porting pending Assembler Technology CPU/RAM Velvet Serial 2 TB RAM ABySS MPI 168 cores x 96 hours SOAP denovo Pthreads 40 cores x 40 hours, > 140 GB RAM (total) Output data Input data Output AExtract links Count in-links: Site/URL frequency Merge w/seen Score and threshold D σ(D) state ΔD Output ΔAExtract links Count in-links: Site/URL frequency Merge w/seen Score and threshold state state 1.) Process Dataflow σ with input D 2.) Create state 3.) Process changes, ΔD, and prior state Updates, does not reprocess. Saves CPU, disk, network and energy! • Newt: Provenance manager • Capture fine-grained provenance in MapReduce jobs • Trace data provenance through multi-staged pipeline • Replay actors with selected inputs • Graph refinement phase • Build graph stage is stateful (saves src,dst information). • 10 MR stages (Contrail) mapped. • Graph refinement stage is stateful and iterative (refines the graph). • 30 MR stages (Contrail) to be mapped. Contrail (Schatz et al. 2009) uses Hadoop – builds a big graph(>3B nodes and >10B edges), iterates, scales but inefficient, slow (1.5 MB input takes 2 hours to assemble) and complex (40+ MR stages )!!!