Exploring Spark for Scalable Metagenomics Analysis: Spark Summit East talk by Zhong Wang
•
1 like•1,368 views
This document discusses using Apache Spark to assemble metagenomes from short read sequencing data. Metagenomes are genomes from microbial communities containing many species. Spark provides an efficient and scalable approach compared to previous methods. The document demonstrates clustering reads from small test datasets in Spark and evaluates performance on real datasets ranging from 20GB to failures at 100GB. While Spark is easy to develop for and efficient, challenges remain in robustness at large scales and optimizing for different problem complexities.
Report
Share
Report
Share
Download to read offline
Recommended
Algorithms and Tools for Genomic Analysis on Spark: Spark Summit East talk by...
Hammer Lab has built several tools for analyzing genomic data on Spark, as well as libraries for more general computations using RDDs; I’ll discuss some of the most interesting applications and algorithms therein:
Guacamole (https://github.com/hammerlab/guacamole) is a somatic variant caller built on Spark; it identifies mutations in cancer genomes in a fraction of the time that comparable tools take.
Pageant (https://github.com/hammerlab/pageant) contains miscellaneous other genomic analyses and a few interesting and novel algorithms for massively-parallel Burrows-Wheeler-Transform and FM-Index construction.
Magic RDDs (https://github.com/hammerlab/magic-rdds) contains some yet more interesting general-purpose algorithms implemented on RDDs.
Finding Needles in Genomic Haystacks with “Wide” Random Forest: Spark Summit ...
Recent advances in genome sequencing technologies and bioinformatics have enabled whole-genomes to be studied at population-level rather then for small number of individuals. This provides new power to whole genome association studies (WGAS
), which now seek to identify the multi-gene causes of common complex diseases like diabetes or cancer.
As WGAS involve studying thousands of genomes, they pose both technological and methodological challenges. The volume of data is significant, for example the dataset from 1000 Genomes project with genomes of 2504 individuals includes nearly 85M genomic variants with raw data size of 0.8 TB. The number of features is enormous and greatly exceeds the number of samples, which makes it challenging to apply traditional statistical approaches.
Random forest is one of the methods that was found to be useful in this context, both because of its potential for parallelization and its robustness. Although there is a number of big data implementations available (including Spark ML) they are tuned for typical dataset with large number of samples and relatively small number of variables, and either fail or are inefficient in the GWAS context especially, that a costly data preprocessing is usually required.
To address these problems, we have developed the RandomForestHD – a Spark based implementation optimized for highly dimensional data sets. We have successfully RandomForestHD applied it to datasets beyond the reach of other tools and for smaller datasets found its performance superior. We are currently applying RandomForestHD, released as part of the VariantSpark toolkit, to a number of WGAS studies.
In the presentation we will introduce the domain of WGAS and related challenges, present RandomForestHD with its design principles and implementation details with regards to Spark, compare its performance with other tools, and finally showcase the results of a few WGAS applications.
Spark Summit EU talk by Erwin Datema and Roeland van Ham
The document discusses KeyGene's use of Apache Spark for high-throughput genomics data analysis. KeyGene is a crop innovation company that analyzes genomic data from thousands of plants to improve crop traits like yield and quality. They previously used conventional HPC clusters for genomics pipelines but found Spark enabled more interactive analysis. KeyGene developed a "Sparkified" genomics pipeline using tools like BWA, GATK and their own Guacamole variant caller. This allowed interactive variant selection and GWAS using Spark SQL queries, demonstrating Spark is well-suited for interactive plant genomics analysis.
Challenges and Opportunities of Big Data Genomics
The document discusses the challenges and opportunities of big data genomics. It notes that the bottleneck in genomics has shifted from data generation to data handling as sequencing capacity doubles every year. While compression can help address the data deluge, throughput from techniques like metagenomics and single-cell sequencing will continue to outpace storage gains. The document then explores solutions for analyzing and storing large genomic datasets through techniques like cloud computing, distributed file systems, and MapReduce frameworks.
A Step to the Clouded Solution of Scalable Clinical Genome Sequencing (BDT308...
Professors Wall and Tonellato of Harvard Medical School in collaboration with Beth Israel Deaconess Medical Center discuss the emerging area of clinical whole genome sequencing analysis and tools. They report on the use of Amazon EC2 and Spot Instances to achieve a robust clinical time processing solution and examine the barriers to and resolution of producing clinical-grade whole genome results in the cloud. They benchmark an AWS solution, called COSMOS, against local computing solutions and demonstrate the time and capacity gains conferred through the use of AWS.
Scalable Genome Analysis With ADAM
This document provides a summary of the Scalable Genome Analysis with ADAM project. ADAM is an open-source, high-performance, distributed platform for genomic analysis that defines a data schema, data layout on disk, and programming interface for distributed processing of genomic data using Spark and Scala. The goal of ADAM is to integrate across terabyte and petabyte-scale datasets to enable the discovery of low frequency genetic variants linked to traits and diseases.
Lightning fast genomics with Spark, Adam and Scala
This document discusses using Apache Spark and ADAM to perform scalable genomic analysis. It provides an overview of genomics and challenges with existing approaches. ADAM uses Apache Spark and Parquet to efficiently store and query large genomic datasets. The document demonstrates clustering genomic data from the 1000 Genomes Project to predict populations, showing ADAM and Spark can handle large genomic workloads. It concludes these tools provide scalable genomic data processing but future work is needed to implement more advanced algorithms.
VariantSpark a library for genomics by Lynn Langit
VariantSpark is a library for scalable genomic analysis that can process large genomic datasets containing millions of variants and thousands of samples. It uses machine learning techniques like k-means clustering and random forests for unsupervised and supervised analysis. VariantSpark can analyze whole genome datasets faster than other methods and scale to process 100% of genomic data. It also integrates with cloud platforms like AWS and Databricks for easy access and demo of its capabilities through Jupyter notebooks.
Recommended
Algorithms and Tools for Genomic Analysis on Spark: Spark Summit East talk by...
Hammer Lab has built several tools for analyzing genomic data on Spark, as well as libraries for more general computations using RDDs; I’ll discuss some of the most interesting applications and algorithms therein:
Guacamole (https://github.com/hammerlab/guacamole) is a somatic variant caller built on Spark; it identifies mutations in cancer genomes in a fraction of the time that comparable tools take.
Pageant (https://github.com/hammerlab/pageant) contains miscellaneous other genomic analyses and a few interesting and novel algorithms for massively-parallel Burrows-Wheeler-Transform and FM-Index construction.
Magic RDDs (https://github.com/hammerlab/magic-rdds) contains some yet more interesting general-purpose algorithms implemented on RDDs.
Finding Needles in Genomic Haystacks with “Wide” Random Forest: Spark Summit ...
Recent advances in genome sequencing technologies and bioinformatics have enabled whole-genomes to be studied at population-level rather then for small number of individuals. This provides new power to whole genome association studies (WGAS
), which now seek to identify the multi-gene causes of common complex diseases like diabetes or cancer.
As WGAS involve studying thousands of genomes, they pose both technological and methodological challenges. The volume of data is significant, for example the dataset from 1000 Genomes project with genomes of 2504 individuals includes nearly 85M genomic variants with raw data size of 0.8 TB. The number of features is enormous and greatly exceeds the number of samples, which makes it challenging to apply traditional statistical approaches.
Random forest is one of the methods that was found to be useful in this context, both because of its potential for parallelization and its robustness. Although there is a number of big data implementations available (including Spark ML) they are tuned for typical dataset with large number of samples and relatively small number of variables, and either fail or are inefficient in the GWAS context especially, that a costly data preprocessing is usually required.
To address these problems, we have developed the RandomForestHD – a Spark based implementation optimized for highly dimensional data sets. We have successfully RandomForestHD applied it to datasets beyond the reach of other tools and for smaller datasets found its performance superior. We are currently applying RandomForestHD, released as part of the VariantSpark toolkit, to a number of WGAS studies.
In the presentation we will introduce the domain of WGAS and related challenges, present RandomForestHD with its design principles and implementation details with regards to Spark, compare its performance with other tools, and finally showcase the results of a few WGAS applications.
Spark Summit EU talk by Erwin Datema and Roeland van Ham
The document discusses KeyGene's use of Apache Spark for high-throughput genomics data analysis. KeyGene is a crop innovation company that analyzes genomic data from thousands of plants to improve crop traits like yield and quality. They previously used conventional HPC clusters for genomics pipelines but found Spark enabled more interactive analysis. KeyGene developed a "Sparkified" genomics pipeline using tools like BWA, GATK and their own Guacamole variant caller. This allowed interactive variant selection and GWAS using Spark SQL queries, demonstrating Spark is well-suited for interactive plant genomics analysis.
Challenges and Opportunities of Big Data Genomics
The document discusses the challenges and opportunities of big data genomics. It notes that the bottleneck in genomics has shifted from data generation to data handling as sequencing capacity doubles every year. While compression can help address the data deluge, throughput from techniques like metagenomics and single-cell sequencing will continue to outpace storage gains. The document then explores solutions for analyzing and storing large genomic datasets through techniques like cloud computing, distributed file systems, and MapReduce frameworks.
A Step to the Clouded Solution of Scalable Clinical Genome Sequencing (BDT308...
Professors Wall and Tonellato of Harvard Medical School in collaboration with Beth Israel Deaconess Medical Center discuss the emerging area of clinical whole genome sequencing analysis and tools. They report on the use of Amazon EC2 and Spot Instances to achieve a robust clinical time processing solution and examine the barriers to and resolution of producing clinical-grade whole genome results in the cloud. They benchmark an AWS solution, called COSMOS, against local computing solutions and demonstrate the time and capacity gains conferred through the use of AWS.
Scalable Genome Analysis With ADAM
This document provides a summary of the Scalable Genome Analysis with ADAM project. ADAM is an open-source, high-performance, distributed platform for genomic analysis that defines a data schema, data layout on disk, and programming interface for distributed processing of genomic data using Spark and Scala. The goal of ADAM is to integrate across terabyte and petabyte-scale datasets to enable the discovery of low frequency genetic variants linked to traits and diseases.
Lightning fast genomics with Spark, Adam and Scala
This document discusses using Apache Spark and ADAM to perform scalable genomic analysis. It provides an overview of genomics and challenges with existing approaches. ADAM uses Apache Spark and Parquet to efficiently store and query large genomic datasets. The document demonstrates clustering genomic data from the 1000 Genomes Project to predict populations, showing ADAM and Spark can handle large genomic workloads. It concludes these tools provide scalable genomic data processing but future work is needed to implement more advanced algorithms.
VariantSpark a library for genomics by Lynn Langit
VariantSpark is a library for scalable genomic analysis that can process large genomic datasets containing millions of variants and thousands of samples. It uses machine learning techniques like k-means clustering and random forests for unsupervised and supervised analysis. VariantSpark can analyze whole genome datasets faster than other methods and scale to process 100% of genomic data. It also integrates with cloud platforms like AWS and Databricks for easy access and demo of its capabilities through Jupyter notebooks.
VariantSpark: applying Spark-based machine learning methods to genomic inform...
Genomic information is increasingly used in medical practice giving rise to the need for efficient analysis methodology able to cope with thousands of individuals and millions of variants. Here we introduce VariantSpark, which utilizes Hadoop/Spark along with its machine learning library, MLlib, providing the means of parallelisation for population-scale bioinformatics tasks. VariantSpark is the interface to the standard variant format (VCF), offers seamless genome-wide sampling of variants and provides a pipeline for visualising results.
To demonstrate the capabilities of VariantSpark, we clustered more than 3,000 individuals with 80 Million variants each to determine the population structure in the dataset. VariantSpark is 80% faster than the Spark-based genome clustering approach, ADAM, the comparable implementation using Hadoop/Mahout, as well as Admixture, a commonly used tool for determining individual ancestries. It is over 90% faster than traditional implementations using R and Python. These benefits of speed, resource consumption and scalability enables VariantSpark to open up the usage of advanced, efficient machine learning algorithms to genomic data.
The package is written in Scala and available at https://github.com/BauerLab/VariantSpark.
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...GigaScience, BGI Hong Kong
Lisa Johnson's talk at the #ICG13 GigaScience Prize Track: Re-assembly, quality evaluation, and annotation of 678 microbial eukaryotic reference transcriptomes. Shenzhen, 26th October 2018BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale
A talk given at the BioBankCloud conference in Feb 2015 about distributed computing in the contexts of genomics and health.
In this one, we exposed what results we obtained exploring the 1000genomes data using ADAM, followed by an introduction to our scalable GA4GH server implementation built using ADAM, Apache Spark and Play Framework 2.
Scaling up genomic analysis with ADAM
Slides from DNAnexus about engineering/system design principles from ADAM, as well as emerging work in long-read OLC assembly.
Hail: SCALING GENETIC DATA ANALYSIS WITH APACHE SPARK: Keynote by Cotton Seed
In 2001, it cost ~$100M to sequence a single human genome. In 2014, due to dramatic improvements in sequencing technology far outpacing Moore’s law, we entered the era of the $1,000 genome. At the same time, the power of genetics to impact medicine has become evident: for example, drugs with supporting genetic evidence have twice the clinical trial success rate. These factors have led to an explosion in the volume of genetic data, in the face of which existing analysis tools are breaking down.
Therefore, we began the open-source Hail project (https://hail.is) to be a scalable platform built on Apache Spark to enable the worldwide genetics community to build, share, and apply new tools. Hail is focused on variant-level (post-read) data; querying genetic data, annotations and sample data; and performing rare and common variant association analyses. Hail has already been used to analyze datasets with hundreds of thousands of exomes and tens of thousands of whole genomes.
We will give an overview of the goals of the Hail project and its architecture. The challenge of efficiently manipulating genetic data in Spark has led to several innovations that may have wider applicability, including an RDD-like abstraction for representing multidimensional data and an OrderedRDD abstraction for ordered data, (for example, data indexed by position in the genome). Finally, we will discuss Hail performance and future directions.
Scaling Genetic Data Analysis with Apache Spark with Jon Bloom and Tim Poterba
In 2001, it cost ~$100M to sequence a single human genome. In 2014, due to dramatic improvements in sequencing technology far outpacing Moore’s law, we entered the era of the $1,000 genome. At the same time, the power of genetics to impact medicine has become evident. For example, drugs with supporting genetic evidence are twice as likely to succeed in clinical trials. These factors have led to an explosion in the volume of genetic data, in the face of which existing analysis tools are breaking down.
As a result, the Broad Institute began the open-source Hail project (https://hail.is), a scalable platform built on Apache Spark, to enable the worldwide genetics community to build, share and apply new tools. Hail is focused on variant-level (post-read) data; querying genetic data, as well as annotations, on variants and samples; and performing rare and common variant association analyses. Hail has already been used to analyze datasets with hundreds of thousands of exomes and tens of thousands of whole genomes, enabling dozens of major research projects.
Fast Variant Calling with ADAM and avocado
Slides from the BioBankCloud Hadoop/NGS Workshop at KTH Kista on 2/19/2015. Discusses early work on variant calling using ADAM and avocado.
Hadoop for Bioinformatics: Building a Scalable Variant Store
Talk at Mount Sinai School of Medicine. Introduction to the Hadoop ecosystem, problems in bioinformatics data analytics, and a specific use case of building a genome variant store backed by Cloudera Impala.
Population-scale high-throughput sequencing data analysis
This document provides an overview of a presentation on population-scale high-throughput sequencing data analysis. It discusses:
1) The background and goals of the CSIRO/Omics Project which aims to investigate colorectal cancer susceptibility using sequencing data from 500 individuals.
2) Methods for processing large-scale NGS data on high-performance computing clusters and cloud infrastructure using the NGSANE framework, which allows processing modules to be run in parallel.
3) Preliminary research outcomes identifying cancer-associated and microbiome changes from analysis of colorectal cancer and control samples.
Scalable Genome Analysis with ADAM
This document discusses scalable genome analysis using ADAM (Apache Spark-based framework). It begins by describing genomes and the goal of analyzing genetic variations. The document then discusses challenges like the large size of genomes and complexity of linking variations to traits. It proposes using ADAM's schema, optimized storage and algorithms to accelerate common access patterns like overlap joins. The document also emphasizes applying biological knowledge like protein grammars to make sense of non-coding variations. Finally, it acknowledges contributions from various institutions that have helped develop ADAM and its ability to enable genome analysis at scale.
Introduction to Bayesian phylogenetics and BEAST
This document provides an overview of a course on Bayesian phylogenetics and the BEAST software package. The course covers introductory topics on Bayesian analysis and BEAST, as well as more advanced analyses including incorporating temporal and trait data. The document outlines the organization and topics to be covered in lectures, including why Bayesian methods are well-suited for pathogen evolution analysis and an introduction to Markov chain Monte Carlo sampling. It also provides information on setting up BEAST analyses using BEAUti, evaluating runs in Tracer, and summarizing runs using LogCombiner and TreeAnnotator.
Scalable up genomic analysis with ADAM
ADAM is an open source platform for scalable genomic analysis that defines a data schema, Scala API, and command line interface. It uses Apache Spark for efficient parallel and distributed processing of large genomic datasets stored in Parquet format. Key features of ADAM include its ability to perform iterative analysis on whole genome datasets while minimizing data movement through Spark. The document also describes using ADAM and PacMin for long read assembly through techniques like minhashing for fast read overlapping and building consensus sequences on read graphs.
2014 sage-talk
This document discusses the challenges of analyzing large datasets from metagenomic shotgun sequencing experiments. It notes that while sequencing costs have decreased significantly, the computational analysis of the massive amounts of data generated still poses major challenges. It introduces the concept of "digital normalization" as an approach to reduce dataset sizes while retaining most of the biological information by removing redundant reads. The document advocates for making analysis tools and datasets openly accessible to help advance understanding of microbial communities from metagenomics studies.
Pathogen phylogenetics using BEAST
This document provides an overview of a lecture on analyzing pathogens using BEAST (Bayesian Evolutionary Analysis Sampling Trees). The lecture covers what makes pathogens special from an evolutionary perspective, an introduction to Bayesian analysis and Markov chain Monte Carlo (MCMC) methods, an overview of the BEAST software package and its components, and a demonstration of running a BEAST analysis. The lecture discusses building phylogenetic trees incorporating sample time data and estimating parameters like substitution rates and population dynamics from molecular sequences using Bayesian methods in BEAST.
Managing Genomes At Scale: What We Learned - StampedeCon 2014
This document summarizes lessons learned from managing large genomic datasets at Monsanto. It discusses how Monsanto uses big data technologies like Hadoop, HBase, and Solr to store and query genomic data at scale. Key lessons include using HBase more like a hashmap than a relational database, denormalizing HBase schemas, and using distributed search technologies like SolrCloud rather than rebuilding Solr indexes. The document provides examples of genomic data formats and architectures used to store, index, and retrieve genomic feature data from petabytes of sequence data.
2014 bangkok-talk
This document discusses opportunities and challenges presented by next-generation DNA sequencing technologies. It begins by introducing the speaker, C. Titus Brown, and their commitment to open science. It then describes the dramatic decreases in cost and increases in scale of DNA sequencing. While this enables sequencing entire genomes and environmental samples, it presents challenges for analysis due to lack of reference genomes and limited computational tools. The document outlines goals for shotgun sequencing analysis and challenges for non-model organisms. It concludes by emphasizing the need for training in data analysis to take advantage of the vast amounts of sequencing data being generated.
Why is Bioinformatics a Good Fit for Spark?
DNA sequencing is producing a wave of data which will change the way that drugs are developed, patients diagnosed, and our understanding of human biology. To fulfill this promise, however, the tools for interpretation and analysis must scale to match the quantity and diversity of "big data genomics."
ADAM is an open-source genomics processing engine, built using Spark, Apache Avro, and Parquet. This talk will discuss some of the advantages that the Spark platform brings to genomics, the benefits of using technologies like Parquet in conjunction with Spark, and the challenges of adapting new technologies for existing tools in bioinformatics.
These are slides for a talk given at the Apache Spark Meetup in Boston on October 20, 2014.
Design for Scalability in ADAM
ADAM is an open source, high performance, distributed platform for genomic analysis built on Apache Spark. It defines a Scala API and data schema using Avro and Parquet to store data in a columnar format, addressing the I/O bottleneck in genomics pipelines. ADAM implements common genomics algorithms as data or graph parallel computations and minimizes data movement by sending code to the data using Spark. It is designed to scale to processing whole human genomes across distributed file systems and cloud infrastructure.
Many Task Applications for Grids and Supercomputers
The document discusses how new supercomputing applications are increasingly focused on "logistical" issues like executing many communication-intensive tasks over large shared datasets, rather than "heroic" computations of a single task. It argues that new programming models and tools are needed to efficiently manage large numbers of tasks, complex data dependencies, and failures at extreme scales of petascale and exascale computers. Examples of applications that could benefit include parameter studies, ensemble simulations, data analysis, and scientific workflows involving millions of tasks.
Novel Techniques & Connections Between High-Pressure Mineral Physics, Microto...
Talk at the EarthCube End-User Domain Workshop for Rock Deformation and Mineral Physics Research.
By Mark Rivers, University of Chicago
Building bioinformatics resources for the global community
1. The document evaluates different methods for inferring relationships between Salmonella samples based on whole genome sequencing data from large databases. It compares k-mer based methods and site-based methods using 18,997 Salmonella isolates from public databases.
2. Site-based methods like NUCmer and MLST produced more accurate results, but require more computing resources when dealing with large databases. K-mer based methods are faster but more sensitive to assembly and contamination issues.
3. While k-mer methods may be useful for initial filtering, site-based methods are superior for accuracy, though challenges remain in applying them to databases containing tens of thousands of samples. Quality control and computing resources are important considerations.
Aug2013 Heidi Rehm integrating large scale sequencing into clinical practice
This document discusses integrating large scale genomic sequencing into clinical practice. It describes how the Laboratory for Molecular Medicine at Partners Healthcare has transitioned from testing a few genes to now offering over 150 tests using next generation sequencing technologies. It also discusses the evolution of testing panels, moving from targeted panels to exome and genome sequencing. Validation of sequencing methods and challenges of variant interpretation are discussed.
More Related Content
What's hot
VariantSpark: applying Spark-based machine learning methods to genomic inform...
Genomic information is increasingly used in medical practice giving rise to the need for efficient analysis methodology able to cope with thousands of individuals and millions of variants. Here we introduce VariantSpark, which utilizes Hadoop/Spark along with its machine learning library, MLlib, providing the means of parallelisation for population-scale bioinformatics tasks. VariantSpark is the interface to the standard variant format (VCF), offers seamless genome-wide sampling of variants and provides a pipeline for visualising results.
To demonstrate the capabilities of VariantSpark, we clustered more than 3,000 individuals with 80 Million variants each to determine the population structure in the dataset. VariantSpark is 80% faster than the Spark-based genome clustering approach, ADAM, the comparable implementation using Hadoop/Mahout, as well as Admixture, a commonly used tool for determining individual ancestries. It is over 90% faster than traditional implementations using R and Python. These benefits of speed, resource consumption and scalability enables VariantSpark to open up the usage of advanced, efficient machine learning algorithms to genomic data.
The package is written in Scala and available at https://github.com/BauerLab/VariantSpark.
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...GigaScience, BGI Hong Kong
Lisa Johnson's talk at the #ICG13 GigaScience Prize Track: Re-assembly, quality evaluation, and annotation of 678 microbial eukaryotic reference transcriptomes. Shenzhen, 26th October 2018BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale
A talk given at the BioBankCloud conference in Feb 2015 about distributed computing in the contexts of genomics and health.
In this one, we exposed what results we obtained exploring the 1000genomes data using ADAM, followed by an introduction to our scalable GA4GH server implementation built using ADAM, Apache Spark and Play Framework 2.
Scaling up genomic analysis with ADAM
Slides from DNAnexus about engineering/system design principles from ADAM, as well as emerging work in long-read OLC assembly.
Hail: SCALING GENETIC DATA ANALYSIS WITH APACHE SPARK: Keynote by Cotton Seed
In 2001, it cost ~$100M to sequence a single human genome. In 2014, due to dramatic improvements in sequencing technology far outpacing Moore’s law, we entered the era of the $1,000 genome. At the same time, the power of genetics to impact medicine has become evident: for example, drugs with supporting genetic evidence have twice the clinical trial success rate. These factors have led to an explosion in the volume of genetic data, in the face of which existing analysis tools are breaking down.
Therefore, we began the open-source Hail project (https://hail.is) to be a scalable platform built on Apache Spark to enable the worldwide genetics community to build, share, and apply new tools. Hail is focused on variant-level (post-read) data; querying genetic data, annotations and sample data; and performing rare and common variant association analyses. Hail has already been used to analyze datasets with hundreds of thousands of exomes and tens of thousands of whole genomes.
We will give an overview of the goals of the Hail project and its architecture. The challenge of efficiently manipulating genetic data in Spark has led to several innovations that may have wider applicability, including an RDD-like abstraction for representing multidimensional data and an OrderedRDD abstraction for ordered data, (for example, data indexed by position in the genome). Finally, we will discuss Hail performance and future directions.
Scaling Genetic Data Analysis with Apache Spark with Jon Bloom and Tim Poterba
In 2001, it cost ~$100M to sequence a single human genome. In 2014, due to dramatic improvements in sequencing technology far outpacing Moore’s law, we entered the era of the $1,000 genome. At the same time, the power of genetics to impact medicine has become evident. For example, drugs with supporting genetic evidence are twice as likely to succeed in clinical trials. These factors have led to an explosion in the volume of genetic data, in the face of which existing analysis tools are breaking down.
As a result, the Broad Institute began the open-source Hail project (https://hail.is), a scalable platform built on Apache Spark, to enable the worldwide genetics community to build, share and apply new tools. Hail is focused on variant-level (post-read) data; querying genetic data, as well as annotations, on variants and samples; and performing rare and common variant association analyses. Hail has already been used to analyze datasets with hundreds of thousands of exomes and tens of thousands of whole genomes, enabling dozens of major research projects.
Fast Variant Calling with ADAM and avocado
Slides from the BioBankCloud Hadoop/NGS Workshop at KTH Kista on 2/19/2015. Discusses early work on variant calling using ADAM and avocado.
Hadoop for Bioinformatics: Building a Scalable Variant Store
Talk at Mount Sinai School of Medicine. Introduction to the Hadoop ecosystem, problems in bioinformatics data analytics, and a specific use case of building a genome variant store backed by Cloudera Impala.
Population-scale high-throughput sequencing data analysis
This document provides an overview of a presentation on population-scale high-throughput sequencing data analysis. It discusses:
1) The background and goals of the CSIRO/Omics Project which aims to investigate colorectal cancer susceptibility using sequencing data from 500 individuals.
2) Methods for processing large-scale NGS data on high-performance computing clusters and cloud infrastructure using the NGSANE framework, which allows processing modules to be run in parallel.
3) Preliminary research outcomes identifying cancer-associated and microbiome changes from analysis of colorectal cancer and control samples.
Scalable Genome Analysis with ADAM
This document discusses scalable genome analysis using ADAM (Apache Spark-based framework). It begins by describing genomes and the goal of analyzing genetic variations. The document then discusses challenges like the large size of genomes and complexity of linking variations to traits. It proposes using ADAM's schema, optimized storage and algorithms to accelerate common access patterns like overlap joins. The document also emphasizes applying biological knowledge like protein grammars to make sense of non-coding variations. Finally, it acknowledges contributions from various institutions that have helped develop ADAM and its ability to enable genome analysis at scale.
Introduction to Bayesian phylogenetics and BEAST
This document provides an overview of a course on Bayesian phylogenetics and the BEAST software package. The course covers introductory topics on Bayesian analysis and BEAST, as well as more advanced analyses including incorporating temporal and trait data. The document outlines the organization and topics to be covered in lectures, including why Bayesian methods are well-suited for pathogen evolution analysis and an introduction to Markov chain Monte Carlo sampling. It also provides information on setting up BEAST analyses using BEAUti, evaluating runs in Tracer, and summarizing runs using LogCombiner and TreeAnnotator.
Scalable up genomic analysis with ADAM
ADAM is an open source platform for scalable genomic analysis that defines a data schema, Scala API, and command line interface. It uses Apache Spark for efficient parallel and distributed processing of large genomic datasets stored in Parquet format. Key features of ADAM include its ability to perform iterative analysis on whole genome datasets while minimizing data movement through Spark. The document also describes using ADAM and PacMin for long read assembly through techniques like minhashing for fast read overlapping and building consensus sequences on read graphs.
2014 sage-talk
This document discusses the challenges of analyzing large datasets from metagenomic shotgun sequencing experiments. It notes that while sequencing costs have decreased significantly, the computational analysis of the massive amounts of data generated still poses major challenges. It introduces the concept of "digital normalization" as an approach to reduce dataset sizes while retaining most of the biological information by removing redundant reads. The document advocates for making analysis tools and datasets openly accessible to help advance understanding of microbial communities from metagenomics studies.
Pathogen phylogenetics using BEAST
This document provides an overview of a lecture on analyzing pathogens using BEAST (Bayesian Evolutionary Analysis Sampling Trees). The lecture covers what makes pathogens special from an evolutionary perspective, an introduction to Bayesian analysis and Markov chain Monte Carlo (MCMC) methods, an overview of the BEAST software package and its components, and a demonstration of running a BEAST analysis. The lecture discusses building phylogenetic trees incorporating sample time data and estimating parameters like substitution rates and population dynamics from molecular sequences using Bayesian methods in BEAST.
Managing Genomes At Scale: What We Learned - StampedeCon 2014
This document summarizes lessons learned from managing large genomic datasets at Monsanto. It discusses how Monsanto uses big data technologies like Hadoop, HBase, and Solr to store and query genomic data at scale. Key lessons include using HBase more like a hashmap than a relational database, denormalizing HBase schemas, and using distributed search technologies like SolrCloud rather than rebuilding Solr indexes. The document provides examples of genomic data formats and architectures used to store, index, and retrieve genomic feature data from petabytes of sequence data.
2014 bangkok-talk
This document discusses opportunities and challenges presented by next-generation DNA sequencing technologies. It begins by introducing the speaker, C. Titus Brown, and their commitment to open science. It then describes the dramatic decreases in cost and increases in scale of DNA sequencing. While this enables sequencing entire genomes and environmental samples, it presents challenges for analysis due to lack of reference genomes and limited computational tools. The document outlines goals for shotgun sequencing analysis and challenges for non-model organisms. It concludes by emphasizing the need for training in data analysis to take advantage of the vast amounts of sequencing data being generated.
Why is Bioinformatics a Good Fit for Spark?
DNA sequencing is producing a wave of data which will change the way that drugs are developed, patients diagnosed, and our understanding of human biology. To fulfill this promise, however, the tools for interpretation and analysis must scale to match the quantity and diversity of "big data genomics."
ADAM is an open-source genomics processing engine, built using Spark, Apache Avro, and Parquet. This talk will discuss some of the advantages that the Spark platform brings to genomics, the benefits of using technologies like Parquet in conjunction with Spark, and the challenges of adapting new technologies for existing tools in bioinformatics.
These are slides for a talk given at the Apache Spark Meetup in Boston on October 20, 2014.
Design for Scalability in ADAM
ADAM is an open source, high performance, distributed platform for genomic analysis built on Apache Spark. It defines a Scala API and data schema using Avro and Parquet to store data in a columnar format, addressing the I/O bottleneck in genomics pipelines. ADAM implements common genomics algorithms as data or graph parallel computations and minimizes data movement by sending code to the data using Spark. It is designed to scale to processing whole human genomes across distributed file systems and cloud infrastructure.
Many Task Applications for Grids and Supercomputers
The document discusses how new supercomputing applications are increasingly focused on "logistical" issues like executing many communication-intensive tasks over large shared datasets, rather than "heroic" computations of a single task. It argues that new programming models and tools are needed to efficiently manage large numbers of tasks, complex data dependencies, and failures at extreme scales of petascale and exascale computers. Examples of applications that could benefit include parameter studies, ensemble simulations, data analysis, and scientific workflows involving millions of tasks.
Novel Techniques & Connections Between High-Pressure Mineral Physics, Microto...
Talk at the EarthCube End-User Domain Workshop for Rock Deformation and Mineral Physics Research.
By Mark Rivers, University of Chicago
What's hot (20)
VariantSpark: applying Spark-based machine learning methods to genomic inform...
VariantSpark: applying Spark-based machine learning methods to genomic inform...
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...
Lisa Johnson at #ICG13: Re-assembly, quality evaluation, and annotation of 67...
BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale
BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale
Hail: SCALING GENETIC DATA ANALYSIS WITH APACHE SPARK: Keynote by Cotton Seed
Hail: SCALING GENETIC DATA ANALYSIS WITH APACHE SPARK: Keynote by Cotton Seed
Scaling Genetic Data Analysis with Apache Spark with Jon Bloom and Tim Poterba
Scaling Genetic Data Analysis with Apache Spark with Jon Bloom and Tim Poterba
Hadoop for Bioinformatics: Building a Scalable Variant Store
Hadoop for Bioinformatics: Building a Scalable Variant Store
Population-scale high-throughput sequencing data analysis
Population-scale high-throughput sequencing data analysis
Managing Genomes At Scale: What We Learned - StampedeCon 2014
Managing Genomes At Scale: What We Learned - StampedeCon 2014
Many Task Applications for Grids and Supercomputers
Many Task Applications for Grids and Supercomputers
Novel Techniques & Connections Between High-Pressure Mineral Physics, Microto...
Novel Techniques & Connections Between High-Pressure Mineral Physics, Microto...
Viewers also liked
Building bioinformatics resources for the global community
1. The document evaluates different methods for inferring relationships between Salmonella samples based on whole genome sequencing data from large databases. It compares k-mer based methods and site-based methods using 18,997 Salmonella isolates from public databases.
2. Site-based methods like NUCmer and MLST produced more accurate results, but require more computing resources when dealing with large databases. K-mer based methods are faster but more sensitive to assembly and contamination issues.
3. While k-mer methods may be useful for initial filtering, site-based methods are superior for accuracy, though challenges remain in applying them to databases containing tens of thousands of samples. Quality control and computing resources are important considerations.
Aug2013 Heidi Rehm integrating large scale sequencing into clinical practice
This document discusses integrating large scale genomic sequencing into clinical practice. It describes how the Laboratory for Molecular Medicine at Partners Healthcare has transitioned from testing a few genes to now offering over 150 tests using next generation sequencing technologies. It also discusses the evolution of testing panels, moving from targeted panels to exome and genome sequencing. Validation of sequencing methods and challenges of variant interpretation are discussed.
EU PathoNGenTraceConsortium:cgMLST Evolvement and Challenges for Harmonization
EU PathoNGenTraceConsortium:cgMLST Evolvement and Challenges for HarmonizationEuropean Centre for Disease Prevention and Control (ECDC)
Dag Harmsen presented on the evolvement and challenges of cgMLST for the harmonization of bacterial genome sequencing and analysis. Key points include:
- cgMLST (core genome multilocus sequence typing) involves identifying and comparing alleles across a fixed set of core genome genes and has been applied to outbreak investigation and global pathogen nomenclature.
- Tools for cgMLST analysis have been developed and improved to work on read, draft, and complete genome levels and allow scalable, additive analysis of single genes to whole genomes.
- Standardizing a hierarchical cgMLST-based approach and developing common nomenclature poses challenges but is important for microbial genotypic surveillance across laboratories and countries. Next Generation Sequencing for Identification and Subtyping of Foodborne Pat...
"Next Generation Sequencing for Identification and Subtyping of Foodborne Pathogens" presentation at the Standards for Pathogen Identification via NGS (SPIN) workshop hosted by National Institute for Standards and Technology October 2014 by Rebecca Lindsey, PhD from Enteric Diseases Laboratory Branch of the CDC.
Tools for Metagenomics with 16S/ITS and Whole Genome Shotgun Sequences
Presented at Cornell Symbiosis symposium. Workflow for processing amplicon based 16S/ITS sequences as well as whole genome shotgun sequences are described. Slides include short description and links for each tool.
DISCLAIMER: This is a small subset of tools out there. No disrespect to methods not mentioned.
Plant genome sequencing and crop improvement
This document summarizes information from a student's assignment on plant genome sequencing techniques. It discusses early phenotypic selection methods and their limitations. It then summarizes different sequencing strategies used for important crop plants like rice, poplar, and Arabidopsis. These include BAC-by-BAC, whole genome shotgun, and various next-generation sequencing platforms. The document also summarizes applications of sequencing including identifying genes related to rice yield and flowering time and using sequencing to improve potato and maize varieties.
20170209 ngs for_cancer_genomics_101
This document provides an overview of next generation sequencing (NGS) for cancer genomics. It discusses how NGS allows sequencing of whole genomes or targeted regions to identify genetic changes between normal and tumor DNA. Analysis involves aligning DNA reads to reference genomes, detecting mutations, and categorizing them as silent, non-silent, oncogenic drivers, or signatures of mutational processes. RNA sequencing can also profile gene expression and detect fusion genes. The data requires specialized bioinformatics analysis to interpret genetic changes driving cancer.
Errors and Limitaions of Next Generation Sequencing
This document discusses some key errors and limitations of next-generation sequencing (NGS). It notes that while NGS has significantly reduced costs and improved throughput, it also has some drawbacks compared to previous sequencing technologies. Specifically, it outlines issues related to low quality bases, PCR errors during amplification, and high error rates that can make rare mutations difficult to detect. Limitations include short read lengths that hamper assembly of repetitive regions, contamination risks, incomplete representation of repeats, difficulties assembling segmental duplications and genes fragmented across scaffolds. The document emphasizes the need for validation of genome assemblies and development of hybrid approaches combining long and short reads to overcome these challenges.
Bioinformática Introdução (Basic NGS)
O documento fornece uma introdução à bioinformática, definindo-a como o campo interdisciplinar que aplica técnicas de informática à análise de informações biológicas. Ele também descreve os principais tipos de dados gerados por sequenciamento de nova geração, como FASTQ, BAM, VCF e BED, e ferramentas de bioinformática como GATK usadas para análise desses dados e identificação de variantes. Por fim, discute bancos de dados públicos como ExAC e ClinVar que fornecem dados de referência para interpreta
Rossen eccmid2015v1.5
Eyes for the invisible: microfluidic paper-based analytics (µPads) and 3rd - generation sequencing in clinical microbiology #ECCMID2015
Speeding up sequencing: Sequencing in an hour enables sample to answer in a w...
Speeding up sequencing: Sequencing in an hour enables sample to answer in a w...Thermo Fisher Scientific
1) Thermo Fisher Scientific developed improvements to targeted and whole genome sequencing workflows that dramatically decrease overall time from sample to data/answer.
2) For a targeted bacterial identification panel, sequencing and analysis can now be completed in approximately 6.5 hours compared to standard times of 2.5 hours for sequencing and 1 hour for analysis.
3) For whole genome sequencing of bacterial isolates, the entire workflow from library preparation to data can now be finished in approximately 11 hours, compared to standard times of 1.5-3.5 hours for library prep, 2-7 hours for sequencing, and 1 hour for analysis.transforming clinical microbiology by next generation sequencing
This document discusses the current use of laboratory diagnosis and the role of next-generation sequencing (NGS). It outlines traditional sequencing methods and some of their limitations. NGS allows for whole genome sequencing, which can be used for rapid diagnosis, antimicrobial resistance detection, and high-resolution epidemiological typing to track disease outbreaks. Specific examples are given of using NGS to simultaneously detect bacterial pathogens and resistance genes from clinical samples and to trace the source of a cholera outbreak in Haiti.
Introduction to Metagenomics Data Analysis - UEB-VHIR - 2013
UEB-VHIR's Metagenomics Training. Session 1. 2013/08/26. An Introduction to Metagenomics Data Analysis. Ferran Briansó (ferran.brianso@vhir.org)
Ngs presentation
Whole Exome and Genome sequencing in Neurological disorders. The document discusses the techniques of next generation sequencing (NGS), including whole exome sequencing (WES) and whole genome sequencing (WGS). It provides examples of studies applying WES/WGS to diagnose neurological disorders, finding diagnostic variants in 25-60% of cases of leukoencephalopathy, limb-girdle muscular dystrophy, cerebellar ataxia, and more. The limitations and clinical applications of WES/WGS are also reviewed.
NGS overview
The document discusses the advantages and future of next-generation sequencing (NGS). It notes that the NGS market has grown rapidly, with costs and runtimes decreasing significantly over time. Current optimization aims to further lower costs and runtimes while increasing ease-of-use. NGS allows for hypothesis-free and versatile experimental design. A diverse set of applications are discussed, including whole genome sequencing, exome sequencing, and metagenome sequencing. The document predicts that new platforms will offer cheaper, quicker, and longer sequencing. Future applications may include single-cell sequencing and direct detection of base modifications.
Metagenomics
Metagenomics is the study of genetic material recovered directly from environmental samples without culturing. This field enables research on uncultured organisms and microbial communities. There are three main metagenomic approaches: biochemical, whole genome shotgun sequencing, and 16s rRNA sequencing. Metagenomics is being applied to study human microbiomes, discover new genes and enzymes, monitor environmental impacts, and characterize uncultured microbes. Future directions include identifying more novel products from uncultured bacteria and improving culture methods and bioinformatics tools.
Ngs intro_v6_public
An Introduction to NGS(Next Generation Sequencing). Part 1. principle, machines and comparative analysis. by François PAILLIER
QIAseq Technologies for Metagenomics and Microbiome NGS Library Prep
In this slide deck, learn about the innovative technologies that form the basis of QIAGEN’s portfolio of QIAseq library prep solutions for metagenomics and microbiome sequencing. Whether your research starts from single microbial cells, 16s rRNA PCR amplicons, or gDNA for whole genome analysis, QIAseq technologies offer tips and tricks for capturing the genomic diversity of your samples in the most unbiased, streamlined way possible.
NGS Applications I (UEB-UAT Bioinformatics Course - Session 2.1.2 - VHIR, Bar...
NGS Applications I (UEB-UAT Bioinformatics Course - Session 2.1.2 - VHIR, Bar...VHIR Vall d’Hebron Institut de Recerca
Course: Bioinformatics for Biomedical Research (2014).
Session: 2.1.2- Next Generation Sequencing. Technologies and Applications. Part II: NGS Applications I.
Statistics and Bioinformatisc Unit (UEB) & High Technology Unit (UAT) from Vall d'Hebron Research Institute (www.vhir.org), Barcelona.NGx Sequencing 101-platforms
The field of next-generation sequencing (NGS) has been experiencing explosive growth over the past several years and shows little sign of slowing down. The increasing capabilities and dramatically lowered costs have expanded NGS's reach beyond that of the human genome into nearly every corner of biological research. An overview of the platforms on the market today, including an assessment of their relative strengths and weaknesses, will be presented. The presentation will conclude with a peek into where the technology is going and what will be available in the future.
Viewers also liked (20)
Building bioinformatics resources for the global community
Building bioinformatics resources for the global community
Aug2013 Heidi Rehm integrating large scale sequencing into clinical practice
Aug2013 Heidi Rehm integrating large scale sequencing into clinical practice
EU PathoNGenTraceConsortium:cgMLST Evolvement and Challenges for Harmonization
EU PathoNGenTraceConsortium:cgMLST Evolvement and Challenges for Harmonization
Next Generation Sequencing for Identification and Subtyping of Foodborne Pat...
Next Generation Sequencing for Identification and Subtyping of Foodborne Pat...
Tools for Metagenomics with 16S/ITS and Whole Genome Shotgun Sequences
Tools for Metagenomics with 16S/ITS and Whole Genome Shotgun Sequences
Errors and Limitaions of Next Generation Sequencing
Errors and Limitaions of Next Generation Sequencing
Speeding up sequencing: Sequencing in an hour enables sample to answer in a w...
Speeding up sequencing: Sequencing in an hour enables sample to answer in a w...
transforming clinical microbiology by next generation sequencing
transforming clinical microbiology by next generation sequencing
Introduction to Metagenomics Data Analysis - UEB-VHIR - 2013
Introduction to Metagenomics Data Analysis - UEB-VHIR - 2013
QIAseq Technologies for Metagenomics and Microbiome NGS Library Prep
QIAseq Technologies for Metagenomics and Microbiome NGS Library Prep
NGS Applications I (UEB-UAT Bioinformatics Course - Session 2.1.2 - VHIR, Bar...
NGS Applications I (UEB-UAT Bioinformatics Course - Session 2.1.2 - VHIR, Bar...
Similar to Exploring Spark for Scalable Metagenomics Analysis: Spark Summit East talk by Zhong Wang
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
BioPig for scalable analysis of big sequencing data
This document introduces BioPig, a Hadoop-based analytic toolkit for large-scale genomic sequence analysis. BioPig aims to provide a flexible, high-level, and scalable platform to enable domain experts to build custom analysis pipelines. It leverages Hadoop's data parallelism to speed up bioinformatics tasks like k-mer counting and assembly. The document demonstrates how BioPig can analyze over 1 terabase of metagenomic data using just 7 lines of code, much more simply than alternative MPI-based solutions. While challenges remain around optimization and integration, BioPig shows promise for scalable genomic analytics on very large datasets.
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
This document provides guidance on sizing MongoDB deployments on AWS for optimal performance. It discusses key considerations for capacity planning like testing workloads, measuring performance, and adjusting over time. Different AWS services like compute-optimized instances and storage options like EBS are reviewed. Best practices for WiredTiger like sizing cache, effects of compression and encryption, and monitoring tools are covered. The document emphasizes starting simply and scaling based on business needs and workload profiling.
Processing 70Tb Of Genomics Data With ADAM And Toil
This document discusses analyzing large genomic datasets with ADAM and Toil. It summarizes the sequencing and analysis process, and how ADAM implemented on Spark can provide horizontal scalability and speedups of 30-50x over traditional tools. Toil is introduced as a pipeline system for massive genomic workflows that can run on thousands of nodes and is resilient to failures. Results show ADAM produces equivalent variants to GATK while being 3.5x faster and 4x cheaper.
Analyzing Data Movements and Identifying Techniques for Next-generation Networks
Jan 28th, 2013 - 10:00 am
UC Davis
Title: Analyzing Data Movements and Identifying Techniques for Next-generation Networks
Abstract: Large bandwidth provided by today’s networks requires careful evaluation in order to eliminate system overheads and to bring anticipated high performance to the application layer. As a part of the Advance Network Initiative (ANI) project, we have conducted a large number of experiments in the initial evaluation of the 100Gbps network prototype.
We needed intense fine-tuning, both in network and application layers, to take advantage of the higher network capacity. Instead of explicit improvements in every application as we keep changing the underlying link technology, we require novel data movement mechanisms and abstract layers for end-to-end processing of data. Based on our experience in 100Gbps network, we have developed an experimental prototype, called MemzNet: Memory-mapped Zero-copy Network Channel. MemzNet def ines new data access methods in which applications map memory blocks for remote data, in contrast to the send/receive semantics. In one of the early demonstrations of 100Gbps network applications, we used the initial implementation of MemzNet that takes the approach of aggregating files into blocks and providing dynamic data channel management. We observed that MemzNet showed better results in terms of performance and efficiency,
than the current state-of-the-art file-centric data transfer tools for the transfer of climate datasets with many small files. In this talk, I will mainly describe our experience in 100Gbps tests and present results from the 100Gbps demonstration. I will briefly explain the ANI testbed environment and highlight future research plans.
Bio: Mehmet Balman is a researcher working as a computer engineer in the Computational Research Division at Lawrence Berkeley National Laboratory. His recent work
particularly deals with efficient data transfer mechanisms, high-performance network protocols, bandwidth reservation, network virtualization, scheduling and resource management for large-scale applications. He received his doctoral degree in computer science from Louisiana State University (LSU) in 2010. He has several years of industrial experience as system administrator and R&D specialist, at various software companies before joining LSU. He also worked as a summer intern in Los Alamos National Laboratory.
MSR 2009
The document proposes using MapReduce as a general framework to support research in mining software repositories (MSR). It describes how MapReduce can provide efficiency, scalability, adaptability and flexibility for common MSR tasks like analyzing large code repositories. A case study of applying MapReduce to the J-REX MSR tool shows significant reductions in running time for large datasets. Minimal programming effort was required and MapReduce could run on various computing environments.
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
Machine Learning at the Limit
John Canny, UC Berkeley
How fast can machine learning and graph algorithms be? In "roofline" design, every kernel is driven toward the limits imposed by CPU, memory, network etc. This can lead to dramatic improvements: BIDMach is a toolkit for machine learning that uses rooflined design and GPUs to achieve two- to three-orders of magnitude improvements over other toolkits on single machines. These speedups are larger than have been reported for *cluster* systems (e.g. Spark/MLLib, Powergraph) running on hundreds of nodes, and BIDMach with a GPU outperforms these systems for most common machine learning tasks. For algorithms (e.g. graph algorithms) which do require cluster computing, we have developed a rooflined network primitive called "Kylix". We can show that Kylix approaches the rooline limits for sparse Allreduce, and empirically holds the record for distributed Pagerank. Beyond rooflining, we believe there are great opportunities from deep algorithm/hardware codesign. Gibbs Sampling (GS) is a very general tool for inference, but is typically much slower than alternatives. SAME (State Augmentation for Marginal Estimation) is a variation of GS which was developed for marginal parameter estimation. We show that it has high parallelism, and a fast GPU implementation. Using SAME, we developed a GS implementation of Latent Dirichlet Allocation whose running time is 100x faster than other samplers, and within 3x of the fastest symbolic methods. We are extending this approach to general graphical models, an area where there is currently a void of (practically) fast tools. It seems at least plausible that a general-purpose solution based on these techniques can closely approach the performance of custom algorithms.
Bio
John Canny is a professor in computer science at UC Berkeley. He is an ACM dissertation award winner and a Packard Fellow. He is currently a Data Science Senior Fellow in Berkeley's new Institute for Data Science and holds a INRIA (France) International Chair. Since 2002, he has been developing and deploying large-scale behavioral modeling systems. He designed and protyped production systems for Overstock.com, Yahoo, Ebay, Quantcast and Microsoft. He currently works on several applications of data mining for human learning (MOOCs and early language learning), health and well-being, and applications in the sciences.
2013 talk at TGAC, November 4
Digital normalization provides a computationally efficient way to filter high-coverage reads from shotgun sequencing data, reducing data size while retaining most of the information needed for downstream analysis. It works by estimating the coverage of each read without using a reference genome and discarding reads above a given coverage cutoff. The method has been shown to significantly decrease memory requirements for de novo assembly of various data types like metagenomes and transcriptomes, while producing similar or improved assembly results. Future work includes developing reference-free methods for analyzing sequencing data in a streaming fashion before or without assembly.
IAP09 CUDA@MIT 6.963 - Guest Lecture: Out-of-Core Programming with NVIDIA's C...
More at http://sites.google.com/site/cudaiap2009 and http://pinto.scripts.mit.edu/Classes/CUDAIAP2009
Computing Outside The Box September 2009
Keynote talk at Parco 2009 in Lyon, France. An updated version of http://www.slideshare.net/ianfoster/computing-outside-the-box-june-2009.
Memory efficient java tutorial practices and challenges
This document summarizes challenges in building memory-efficient Java applications and common patterns of memory usage. It discusses how object representation and collection choices can significantly impact memory usage, with overhead sometimes accounting for 50-90% of memory consumption. The document provides examples of how data type modeling decisions, such as high levels of delegation, large base classes, and unnecessary fields, can lead to high memory overhead. It emphasizes measuring and understanding memory usage at the data type and collection level in order to make informed design tradeoffs.
Ch22 parallel d_bs_cs561
1) Parallelism can significantly speed up data access and queries by dividing problems across multiple machines. For example, scanning 1TB in parallel across 1000 machines would take 1.5 minutes instead of 1.2 days with a single machine.
2) There are two main types of parallelism in database management systems - pipeline parallelism where each machine performs one step in a process, and partition parallelism where each machine works on a different portion of data.
3) Shared-nothing architectures where each machine has its own processors, memory and disks are best for parallel databases as they are cheap, scalable, and allow each machine to work independently on partitioned data.
Scaling Genomic Analyses
ADAM is an open source, high performance, distributed platform for genomic analysis that defines a data schema and layout on disk using Parquet and Avro, integrates with Spark's Scala and Java APIs, and provides a command line interface. ADAM achieves linear scalability out to 128 nodes for most tasks and provides a 2-4x performance improvement over other tools like GATK and samtools. The platform includes various tools like avocado for efficient local variant calling via de Bruijn graph reassembly of sequencing reads.
Future Architectures for genomics
This talk details some of the work we are doing looking at new CPU and storage architectures to support sequencing and bioinformatics.
ASE2010
This document reports on scaling tools for mining software repositories (MSR) studies using MapReduce. It finds that MapReduce can effectively scale three large MSR studies - a software evolution study, code clone detection, and log analysis - to larger datasets and clusters of up to 28 machines. The main challenges in migrating MSR studies to MapReduce are the locality and granularity of the analysis, locating a suitable cluster, managing large datasets, and handling errors.
Scaling metagenome assembly
The document discusses scaling metagenome assembly to very large datasets. It proposes two approaches: 1) Partitioning the de Bruijn graph to divide the data into smaller, independent components that can be assembled in parallel. However, partitioning faces the same challenges of scaling to large datasets as other assemblers. 2) Digital normalization, which "squashes" redundant high coverage reads in a single pass to reduce the total data while preserving sequence content and gene coverage. Initial assemblies of field samples demonstrate the approaches can assemble petabase-scale datasets on a single compute node. Future work involves optimizing partitions, scaling implementations, and integrating other data types.
Next-generation sequencing format and visualization with ngs.plot
Lecture given at the department of neuroscience, Icahn school of medicine at Mount Sinai. ngs.plot has been published in BMC genomics. Link: http://www.biomedcentral.com/1471-2164/15/284
BDI- The Beginning (Big data training in Coimbatore)
The main objective of “Big Data intelligence” is to understand all of us better to predict the future. Be it 4 billion google queries a day or 1 billion FB users, we need smarter AI algorithms to learn and connect the dots from the ocean of data. With massive parallelism and Map-Reduce techniques, millions of servers take us one step closer to the “Turing’s Intelligent machine”. Near AI success stories are google, facebook, twitter, youtube and Amazon. Let's begin our journey by knowing big hype, big dreams of 50's , big laws, big growth and basic operations to extract big data intelligence.For more information on Big Data training in coimbatore, please visit https://bigzettab.wordpress.com/ . - Prof. Ashok.R, +91-9943900101, ashok@zettab.com.
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.
Making powerful science: an introduction to NGS data analysis
This slide deck is from the Botnar Research Centre introduction to NGS sequencing workshop 2021- an overview of the theoretical concepts behind sequencing data analysis are given
Similar to Exploring Spark for Scalable Metagenomics Analysis: Spark Summit East talk by Zhong Wang (20)
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
BioPig for scalable analysis of big sequencing data
BioPig for scalable analysis of big sequencing data
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
Processing 70Tb Of Genomics Data With ADAM And Toil
Processing 70Tb Of Genomics Data With ADAM And Toil
Analyzing Data Movements and Identifying Techniques for Next-generation Networks
Analyzing Data Movements and Identifying Techniques for Next-generation Networks
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
IAP09 CUDA@MIT 6.963 - Guest Lecture: Out-of-Core Programming with NVIDIA's C...
IAP09 CUDA@MIT 6.963 - Guest Lecture: Out-of-Core Programming with NVIDIA's C...
Memory efficient java tutorial practices and challenges
Memory efficient java tutorial practices and challenges
Next-generation sequencing format and visualization with ngs.plot
Next-generation sequencing format and visualization with ngs.plot
BDI- The Beginning (Big data training in Coimbatore)
BDI- The Beginning (Big data training in Coimbatore)
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Making powerful science: an introduction to NGS data analysis
Making powerful science: an introduction to NGS data analysis
More from Spark Summit
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Graph is on the rise and it’s time to start learning about scalable graph analytics! In this session we will go over two Spark-based Graph Analytics frameworks: Tinkerpop and GraphFrames. While both frameworks can express very similar traversals, they have different performance characteristics and APIs. In this Deep-Dive by example presentation, we will demonstrate some common traversals and explain how, at a Spark level, each traversal is actually computed under the hood! Learn both the fluent Gremlin API as well as the powerful GraphFrame Motif api as we show examples of both simultaneously. No need to be familiar with Graphs or Spark for this presentation as we’ll be explaining everything from the ground up!
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. In cooperation with our partner, NEC Laboratories America, we have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, parameters and features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. The evaluation with real open data demonstrates that our system can explore hundreds of predictive models and discovers the most accurate ones in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints. This talk will cover the presentation already shown on Spark Summit SF’17 (#SFds5) but from more technical perspective.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
With the rapid growth of available datasets, it is imperative to have good tools for extracting insight from big data. The Spark ML library has excellent support for performing at-scale data processing and machine learning experiments, but more often than not, Data Scientists find themselves struggling with issues such as: low level data manipulation, lack of support for image processing, text analytics and deep learning, as well as the inability to use Spark alongside other popular machine learning libraries. To address these pain points, Microsoft recently released The Microsoft Machine Learning Library for Apache Spark (MMLSpark), an open-source machine learning library built on top of SparkML that seeks to simplify the data science process and integrate SparkML Pipelines with deep learning and computer vision libraries such as the Microsoft Cognitive Toolkit (CNTK) and OpenCV. With MMLSpark, Data Scientists can build models with 1/10th of the code through Pipeline objects that compose seamlessly with other parts of the SparkML ecosystem. In this session, we explore some of the main lessons learned from building MMLSpark. Join us if you would like to know how to extend Pipelines to ensure seamless integration with SparkML, how to auto-generate Python and R wrappers from Scala Transformers and Estimators, how to integrate and use previously non-distributed libraries in a distributed manner and how to efficiently deploy a Spark library across multiple platforms.
Next CERN Accelerator Logging Service with Jakub Wozniak
The Next Accelerator Logging Service (NXCALS) is a new Big Data project at CERN aiming to replace the existing Oracle-based service.
The main purpose of the system is to store and present Controls/Infrastructure related data gathered from thousands of devices in the whole accelerator complex.
The data is used to operate the machines, improve their performance and conduct studies for new beam types or future experiments.
During this talk, Jakub will speak about NXCALS requirements and design choices that lead to the selected architecture based on Hadoop and Spark. He will present the Ingestion API, the abstractions behind the Meta-data Service and the Spark-based Extraction API where simple changes to the schema handling greatly improved the overall usability of the system. The system itself is not CERN specific and can be of interest to other companies or institutes confronted with similar Big Data problems.
Powering a Startup with Apache Spark with Kevin Kim
In Between (A mobile App for couples, downloaded 20M in Global), from daily batch for extracting metrics, analysis and dashboard. Spark is widely used by engineers and data analysts in Between, thanks to the performance and expendability of Spark, data operating has become extremely efficient. Entire team including Biz Dev, Global Operation, Designers are enjoying data results so Spark is empowering entire company for data driven operation and thinking. Kevin, Co-founder and Data Team leader of Between will be presenting how things are going in Between. Listeners will know how small and agile team is living with data (how we build organization, culture and technical base) after this presentation.
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
In many cases, Big Data becomes just another buzzword because of the lack of tools that can support both the technological requirements for developing and deploying of the projects and/or the fluency of communication between the different profiles of people involved in the projects.
In this talk, we will present Moriarty, a set of tools for fast prototyping of Big Data applications that can be deployed in an Apache Spark environment. These tools support the creation of Big Data workflows using the already existing functional blocks or supporting the creation of new functional blocks. The created workflow can then be deployed in a Spark infrastructure and used through a REST API.
For better understanding of Moriarty, the prototyping process and the way it hides the Spark environment to the Big Data users and developers, we will present it together with a couple of examples based on a Industry 4.0 success cases and other on a logistic success case.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Data lineage tracking is one of the significant problems that financial institutions face when using modern big data tools. This presentation describes Spline – a data lineage tracking and visualization tool for Apache Spark. Spline captures and stores lineage information from internal Spark execution plans and visualizes it in a user-friendly manner.
Goal Based Data Production with Sim Simeonov
Since the invention of SQL and relational databases, data production has been about specifying how data is transformed through queries. While Apache Spark can certainly be used as a general distributed query engine, the power and granularity of Spark’s APIs enables a revolutionary increase in data engineering productivity: goal-based data production. Goal-based data production concerns itself with specifying WHAT the desired result is, leaving the details of HOW the result is achieved to a smart data warehouse running on top of Spark. That not only substantially increases productivity, but also significantly expands the audience that can work directly with Spark: from developers and data scientists to technical business users. With specific data and architecture patterns spanning the range from ETL to machine learning data prep and with live demos, this session will demonstrate how Spark users can gain the benefits of goal-based data production.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This work presents new efficient and scalable matrix processing and optimization techniques based on Spark. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix techniques inside the Spark SQL, and optimize the matrix execution plan based on Spark SQL Catalyst. We conduct case studies on a series of ML models and matrix computations with special features on different datasets. These are PageRank, GNMF, BFGS, sparse matrix chain multiplications, and a biological data analysis. The open-source library ScaLAPACK and the array-based database SciDB are used for performance evaluation. Our experiments are performed on six real-world datasets are: social network data ( e.g., soc-pokec, cit-Patents, LiveJournal), Twitter2010, Netflix recommendation data, and 1000 Genomes Project sample. Experiments demonstrate that our proposed techniques achieve up to an order-of-magnitude performance.
More from Spark Summit (20)
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Recently uploaded
一比一原版马来西亚博特拉大学毕业证（upm毕业证）如何办理
原版一模一样【微信：741003700 】【马来西亚博特拉大学毕业证（upm毕业证）成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理马来西亚博特拉大学毕业证（upm毕业证）【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理马来西亚博特拉大学毕业证（upm毕业证）【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理马来西亚博特拉大学毕业证（upm毕业证）【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理马来西亚博特拉大学毕业证（upm毕业证）【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
Gran libro. Lo recomiendo
一比一原版格里菲斯大学毕业证(Griffith毕业证书)学历如何办理
原版制作【微信号:BYZS866】【格里菲斯大学毕业证(Griffith毕业证书)】【微信号:BYZS866】《成绩单、外壳、雅思、offer、真实留信官方学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路）我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信号BYZS866】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信号BYZS866】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
Build applications with generative AI on Google Cloud
We will explore Vertex AI - Model Garden powered experiences, we are going to learn more about the integration of these generative AI APIs. We are going to see in action what the Gemini family of generative models are for developers to build and deploy AI-driven applications. Vertex AI includes a suite of foundation models, these are referred to as the PaLM and Gemini family of generative ai models, and they come in different versions. We are going to cover how to use via API to: - execute prompts in text and chat - cover multimodal use cases with image prompts. - finetune and distill to improve knowledge domains - run function calls with foundation models to optimize them for specific tasks. At the end of the session, developers will understand how to innovate with generative AI and develop apps using the generative ai industry trends.
一比一原版多伦多大学毕业证(UofT毕业证书)学历如何办理
办理【微信号:176555708】【办理(UofT毕业证书)】【微信号:176555708】《成绩单、外壳、offer、真实留信官方学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路）我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信号:176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信号:176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
To honor ten years of PyData London, join Dr. Rebecca Bilbro as she takes us back in time to reflect on a little over ten years working as a data scientist. One of the many renegade PhDs who joined the fledgling field of data science of the 2010's, Rebecca will share lessons learned the hard way, often from watching data science projects go sideways and learning to fix broken things. Through the lens of these canon events, she'll identify some of the anti-patterns and red flags she's learned to steer around.
一比一原版悉尼大学毕业证如何办理
原版一模一样【微信：741003700 】【悉尼大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理悉尼大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理悉尼大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理悉尼大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理悉尼大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
一比一原版南昆士兰大学毕业证如何办理
原版一模一样【微信：741003700 】【南昆士兰大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理南昆士兰大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理南昆士兰大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理南昆士兰大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理南昆士兰大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
社内勉強会資料_Hallucination of LLMs .
社内エンジニア・リサーチャー勉強会の発表資料「Hallucination of LLMs」を公開しました！
大規模言語モデル(LLM)に起こるハルシネーションのメカニズムと、それらを軽減する方法について紹介しています。
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
原版办【微信号:BYZS866】【兰加拉学院毕业证(Langara毕业证书)】【微信号:BYZS866】《成绩单、外壳、雅思、offer、真实留信官方学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路）我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信号BYZS866】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信号BYZS866】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
原版一模一样【微信：6496090 】【(uob毕业证书)伯明翰大学毕业证成绩单】【微信：6496090 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微6496090
【主营项目】
一.毕业证【q微6496090】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微6496090】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理(uob毕业证书)伯明翰大学毕业证【微信：6496090 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理(uob毕业证书)伯明翰大学毕业证【微信：6496090 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理(uob毕业证书)伯明翰大学毕业证【微信：6496090 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理(uob毕业证书)伯明翰大学毕业证【微信：6496090 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
一比一原版英属哥伦比亚大学毕业证(UBC毕业证书)学历如何办理
原版办【微信号:BYZS866】【英属哥伦比亚大学毕业证(UBC毕业证书)】【微信号:BYZS866】《成绩单、外壳、雅思、offer、真实留信官方学历认证（永久存档/真实可查）》采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路）我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信号BYZS866】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信号BYZS866】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
学校原件一模一样【微信：741003700 】《(unimelb毕业证书)墨尔本大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
8 things to know before you start to code in 2024
Things I wish I knew before I started coding.
Simple best practices that will save you time and effort in 2024.
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
原版一模一样【微信：741003700 】【(lbs毕业证书)伦敦商学院毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理(lbs毕业证书)伦敦商学院毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理(lbs毕业证书)伦敦商学院毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理(lbs毕业证书)伦敦商学院毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理(lbs毕业证书)伦敦商学院毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
一比一原版加拿大麦吉尔大学毕业证（mcgill毕业证书）如何办理
原版一模一样【微信：741003700 】【加拿大麦吉尔大学毕业证（mcgill毕业证书）成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理加拿大麦吉尔大学毕业证（mcgill毕业证书）【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理加拿大麦吉尔大学毕业证（mcgill毕业证书）【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理加拿大麦吉尔大学毕业证（mcgill毕业证书）【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理加拿大麦吉尔大学毕业证（mcgill毕业证书）【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
Telemetry Solution for Gaming (AWS Summit'24)
Discover the cutting-edge telemetry solution implemented for Alan Wake 2 by Remedy Entertainment in collaboration with AWS. This comprehensive presentation dives into our objectives, detailing how we utilized advanced analytics to drive gameplay improvements and player engagement.
Key highlights include:
Primary Goals: Implementing gameplay and technical telemetry to capture detailed player behavior and game performance data, fostering data-driven decision-making.
Tech Stack: Leveraging AWS services such as EKS for hosting, WAF for security, Karpenter for instance optimization, S3 for data storage, and OpenTelemetry Collector for data collection. EventBridge and Lambda were used for data compression, while Glue ETL and Athena facilitated data transformation and preparation.
Data Utilization: Transforming raw data into actionable insights with technologies like Glue ETL (PySpark scripts), Glue Crawler, and Athena, culminating in detailed visualizations with Tableau.
Achievements: Successfully managing 700 million to 1 billion events per month at a cost-effective rate, with significant savings compared to commercial solutions. This approach has enabled simplified scaling and substantial improvements in game design, reducing player churn through targeted adjustments.
Community Engagement: Enhanced ability to engage with player communities by leveraging precise data insights, despite having a small community management team.
This presentation is an invaluable resource for professionals in game development, data analytics, and cloud computing, offering insights into how telemetry and analytics can revolutionize player experience and game performance optimization.
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
We are pleased to share with you the latest VCOSA statistical report on the cotton and yarn industry for the month of March 2024.
Starting from January 2024, the full weekly and monthly reports will only be available for free to VCOSA members. To access the complete weekly report with figures, charts, and detailed analysis of the cotton fiber market in the past week, interested parties are kindly requested to contact VCOSA to subscribe to the newsletter.
Recently uploaded (20)
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
PyData London 2024: Mistakes were made (Dr. Rebecca Bilbro)
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
Overview IFM June 2024 Consumer Confidence INDEX Report.pdf
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
[VCOSA] Monthly Report - Cotton & Yarn Statistics March 2024
Exploring Spark for Scalable Metagenomics Analysis: Spark Summit East talk by Zhong Wang
- 1. Explore Spark for Metagenome assembly Zhong Wang, Ph.D. Group Lead, DOE Joint Genome Institute Lawrence Berkeley National Lab
- 2. Metagenome is the genome of a microbial community
- 3. Microbial communities are “dark matters” Number of Species Cow ～6000 Human ～1000 Soil, >100000 >90% of the species haven’t been seen before
- 4. Metagenome sequencing Harvest microbes Extract DNA Shear, & Sequencing Assembly Short Reads Reconstructed genomes Microbes Genomic DNA
- 5. Metagenome assembly Library of Books Shredded Library “reconstructed”Library Genome ~= Book Metagenome ~=Library Sequencing ~=sampling the pieces
- 6. Scale is an enemy 1 10 100 1,000 10,000 100,000 1,000,000 Common Human Cow Soil Gigabytes
- 7. Complexity is another… • Data complexity – Contamination – Number of microbial species – Species abundance distribution – Sequencing errors • Algorithm complexity – Multiple steps, each has different time/space characteristics
- 8. The Ideal Solution q Easy to develop q Robust q Scale to big data q Efficient
- 9. 2009: Special Hardware Input/Output (IO) Memory $1M Only scale up to ~100Gb Jeremy Brand @JGI FPGA @Convey
- 10. 2010: MP/MPI on supercomputers • Experienced software engineers • Six months of development time • One task fails, all tasks fail Problems: Fast, scalable Rob Egan @JGI MPI version 412 Gb, 4.5B reads 2.7 hours on 128x24 cores NESRC Supercomputer
- 11. 2013: Hadoop/BioPig x x Karan Bhatia, Henrik Nordberg, Kai Wang BioPIG
- 12. Challenges in application • 2-3 orders of magnitude slower than MPI • IO optimization, e.g., reduce data copying • Some problems do not easily fit into map/reduce framework, e.g., graph-based algorithms • Runs on AWS, but cost $$$ if not optimized
- 13. Addressing big data: Apache Spark • New scalable programming paradigm • Compatible with Hadoop-supported storage systems • Improves efficiency through: • In-memory computing primitives • General computation graphs • Improves usability through: • Rich APIs in Java, Scala, Python • Interactive shell q Scale to big data q Efficient q Easy to develop q Robust?
- 14. Goal: Metagenome read clustering • Clustering reads based on their genome of origin can reduce metagenome problem to single-genome problem • Ideally scale up to TB data sizes
- 15. Algorithm • highly frequent k- mers (n-grams) Contamination prediction • K-mer generation • Filter out low frequency k-mers (noise) K-mer generation and filtering • Compute edge weight • Remove low weight edges Read graph generation and filtering • Connected component • Power-iteration clustering Graph partition
- 16. Platforms we run spark • Standalone Spark on single large memory server • On-demand Spark cluster over HPC • AWS Elastic Map Reduce (EMR)
- 17. Testing the accuracy of the algorithm with a small toy test dataset • Species: – 6 bacterial species (10kb from each) – Synthetic communities with random proportions of each genome, reads drawn from single genome sequencing projects (noisy) – Ideal situation (no shared sequences between genomes, sufficient sequencing coverage): Reads of the same color belong to the same genome
- 18. Real world datasets Dataset Number of spieces Sampling depth Soil metagenome High Low Cow rumen metagenome Medium Low to medium Maize transcriptome (“fake metagenome”) Low High
- 19. Data grows during analysis K-mers Full graph Filtered graph Graph is ~200x larger than input data
- 20. Tuning parallelism for load balance K-mer generation/filtering: partition_size = 16Mb K-mer generation/filtering: partition_size = 1.3Mb
- 21. Tuning parallelism for performance Optimizing parameters can reduce total running time from > 90 min to 20 min Decreasing partition size for graph construction
- 22. Scale well on small data 1 2 4 8 Contamination prediction time K-mer generation and filter time Read graph generation and filter time Connected Component computation time Total elapsed time GB On 50x r3-xlarge Walltime
- 23. Performance over different datasets Data complexity is a big driving factor for compute time <0.5 hours ~5 hours All three are 20GB datasets
- 24. Cost on AWS EMR spot instances Projected cost for a typical 1TB dataset: ~$500 20GB datasets
- 25. Scaling up to 100Gb: failed Read graph generation/filtering Reducing partition numbers for graph partitioning
- 26. Potential solutions • Avoid shuffling: – generate the graph, save to disk, then merge partitions outside of Spark. • Size-specific parameters – Larger datasets may not use parallelism parameters optimized for smaller ones • Your inputs…
- 27. Overall impression of Spark ü Easy to develop o Scala/python API o Databricks notebook ü Efficient o Much, much faster than Hadoop o High cluster utilization rate ? Robust o Platform dependent o 30% failure rate on AWS ? Scale o Problem specific o Intermediate data size may change during running o Problem complexity may grow with scale
- 28. Acknowledgements Spark Team Xiandong Meng Jordan Hoffman@Harvard Lisa Gerhardt , Evan Racah @ NERSC Gary Jung, Greg Kurtzer Bernard Li,Yong Qin @ HPC
- 29. Thank You.