Downloaded 36 times
Uploaded bySri Ambati
H2O World - Sparkling water on the Spark Notebook: Interactive Genomes Clustering - Xavier Tordoir
Spark is a distributed computing framework that can handle large scale data processing. The Spark notebook provides an interactive environment for working with Spark. ADAM is a data format and API for genomic data on Spark that optimizes for large datasets. Sparkling Water integrates H2O machine learning with Spark to enable techniques like deep learning on genomic data in a distributed manner using the Spark notebook. Data scientists and developers can collaborate using these tools to access, manipulate, and analyze massive genomic datasets.
More Related Content
Similar to H2O World - Sparkling water on the Spark Notebook: Interactive Genomes Clustering - Xavier Tordoir
H2O World - Sparkling water on the Spark Notebook: Interactive Genomes Clustering - Xavier Tordoir
- 1. Sparkling Water onthe Spark Notebook: Interactive Genomes clustering Why you must care, by Data Fellas Xavier Tordoir xtordoir@data-fellas.guru @xtordoir
- 2. ● Apache Spark ●Interactivity: Spark notebook ● Genomics on Spark: ADAM ● Data exploitation ● H2O w/ Spark: Sparkling water ● Show time ● Streamlining dev/deployment Lineup Can’t wait!
- 3. Data Fellas Andy Petrella Maths Geospatial DistributedComputing Spark Notebook Trainer Spark/Scala Machine Learning Xavier Tordoir Physics Bioinformatics Distributed Computing Scala (& Perl) trainer Spark Machine Learning
- 4. Distributed computing framework LargeScale Data Processing engine I play BIG! What is Apache Spark?
- 5. Distributed computing framework LargeScale Data Processing engine ● SQL & Dataframes ● Streaming ● Graph Processing ● Machine Learning With all colors! What is Apache Spark?
- 6. Distributed computing framework LargeScale Data Processing engine ● Optimize memory usage (FAST) ● Optimize computation execution (Complex tasks) ● Easy programming model Checking in cache If I remember... What is Apache Spark?
- 7. Distributed computing framework LargeScale Data Processing engine ● Interactive ● @ any scale http://spark-notebook.io Laurel? HArdy? Anyone? What is Apache Spark?
- 8. ● Scala (types,production quality) ● Reactive&pluggable charts API (scala = no.js) ● easy install, no deps. ● multiple sparkContext out of the box. What is Apache Spark?
- 9. http://bdgenomics.org/ ADAM Project (UCBerkeley): ● Data format (schema, compact, distributed): avro + parquet ● API (Reads, Variants, Genotypes, …) I, ADAM Genomics with Spark?
- 10. 1000 genomes: http://www.1000genomes.org/ ~1000samples ~30M Genotypes per sample (features) Genomics The data Please, don’t mind the colors...
- 11. Genomics The data So… that’swhat separates us huh?
- 12. 1000 genomes: http://www.1000genomes.org/ ~1000samples Few samples => Machine Learning Genomics The data Woooow, really, you must be kidding me… ahahahahah
- 13. 1000 genomes: http://www.1000genomes.org/ ~1000samples ~30M Genotypes per sample (features) Few samples => Machine Learning Lots of Data => Distributed computing Genomics The data Oh… damned… hum huh
- 14. Population stratification w/ Deeplearning?H2O From the spark notebook? Sparkling water Genomics The problem Here I need some water.
- 15. Memory implementation of“Map-Reduce” Highly optimised structures for the JVM blazing fast convergent models H2O Higher API
- 16. H2O Sparkling: in-memory dataexchange I remember things better with two copies in memory. http://h2o.ai/product/sparkling-water/
- 17. Showtime! press play... There’s anotebook for that
- 18. “Create” Cluster Find sources(context, quality, semantic, …) Connect to sources (structure, schema/types, …) Create distributed data pipeline/Model Tune accuracy Tune performances Write results to Sinks Access Layer User Access Shar3 (Data Fellas) ops data ops data sci sci ops sci ops data web ops data web ops data sci
- 19. Shar3 (Data Fellas) Analysis Production DistributionRendering Discovery Catalog Project Generator MicroService / Binary format Schema for output Metadata
- 20. Spark and theNotebook are interactive and leverage distributed computing infrastructure ADAM is an optimized storage format for Massive genomic data Spark provides tools to manipulate data and works w/ other libraries like H2O Data scientists and application developers can work together Summary Wake up, we’re back!
- 21. Acknowledgements Frank Nothaft Matt Massie NeilFergusson Vinod & Michal Thank you For your attention! Questions? And now let’s talk.