VariantSpark a library for genomics by Lynn Langit
•
1 like•164 views
Abstract:- See the VariantSpark library in action on a Databricks Jupyter notebook.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to VariantSpark a library for genomics by Lynn Langit
Similar to VariantSpark a library for genomics by Lynn Langit (20)
More from Data Con LA
More from Data Con LA (20)
Recently uploaded
Recently uploaded (20)
VariantSpark a library for genomics by Lynn Langit
- 1. VariantSpark: a library for Genomics Transformational Bioinformatics | Denis C. Bauer | @allPowerde Lynn Langit
- 3. Natalie Twine Transformational Bioinformatics Team Transformational Bioinformatics | Denis C. Bauer | @allPowerde Denis Bauer Oscar Luo Rob Dunne Piotr SzulAidan O’BrienLaurence Wilson Adrian White Mia Champion Gaetan Burgio Collaborators David Levy News Software Dan Andrews Kaitao Lai Kaylene Simpson Iva Nikolic Ian Blair Kelly Williams
- 4. BMC Genomics 2015, 16:1052 PMID: 26651996 (IF=4) Cited 4 VariantSpark | Denis C. Bauer @allPowerde
- 5. Unsupervised ML : K-Means www.cloudaccess.eu 1000 x 40 Million variants Matrix * k-means Predict super population 4 14 ethnic groups and s u p e r populations VariantSpark | Denis C. Bauer @allPowerde * VariantSpark can also process phase 3 data: 3000 individuals and 80 million variants
- 6. Comparing k-means to other implementations 0 1000 2000 Python R H adoop Adam AD M IXTU R E VariantSpark method timeinseconds task binary−conversion clustering pre−processing 103 75 29 28 18 4 min VariantSpark | Denis C. Bauer @allPowerde
- 7. Supervised ML: Wide Random Forests Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 8. Transformational Bioinformatics | Denis C. Bauer | @allPowerde Genomic Research Workflow https://www.projectmine.com/about/ Focus
- 9. Performance – Faster and More Accurate VariantSpark is the only method to scale to 100% of the genome Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 10. Scaling to 50 M variable and 10 K samples Transformational Bioinformatics | Denis C. Bauer | @allPowerde 100K trees: 5 – 50h AWS: ~$215.50 100K trees: 200 – 2000h AWS: ~ $ 8620.00 • Yarn Cluster (12 workers) • 16 x Intel Xeon E5-2660@2.20GHz CPU • 128 GB of RAM • Spark 1.6.1 on YARN • 128 executors • 6GB / executor (0.75TB) • Synthetic dataset (mtry = 0.25) Whole Genome Range GWAS Range
- 12. Databricks & VariantSpark via a Jupyter notebook
- 13. Solving Important Questions… Cancer genomics?
- 14. DEMO: Who is a Hipster?
- 15. • Quickly access a managed Spark cluster - AWS EC2 / spot instances • Link to your data and perform whole genome analysis in real-time VariantSpark & Databricks Notebooks Transformational Bioinformatics | Denis C. Bauer | @allPowerde Jupyter Notebook Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 16. Joint-loci association test Hipster-Index = ((2 + GT[B6]) * (1.5 + GT[R1])) + ((0.5 + GT[C2]) * (1 + GT[B2])) Label = 1 if Hipster-Index>10 Genomic profile Label Samples(n=2500) Transformational Bioinformatics | Denis C. Bauer | @allPowerde
- 17. VariantSpark: a library for Genomics Transformational Bioinformatics | Denis C. Bauer | @allPowerde Lynn Langit
Editor's Notes
- http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195
- http://www.cloudaccess.eu/blog/wp-content/uploads/2014/06/genetic_roots.png
- Chromosome 22; VM on Microsoft Azure with A7 Linux instance and 8 cores, 56GB memory running Ubuntu.
- https://academics.cloud.databricks.com/#notebook/170398/command/170419