Download to read offline
Uploaded byLynn Langit
VariantSpark on AWS
The document discusses Variantspark, a bioinformatics tool developed for AWS that enables efficient genomic variant analysis using supervised machine learning techniques. It describes its capabilities, performance metrics, and cloud data pipeline configurations, as well as the CSIRO team's efforts to create reproducible environments for researchers. Future development plans include comparing Variantspark's performance on different cloud platforms and optimizing workflow tools.
More Related Content
Similar to VariantSpark on AWS
VariantSpark on AWS
- 1. Lynn Langit forCSIRO Bioinformatics VariantSpark on AWS
- 2. Denis Bauer, PhD Oscar Luo, PhD RobDunne, PhD Piotr SzulAidan O’BrienLaurence Wilson, PhD Adrian White Andy Hindmarch David Levy Dan Andrews Kaitao Lai, PhD Arash Bayat PhD John Hildebrandt Mia Chapman Ian Blair Kelly Williams Jules Damji Gaetan Burgio Lynn Langit Jim Counts Matthew Jones Natalie Twine, PhD Prabha Pillay Transformational Bioinformatics Team www.csiro.au Denis C. Bauer | @allPowerde
- 3. BMC Genomics 2015,16:1052 PMID: 26651996 (IF=4) Cited 4 Denis C. Bauer | @allPowerde
- 4. VariantSpark works withVCF Data
- 5. Supervised ML: WideRandom Forests
- 6. Custom Splits • Horizontal •Vertical Gini Scoring • Local • Global
- 7. Performance – Fasterand More Accurate VariantSpark can scale to 100% of the genome low Accuracy high lowSpeedhigh
- 8. Scaling to 50Mvariables & 10K samples 100K trees: 5 – 50h AWS: ~$215.50 100K trees: 200 – 2000h AWS: ~ $ 8620.00 • Yarn Cluster • 12 workers • 16 x Intel CPUs • Xeon E5-2660@2.20GHz • 128 GB RAM • Spark 1.6.1 • 128 executors • 6GB / executor 0.75TB • Synthetic dataset Whole Genome Range GWAS Range
- 9. Building a CloudData Pipeline Spark •IaaS, PaaS, SaaS Vendors •Alibaba, AWS, GCP…
- 10.
- 11.
- 12. Moving to theCloud – SaaS / Databricks
- 13. Synthetic Phenotype: Whois a Bondi Hipster?
- 14.
- 15. Hello VariantSpark viaHipster-Index BUILDS Community
- 16. Try it outSaaS: VariantSpark Notebook
- 17. Transformational Bioinformatics |@allPowerde
- 18. Spark Server ClusterPipeline Pattern Jupyter Notebook Data Lake
- 19.
- 20. Try it out:VariantSpark on AWS EMR
- 21. Try it out:VariantSpark on AWS EKS
- 22. Apache Spark 2.3+with Kubernetes
- 23. Try it out:VariantSpark on AWS EKS
- 24. Try it out:VariantSpark on AWS EKS
- 25. CSIRO Team TrainsOther Researchers Team creates reproducible cloud environments • AWS CloudFormation Templates for EMR • Setup screencasts for Databricks and AWS • Scripts and recommended parameters
- 26. Next Steps • VariantSparkon GCP • Use GCP DataProc – compare to AWS EMR • Use GCP GKE – compare to AWS EKS (K8) • VariantSpark on Terra.bio • First optimize container for GCP raw compute • Write WDL for VariantSpark tool/workflow • Publish on Dockstore as Tool/Workflow • Publish example Jupyter notebooks • Publish example Terra.bio VariantSpark workflow
- 27. Bioinformatics Tools onAWS Lynn Langit for CSIRO Bioinformatics
Editor's Notes
- #3 VariantSpark - https://bioinformatics.csiro.au/variantspark GT-Scan Suite - https://gt-scan.csiro.au/
- #4 https://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-015-2269-7
- #5 http://www.internationalgenome.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-40/
- #6 https://github.com/aehrc/VariantSpark
- #13 https://academics.cloud.databricks.com/#notebook/170398/command/170419
- #14 Photo from - http://www.drjasonfox.com/
- #15 https://aehrc.github.io/VariantSpark/notebook-examples/VariantSpark_HipsterIndex.html
- #17 https://docs.databricks.com/applications/genomics/variant-spark.html
- #18 https://aehrc.com/als-genomic-research/
- #19 Quickly access a managed Spark cluster - AWS EC2 / spot instances Link to your data and perform whole genome analysis in real-time
- #21 https://medium.com/@lynnlangit/scaling-custom-machine-learning-on-aws-part-2-emr-6dfc3cd91a1f
- #24 https://databricks.com/blog/2017/07/26/breaking-the-curse-of-dimensionality-in-genomics-using-wide-random-forests.html https://github.com/jamesrcounts/VariantSpark/tree/spark2.3/kubernetes
- #25 https://databricks.com/blog/2017/07/26/breaking-the-curse-of-dimensionality-in-genomics-using-wide-random-forests.html https://github.com/jamesrcounts/VariantSpark/tree/spark2.3/kubernetes