Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Scaling Galaxy on GCP
LynnLangit
Cloud and Data Architect
Google Developer Cloud Expert, AWS Community Hero, Microsoft D...
2
Agenda
• Scaling Up
• Virtual Machines
• Hello Galaxy
• Adding Tools to Galaxy
• Genomic Data on GCP
• Scaling Out
• Doc...
3
Google Cloud in Australia
Data center here in 2017
4
Galaxy on GCP – Scale Up
5Google Cloud Platform 5
Demo 1- Hello Galaxy on Google Cloud
6Google Cloud Platform 6
Demo 2 - Adding Galaxy Tools
77
• Cloud Storage (file) buckets
• Source data
• Compute Engine Virtual Machines
• Virtual Machine Image files
• External...
88
• Re-size Virtual Machines
• Attach more persistent disks
• Update base image
• Monitor with Stackdriver
Scale Up Patte...
99
1010
Genomic Data
• Files at GCS
• gs://genomics-public-data
• Query via BigQuery
• https://bigquery.cloud.google.com/quer...
1111
12
Galaxy on GCP – Scale Out
1313
GCP Docker Container Services
• Cloud Storage
• Container Engine / Docker
Key Concepts:
-- Container configuration as...
1414
1515
• Docker Container Cluster
• Kubernetes manager
• Container orchestration
Scale Out Patterns
1616
GCP Serverless Services
• Cloud Functions
• Microservices
Key Concepts:
-- Function configuration as code
-- Fast, ch...
17
Galaxy on GCP – Advanced Pipelines
18Google Cloud Platform 18
Demo 3 – Using the Google Genomics API &
BigQuery
1919
BigQuery
• ANSI SQL Queries
• Query-as-a-service
Key Concepts:
-- SQL query configuration as code
-- Fast, cheap scal...
20
Private Datasets Public Datasets
Variant Analysis
MSSNG Autism
Cloud Storage
Scientist
High
Throughput
Genome
Sequencer...
21
Ingest
Elastic Cluster
Storage
Analytics
Carrier
Interconnect
High
Throughput
Genome
Sequencer
s
Scientist
Raw Datafile...
2222
• Cloud Storage / Public datasets on
GCP
• Big Query
• Cloud Dataflow
• Genomics API
Advanced GCP Pipelines Core
Prod...
23
Resources
2424
• Cloud Storage (files) -- here
• Compute Engine (VMs) -- here
• Container Engine (Docker) -- here
• Big Query (SQL) ...
25
More about Google Cloud Services
26
Compute
Compute
Engine
App
Engine
Container
Engine
Container
Registry
Cloud
Functions
Networking
Cloud Virtual
Network
...
27
Management Tools
Stackdriver Monitoring Logging
Error
Reporting
Trace Debugger
Deployment
Manager
Cloud
Endpoints
Cloud...
28
29
30
GCE Persistence Options – Disks,
etc.… Created From Notes
Image GCS File or Disk File path <bucket>/<folder>/<file>
Dis...
31
Upcoming SlideShare
Loading in …5
×

Scaling Galaxy on Google Cloud Platform

3,920 views

Published on

Deck from Galaxy 2017 conference in Australia on scaling Galaxy bioinformatics tools via GCP or Google Cloud Platform

Published in: Health & Medicine
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Is there also a link to the youtube video? The upper left corner of your slide said YouTube video inside, but I'm not quite sure which slide does provide the link or has the video embeded.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scaling Galaxy on Google Cloud Platform

  1. 1. 1 Scaling Galaxy on GCP LynnLangit Cloud and Data Architect Google Developer Cloud Expert, AWS Community Hero, Microsoft Data Platform MVP
  2. 2. 2 Agenda • Scaling Up • Virtual Machines • Hello Galaxy • Adding Tools to Galaxy • Genomic Data on GCP • Scaling Out • Docker Containers • Google Persistent Disks • Pipelines • Google Genomics APIs • Big Query Galaxy on Google Cloud Platform
  3. 3. 3 Google Cloud in Australia Data center here in 2017
  4. 4. 4 Galaxy on GCP – Scale Up
  5. 5. 5Google Cloud Platform 5 Demo 1- Hello Galaxy on Google Cloud
  6. 6. 6Google Cloud Platform 6 Demo 2 - Adding Galaxy Tools
  7. 7. 77 • Cloud Storage (file) buckets • Source data • Compute Engine Virtual Machines • Virtual Machine Image files • External VM persistent hard disks with your source data GCP Virtual Machine Services Key Concepts: -- VM configuration as code -- Fast, cheap scalable VMs
  8. 8. 88 • Re-size Virtual Machines • Attach more persistent disks • Update base image • Monitor with Stackdriver Scale Up Patterns
  9. 9. 99
  10. 10. 1010 Genomic Data • Files at GCS • gs://genomics-public-data • Query via BigQuery • https://bigquery.cloud.google.com/queries/genomics-public- data • Code via Genomics API • Implements Global Alliance for Genomics and Health APIs • Genome browser - https://gabrowse.appspot.com • Google Genomics example code on GitHub
  11. 11. 1111
  12. 12. 12 Galaxy on GCP – Scale Out
  13. 13. 1313 GCP Docker Container Services • Cloud Storage • Container Engine / Docker Key Concepts: -- Container configuration as code -- Fast, cheap scalable Docker Containers
  14. 14. 1414
  15. 15. 1515 • Docker Container Cluster • Kubernetes manager • Container orchestration Scale Out Patterns
  16. 16. 1616 GCP Serverless Services • Cloud Functions • Microservices Key Concepts: -- Function configuration as code -- Fast, cheap scalable Microservices
  17. 17. 17 Galaxy on GCP – Advanced Pipelines
  18. 18. 18Google Cloud Platform 18 Demo 3 – Using the Google Genomics API & BigQuery
  19. 19. 1919 BigQuery • ANSI SQL Queries • Query-as-a-service Key Concepts: -- SQL query configuration as code -- Fast, cheap scalable SQL Queries
  20. 20. 20 Private Datasets Public Datasets Variant Analysis MSSNG Autism Cloud Storage Scientist High Throughput Genome Sequencers 1000 Genomes Cloud Storage Patient Data Cloud Storage Illumina Platform Cloud Storage Ref Genomes Cloud Storage TCGA Cloud Storage Analytics Online Analytics BigQuery Batch Analytics Cloud Dataflow Lab Notebooks Cloud Datalab Data Ingest Genomics BAM FASTQ
  21. 21. 21 Ingest Elastic Cluster Storage Analytics Carrier Interconnect High Throughput Genome Sequencer s Scientist Raw Datafiles Cloud Storage Processed Data Cloud Storage Metadata Cloud SQL Lab notebooks Cloud Datalab HPC Cluster Compute Engine 10 Nodes Ingest Server Compute Engine Online Analytics BigQuery Cloud Load Balancing Cloud Network Genomics, Secondary Analysis
  22. 22. 2222 • Cloud Storage / Public datasets on GCP • Big Query • Cloud Dataflow • Genomics API Advanced GCP Pipelines Core Products Key Concepts: -- Pipeline configuration as code -- Fast, cheap scalable cloud services
  23. 23. 23 Resources
  24. 24. 2424 • Cloud Storage (files) -- here • Compute Engine (VMs) -- here • Container Engine (Docker) -- here • Big Query (SQL) -- here • Cloud Dataflow (pipelines) -- here • Genomics API-- here • Genomics Cookbook– here • Public datasets on GCP-- here • Google’s Genomic code samples – here • Lynn’s GitHub code samples -- here Resources
  25. 25. 25 More about Google Cloud Services
  26. 26. 26 Compute Compute Engine App Engine Container Engine Container Registry Cloud Functions Networking Cloud Virtual Network Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Cloud Platform Security Storage and Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Persistent Disk Machine Learning Cloud Machine Learning Vision API Speech API Natural Language API Translation API Google Cloud Platform Services Part One Jobs API
  27. 27. 27 Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Deployment Manager Cloud Endpoints Cloud Console Developer Tools Cloud SDK Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Google Plug-in for Eclipse Cloud Test Lab Google Cloud Platform Services Part Two Cloud Shell Cloud Mobile App Billing App Cloud APIs
  28. 28. 28
  29. 29. 29
  30. 30. 30 GCE Persistence Options – Disks, etc.… Created From Notes Image GCS File or Disk File path <bucket>/<folder>/<file> Disk must detached from VM Snapshot Disk or Instance (boot) Can create an Instance FROM a Snapshot Persistent Disk Image –or- Snapshot –or- Blank Blank disk must be formatted Can create an Instance or Snapshot FROM a Disk Bucket GCS console for file Access via path gs://<bucketName>/<fileName> VM Instance Boot Disk Image –or- Snapshot –or- Disk Images -> OS, Application or Custom Image N/A From Saved Disk VM Instance Additional Disk Local Scratch –or- Standard persistent –or- SSD persistent Max 8 at 375 GB each. 500 GB  64 TB Read/Write or Read Only Attach up to 16 Disks* per VM
  31. 31. 31

×