SlideShare a Scribd company logo
Building and provisioning
genomics platforms on the
world’s clouds
Enis Afgan
Johns Hopkins University
Galaxy Project
April 2016, University of Heidelberg
World’s clouds
AWS
AWS (coming soon)
Google Compute Engine
Chameleon
Jetstream
NeCTAR
Azure
Capacity without end-to-end
solution
How to appropriately utilize
clouds?
VM Platform Service
Standalone VM
Pre-configured server that is readily available.
Pros
Easy to build; easy to deploy
Low cloud infrastructure requirements ⟶ Transferable
Cons
Limited capacity (compute and storage)
See it in action
wiki.galaxyproject.org/Cloud/Jetstream
Scalable platform
Set up a virtual cluster across multiple VMs with app services.
Pros
Dynamically scale compute and storage
Higher-level services: persistent storage, sharing, multi-
application
Cons
Complicated build; considerable infrastructure requirements
See it in action
wiki.galaxyproject.org/CloudMan
Scalable platform (cont)
Data analysis spans more than one application (even if that is
Galaxy).
Meet Genomics Virtual Lab (GVL)
Pros
Versatile platform built on
the scalable CloudMan cluster
Includes common tutorials
Cons
Demanding to build
Calls for more customization
See it in action
genome.edu.au
Ready-to-use service
Use cloud resources from an always-on, public service
Pros
Visit a URL and start computing – no setup required
Cons
User quotas still apply
It’s still a public service: no user customization
See it in action
usegalaxy.org (bwa, bowtie2 – more coming)
There’s a lot of clouds out there!
AWS
AWS (coming soon)
Google Compute Engine
Chameleon
Jetstream
NeCTAR
Azure
How to appropriately utilize many
clouds?
VM Platform Service
Build system
Adjustable build system
Automate the process of building each component
Codify knowledge about the system ⟶ easier to reproduce
We use Ansible as the technology of choice
Compose systems from configurable and reusable roles
Galaxy-Kickstarter
Playbook
artbio.github.io/ansible-artimed/
Galaxy-CloudMan
Playbook
github.com/galaxyproject/
galaxy-cloudman-playbook
Use-Galaxy
Playbook
github.com/galaxyproject/
usegalaxy-playbook
Many clouds AND many solutions
!?!
launch.genome.edu.au ; use.jetstream-cloud.org ; launch.usegalaxy.org
CloudBridge (future)
A Simple Cross-Cloud Python Library
1. Offer a uniform API irrespective of the underlying provider
2. Provide a set of conformance tests for all supported clouds
3. Focus on mature clouds with a required minimal set of features
4. Be as thin as possible
Support for AWS and OpenStack exists; Google Cloud under
development
cloudbridge.readthedocs.org
CloudLaunch (future)
A centralized launcher for any app and any cloud.
User configurable applications and clouds; view and launch
shared instances; multi-cloud dashboard view
github.com/galaxyproject/cloudlaunch
github.com/galaxyproject/cloudlaunch-u
CloudMan (future)
Resource manager with configurable service layer
• Pull away from low-level application service management
• Leverage containers to supply services
• Allow runtime service and configuration changes
• Run on any infrastructure, including high-level services, such as
ECS, or Docker API
Goal: Launch a (template-based) CloudMan platform and add
application services as desired from Dockerhub or similar while
resource provisioning is automatically handled.
Galaxy ObjectStore (future)
Allow uniform any-Galaxy computing (i.e., make Galaxy instances
interchangeable and disposable)
• Galaxy implements an ObjectStore interface as an abstraction to
data
• Leverage it to expand user data storage and allow any Galaxy
to connect to a user’s bucket
• Use ObjectStore for reference data (simplify builds)
• Still will need to deal with the database dependency
The endgame?
launch.usegalaxy.org
ObjectStore
CloudBridge
CloudMan
A P P L I C A T I O N S
Building your own cloud?
Make it easy
For end-users to register and get onboard (very simple auth)
For deployers to interface with the cloud (adopt ‘standards’)
Develop capacity and usage plans
Go for monthly-reset, merit-based Allocation Units (AUs)
Design for flexibility
Users need more storage? Different instance types?
Create champion teams
Bring them onboard early to deploy target apps; give them $$$
Start with good documentation
Technical but not overly detailed (look at AWS)
Be open; add great, interactive support
Design a training program
For application developers and end users; build a community
Acknowledgments
Want more Galaxy?
gcc2016.iu.edu
usegalaxy.org cloud-bursting
usegalaxy.org
CVMFS
NFS
job_conf.xml

More Related Content

What's hot

Serverless and Design Patterns In GCP
Serverless and Design Patterns In GCPServerless and Design Patterns In GCP
Serverless and Design Patterns In GCP
Oliver Fierro
 
Using ansible to manage cloud platform by Accelerite
Using ansible to manage cloud platform by AcceleriteUsing ansible to manage cloud platform by Accelerite
Using ansible to manage cloud platform by Accelerite
Madan Ganesh Velayudham
 
Apache CloudStack 4.2: A First Look
Apache CloudStack 4.2: A First LookApache CloudStack 4.2: A First Look
Apache CloudStack 4.2: A First LookShanker Balan
 
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
DevOps.com
 
Deploy in scale with docker, coreos, kubernetes and apache stratos
Deploy in scale with docker, coreos, kubernetes and apache stratosDeploy in scale with docker, coreos, kubernetes and apache stratos
Deploy in scale with docker, coreos, kubernetes and apache stratos
WSO2
 
Cloud infrastructure on Apache Mesos
Cloud infrastructure on Apache MesosCloud infrastructure on Apache Mesos
Cloud infrastructure on Apache Mesos
Ahmed Bacha
 
Architecting Enterprise Applications In The Cloud
Architecting Enterprise Applications In The CloudArchitecting Enterprise Applications In The Cloud
Architecting Enterprise Applications In The Cloud
Amazon Web Services
 
GCP CloudRun Overview
GCP CloudRun OverviewGCP CloudRun Overview
GCP CloudRun Overview
Oliver Fierro
 
How to migrate workloads to the google cloud platform
How to migrate workloads to the google cloud platformHow to migrate workloads to the google cloud platform
How to migrate workloads to the google cloud platform
actualtechmedia
 
Getting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMRGetting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMR
Arun Sirimalla
 
Google cloud platform
Google cloud platformGoogle cloud platform
Google cloud platform
Piyumi Niwanthika Herath
 
A Tour of Google Cloud Platform
A Tour of Google Cloud PlatformA Tour of Google Cloud Platform
A Tour of Google Cloud Platform
Colin Su
 
SaaSification in Action. Attracting Software Vendors with Easy Transformation
SaaSification in Action. Attracting Software Vendors with Easy TransformationSaaSification in Action. Attracting Software Vendors with Easy Transformation
SaaSification in Action. Attracting Software Vendors with Easy Transformation
Jelastic Multi-Cloud PaaS
 
How to Migrate a Web App to AWS
How to Migrate a Web App to AWSHow to Migrate a Web App to AWS
How to Migrate a Web App to AWS
Global Knowledge Training
 
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
Big Data Day LA 2015 -  Lessons learned from scaling Big Data in the Cloud by...Big Data Day LA 2015 -  Lessons learned from scaling Big Data in the Cloud by...
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
Data Con LA
 
Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!
Krishna-Kumar
 
Cloud computing's truly open silver lining: OpenStack
Cloud computing's truly open silver lining: OpenStackCloud computing's truly open silver lining: OpenStack
Cloud computing's truly open silver lining: OpenStackAsociatia ProLinux
 
A practical approach to provisioning resources in azure
A practical approach to provisioning resources in azureA practical approach to provisioning resources in azure
A practical approach to provisioning resources in azure
Morten Christensen
 
Crash Course in Cloud Computing
Crash Course in Cloud ComputingCrash Course in Cloud Computing
Crash Course in Cloud Computing
All Things Open
 
A quick introduction to AKS
A quick introduction to AKSA quick introduction to AKS
A quick introduction to AKS
Alessandro Melchiori
 

What's hot (20)

Serverless and Design Patterns In GCP
Serverless and Design Patterns In GCPServerless and Design Patterns In GCP
Serverless and Design Patterns In GCP
 
Using ansible to manage cloud platform by Accelerite
Using ansible to manage cloud platform by AcceleriteUsing ansible to manage cloud platform by Accelerite
Using ansible to manage cloud platform by Accelerite
 
Apache CloudStack 4.2: A First Look
Apache CloudStack 4.2: A First LookApache CloudStack 4.2: A First Look
Apache CloudStack 4.2: A First Look
 
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
Business Continuity with Microservices-Based Apps and DevOps: Learnings from ...
 
Deploy in scale with docker, coreos, kubernetes and apache stratos
Deploy in scale with docker, coreos, kubernetes and apache stratosDeploy in scale with docker, coreos, kubernetes and apache stratos
Deploy in scale with docker, coreos, kubernetes and apache stratos
 
Cloud infrastructure on Apache Mesos
Cloud infrastructure on Apache MesosCloud infrastructure on Apache Mesos
Cloud infrastructure on Apache Mesos
 
Architecting Enterprise Applications In The Cloud
Architecting Enterprise Applications In The CloudArchitecting Enterprise Applications In The Cloud
Architecting Enterprise Applications In The Cloud
 
GCP CloudRun Overview
GCP CloudRun OverviewGCP CloudRun Overview
GCP CloudRun Overview
 
How to migrate workloads to the google cloud platform
How to migrate workloads to the google cloud platformHow to migrate workloads to the google cloud platform
How to migrate workloads to the google cloud platform
 
Getting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMRGetting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMR
 
Google cloud platform
Google cloud platformGoogle cloud platform
Google cloud platform
 
A Tour of Google Cloud Platform
A Tour of Google Cloud PlatformA Tour of Google Cloud Platform
A Tour of Google Cloud Platform
 
SaaSification in Action. Attracting Software Vendors with Easy Transformation
SaaSification in Action. Attracting Software Vendors with Easy TransformationSaaSification in Action. Attracting Software Vendors with Easy Transformation
SaaSification in Action. Attracting Software Vendors with Easy Transformation
 
How to Migrate a Web App to AWS
How to Migrate a Web App to AWSHow to Migrate a Web App to AWS
How to Migrate a Web App to AWS
 
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
Big Data Day LA 2015 -  Lessons learned from scaling Big Data in the Cloud by...Big Data Day LA 2015 -  Lessons learned from scaling Big Data in the Cloud by...
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
 
Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!Kubernetes Application Deployment with Helm - A beginner Guide!
Kubernetes Application Deployment with Helm - A beginner Guide!
 
Cloud computing's truly open silver lining: OpenStack
Cloud computing's truly open silver lining: OpenStackCloud computing's truly open silver lining: OpenStack
Cloud computing's truly open silver lining: OpenStack
 
A practical approach to provisioning resources in azure
A practical approach to provisioning resources in azureA practical approach to provisioning resources in azure
A practical approach to provisioning resources in azure
 
Crash Course in Cloud Computing
Crash Course in Cloud ComputingCrash Course in Cloud Computing
Crash Course in Cloud Computing
 
A quick introduction to AKS
A quick introduction to AKSA quick introduction to AKS
A quick introduction to AKS
 

Viewers also liked

Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khaleel shaik
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
Bryan Donaldson
 
Làm thế nào để sống chung với bệnh khớp
Làm thế nào để sống chung với bệnh khớpLàm thế nào để sống chung với bệnh khớp
Làm thế nào để sống chung với bệnh khớpwilliam728
 
Características do verdadeiro adorador
Características do verdadeiro adoradorCaracterísticas do verdadeiro adorador
Características do verdadeiro adorador
Renato Cordeiro
 
Community Helpers Themed Unit
Community Helpers Themed UnitCommunity Helpers Themed Unit
Community Helpers Themed UnitNicole Pilarz
 
Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khaleel shaik
 
edu 311 fact and opinion Lesson Plan
edu 311 fact and opinion Lesson Planedu 311 fact and opinion Lesson Plan
edu 311 fact and opinion Lesson PlanNicole Pilarz
 

Viewers also liked (9)

KhalilCV UPDATED
KhalilCV UPDATEDKhalilCV UPDATED
KhalilCV UPDATED
 
Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Làm thế nào để sống chung với bệnh khớp
Làm thế nào để sống chung với bệnh khớpLàm thế nào để sống chung với bệnh khớp
Làm thế nào để sống chung với bệnh khớp
 
Addition Detective
Addition DetectiveAddition Detective
Addition Detective
 
Características do verdadeiro adorador
Características do verdadeiro adoradorCaracterísticas do verdadeiro adorador
Características do verdadeiro adorador
 
Community Helpers Themed Unit
Community Helpers Themed UnitCommunity Helpers Themed Unit
Community Helpers Themed Unit
 
Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)Khalil MB@CV (1) (1) (1)
Khalil MB@CV (1) (1) (1)
 
edu 311 fact and opinion Lesson Plan
edu 311 fact and opinion Lesson Planedu 311 fact and opinion Lesson Plan
edu 311 fact and opinion Lesson Plan
 

Similar to Building and provisioning genomics platforms on the world’s clouds

Cloudjiffy vs Amazon ECS
Cloudjiffy vs Amazon ECSCloudjiffy vs Amazon ECS
Cloudjiffy vs Amazon ECS
Sharma Aashish
 
Cloudjiffy vs Amazon Elastic Beanstalk
Cloudjiffy vs Amazon Elastic BeanstalkCloudjiffy vs Amazon Elastic Beanstalk
Cloudjiffy vs Amazon Elastic Beanstalk
Sharma Aashish
 
CloudStack vs Openstack
CloudStack vs OpenstackCloudStack vs Openstack
CloudStack vs Openstack
Madan Ganesh Velayudham
 
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
NETWAYS
 
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar Neelamegam
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar NeelamegamOpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar Neelamegam
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar NeelamegamOpenNebula Project
 
A Journey To The Cloud - An Introduction To Cloud Computing
A Journey To The Cloud - An Introduction To Cloud Computing A Journey To The Cloud - An Introduction To Cloud Computing
A Journey To The Cloud - An Introduction To Cloud Computing
Haim Ateya
 
Cloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
Cloud Expo East 2013: Essential Open Source Software for Building the Open CloudCloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
Cloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
Mark Hinkle
 
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the CloudLinux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
Mark Hinkle
 
An Introduction To Infarstructures For Cloud Computing V0.2
An Introduction To Infarstructures For Cloud Computing V0.2An Introduction To Infarstructures For Cloud Computing V0.2
An Introduction To Infarstructures For Cloud Computing V0.2Ignacio M. Llorente
 
Cloud1 Computing 01
Cloud1 Computing 01Cloud1 Computing 01
Cloud1 Computing 01
Heartin Jacob
 
Velocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ NetflixVelocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ Netflix
aspyker
 
AWS Webcast - Best Practices in Architecting for the Cloud
AWS Webcast - Best Practices in Architecting for the CloudAWS Webcast - Best Practices in Architecting for the Cloud
AWS Webcast - Best Practices in Architecting for the Cloud
Amazon Web Services
 
Managing Your Private Cloud with RightScale
Managing Your Private Cloud with RightScaleManaging Your Private Cloud with RightScale
Managing Your Private Cloud with RightScale
RightScale
 
What is cloud computing
What is cloud computingWhat is cloud computing
What is cloud computing
Brian Bullard
 
Cloud Deployment Toolkit
Cloud Deployment ToolkitCloud Deployment Toolkit
Cloud Deployment Toolkit
Bret Piatt
 
Searching The Cloud - The eclipseRT Umbrella
Searching The Cloud - The eclipseRT UmbrellaSearching The Cloud - The eclipseRT Umbrella
Searching The Cloud - The eclipseRT Umbrella
Markus Knauer
 
Drilett aws vpc_presentation_shared
Drilett aws vpc_presentation_sharedDrilett aws vpc_presentation_shared
Drilett aws vpc_presentation_sharedDavid Rilett
 
Project COLA: Use Case to create a scalable application in the cloud based on...
Project COLA: Use Case to create a scalable application in the cloud based on...Project COLA: Use Case to create a scalable application in the cloud based on...
Project COLA: Use Case to create a scalable application in the cloud based on...
Project COLA
 
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud ComputingLinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
Mark Hinkle
 
Eucalyptus: Open Source for Cloud Computing
Eucalyptus: Open Source for Cloud ComputingEucalyptus: Open Source for Cloud Computing
Eucalyptus: Open Source for Cloud Computing
clive boulton
 

Similar to Building and provisioning genomics platforms on the world’s clouds (20)

Cloudjiffy vs Amazon ECS
Cloudjiffy vs Amazon ECSCloudjiffy vs Amazon ECS
Cloudjiffy vs Amazon ECS
 
Cloudjiffy vs Amazon Elastic Beanstalk
Cloudjiffy vs Amazon Elastic BeanstalkCloudjiffy vs Amazon Elastic Beanstalk
Cloudjiffy vs Amazon Elastic Beanstalk
 
CloudStack vs Openstack
CloudStack vs OpenstackCloudStack vs Openstack
CloudStack vs Openstack
 
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
OpenNebula Conf 2014 | Cloud Automation for OpenNebula by Kishorekumar Neelam...
 
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar Neelamegam
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar NeelamegamOpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar Neelamegam
OpenNebulaConf 2014 - Cloud Automation for OpenNebula - Kishorekumar Neelamegam
 
A Journey To The Cloud - An Introduction To Cloud Computing
A Journey To The Cloud - An Introduction To Cloud Computing A Journey To The Cloud - An Introduction To Cloud Computing
A Journey To The Cloud - An Introduction To Cloud Computing
 
Cloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
Cloud Expo East 2013: Essential Open Source Software for Building the Open CloudCloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
Cloud Expo East 2013: Essential Open Source Software for Building the Open Cloud
 
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the CloudLinux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud
 
An Introduction To Infarstructures For Cloud Computing V0.2
An Introduction To Infarstructures For Cloud Computing V0.2An Introduction To Infarstructures For Cloud Computing V0.2
An Introduction To Infarstructures For Cloud Computing V0.2
 
Cloud1 Computing 01
Cloud1 Computing 01Cloud1 Computing 01
Cloud1 Computing 01
 
Velocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ NetflixVelocity NYC 2016 - Containers @ Netflix
Velocity NYC 2016 - Containers @ Netflix
 
AWS Webcast - Best Practices in Architecting for the Cloud
AWS Webcast - Best Practices in Architecting for the CloudAWS Webcast - Best Practices in Architecting for the Cloud
AWS Webcast - Best Practices in Architecting for the Cloud
 
Managing Your Private Cloud with RightScale
Managing Your Private Cloud with RightScaleManaging Your Private Cloud with RightScale
Managing Your Private Cloud with RightScale
 
What is cloud computing
What is cloud computingWhat is cloud computing
What is cloud computing
 
Cloud Deployment Toolkit
Cloud Deployment ToolkitCloud Deployment Toolkit
Cloud Deployment Toolkit
 
Searching The Cloud - The eclipseRT Umbrella
Searching The Cloud - The eclipseRT UmbrellaSearching The Cloud - The eclipseRT Umbrella
Searching The Cloud - The eclipseRT Umbrella
 
Drilett aws vpc_presentation_shared
Drilett aws vpc_presentation_sharedDrilett aws vpc_presentation_shared
Drilett aws vpc_presentation_shared
 
Project COLA: Use Case to create a scalable application in the cloud based on...
Project COLA: Use Case to create a scalable application in the cloud based on...Project COLA: Use Case to create a scalable application in the cloud based on...
Project COLA: Use Case to create a scalable application in the cloud based on...
 
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud ComputingLinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing
 
Eucalyptus: Open Source for Cloud Computing
Eucalyptus: Open Source for Cloud ComputingEucalyptus: Open Source for Cloud Computing
Eucalyptus: Open Source for Cloud Computing
 

Recently uploaded

Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 

Recently uploaded (20)

Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 

Building and provisioning genomics platforms on the world’s clouds

  • 1. Building and provisioning genomics platforms on the world’s clouds Enis Afgan Johns Hopkins University Galaxy Project April 2016, University of Heidelberg
  • 2. World’s clouds AWS AWS (coming soon) Google Compute Engine Chameleon Jetstream NeCTAR Azure
  • 4. How to appropriately utilize clouds? VM Platform Service
  • 5. Standalone VM Pre-configured server that is readily available. Pros Easy to build; easy to deploy Low cloud infrastructure requirements ⟶ Transferable Cons Limited capacity (compute and storage) See it in action wiki.galaxyproject.org/Cloud/Jetstream
  • 6. Scalable platform Set up a virtual cluster across multiple VMs with app services. Pros Dynamically scale compute and storage Higher-level services: persistent storage, sharing, multi- application Cons Complicated build; considerable infrastructure requirements See it in action wiki.galaxyproject.org/CloudMan
  • 7. Scalable platform (cont) Data analysis spans more than one application (even if that is Galaxy). Meet Genomics Virtual Lab (GVL) Pros Versatile platform built on the scalable CloudMan cluster Includes common tutorials Cons Demanding to build Calls for more customization See it in action genome.edu.au
  • 8. Ready-to-use service Use cloud resources from an always-on, public service Pros Visit a URL and start computing – no setup required Cons User quotas still apply It’s still a public service: no user customization See it in action usegalaxy.org (bwa, bowtie2 – more coming)
  • 9. There’s a lot of clouds out there! AWS AWS (coming soon) Google Compute Engine Chameleon Jetstream NeCTAR Azure
  • 10. How to appropriately utilize many clouds? VM Platform Service Build system
  • 11. Adjustable build system Automate the process of building each component Codify knowledge about the system ⟶ easier to reproduce We use Ansible as the technology of choice Compose systems from configurable and reusable roles Galaxy-Kickstarter Playbook artbio.github.io/ansible-artimed/ Galaxy-CloudMan Playbook github.com/galaxyproject/ galaxy-cloudman-playbook Use-Galaxy Playbook github.com/galaxyproject/ usegalaxy-playbook
  • 12. Many clouds AND many solutions !?! launch.genome.edu.au ; use.jetstream-cloud.org ; launch.usegalaxy.org
  • 13. CloudBridge (future) A Simple Cross-Cloud Python Library 1. Offer a uniform API irrespective of the underlying provider 2. Provide a set of conformance tests for all supported clouds 3. Focus on mature clouds with a required minimal set of features 4. Be as thin as possible Support for AWS and OpenStack exists; Google Cloud under development cloudbridge.readthedocs.org
  • 14. CloudLaunch (future) A centralized launcher for any app and any cloud. User configurable applications and clouds; view and launch shared instances; multi-cloud dashboard view github.com/galaxyproject/cloudlaunch github.com/galaxyproject/cloudlaunch-u
  • 15. CloudMan (future) Resource manager with configurable service layer • Pull away from low-level application service management • Leverage containers to supply services • Allow runtime service and configuration changes • Run on any infrastructure, including high-level services, such as ECS, or Docker API Goal: Launch a (template-based) CloudMan platform and add application services as desired from Dockerhub or similar while resource provisioning is automatically handled.
  • 16. Galaxy ObjectStore (future) Allow uniform any-Galaxy computing (i.e., make Galaxy instances interchangeable and disposable) • Galaxy implements an ObjectStore interface as an abstraction to data • Leverage it to expand user data storage and allow any Galaxy to connect to a user’s bucket • Use ObjectStore for reference data (simplify builds) • Still will need to deal with the database dependency
  • 18. Building your own cloud? Make it easy For end-users to register and get onboard (very simple auth) For deployers to interface with the cloud (adopt ‘standards’) Develop capacity and usage plans Go for monthly-reset, merit-based Allocation Units (AUs) Design for flexibility Users need more storage? Different instance types? Create champion teams Bring them onboard early to deploy target apps; give them $$$ Start with good documentation Technical but not overly detailed (look at AWS) Be open; add great, interactive support Design a training program For application developers and end users; build a community

Editor's Notes

  1. Standards: expose API, reliable/adopted middleware,