Download as PDF, PPTX
![3. Job mapper function
Determine job destination at runtime
import pyslurm
def cloud_burst():
n = pyslurm.node()
nodes_state = n.get()
available_nodes = []
for node in nodes_state.itervalues():
if node['total_cpus'] > 0:
available_nodes.append(node)
if not available_nodes:
return 'pulsar_nectar_galaxy'
return 'drmaa_runner’
job destination](https://image.slidesharecdn.com/burstingpresentation-150213085810-conversion-gate02/85/Enabling-Cloud-Bursting-for-Life-Sciences-within-Galaxy-12-320.jpg)
Uploaded byEnis Afgan
Enabling Cloud Bursting for Life Sciences within Galaxy
The document outlines the implementation of cloud bursting for life sciences using the Galaxy data analysis framework, which integrates various tools and resources to facilitate data analysis. It discusses the mechanics of cloud bursting, user configurations, job destination management, and the role of pulsar in handling job execution and data staging. Additionally, it provides insights into job performance metrics and resource utilization within the Galaxy environment.
Enabling Cloud Bursting for Life Sciences within Galaxy
- 1. Enabling Cloud Burstingfor Life Sciences within Galaxy Enis Afgan Johns Hopkins University Galaxy Team Slides available at bit.ly/gxy-bursting
- 2. What is • Adata analysis and integration tool • A (free for everyone) web service integrating a wealth of tools, compute resources, terabytes of reference data and permanent storage • Open source software that makes integrating your own tools and data and customizing for your own site simple ?
- 3. usegalaxy.org or any of theother 60+ public servers $ hg clone bitbucket.org/ galaxy/galaxy-dist $ sh run.sh
- 5. Galaxy /Tools /Data /Indices DB Compute resources Galaxy Galaxy Galaxy RNA-Seq Assembly Quality Control (QC) Local Federated Galaxy Object Store interface DB IndicesA Data A Tools A S3, Swift Pulsar Indices B Data B Tools B Local Pulsar Indices C Data C Tools C Artifact & job provenance RNA-Seq, Assembly, QC GalaxyGalaxy CloudMan
- 6. Focus on CloudBursting Peak usage scenarios Resource heterogeneity Software licensing Software installation restrictions National cyber infrastructure resource access Per-user, merit-based resource access
- 7. Burst Triggers When? Resource capacity Jobrequirements Data locality System configuration User preferences Where? Remote resource availability Cost
- 8. Burst Architecture 1. Galaxydynamic job destination framework 2. Galaxy CloudMan cluster with Pulsar 3. A job destination mapper function CloudMan Pulsar CloudMan Pulsar Local DRM Galaxy <dynamic)job) destination) framework)/> f(mapper)
- 9. Pulsar A standalone jobmanager server for Galaxy Can be deployed on dedicated or transient servers (even MS Windows!) Handles data staging and remote job execution Pulsarjob Stage data Submit job Monitor job Send back the data
- 10. 1. Galaxy dynamicjob destination framework Define job execution properties • Runners: local, Slurm, HTCondor, DRMAA, Pulsar, … • Destinations: resource & job properties (e.g., DRM queue, wall time)
- 11. 2. CloudMan withPulsar A. Launch a Galaxy on the Cloud instance B. Enable Pulsar service C. Add the instance as a destination in job config Tool availability • Direct tool install • Docker images
- 12. 3. Job mapperfunction Determine job destination at runtime import pyslurm def cloud_burst(): n = pyslurm.node() nodes_state = n.get() available_nodes = [] for node in nodes_state.itervalues(): if node['total_cpus'] > 0: available_nodes.append(node) if not available_nodes: return 'pulsar_nectar_galaxy' return 'drmaa_runner’ job destination
- 13.
- 14. An outcome? 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 0 100 200 300 400 500 600 700 800 900 1000 2013-41 2013-43 2013-45 2013-47 2013-49 2013-51 2013-53 2014-02 2014-04 2014-06 2014-08 2014-10 2014-12 2014-14 2014-16 2014-18 2014-20 2014-22 2014-24 2014-26 2014-28 2014-30 2014-32 2014-34 2014-36 2014-38 2014-40 2014-42 2014-44 2014-46 2014-48 2014-50 2014-52 2015-01 2015-03 Jobsruntocompletion(count) Averagewaittime(minutes) Week Averagewait Jobs run to completion usegalaxy.org Start bursting No job wait More jobs
- 15.