GCC 2014 scriptable workshop

300
-1

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
300
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

GCC 2014 scriptable workshop

  1. 1. Outline 1.  Working with CloudMan 2.  Deploying bioinformatics tools with CloudBioLinux 3.  Deploying a genome assembly and annotation pipeline with Fabric
  2. 2. Working with CloudMan Enis Afgan @ GCC 2014, Baltimore
  3. 3. Researchers ` 100sGB 100+
  4. 4. Why the cloud? Infrastructure Customization
  5. 5. How to use the cloud? 1. Get an account on the supported cloud 2. Start a master instance via the cloud web console or CloudLaunch 3. Use CloudMan s web interface on the master instance to manage the platform 4. Use or customize Galaxy
  6. 6. YOUR TURN
  7. 7. Launch an instance 1.  Visit biocloudcentral.org 2.  Enter the access key and secret key provided by Dave Clements on 6/25 3.  Provide your email address 4.  Use your initials as the cluster name 5.  Set any password (and remember it) 6.  Keep Large instance type 7.  Start your instance Wait for the instance to start (~2-3 minutes) For more details, see wiki.galaxyproject.org/CloudMan
  8. 8. Launch an instance 1.  Visit usegalaxy.org/cloudlaunch 2.  Enter your access key and secret key (credits provided by Dave Clements on 6/25) 3.  Choose New cluster 4.  Set any name as the cluster name 5.  Set any password 6.  Keep cloudman_keypair 7.  Keep Large instance type 8.  Launch your instance Wait for the instance to start (~2-3 minutes) For more details, see wiki.galaxyproject.org/CloudMan
  9. 9. Manual scaling •  Explicitly add 1 worker node to your cluster •  Node type corresponds to node processing capacity •  Research use of Spot instances
  10. 10. Auto-scaling
  11. 11. Using an S3 bucket as a data source
  12. 12. Accessing an instance over ssh Use the terminal (or install Secure Shell for Chrome extension) SSH using user ubuntu and the password you chose when launching an instance: [local machine]$ ssh ubuntu@<instance IP address>
  13. 13. Once logged in •  You have full system access to your instance, including sudo; use it as any other system •  CloudMan-managed file systems are located under /mnt •  Can submit jobs to SGE job manager via the standard qsub command
  14. 14. Customize your instance - install a new tool $ cd /mnt/galaxy/export $ wget http://heanet.dl.sourceforge.net/ project/dnaclust/parallel_release_3/ dnaclust_linux_release3.zip $ unzip dnaclust_linux_release3.zip
  15. 15. Use the new tool in the cluster mode 1.  Create a new sample shell file to run the tool; call it job_script.sh with the following content: #$ -cwd /mnt/galaxy/export/dnaclust_linux_release3/dnaclust -l -s 0.9 /mnt/workshop-data/mtDNA.fasta 2.  Submit single job to SGE queue qsub job_script.sh 3.  Check the queue: qstat -f 4.  Job output will be in the local directory in file job_script.sh.o# 5.  Submit the same job a number of times: qsub job_script.sh (*10) watch qstat ‒f 1.  See all jobs lined up 6.  [optional] See auto-scaling in action (if enabled) [1.5-2 mins]
  16. 16. Sharing-an-Instance •  Share the entire CloudMan platform •  Includes all of user data and even the customizations •  Publish a self-contained analysis •  Make a note of the share-string and send it to your neighbor
  17. 17. UNDERPINNING CONCEPTS Technical details
  18. 18. CloudMan core Tool as a service - cloud resource management - multi-cloud interface - service management and monitoring - user interaction - system state - well-defined interface / API - web interface - self-contained service definition - implements the default service interface - implements optional service hooks - service dependencies - automatic dependency management - dynamic loading
  19. 19. Launcher / Management Console Application(s) (eg, Galaxy) 1° 2° 3° 6°, 8° 9° Persistent data repository Start CloudMan Setup services 5° 4° 7° 10° CM-w CM-w CM-w ... FS 1 FS ... Block storage Contextualize image CloudMan MI CloudMan MI ... CloudMan MI CloudMan machine image 11° S3/Swift CloudMan master instance Snap1 Snap.. Managed FS Inst. storage or or FS archive
  20. 20. What do you get? •  Cluster-in-the-cloud: SGE •  Galaxy on the Cloud + control •  Customizable: tools, configs, data •  Sharable •  Extensible
  21. 21. Supported cloud middleware 1.  Amazon Web Services 2.  OpenStack 3.  Eucalyptus 4.  OpenNebula
  22. 22. Building •  Leverage CloudBioLinux build framework •  Via GVL flavor •  Base CloudMan machine image •  Tools and data •  There are also more specific CBL flavors available •  cloudman github.com/afgane/gvl_flavor
  23. 23. Deploying •  Integrated with BioCloudCentral.org app •  Use the public one, deploy your own or run locally •  BCC supports multiple clouds
  24. 24. Launcher / Management Console Application(s) (eg, Galaxy) 1° 2° 3° 6°, 8° 9° Persistent data repository Start CloudMan Setup services 5° 4° 7° 10° CM-w CM-w CM-w ... FS 1 FS ... Block storage Contextualize image CloudMan MI CloudMan MI ... CloudMan MI CloudMan machine image 11° S3/Swift CloudMan master instance Snap1 Snap.. Managed FS Inst. storage or or FS archive Troubleshooting /mnt/galaxy[Indices] /mnt/cm/paster.log cm-<hash> /usr/bin/ec2autorun.log /tmp/cm/cm_boot.log /mnt/cm/paster.log 2 1 3
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×