Sge

982 views

Published on

A lab talk on Sun Grid Engine.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
982
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Sge

  1. 1. Scaling, Grid Engine and Running UIMA on the Cluster<br />Chris Roeder 11/2010<br />
  2. 2. The Scaling Problem<br />“Does the solution scale?” asks if larger versions of the problem (often more data) can be dealt with by a given piece of software.<br />“Scaling” is a loose collection of techniques to improve or implement a solution’s scalability. <br />The choice of techniques depends on the critical resource: cpu, memory or i/o and how easily the task is broken into pieces.<br />This talk focusses on Scaling as it applies to UIMA NLP processing (not withstanding OpenDMAPv2).<br />It is a work in progress.<br />
  3. 3. Scaling NLP<br />Processing a file is independent of processing another file:Text in, annotations out.<br />Multi-threaded<br />More than one thread of execution in one process<br /> pipelines share memory and can step on each other.<br />Ex. Stanford crashes because of concurrency issues<br />“was not an issue in 2001”<br /> <casProcessors casPoolSize=“4" processingUnitThreadCount=“2"><br />Multi-process<br />Separate JVM’s, each with a single thread<br />Memory is not shared, no crushed toes<br /><casProcessors casPoolSize="3" processingUnitThreadCount=“1"><br />Overhead of repeated JVM and pipeline does cost, but it works.<br />Many machines<br />More memory, more cores<br />Independence means they won’t miss being on the same machine<br />Independent machines (Cluster) are cheaper than integrated (Enki)<br />
  4. 4. Hardware<br />Local Cluster (Colfax)<br />A rack of machines with software (SGE) to integrate<br />Integrated CPUs (Enki)<br />Much like a rack, but motherboards are tied together and can share memory<br />Gigabit ethernet delivers on the order of 300Mb/sec<br />Motherboard runs up to 4.8GB/sec<br />Virtual Cluster<br />Virtualization software allows for a single machine to appear as many, offers flexibility, security<br />Cloud<br />A virtual cluster on the net: Amazon EC2<br />
  5. 5. Hardware: CCP’s Colfax Cluster<br />Runs Linux (Fedora/Red Hat)<br />6 machines (amc-colfax, amc-colfaxnd[1-5])<br />2 cpus (Intel), 4 cores each, 48 cores total<br />Intel motherboard<br />16GB memory each, 96 GB total<br />5TB shared (over NFS) disk array, RAID5<br />Named after the assembler: Colfax International<br />
  6. 6. (Sun|Oracle) Grid Engine (SGE)<br />Manages a queue of jobs, optimizing resources utilization<br />Starts individual processes for a job<br />Often used with Message Passing Interface (MPI) for processes that cooperate<br />Used here to start “Array Jobs”<br />Each job processes a portion of a large array of work to be done. <br />
  7. 7. SGE Job<br />An SGE job is a script and a command line<br />Command line specifies resources for scheduling<br />Memory<br />others<br />Script is run once for each process started<br />Is not pure shell, but more/less a shell script (next slide)<br />Job is assigned an ID number<br />
  8. 8. more/less a shell script?<br />Put these lines at top for SGE:<br />#$ -N stanford_out<br />Standard out goes to a file with this prefix<br />#$ -S /bin/bash<br />The shell to use (no “she-bang”: #!/bin/sh)<br />#$ -cwd<br />Runs from the current directory<br />#$ -j y<br />Merge stdout and stderr to one file<br />
  9. 9. Submit a Job: qsub<br />Qsub –t 1-200000:20000 sge_stanford_out.sh<br />-t Index Range<br />Do array items from 1 to 200 thousand, by 20k: 10 processes<br />Do this with the sge_stanford_out.sh script<br />How does the script know what files to process?<br />$SGE_TASK_ID (first file number to run)<br />$SGE_TASK_STEPSIZE<br />A task will get values of 0,19999,20000 for example<br />
  10. 10. Sge_stanford_out.sh<br />Will evolve into generic UIMA job submission script<br />Script modifies a template CPE file, creates a CPE for each process<br />CPE specifies starting document number and number to process<br />http://wikis.sun.com/display/gridengine62u2/How+to+Submit+an+Array+Job+From+the+Command+Line<br />[roederc@amc-colfax sge_scripts]$ qsub -t 1-50:3 sge_stanford_out.sh<br />Your job-array 130.1-50:3 ("stanford_out") has been submitted<br />
  11. 11. qstat<br />[roederc@amc-colfax sge_scripts]$ qstat<br />job-ID prior name user state submit/start at queue slots ja-task-ID <br />-----------------------------------------------------------------------------------------------------------------<br /> 130 0.00000 stanford_o roederc qw 11/02/2010 12:39:01 1 1-49:3<br />[roederc@amc-colfax sge_scripts]$ qmon<br />[roederc@amc-colfax sge_scripts]$ qstat<br />job-ID prior name user state submit/start at queue slots ja-task-ID <br />-----------------------------------------------------------------------------------------------------------------<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd4.ucdenver.p 1 4<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd2.ucdenver.p 1 7<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd5.ucdenver.p 1 10<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd3.ucdenver.p 1 13<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd1.ucdenver.p 1 16<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd5.ucdenver.p 1 19<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd2.ucdenver.p 1 22<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd4.ucdenver.p 1 25<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfax.ucdenver.pvt 1 28<br /> 130 0.55500 stanford_o roederc r 11/02/2010 12:39:10 all.q@amc-colfaxnd3.ucdenver.p 1 31<br />
  12. 12. Qdel command<br />Use to kill a command<br />Qdel <job num><br />
  13. 13. Failures? <br />Q:What if a job fails?<br />(A: it stops)<br />Open problem <br />For now, that process dies leaving unprocessed jobs<br />Need to cull unprocessed files and try again<br />Usually not enough memory<br />Future: db-driven collection reader with cas-consumer that reports completion <br />
  14. 14. Example 1:<br />Distribute a simple script on cluster:<br />Test_sge.sh<br />Qsub test_sge.sh<br />Runs it once<br />Qsub test_sge.sh –t 1-5:1<br />Runs it five times<br />Qsub test_sge.sh –t 100-500:100<br />Also runs it five times<br />Gives index starts spaced by 100<br />
  15. 15. Example 2:Run UIMA on Cluster<br />Sge_stanford_out.sh:<br />Calls a script with a template CPE and index range: <br />run_cpe_cluster_stanford_out.sh<br />Modifies CPE template, creating a CPE for each sub-range<br />Sets up environment, calls SimpleRunCPE (java)<br />Note temp_cpe_<n>.xml in ../desc/cpe<br />Start a number of terminals, run “top” in each to see cpu and memory usage.<br />
  16. 16. Hadoop<br />Inspired by Lisp’s map/reduce<br />Map: apply a function to each element of a hash<br />Reduce: combine hashes into one<br />Known for optimizing by moving processing rather than data<br />Similar code used by Google. <br />Hadoop is open source, used by Yahoo, Amazon.<br />Specialized interfaces make it more suited to greenfield development<br />
  17. 17. What about “The Cloud”<br />Amazon’s Elastic Compute Cloud (EC2) is a cluster on the internet that can be rented by the hour<br />Very Dynamic<br />Set up nodes when you start using them <br />Expect them to dissapper when you stop<br />Must have machine configuration management sussed. You have to re-install everything.<br />Use S3 for long-term storage<br />Starts at $0.10/hour<br />
  18. 18. Colfax Cluster<br />6 CPUs<br />5TB disk array<br />
  19. 19. Enki<br />8TB RAID<br />CPU<br />

×