Introduction to LAVA Workload Scheduler

5,253 views

Published on

Training slide for workshop
"Introduction to LAVA Workload Scheduler"
@NCSEC2009 BKK THILAND

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,253
On SlideShare
0
From Embeds
0
Number of Embeds
38
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Introduction to LAVA Workload Scheduler

  1. 1. Introduction to LAVA workload scheduler High Performance Computing and Networking Center (HPCNC) Kasetsart University In collaboration with Innovative Extremist (INOX) Co.,Ltd. and Platform Computing Inc.
  2. 2. Outline <ul><li>Introduction to HPC, cluster and workload scheduler
  3. 3. LAVA workload scheduler
  4. 4. Installing and configuring LAVA Cluster
  5. 5. Workshop : Using LAVA </li></ul>
  6. 6. Introduction to HPC, Cluster and Workload Scheduler
  7. 7. Cluster Computing <ul><li>Cluster computing is a technology related to the building of high performance scalable computing system from a collection of small computing system and high speed interconnection network </li></ul>
  8. 8. Why now? <ul><li>Maturity of many enabling technologies </li><ul><li>Low to medium cost high speed network </li><ul><li>Gigabit Ethernet, Myrinet, InfiniBand </li></ul><li>Powerful operating systems such as Windows, Linux, UNIX
  9. 9. Parallel Programming Systems which is portable and efficient
  10. 10. MPI (LAM, MPICH)
  11. 11. Software library that ease the application development e.g. Scalapack, Plapack, PetSc </li></ul></ul>
  12. 12. PC also rule the world <ul><li>Impact of PC technology </li><ul><li>Intel Pentium can deliver supercomputing performance at low cost
  13. 13. PC mass market nature drive the price down while performance increase rapidly
  14. 14. Cluster nature make it easy to capitalize on PC new technology right away </li></ul></ul>
  15. 15. Why? Price Performance!
  16. 16. Goal of Clustering <ul><li>High-performance clustering </li><ul><li>Link many computers together to team up and finish problem fasters by having multiple computer working on the same problem independently </li></ul></ul>
  17. 17. Goal of Clustering <ul><li>High-availability clustering </li><ul><li>make more reliable computer system by having many computers working together and takeover when any of them fail </li></ul></ul>
  18. 18. Applications <ul><li>Scientific computing </li><ul><li>CAD/CAM
  19. 19. Bioinformatics
  20. 20. Large scale financial analysis
  21. 21. Simulation
  22. 22. Drug Design
  23. 23. Automobile Design ( Crash Simulation) </li></ul><li>IT infrastructure </li><ul><li>Scalable web server, Search engine
  24. 24. (Google use more than 10000 node servers) </li></ul><li>Entertainment </li><ul><li>Rendering - On-line Gaming </li></ul></ul>
  25. 25. Molecular Dynamic Simulation <ul><li>Drug Discovery using molecular docking </li><ul><li>Avian Flu
  26. 26. HIV </li></ul><li>Analyzing property of Chemical compound </li></ul>
  27. 27. Graphics Rendering and Special Effect <ul><li>Rendering </li><ul><li>Generating 3D image from model </li></ul><li>Problem </li><ul><li>Rendering is a time consuming process especially for complex and realistic scene
  28. 28. Massive number of rendering job needed to be done to create a movie </li></ul></ul>
  29. 29. Cluster Software Architecture HTC HPC HPTC
  30. 30. High Throughput Computing <ul><li>High throughout, not high performance </li><ul><li>Complete most number of jobs in shortest amount of time </li></ul><li>Serial, parametric (usually), non-parallelized code </li><ul><li>Solve them on multiple processors at the same time, varying input parameters </li></ul><li>Example </li><ul><li>BLAST, Monte Carlo simulation </li></ul><li>Use of Load Schedulers </li><ul><li>Condor, Codine, LSF, Sun Grid Engine, SQMS </li></ul></ul>
  31. 31. High Throughput Computing (con) <ul><li>Pros and Cons </li><ul><li>Easy to get started. Use the sequential code in C or Fortran.
  32. 32. Excellence for many type of applications such as </li><ul><li>Parametric computing: Running the same computation with multiple data set
  33. 33. Distributed application such as massive rendering in animation industry </li></ul><li>Excellence when model can fit well in memory of a single computer </li><ul><li>No communication at all </li></ul></ul></ul>
  34. 34. High Performance Computing <ul><li>Maximum performance, not maximum throughput
  35. 35. Use of specialised codes, libraries </li><ul><li>MPI (Message Passing Interface)
  36. 36. Parallel Maths Libraries (ScaLapack) </li></ul><li>Solve large problem by breaking it in to a number of small problems (data or task partitioning), then solve them on distributed, multiple processors at the same time.
  37. 37. Pros and Cons </li><ul><li>Difficult since a parallel program must be developed
  38. 38. Good when </li><ul><li>Problem is larger than memory size of a single machines
  39. 39. Speedup for a single instance of problem is needed </li></ul></ul></ul>
  40. 40. Advantages and Challenges <ul><li>Advantages </li><ul><li>Highly scalable, light weight, easy setup
  41. 41. Plenty of free software </li></ul><li>Challenges </li><ul><li>Require a very highly trained, skill personal to maintain the system
  42. 42. No powerful software development environment
  43. 43. Low compatibility with many enterprise computing environment </li></ul></ul>
  44. 44. Parallel Application Development <ul><li>Shared memory – data is exchanged using memory reference
  45. 45. Message passing – data is exchanged by sending/receiving messages between processors </li></ul>
  46. 46. Workload Scheduler <ul><li>Or “Job scheduler” or “Load scheduler”
  47. 47. Main role of distributed computing
  48. 48. Allow users to share computing resources and time sharing </li><ul><li>Unify resources in the cluster in to a shared resource pool
  49. 49. Control shared resource usage for multiple users </li><ul><li>Job queue
  50. 50. Scheduling Policies </li></ul><li>Utilize resources efficiently
  51. 51. Hide the complexity of using cluster's computing resources by submitting job to the scheduler </li></ul></ul>
  52. 52. Key Features <ul><li>Resources Control </li><ul><li>Where are the resources?
  53. 53. How many we can use?
  54. 54. By whom? </li></ul><li>Job queue </li><ul><li>Classify users sharing, waiting queue.
  55. 55. Apply scheduling policy / resource pool </li></ul><li>User Interface </li><ul><li>Job control </li><ul><li>Submit / Suspend / Resume / Delete </li></ul><li>Monitoring job status </li></ul></ul>
  56. 56. Implementations <ul><li>SGE / N1 Grid Engine
  57. 57. Platform LSF / Platform LAVA
  58. 58. PBS / PBS Pro
  59. 59. Torque
  60. 60. Maui
  61. 61. MS Job scheduler </li></ul>
  62. 62. LAVA Workload Scheduler
  63. 63. LAVA Workload Scheduler <ul><li>An open source entry-level workload scheduler
  64. 64. Designed to meet a wide range of workload scheduling needs for clusters up to 512-nodes
  65. 65. Features </li><ul><li>Scalability
  66. 66. Reliability
  67. 67. Parallel Job Scheduling
  68. 68. Complete Job History
  69. 69. Interactive Jobs
  70. 70. Job Arrays
  71. 71. Job Dependency
  72. 72. Job Migration </li></ul></ul>
  73. 73. Components (mbatchd) (sbatchd) (sbatchd) (sbatchd) LAVA Base LIM (Load Information Manager) RES (Remote Execution Service) Computer LSBATCH
  74. 74. Installing and Configuring
  75. 75. Installing LAVA <ul><li>First, We need a cluster (or servers) </li></ul>
  76. 76. Installing LAVA <ul><li>Manual installation </li><ul><li>Setup a cluster
  77. 77. Install LAVA from source </li><ul><li>Download source code from HPCCommunity Website
  78. 78. Extract, compile and install </li><ul><li>tar zxvf lava-tarball.tar.gz
  79. 79. ./configure
  80. 80. make && make install </li></ul></ul></ul></ul>
  81. 81. Installing LAVA <ul><li>Using cluster distribution </li><ul><li>LAVA Kit for KUSU
  82. 82. LAVA Roll for ROCKS Cluster </li></ul><li>Advantages </li><ul><li>Auto configuration tools
  83. 83. Can scale nodes without editing config files </li></ul></ul>
  84. 84. Configuring LAVA <ul><li>Environments
  85. 85. LAVA Base (lsf)
  86. 86. Batch scheduler (lsbatch) </li></ul>
  87. 87. Configuration <ul><li>Environments
  88. 88. Setting in /etc/profiles.d/lava.sh </li><ul><li>LSF_VERSION=1.0
  89. 89. LSF_TOP=/usr
  90. 90. LSF_BINDIR=/usr/bin
  91. 91. LSF_SERVERDIR=/usr/sbin
  92. 92. LSF_LIBDIR=/usr/lib
  93. 93. LSF_ENVDIR=/etc/lava/conf </li></ul></ul>
  94. 94. Configuration <ul><li>LAVA Base (LSF) </li><ul><li>Local information and execution process
  95. 95. Config file usually located at $LSF_ENVDIR = /etc/lava/conf/
  96. 96. Files </li><ul><li>lsf.conf * (Installation and operation of Lava)
  97. 97. lsf.cluster.lava * (general configuration, nodes, parameters)
  98. 98. lsf.task (type of tasks)
  99. 99. lsf.shared (default parameters)
  100. 100. hosts * (list of known hosts and IPs) </li></ul></ul><li>Mark (*) is configuration needed on master and slave nodes </li></ul>
  101. 101. Configuration files <ul><li>Batch scheduler (LSBATCH) </li><ul><li>Central batch scheduler of the cluster
  102. 102. Config file usually located at $LSF_ENVDIR/lsbatch/lava/configdir/
  103. 103. Files </li><ul><li>lsb.hosts (list of nodes and parameters)
  104. 104. lsb.modules (list of plugin modules)
  105. 105. lsb.params (batch scheduler parameters)
  106. 106. lsb.queues (job queue name & properties)
  107. 107. lsb.users (list of allowed users) </li></ul></ul></ul>
  108. 108. Start the LAVA <ul><li>Start/Stop the LAVA service on every nodes </li><ul><li>/etc/init.d/lava start
  109. 109. /etc/init.d/lava stop </li></ul><li>For individual service </li></ul>
  110. 110. Controlling Queues <ul><li>Adding Job Queue </li><ul><li>Edit lsb.queues to add the new queue definition.
  111. 111. Copy another queue definition from this file as a starting point and change the QUEUE_NAME of the copied queue.
  112. 112. Save the changes to lsb.queues.
  113. 113. Run badmin reconfig to reconfigure mbatchd.
  114. 114. Adding a queue does not affect pending or running jobs. </li></ul></ul>
  115. 115. Controlling Queues <ul><li>Removing Queue </li><ul><li>Close the queue to prevent any new jobs from being submitted using command badmin qclose QUEUE_NAME
  116. 116. Move all pending and running jobs into another queue using command bswitch -q Q_FROM Q_TO 0
  117. 117. Edit lsb.queues and remove or comment out the definition for the queue you want to remove.
  118. 118. Save the changes to lsb.queues.
  119. 119. Run badmin reconfig to reconfigure mbatchd. </li></ul></ul>
  120. 120. Using LAVA
  121. 121. LAVA Commands <ul><li>LSBATCH (starts with 'b') </li><ul><li>badmin*
  122. 122. bbot
  123. 123. bchkpnt
  124. 124. bhist
  125. 125. bhosts
  126. 126. bjobs
  127. 127. bkill
  128. 128. bmgroup
  129. 129. bmig
  130. 130. bmod
  131. 131. bparams </li></ul></ul><ul><li>LAVA base (starts with 'ls') </li><ul><li>lsadmin*
  132. 132. lsacct
  133. 133. lseligible
  134. 134. lshosts
  135. 135. lsid
  136. 136. lsinfo
  137. 137. lsload
  138. 138. lsloadadj
  139. 139. lsmon
  140. 140. lsplace
  141. 141. lsrcp </li></ul></ul><ul><ul><li>bpeek
  142. 142. bqueues
  143. 143. brequeue
  144. 144. brestart
  145. 145. bresume
  146. 146. brun
  147. 147. bstop
  148. 148. bsub
  149. 149. bswitch
  150. 150. btop
  151. 151. bugroup
  152. 152. busers </li></ul></ul>
  153. 153. Administrative Commands <ul><li>lsadmin
  154. 154. badmin </li></ul>
  155. 155. System information <ul><li>bhosts / bqueues -l
  156. 156. bqueues / bqueues -l
  157. 157. bparams / bparams -l
  158. 158. bmgroup
  159. 159. bugroup
  160. 160. busers [all] </li></ul><ul><li>lsid
  161. 161. lsinfo
  162. 162. lshosts
  163. 163. lsload
  164. 164. lsmon
  165. 165. lsacct </li></ul>
  166. 166. Monitoring Jobs <ul><li>bjobs
  167. 167. bjobs <job id>
  168. 168. bjobs -a </li><ul><li>show all state (include EXIT, DONE) </li></ul><li>bjobs -r / -p / -s </li><ul><li>show only running / pending / suspended jobs </li></ul><li>bjobs -u user1 / bjobs -u all </li><ul><li>show only user1 / show all users </li></ul><li>bjobs -l </li><ul><li>show more detail </li></ul></ul>
  169. 169. View Job History <ul><li>bhist
  170. 170. bhist -l </li><ul><li>long detail </li></ul></ul>
  171. 171. Submitting a Job <ul><li>Use command </li><ul><li>bsub [-option]/path/to/command args </li></ul><li>If you do not specify any options, the job is submitted to the default queue configured by the Lava administrator (usually thenormal queue)
  172. 172. Example </li><ul><li>$ bsub my_job
  173. 173. Job <1234> is submitted to default queue <normal> </li></ul><li>In the above example, 1234 is the job ID assigned to this job, and normal is the nameof the default job queue. </li></ul>
  174. 174. Submitting a Script <ul><li>Any command or script you can execute from a shell prompt can be submitted to Lava for batch execution.
  175. 175. Create file myscript
  176. 176. chmod u+x myscript
  177. 177. bsub < myscript </li></ul>#!bin/sh #BSUB -q test #BSUB -o outfile -R &quot;mem>10&quot; myjob arg1 arg2 #BSUB -J myjob
  178. 178. Submitting a job to specific hosts <ul><li>To indicate that a job must run on one of the specified hosts, use the to a single host bsub -m &quot;hostA hostB ...&quot; option.
  179. 179. By specifying a single host, your job will wait until that host is available and then run on that host. </li><ul><li>$ bsub -q idle -m &quot;hostA hostD hostB&quot; myjob </li></ul><li>Or select by specific resources </li><ul><li>$ bsub -R &quot;hname!=hostb && type==LINUX86&quot; myjob </li></ul></ul>
  180. 180. <ul>Running Parallel Jobs </ul><ul><li>To submit a parallel job, use bsub -n and specify multiple processors. </li><ul><li>$ bsub -n 4 myjob </li></ul><li>This command submits myjob as a parallel job. The job is started when 4 job slots are available.
  181. 181. Job slot limits for parallel jobs </li><ul><li>A job slot is the basic unit of processor allocation in Lava. A sequential job uses one job slot. A parallel job that has N components (tasks) uses N job slots, which can span multiple hosts. </li></ul></ul>
  182. 182. Modify Jobs <ul><li>Use bmod command to modify job submission parameters on pending job </li><ul><li>bmod -b 2:00 101
  183. 183. change the start time of job 101 to 2:00 a.m </li></ul><li>To reset an option to its default submitted value (undo a bmod), append the n character to the option name </li><ul><li>bmod -bn 101 </li></ul></ul>
  184. 184. <ul>Killing a job </ul><ul><li>Killing a job </li><ul><li>bkill command cancels pending batch jobs and sends signals to running jobs.
  185. 185. By default, bkill sends SIGINT, SIGTERM and then SIGKILL signal to running jobs
  186. 186. Example </li><ul><li>bkill 3421
  187. 187. Job <3421> is being terminated </li></ul><li>bkill -r command removes a job from the system without waiting for the job to terminate in the operating system and mark as EXIT status </li></ul></ul>
  188. 188. Suspend and Resume a job <ul><li>Suspend </li><ul><li>Using command bstop job_ID
  189. 189. Your job goes into USUSP state if the job is already started, or into PSUSP state if it is pending. </li><ul><li>bstop 3421
  190. 190. Job <3421> is being stopped </li></ul></ul><li>Resume </li><ul><li>Run bresume job_ID </li><ul><li>bresume 3421
  191. 191. Job <3421> is being resumed </li></ul></ul></ul>
  192. 192. Requeuing and Rerunning Jobs <ul><li>You can kill and requeue a job while it is running or when it is suspended. Use the brequeue command to requeue the job </li><ul><li>brequeue -u user5 45 67 90 </li></ul><li>To enable automatic job rerun, submit the job with the re-runnable option bsub -r. If the execution host fails, Lava wil dispatch the job to another host. Youlll receive an email informing you of the host failure and the requeuing of the job. </li><ul><li>bsub -r my_job </li></ul></ul>
  193. 193. Moving Jobs <ul><li>Moving a job to the bottom of a queue </li><ul><li>bbot job_id </li></ul><li>Moving a job to the top of a queue </li><ul><li>btop job_id </li></ul></ul>
  194. 194. Switching jobs from one queue to another <ul><li>Switch a single job </li><ul><li>Use bswitch to move pending and running jobs from queue to queue. </li><ul><li>bswitch priority 5309
  195. 195. Job <5309> is switched to queue <priority> </li></ul></ul><li>Switch all jobs </li><ul><li>Use bswitch -q from_queue to_queue 0
  196. 196. The job ID number 0 specifies that all jobs will be moved </li><ul><li>bswitch -q night idle 0 </li></ul></ul></ul>
  197. 197. END OF SESSION Thank you
  198. 198. Reference <ul><li>Lava User Guide http://www.hpccommunity.org/exdata/lava/docs/lava_using.pdf </li></ul>

×