The document provides instructions for interacting with and submitting jobs to a high performance computing (HPC) system. It begins by demonstrating how to log in to the system using SSH and describes the information displayed upon login. It then shows how to create and edit files using nano, run code interactively on a node using idev, and submit batch jobs using SLURM. The user creates Python scripts to print "Hello World" sequentially and in parallel, runs them interactively, and submits the parallel version as a batch job to demonstrate these workflows. Monitoring and management commands like squeue and scancel are also introduced.

1. PRESENTED BY:
HPC Exercises
Interacting with High Performance
Computing Systems
6/7/18 1
VirginiaTrueheart, MSIS
Texas Advanced Computing Center
vtrueheart@tacc.utexas.edu

2. Logging In
• In order to access the TACC machines you will need to login using a
terminal or SSH client
• SSH is an encrypted network protocol to access a secure system
over an unsecured network
• The following example is for XSEDE Login
• You can also log directly into the TACC machine but your password may be
different.

3. Logging In (Mac Terminal)
$ ssh –l <username> login.xsede.org
Please login to this system using your XSEDE username and password:
password:
Duo two-factor login for <username>
Enter a passcode or select one of the following options
1. Duo Push to XXX-XXX-XXXX
2. Phone call to XXX-XXX-XXXX
Passcode or option (1-2):
6/7/18 3

4. Logging In pt. 2
# For example, to login to the Comet system at SDSC, enter: gsissh comet
#
# Email help@xsede.org if you require assistance in the use of this system.
[username@ssohub ~]$ gsissh stampede2
6/7/18 4

5. Interacting with the System
After logging in you will be able to see the TACC Info box which will tell
you what projects you are associated with and how much of the file
system you have used.

6. Welcome to Stampede2, *please* read these important system notes:
--> Stampede2, Phase 2 Skylake nodes are now available for jobs
--> Stampede2 user documentation is available at:
https://portal.tacc.utexas.edu/user-guides/stampede2
----------------------- Project balances for user vtrue -----------------------
| Name Avail SUs Expires | |
| A-ccsc 189624 2018-12-31 | |
------------------------- Disk quotas for user vtrue --------------------------
| Disk Usage (GB) Limit %Used File Usage Limit %Used |
| /home1 1.9 10.0 19.43 39181 200000 19.59 |
| /work 311.8 1024.0 30.45 225008 3000000 7.50 |
| /scratch 0.0 0.0 0.00 4 0 0.00 |
-------------------------------------------------------------------------------
6/7/18 6

7. Creating a File
Inline text editors can be very useful when interacting with the system
so lets use a very simple one (nano) to create a file that we can use to
execute some code.

8. Create a File
login1.stampede2$ cd $WORK
login1.stampede2$ pwd
/work/03658/vtrue/stampede2
login1.stampede2$ nano helloWorld.py
6/7/18 8

9. Editing a File
• You should now be sitting in the editing environment of the
helloWorld.py file.
• You can type in the code found on the next slide in order to create the
contents of the file.
• Type Ctrl+X to exit and type Y to save the file when you are prompted.

10. 6/7/18 10
#!/usr/bin/env python
"""
Hello World
"""
import datetime as DT
today = DT.datetime.today()
print "Hello World! Today is:"
print today.strftime("%d %b %Y")
A Very Small File

11. Executing Our File
• It is prohibited to run code on the login nodes as they are a shared
resource.
• In order to run this little code we have written we will first need to
start an idev (interactive development) session.

12. helloWorld.py
staff.stampede2(1005)$ idev
-> Checking on the status of development queue. OK
-> Defaults file : ~/.idevrc
-> System : stampede2
-> Queue : development (idev default )
[...]
c455-012[knl](1019)$
6/7/18 12

13. helloWorld.py
staff.stampede2(1005)$ idev
-> Checking on the status of development queue. OK
-> Defaults file : ~/.idevrc
-> System : stampede2
-> Queue : development (idev default )
[...]
c455-012[knl](1019)$ python helloWorld.py
Hello World! Today is:
17 Jun 2018
c455-012[knl](1020)$
6/7/18 13

14. Do More with Our File
• Now that we see the helloWorld.py file will run on the compute node
and produce output let’s test out the parallel aspects of running on a
node. Namely, accessing all of the cores on the node.
• Type ‘nano helloWorld.py’ to reopen your file and begin editing it
again.
• Input the python code you see on the next slide and then do Ctrl+X
again to save your changes.

15. #!/usr/bin/env python
"""
Parallel Hello World
"""
from mpi4py import MPI
import sys
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
name = MPI.Get_processor_name()
sys.stdout.write(
"Hello, World! I am process %d of %d on %s.n"
% (rank, size, name))
6/7/18 15

16. Running in Parallel
• Even though this is still a python code, we are now taking advantage
of the parallel capabilities of the node
• As such you need to start your code by using ‘ibrun’ instead of just
typing ‘python’
• When you run this code you will receive feedback from each core on
the node.

17. Parallel helloWorld.py
• c455-012[knl](1019)$ ibrun python helloParallel.py
• TACC: Starting up job 1595632
• TACC: Starting parallel tasks...
• Hello, World! I am process 1 of 68 on c456-042.stampede2.tacc.utexas.edu.
• Hello, World! I am process 49 of 68 on c456-042.stampede2.tacc.utexas.edu.
• Hello, World! I am process 66 of 68 on c456-042.stampede2.tacc.utexas.edu.
• Hello, World! I am process 67 of 68 on c456-042.stampede2.tacc.utexas.edu.
• Hello, World! I am process 64 of 68 on c456-042.stampede2.tacc.utexas.edu.
• ...
• TACC: Shutdown complete. Exiting.
6/7/18 17

18. Exiting an idev Session
• Great! Now that you can see what interactive jobs look like we can go
on to more advanced job submission.
• To leave an idev session simply type ‘exit’. Let’s do that now.

19. Submitting a Job
• The previous two examples ran a job interactively. This means you
could be on the node and see the output generated as it happened.
• This isn’t always practical though so we need a way to submit jobs
and then leave them to the system to run whenever nodes become
available.
• To do this we’ll take advantage of the SLURM system.

20. Create a New Nano File
• Create a new file called batchJob.sh
• Input the code found on the next slide and save the file.

21. An Example SLURM Batch File
#!/bin/bash
#SBATCH -J myJob # Job name
#SBATCH -o myJob.o%j # Name of stdout output file
#SBATCH -e myJob.e%j # Name of stderr error file
#SBATCH -p development # Queue (partition) name
#SBATCH -N 1 # Total # of nodes
#SBATCH -n 68 # Total # of mpi tasks
#SBATCH -t 00:05:00 # Run time (hh:mm:ss)
#SBATCH -A myproject # Allocation name (req'd if you have more than 1)
#SBATCH --mail-user=hkang@austin.utexas.edu
#SBATCH --mail-type=all # Send email at begin and end of job
# Other commands must follow all #SBATCH directives...
module list
pwd
date
# Launch code...
ibrun python helloParallel.py
6/7/18 21

22. Submitting the Batch Job
• To submit the SLURM Batch job use the following command
• sbatch batchJob.sh
• This should generate a text output to let you know about the
parameters of the job and then provide you with a Job ID once the
job has successfully been admitted to the queues.

23. Checking the Status of a Job
• Once a job is submitted to the queues, you can’t see it running the
way you could when you were running interactively.
• Instead we use monitoring commands to see what state our job is in.
• The command ’squeue’ is very useful for this and has several flags
available to control it’s output. Let’s use the –u flag to see what all
jobs under our username are doing.

24. staff.stampede2(1009)$ squeue -u vtrue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1604426 development idv20717 vtrue R 16:57 1 c455-001
staff.stampede2(1010)$ scontrol show job=1604426
JobId=1604426 JobName=idv20717
UserId=vtrue(829572) GroupId=G-815499(815499) MCS_label=N/A
Priority=400 Nice=0 Account=A-ccsc QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:18:08 TimeLimit=00:30:00 TimeMin=N/A
SubmitTime=2018-06-09T21:27:33 EligibleTime=2018-06-09T21:27:33
StartTime=2018-06-09T21:27:36 EndTime=2018-06-09T21:57:36 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
LastSchedEval=2018-06-09T21:27:36
...
6/7/18 24

25. Cleaning Up Jobs
• When your batch job has finished running it will automatically be
cleared from the queues.
• Your output will be in the folder you pointed it to within your batch
job file.
• If for some reason you wish to cancel your job while it is still running,
you can do so with ‘scancel –jobID>’