Audience Level Intermediate Synopsis The traditional user experience for High Performance Computing (HPC) centers around the command line, and the intricacies of the underlying hardware. At the same time, scientific software is moving towards the cloud, leveraging modern web-based frameworks, allowing rapid iteration, and a renewed focus on portability and reproducibility. This software still has need for the huge scale and specialist capabilities of HPC, but leveraging these resources is hampered by variation in implementation between facilities. Differences in software stack, scheduling systems and authentication all get in the way of developers who would rather focus on the research problem at hand. This presentation reviews efforts to overcome these barriers. We will cover container technologies, frameworks for programmatic HPC access, and RESTful APIs that can deliver this as a hosted solution. Speaker Bio Dr. David Perry is Compute Integration Specialist at The University of Melbourne, working to increase research productivity using cloud and HPC. David chairs Australia’s first community-owned wind farm, Hepburn Wind, and is co-founder/CTO of BoomPower, delivering simpler solar and battery purchasing decisions for consumers and NGOs.

1. SUPERCOMPUTINGBYAPI:
CONNECTINGWEBAPPSTOHPC
Dr. David Perry
Compute Integration Specialist, University of Melbourne

3. Worker
Nodes
Database
Web
Server
Virtual Laboratory

4. Worker
Nodes
Database
Web
Server
Login
Node
Compute
Nodes
Virtual Laboratory Supercomputer

5. YAYSUPERCOMPUTERS!

6. THEPROBLEM
Each HPC cluster has its own:
Scheduler
Software/OS
Hardware

7. THEDREAM
Write once, run anywhere.
No platform dependencies.
Consistent RESTful API for ... everything.

8. SOLUTIONS!
(sort of)

9. TODAY:
1. HPC APIs
2. Containers

10. THEIDEALHPCAPI:
Consistent interface across schedulers
Manages les
Work across system boundaries
Doesn't require changes to HPC cluster (no new
software, network ports, or security risks)
Multiple language bindings/wrappers

11. DRMAA

12. import drmaa
# Create session
s = drmaa.Session()
s.initialize()
# Create job
jt = s.createJobTemplate()
jt.remoteCommand = "echo 'hello'"
jt.nativeSpecification = "--mincpus=2"
jt.hardWallclockTimeLimit = '1:00:00'
# Run it
jobid = s.runJob(jt)
print('Your job has been submitted with ID %s' % jobid)
# Wait for it to complete
retval = s.wait(jobid, drmaa.Session.TIMEOUT_WAIT_FOREVER)
print('Job: {0} finished with status {1}'.format(retval.jobId, retval.hasEx
s.exit()

13. Scheduler
A Scheduler
B
Scheduler
C
Features
supported by
DRMAA

15. Good: Supported by almost all schedulers.
Bad: Unfriendly, local access only, limited scheduler
feature support, no longer under active
development.

16. SAGA

17. import saga
import os
# Run job using SAGA
ctx = saga.Context("ssh")
ctx.user_id = 'perryd'
os.environ['SAGA_PTY_SSH_TIMEOUT'] = '60'
session = saga.Session()
session.add_context(ctx)
js = saga.job.Service("slurm+ssh://spartan.hpc.unimelb.edu.au/", session=se
jd = saga.job.Description()
jd.executable = "echo 'hello' > hello.out"
jd.wall_time_limit = 5 # minutes
# Create and submit job, wait for it to finish.
myjob = js.create_job(jd)
myjob.run()
print 'Job Running'
myjob.wait()
print('Job %s finished with status %s' % (myjob.id, myjob.exit_code))

18. # Fetch output files
output = 'file://localhost/tmp/'
source = 'sftp://spartan.hpc.unimelb.edu.au/home/perryd/hello.out'
saga.filesystem.File(source, session=session).copy(output)
print('Remote file contents:')
print(open('/tmp/hello.out').read())

19. Good: Supports popular schedulers, works over
SSH, nothing to install on cluster, handles le
transfers.
Bad: Still not a web API.

20. AGAVE

22. Via RESTful API:
Execution & Storage Systems
Monitoring
Metadata
Permissions
History
Events

23. Demo

24. Agave ToGo
https://togo.agaveapi.co

25. Good: Hosted. RESTful, OpenAPI-compliant. Does
everything.
Bad: Hosted. RESTful, OpenAPI-compliant. Does
everything.

26. On to containers...

27. Why?

28. What versions of Bowtie are available?
At Melbourne:
At Monash:
At NCI:
Bowtie2/2.2.5-GCC-4.9.2
Bowtie2/2.2.5-intel-2016.u3
Bowtie2/2.2.9-GCC-4.9.2
Bowtie2/2.2.9-intel-2016.u3
bowtie/1.1.2
bowtie2/2.2.8
bowtie/1.2.0
bowtie2/2.1.0
bowtie2/2.2.5
bowtie2/2.2.9
bowtie2/2.3.1

33. SINGULARITY
Image-based (just a big le with everything in it)
Flat network/hardware access
Volume mounts similar to Docker

34. DEMO

35. 1. Get or create a container.
$ sudo singularity create -s 6000 my_container.img
$ sudo singularity bootstrap my_container.img ubuntu.def
$ sudo singularity shell -w my_container.img
my_container.img> # Do stuff in a container

36. $ sudo singularity create -s 6000 digits_docker.img
$ sudo singularity --verbose import digits_docker.img
docker://nvidia/digits:latest

37. 2. Run your container.
$ singularity exec -B /tmp:/jobs digits_docker.img
bash -c "export DIGITS_JOBS_DIR=/jobs && python -m digits"

38. As a HPC job:
#!/bin/bash
#SBATCH --nodes 1
#SBATCH --cpus-per-task=12
#SBATCH --partition gpu
#SBATCH --gres=gpu:4
#SBATCH --time 02:00:00
LOGIN_PORT=$(shuf -i 2000-65000 -n 1)
DIGITS_PORT=5000
module load Singularity
ssh -N -f -R $LOGIN_PORT:localhost:$DIGITS_PORT $SLURM_SUBMIT_HOST
echo "Forwarding to port:"
echo $LOGIN_PORT
singularity exec -B /tmp:/jobs -B /tmp:/scratch digits_docker.img bash -c

39. CAVEATS
Hardware/architecture dependencies still there.
Beware the golden image.

40. CONCLUSION
Supercomputer-enable your web app!
But can't ignore details of each supercomputer.
Tools out there to make life a bit easier.

41. MOREEXPLORATION
Project looking at:
APIs (inc. local Agave deployment)
Virtual Laboratory to HPC Single Sign-on
Knowledge Sharing

42. ACKNOWLEDGEMENTS
Nectar
VL managers & developers
Authors of SAGA, DRMAA and Agave
Lev Lafayette & Daniel Tosello