July 1st,2016 - v.01

1. NSCC High
Performance
Computing Cluster
Introduction
[1-July-2016]

2. • Introduction to NSCC
• About HPC
• More about NSCC HPC cluster
• PBS Pro (Scheduler)
• Compilers and Libraries
• Developer Tools
• Co-processor / Accelerators
• Environment Modules
• Applications
• User registration procedures
• Feedback
2
The Discussion

3. 3
Introduction to NSCC

4. • State-of-the-art national facility with computing, data
and resources to enable users to solve science and
technological problems, and stimulate industry to use
computing for problem solving, testing designs and
advancing technologies.
• Facility will be linked by high bandwidth networks to
connect these resources and provide high speed access
to users anywhere and everyone.
Introduction:
The National Supercomputing Centre (NSCC)
4

5. Introduction: Vision & Objectives
Vision: “Democratising Access to Supercomputing”
5
Making Petascale Supercomputing accessible to the
ordinary researcher1
Bringing Petascale Computing and Storage and
Gigabit speed networking to the ordinary person2
Supporting National R&D Initiatives1
Objectives of NSCC
Attracting Industrial Research Collaborations2
Enhancing Singapore’s Research Capabilities3

6. 6
What is HPC?

7. 7
What is HPC?
• Term HPC stands for High Performance Computing or High
Performance Computer
• Tightly coupled personal computers with high speed
interconnect
• Measured in FLOPS (FLoating point Operations Per Second)
• Architectures
– NUMA (Non-uniform memory access)

8. Major Domains where HPC is used
Engineering
Analysis
• Fluid
Dynamics
• Materials
Simulation
• Crash
simulations
• Finite
Element
Analysis
Scientific
Analysis
• Molecular
modelling
• Computational
Chemistry
• High energy
physics
• Quantum
Chemistry
Life Sciences
• Genomic
Sequencing
and Analysis
• Protein
folding
• Drug design
• Metabolic
modelling
Seismic
analysis
• Reservoir
Simulations
and modelling
• Seismic data
processing
8

9. Major Domains where HPC is used
Chip design &
Semiconductor
• Transistor
simulation
• Logic Simulation
• Electromagnetic
field solver
Computational
Mathematics
• Monte-Carlo
methods
• Time stepping
and parallel time
algorithms
• Iterative
methods
Media and
Animation
• VFX and
visualization
• Animation
Weather
research
• Atmospheric
modelling
• Seasonal time-
scale research
• -
Major Domains where HPC is used
9

10. Major Domains where HPC is used
• And More
– Bigdata
– Information Technology
– Cyber security
– Banking and Finance
– Data mining
10

11. 11
Introduction to NSCC HPC
Cluster

12. Executive Summary
• 1 Petaflop System
– About 1300 nodes
– Homogeneous and Heterogeneous architectures
• 13 Petabytes of Storage
– One of the Largest and state of the art Storage architecture
• Research and Industry
– A*STAR, NUS, NTU, SUTD
– And many more commercial and academic organizations
12

13. HPC Stack in NSCC
Mellanox 100 Gbps Network
Intel Parallel
studio
Allinea Tools
PBSPro
Scheduler
Lustre & GPFS
HPC Application software
Operating System
RHEL 6.6 and CentOS 6.6
Fujitsu x86 Servers NVidia Tesla K40 GPUDDN Storage
Application
Modules
13

14. 14
NSCC Supercomputer Architecture
Base Compute Nodes (1160 nodes) Accelerated Nodes (128 nodes)
Parallel File system /
Tiered storage
InfiniBand network - Fully non-
blocking
Ethernet NW
GIS FAT node
NUS Peripheral
Servers
NTU Peripheral
Servers
NSCC Peripheral
Servers
NSCC Direct
users
VPN

15. Login architecture
15
Login
cluster
80Gb/s Link
NSCC cluster

16. 16
Customized
Solution

17. 17
Genomic Institute of
Singapore (GIS)
National
Supercomputing
Center (NSCC)
2km
Connection between GIS and NSCC
Large memory
node (1TB),
Ultra high speed
500Gbps
enabled
2012:
300 Gbytes/week
2015:
4300 Gbytes/week
x 14

18. NGSP Sequencers at B2
(Illumina + PacBio)
NSCC
Gateway
STEP 2: Automated
pipeline analysis once
sequencing completes.
Processed data resides in
NSCC
500Gbps
Primary
Link
Data Manager
STEP 3: Data manager index
and annotates processed data.
Replicate metadata to GIS.
Allowing data to be search and
retrieved from GIS
Data ManagerCompute Tiered Storage
POLARIS, Genotyping &
other Platforms in L4~L8
Tiered Storage
STEP 1: Sequencers
stream directly to
NSCC Storage
(NO footprint in GIS)
Compute
1 Gbps per
sequencer
10 Gbps
1 Gbps per
machine
100 Gbps
10 Gbps
A*CRC-NSCC
GIS
A*CRC: A*Star Computational Resource Center
GIS: Genome Institute of Singapore
Direct streaming of Sequence Data from GIS
to remote Supercomputer in NSCC
2km

19. The Hardware
EDR Interconnect
• Mellanox EDR Fat Tree
within cluster
• InfiniBand connection
to all end-points (login
nodes) at three
campuses
• 40/80/500 Gbps
throughput network
extend to three
campuses
(NUS/NTU/GIS)
Over13PB Storage
• HSM Tiered, 3 Tiers
• I/O 500 GBps flash
burst buffer , 10x
Infinite Memory
Engine (IME)
~1 PFlops System
• 1,288 nodes (dual
socket, 12 cores/CPU
E5-2690v3)
• 128 GB DDR4 / node
• 10 Large memory
nodes (1x6TB, 4x2TB,
5x 1TB)
19

20. Compute nodes
20
• Large Memory Nodes
– 9 Nodes configured with high memory
– FUJITSU Server PRIMERGY RX4770 M2
– Intel(R) Xeon(R) CPU E7-4830 v3 @
2.10GHz
– 4 x 1 TB, 4x 2 TB, and 1x 6 TB Memory
configuration
– EDR Infiniband
• Standard Compute nodes
– 1160 nodes
– Fujitsu Server PRIMERGY CX2550 M1
– 27840 CPU Cores
– Intel(R) Xeon(R) CPU E5-2690 v3 @
2.60GHz
– 128 GB / Server
– EDR InfiniBand
– Liquid cooling system

21. Accelerate your computing
Accelerators nodes
• 128 nodes with NVIDIA GPUs (identical to the compute
nodes)
• NVIDIA K40 (2880 cores)
• 368,640 total GPU cores
Visualization nodes
• 2 nodes Fujitsu Celsius R940 graphic workstations
• Each with 2 x NVIDIA Quadro K4200
• NVIDIA Quadro Sync support
21

22. NSCC Data Centre – Green features
Warm water cooling for CPUs
– First free-cooling system in Singapore and
South-East Asia.
– Water is maintained at a temperature of
40ºC. Enters the racks at 40ºC, exits the
racks at 45ºC.
– Equipment placed in a technical floor(18th)
cool down the water down only using fans.
– The system can easily be extended for
future expansion.
Green features of Data Centre
– PUE of 1.4 (average for Singapore is above
2.5)
22
Cool-Central® Liquid Cooling
technology

23. Parallel file system
• Components
– Burst Buffer
• 265 TB Burst Buffer
• 500 GB/s throughput
• Infinite Memory Engine (IME)
– Scratch
• 4 PB scratch storage
• 210 GB/s
• SFA12KX EXAScalar storage
• Lustre file system
– home and secure
• 4 PB Persistent storage
• GridScalar storage
• 100 GB/s throughput
• IBM Spectrum Scale (formerly GPFS)
– Archive storage
• 5 PB storage
• Archive purpose only
• WOS based archive system
23

24. IME Architecture
24

25. Tiered File system
25

26. NSCC Storage
26
Tier0
BurstBuffer
Tier0
ScratchFS
Tier1
HomeFS
Tier1
ProjectFS
Tier2
Archive
265 TB
500 GB/s
4 PB
210 GB/s
4 PB
100 GB/s
WOS Active
Archive
Infinite Memory
Engine GRIDScaler
GPFS® Storage
HSM
5PB
20TB/h
EXAScaler Lustre® Storage

27. Software Stack
Operating
System
CentOS 6.6
Scheduler
PBS Pro
Compilers
GCC
Intel Parallel Studio
Libraries
GNU, Intel MKL
Allinea tools
GPGPU CUDA
Toolkit 7.5
Environment
Modules
27

28. PBS Professional (Job Scheduler)
28

29. Why PBS Professional (Scheduler)?
29
 Workload management solution that maximizes the efficiency and
utilization of high-performance computing (HPC) resources and
improves job turnaround
Robust Workload
Management
 Floating licenses
 Scalability, with flexible queues
 Job arrays
 User and administrator interface
 Job suspend/resume
 Application checkpoint/restart
 Automatic file staging
 Accounting logs
 Access control lists
Advanced Scheduling
Algorithms
 Resource-based scheduling
 Preemptive scheduling
 Optimized node sorting
 Enhanced job placement
 Advance & standing reservations
 Cycle harvesting across workstations
 Scheduling across multiple complexes
 Network topology scheduling
 Manages both batch and interactive
work
 BackfillingReliability, Availability and Scalability
 Server failover feature
 Automatic job recovery
 System monitoring
 Integration with MPI solutions
 Tested to manage 1,000,000+ jobs per day
 Tested to accept 30,000 Jobs per minute
 EAL3+ security
 Checkpoint support

30. Process Flow of a PBS Job
1. User submits job
2. PBS server returns a job ID
3. PBS scheduler requests a list of resources from the server *
4. PBS scheduler sorts all the resources and jobs *
5. PBS scheduler informs PBS server which host(s) that job can run on *
6. PBS server pushes job script to execution host(s)
7. PBS MoM executes job script
8. PBS MoM periodically reports resource usage back to PBS server *
9. When job is completed PBS MoM copies output and error files
10. Job execution completed/user notification sent
HOST A HOST B HOST C
PBS SCHEDULER
PBS SERVER
pbsworks
ncpus
mem
host
pbsworks on HOST A
pbsworks
Note: * This information is for debugging purposes
only. It may change in future releases.
30
Cluster Network

31. Compute Manager GUI: Job Submission Page
• Applications panel
– Displays the applications available on the registered PAS server
• Submission Form panel
– Displays a job submission form for the application selecting the Applications panel
• Directory Structure panel
– Displays the directory structure of the location specified in the Address box
– Files panel
– Displays the contents of the directory, files, and subdirectories selected in the Directory Structure panel
31
Directory Structure
Files
Applications

32. Job Queues & Scheduling Policies
32
External
Queue Name
Internal
Queue Name
Walltime
limit
Other limits Remarks
largemem 24 Hours
To be decided For Jobs requiring more that
4GB per core.
normal dev 1 Hours
2 standard
nodes per user
High priority queue for testing
and development works.
small 24 Hours
Up to 24 cores
per job
For jobs that do not require
more than one node.
medium 24 Hours
Up to limit as
per prevailing
policies
For standard job runs
requiring more than one
node.
long 120 Hours
1 node per
user
Low priority queue for jobs
that cannot be checkpointed.
gpu gpunormal 24 Hours
Up to limit as
per prevailing
policies
For “normal” jobs which
require GPU.
gpulong 240 Hours
Up to limit as
per prevailing
policies
Low priority queue for GPU
jobs which cannot be
checkpointed.

33. Job Queues & Scheduling Policies
33
External
Queue Name
Internal
Queue Name
Walltime
limit
Other limits Remarks
iworkq 8 Hours
1 node per
user
For visualisation.
ime
(look for it in
the near
future)
24 Hours
Up to limit as
per prevailing
policies
For users who wish to
experiment with DDN's IME
burst buffer which offers up
to 500GB/s of transfer speed
* Users only need to specify the 'External Queue' for job submission. Jobs will be routed
to the internal queue depending on the job resource requirements.

34. Compilers & Libraries
34

35. 35
Compilers and Libraries at a glance

36. Parallel programming OpenMP
• Available compilers (gcc/gfortran/icc/ifort)
– OpenMP (not openmpi, Used mainly in SMP programming)
• OpenMP (Open Multi-Processing)
• OpenMP is an approach and OpenMPI is an implementation of MPI
• An API for shared-memory parallel programming in C/C++ and Fortran
• Parallelization in OpenMP achieved through threads
• Programming OpenMP is easier as it involves only pragma directive
• OpenMP program cannot communicate to the processor over network
• Different stages of the program uses different number of threads
• A typical approach is demonstrated through the below image
36

37. Parallel Programming MPI
• MPI
– MPI stands for Messaging Passing Interface
– MPI is a library specification
– MPI implementation is typically a wrapper to standard
compilers such as C/Fortran/Java/Python
– Typically used in Distributed memory communication
37

38. 38
Developer Tools

39. 39
Allinea DDT
• DDT – Distributed Debugging tool from Allinea
• Graphical interface for debugging
– Serial applications/codes
– OpenMP applications/codes
– MPI applications/codes
– CUDA applications/codes
• You control the pace of the code execution and examine
execution flow and variables
• Typical Scenario
– Set a point in your code where you want execution to stop
– Let your code run until the point is reached
– Check the variables of concern

40. 40
Allinea MAP
• MAP – Application Profiling tool from Allinea
• Graphical interface for profilling
– Serial applications/codes
– OpenMP applications/codes
– MPI applications/codes

41. 41
Allinea MAP
• Running your code with MAP
– $ module load impi/5.1.2
– $ mpiicc -g -O0 -o wave_c wave_c.c
– $ module load map/a.b.c
– $ map mpiexec –n 4 ./wave_c 20

42. 42
Allinea MAP

43. 43
Co-processor / Accelerators

44. GPU
• GPUs – Graphic Processing Units were initially made to
render better graphics performance
• With the amount of research put on GPUs, it was
identified that GPUs can perform better with Floating
Point Operations as well
• The term GPU changed to GPGPUs (General Purpose
GPUs)
• CUDA Toolkit includes compiler, math libraries, tools, and
debuggers
44

45. GPU in NSCC
• GPU Configuration
– Total 128 GPU nodes
– Each server with 1 Tesla K40 GPU
– 128 GB host memory per server
– 12GB device memory
– 2880 CUDA Cores
• Connect to GPU server
– To compile GPU application:
• Submit interactive job requesting for GPU resource
• Compile job using NVCC compiler
– To submit GPU job
• Flexible to among qsub for login nodes
• OR login to compute manager
45

46. 46
Environment Modules

47. What is Environment modules
• Environment modules helps to dynamically load/unload
environment variables such as PATH, LD_LIBRARY_PATH,
etc.,
• Environment modules are based on module files which
are written in TCL language
• Environment modules are shell independent
• Helpful to maintain different version of same software
• Flexibility to create module files by the users
47

48. Applications
48

49. Molecular Dynamics
Computational Chemistry
Compatible Applications
49

50. Compatible Applications
Engineering Applications
Quasiparticle calculationQuantum Chemistry
Numerical Analysis Weather research
50

51. August 27, 2015 51
https://help.nscc.sg/software-list/

52. Managed Services offered
53
• Computational resources
• Storage management
Infrastructure Services
• Hardware break fix
• Software incident resolution
Incident Resolution
• Data management
• Job management
• Software installation etc.,
General Service Requests
• Code Optimization
• Special queue configuration, etc.
Specialized Service Requests
• Introductory class
• Code optimization techniques
• Parallel Profiling etc.
Training Services
• Portal/e-Mail/Phone
• Request for a service via portal
• Interactive Job submission portal
Helpdesk

53. Where is NSCC
• NSCC Petascale
supercomputer in
Connexis building
• 40Gbps links extended to
NUS, NTU and GIS
• Login nodes are placed in
NUS, NTU and GIS
datacenters
• Access to NSCC is just
like your local HPC
system
54
1 Fusionopolis Way, Level-17 Connexis South
Tower, Singapore 138632

54. Supported Login methods
• How do I login
– SSH
From a Windows PC use Putty or any standard SSH client software hostname is
nscclogin.nus.edu.sg, use NSCC Credentials
From Linux machine, use ssh username@nus.nscc.sg
From MAC, open terminal and ssh username@nus.nscc.sg
– File Transfer
SCP or any other secure shell file transfer software from Windows
Use the command scp to transfer files from MAC/Linux
– Compute Manager / Display Manager
Open any standard web browser
In the address bar, type https://nusweb.nscc.sg
Use NSCC credentials to login
– Outside campus
Connect to Campus VPN to gain above mentioned services
55

55. NSCC HPC Support (Proposed to be available by 15th Mar)
• Corporate Info – web portal
http://nscc.sg
• NSCC HPC web portal
http://help.nscc.sg
• NSCC support email
help@nscc.sg
• NSCC Workshop portal
http://workshop.nscc.sg
56

56. 57
Help us improve. Take the online survey!
Visit: http://workshop.nscc.sg >> Survey

57. Help portal
58
FAQs of
NSCC
Enroll to
NSCC
https://help.nscc.sg/

58. Registration Procedures
59

59. Registration Procedure
60

60. Web Site : http://nscc.sg
Helpdesk : https://help.nscc.sg
Email : help@nscc.sg
Phone : +65 6645 3412
61

62. User Enrollment
Instructions:
• Open https://help.nscc.sg
• Navigate User services -> Enrollment
• Click on Login
• Select your organization (NUS/NTU/A*Star) from the
drop down
• Input your credentials
Ref: https://help.nscc.sg -> User Guides -> User Enrollment guide
63

63. Login to NSCC Login nodes
• Download Putty form internet
• Open Putty
• Type login server name (login.nscc.sg)
• Input your credentials to login
64

64. Compute manager
• Open Web Browser (Firefox or IE)
• Type https://nusweb.nscc.sg / https://ntuweb.nscc.sg /
https://loginweb-astar.nscc.sg
• Use your credentials to login
• Submit a sample job
65

65. Transfer files
• Use FileZilla to transfer files
66

66. Creating PBS Job submission script
• Use the below sample script
cat submit.pbs
#!/bin/bash
#PBS -q dev
#PBS -l select=1:ncpus=24:mpiprocs=24
#PBS -l place=scatter
cd ${PBS_O_WORKDIR}
sleep 30
qsub submit.pbs
67

67. Environment module
• Open Putty
• Type module avail
• Type module load
68

68. Compiling simple C Program
• Use putty to login
• Create helloworld.c
#include<stdio.h>
void main()
{
printf("Helloworldn");
}
• Use module load composerxe/2016.1.150
• Type icc heloworld.c -o helloworld.o
69

69. Submit job
cat submit.pbs
#!/bin/bash
#PBS -q dev
#PBS -l select=1:ncpus=1
cd ${PBS_O_WORKDIR}
./helloworld.o
70

70. Compiling mpi C Program
• Use putty to login
• Create helloworld.c
#include <mpi.h>
#include <stdio.h>
#include <string.h>
#include <mpi.h>
#include <stdio.h>
#include <unistd.h>
int main(int argc, char **argv)
{
int rank;
char hostname[256];
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
gethostname(hostname,255);
printf("Hello world! I am process number: %d on host %sn", rank,
hostname);
MPI_Finalize();
return 0;
}
• Use module load composerxe/2016.1.150
• Type icc heloworld.c -o mpihello.o
71

71. Submit job
cat submit.pbs
#!/bin/bash
#PBS -q dev
#PBS -l select=1:ncpus=24:mpiprocs=24
#PBS –l place=scatter
cd ${PBS_O_WORKDIR}
mpirun ./mpihello.o
72

72. Submit pre-compiled applicatin
73
cat submit.pbs
#!/bin/bash
#PBS -q dev
#PBS -l select=1:ncpus=24:mpiprocs=24
#PBS –l place=scatter
cd ${PBS_O_WORKDIR}
mpirun ./mpihello.o

73. Using Scratch space
#!/bin/bash
#PBS -N My_Job
# Name of the job
#PBS -l select=1:ncpus=24:mpiprocs=24
# Setting number of nodes and CPUs to use
#PBS -W sandbox=private
# Get PBS to enter private sandbox
#PBS -W stagein=file_io@wlm01:/home/adm/sup/fsg1/<my input directory>
# Directory name where all the input files are alvailable
# files in the input directory will be copied to scratch space creating a directory file_io
#PBS -W stageout=*@wlm01:/home/adm/sup/fsg1/<myoutput directory>
# Output directory path in my home directory
# Once the job is finished, the files from file_io in scratch will be copied back to <myoutput
directory>
#PBS -q normal
cd ${PBS_O_WORKDIR}
echo " PBS_WORK_DIR is : $PBS_O_WORKDIR"
echo "PBS JOB DIR is: $PBS_JOBDIR"
#Notice that the output of pwd will be in lustre scratch space
echo "PWD is : `pwd`"
sleep 30
#mpirun ./a.out < input_file > output_file
74