NSCC Training - Introductory Class
•
1 like•3,761 views
The National Supercomputing Centre (NSCC) operates a 1 petaflop supercomputing cluster with 1300 nodes for research and industry users. It has 13 petabytes of storage including a 265TB burst buffer and uses a Mellanox 100Gbps network. The cluster scheduler is PBS Pro and supports applications in areas like engineering, science, life sciences and more.
![NSCC High
Performance
Computing Cluster
Introduction
[16-May-2016]](https://image.slidesharecdn.com/nscctrainingintroductoryclassmay16v-160518101611/85/NSCC-Training-Introductory-Class-1-320.jpg)
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Viewers also liked
Viewers also liked (20)
Similar to NSCC Training - Introductory Class
Similar to NSCC Training - Introductory Class (20)
Recently uploaded
Recently uploaded (20)
NSCC Training - Introductory Class
- 2. • Introduction to NSCC • About HPC • More about NSCC HPC cluster • PBS Pro (Scheduler) • Compilers and Libraries • DeveloperTools • Co-processor / Accelerators • Environment Modules • Applications • User registration procedures • Feedback 2 The Discussion
- 4. • State-of-the-art national facility with computing, data and resources to enable users to solve science and technological problems, and stimulate industry to use computing for problem solving, testing designs and advancing technologies. • Facility will be linked by high bandwidth networks to connect these resources and provide high speed access to users anywhere and everyone. Introduction: The National Supercomputing Centre (NSCC) 4
- 5. Introduction:Vision & Objectives Vision:“Democratising Access to Supercomputing” 5 Making Petascale Supercomputing accessible to the ordinary researcher 1 Bringing Petascale Computing and Storage and Gigabit speed networking to the ordinary person 2 Supporting National R&D Initiatives1 Objectives of NSCC Attracting Industrial Research Collaborations2 Enhancing Singapore’s Research Capabilities3
- 7. 7 What is HPC? • Term HPC stands for High Performance Computing or High Performance Computer • Tightly coupled personal computers with high speed interconnect • Measured in FLOPS (FLoating point Operations Per Second) • Architectures – NUMA (Non-uniform memory access)
- 8. Major Domains where HPC is used Engineering Analysis • Fluid Dynamics • Materials Simulation • Crash simulations • Finite Element Analysis Scientific Analysis • Molecular modelling • Computational Chemistry • High energy physics • Quantum Chemistry Life Sciences • Genomic Sequencing and Analysis • Protein folding • Drug design • Metabolic modelling Seismic analysis • Reservoir Simulations and modelling • Seismic data processing 8
- 9. Major Domains where HPC is used Chip design & Semiconductor • Transistor simulation • Logic Simulation • Electromagnetic field solver Computational Mathematics • Monte-Carlo methods • Time stepping and parallel time algorithms • Iterative methods Media and Animation • VFX and visualization • Animation Weather research • Atmospheric modelling • Seasonal time- scale research • - Major Domains where HPC is used 9
- 10. Major Domains where HPC is used • And More – Bigdata – Information Technology – Cyber security – Banking and Finance – Data mining 10
- 11. 11 Introduction to NSCC HPC Cluster
- 12. Executive Summary • 1 Petaflop System – About 1300 nodes – Homogeneous and Heterogeneous architectures • 13 Petabytes of Storage – One of the Largest and state of the art Storage architecture • Research and Industry – A*STAR, NUS, NTU, SUTD – And many more commercial and academic organizations 12
- 13. HPC Stack in NSCC Mellanox 100 Gbps Network Intel Parallel studio Allinea Tools PBSPro Scheduler Lustre & GPFS HPC Application software Operating System RHEL 6.6 and CentOS 6.6 Fujitsu x86 Servers NVidia Tesla K40 GPUDDN Storage Application Modules 13
- 14. 14 NSCC Supercomputer Architecture Base Compute Nodes (1160 nodes) Accelerated Nodes (128 nodes) Parallel File system / Tiered storage InfiniBand network - Fully non- blocking Ethernet NW GIS FAT node NUS Peripheral Servers NTU Peripheral Servers NSCC Peripheral Servers NSCC Direct users VPN
- 17. 17 Genomic Institute of Singapore (GIS) National Supercomputing Center (NSCC) 2km Connection between GIS and NSCC Large memory node (1TB), Ultra high speed 500Gbps enabled 2012: 300 Gbytes/week 2015: 4300 Gbytes/week x 14
- 18. NGSP Sequencers at B2 (Illumina + PacBio) NSCC Gateway STEP 2: Automated pipeline analysis once sequencing completes. Processed data resides in NSCC 500Gbps Primary Link Data Manager STEP 3: Data manager index and annotates processed data. Replicate metadata to GIS. Allowing data to be search and retrieved from GIS Data ManagerCompute Tiered Storage POLARIS, Genotyping & other Platforms in L4~L8 Tiered Storage STEP 1: Sequencers stream directly to NSCC Storage (NO footprint in GIS) Compute 1 Gbps per sequencer 10 Gbps 1 Gbps per machine 100 Gbps 10 Gbps A*CRC-NSCC GIS A*CRC: A*Star Computational Resource Center GIS: Genome Institute of Singapore Direct streaming of Sequence Data from GIS to remote Supercomputer in NSCC 2km
- 19. The Hardware EDR Interconnect • Mellanox EDR Fat Tree within cluster • InfiniBand connection to all end-points (login nodes) at three campuses • 40/80/500 Gbps throughput network extend to three campuses (NUS/NTU/GIS) Over13PB Storage • HSM Tiered, 3 Tiers • I/O 500 GBps flash burst buffer , 10x Infinite Memory Engine (IME) ~1 PFlops System • 1,288 nodes (dual socket, 12 cores/CPU E5-2690v3) • 128 GB DDR4 / node • 10 Large memory nodes (1x6TB, 4x2TB, 5x 1TB) 19
- 20. Compute nodes 20 • Large Memory Nodes – 9 Nodes configured with high memory – FUJITSU Server PRIMERGY RX4770 M2 – Intel(R) Xeon(R) CPU E7-4830 v3 @ 2.10GHz – 4 x 1TB, 4x 2TB, and 1x 6TB Memory configuration – EDR Infiniband • Standard Compute nodes – 1160 nodes – Fujitsu Server PRIMERGY CX2550 M1 – 27840 CPU Cores – Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz – 128 GB / Server – EDR InfiniBand – Liquid cooling system
- 21. Accelerate your computing Accelerators nodes • 128 nodes with NVIDIA GPUs (identical to the compute nodes) • NVIDIA K40 (2880 cores) • 368,640 total GPU cores Visualization nodes • 2 nodes Fujitsu Celsius R940 graphic workstations • Each with 2 x NVIDIA Quadro K4200 • NVIDIA Quadro Sync support 21
- 22. NSCC Data Centre – Green features Warm water cooling for CPUs – First free-cooling system in Singapore and South-East Asia. – Water is maintained at a temperature of 40ºC. Enters the racks at 40ºC, exits the racks at 45ºC. – Equipment placed in a technical floor(18th) cool down the water down only using fans. – The system can easily be extended for future expansion. Green features of Data Centre – PUE of 1.4 (average for Singapore is above 2.5) 22 Cool-Central® Liquid Cooling technology
- 23. Parallel file system • Components – Burst Buffer • 265TB Burst Buffer • 500 GB/s throughput • Infinite Memory Engine (IME) – Scratch • 4 PB scratch storage • 210 GB/s • SFA12KX EXAScalar storage • Lustre file system – home and secure • 4 PB Persistent storage • GridScalar storage • 100 GB/s throughput • IBM Spectrum Scale (formerly GPFS) – Archive storage • 5 PB storage • Archive purpose only • WOS based archive system 23
- 26. NSCC Storage 26 Tier0 BurstBuffer Tier0 ScratchFS Tier1 HomeFS Tier1 ProjectFS Tier2 Archive 265 TB 500 GB/s 4 PB 210 GB/s 4 PB 100 GB/s WOS Active Archive Infinite Memory Engine GRIDScaler GPFS® Storage HSM 5PB 20TB/h EXAScaler Lustre® Storage
- 27. Software Stack Operating System CentOS 6.6 Scheduler PBS Pro Compilers GCC Intel Parallel Studio Libraries GNU, Intel MKL Allinea tools GPGPU CUDA Toolkit 7.5 Environment Modules 27
- 28. PBS Professional (Job Scheduler) 28
- 29. Why PBS Professional (Scheduler)? 29 Workload management solution that maximizes the efficiency and utilization of high-performance computing (HPC) resources and improves job turnaround RobustWorkload Management Floating licenses Scalability, with flexible queues Job arrays User and administrator interface Job suspend/resume Application checkpoint/restart Automatic file staging Accounting logs Access control lists Advanced Scheduling Algorithms Resource-based scheduling Preemptive scheduling Optimized node sorting Enhanced job placement Advance & standing reservations Cycle harvesting across workstations Scheduling across multiple complexes Network topology scheduling Manages both batch and interactive work Backfilling Reliability,Availability and Scalability Server failover feature Automatic job recovery System monitoring Integration with MPI solutions Tested to manage 1,000,000+ jobs per day Tested to accept 30,000 Jobs per minute EAL3+ security Checkpoint support
- 30. Process Flow of a PBS Job 1. User submits job 2. PBS server returns a job ID 3. PBS scheduler requests a list of resources from the server * 4. PBS scheduler sorts all the resources and jobs * 5. PBS scheduler informs PBS server which host(s) that job can run on * 6. PBS server pushes job script to execution host(s) 7. PBS MoM executes job script 8. PBS MoM periodically reports resource usage back to PBS server * 9.When job is completed PBS MoM copies output and error files 10. Job execution completed/user notification sent HOST A HOST B HOST C PBS SCHEDULER PBS SERVER pbsworks ncpus mem host pbsworks on HOST A pbsworks Note: * This information is for debugging purposes only. It may change in future releases. 30 Cluster Network
- 31. Compute Manager GUI: Job Submission Page • Applications panel – Displays the applications available on the registered PAS server • Submission Form panel – Displays a job submission form for the application selecting the Applications panel • Directory Structure panel – Displays the directory structure of the location specified in the Address box – Files panel – Displays the contents of the directory, files, and subdirectories selected in the Directory Structure panel 31 Directory Structure Files Applications
- 32. Job Queues & Scheduling Policies 32 Queue Name Queue type Job run time limit No of cores available Description Long Batch 240 Hours 1024 Jobs are expected to run longer time Development Interactive 24 Hours 48 Coding, profiling and debugging Normal Default Batch 3 Days 27000 Default queue Large Memory Batch - 360 Jobs dispatched based on memory requirement GPU GPU batch - 368,640 (CUDA) Specific for GPU jobs Visualization Interactive 8 Hours 1 High end graphics card Production Batch - 480 Cores GIS queue
- 34. 34 Compilers and Libraries at a glance
- 35. Parallel programming OpenMP • Available compilers (gcc/gfortran/icc/ifort) – OpenMP (not openmpi, Used mainly in SMP programming) • OpenMP (Open Multi-Processing) • OpenMP is an approach and OpenMPI is an implementation of MPI • An API for shared-memory parallel programming in C/C++ and Fortran • Parallelization in OpenMP achieved through threads • Programming OpenMP is easier as it involves only pragma directive • OpenMP program cannot communicate to the processor over network • Different stages of the program uses different number of threads • A typical approach is demonstrated through the below image 35
- 36. Parallel Programming MPI • MPI – MPI stands for Messaging Passing Interface – MPI is a library specification – MPI implementation is typically a wrapper to standard compilers such as C/Fortran/Java/Python – Typically used in Distributed memory communication 36
- 38. 38 Allinea DDT • DDT – Distributed Debugging tool from Allinea • Graphical interface for debugging – Serial applications/codes – OpenMP applications/codes – MPI applications/codes – CUDA applications/codes • You control the pace of the code execution and examine execution flow and variables • Typical Scenario – Set a point in your code where you want execution to stop – Let your code run until the point is reached – Check the variables of concern
- 39. 39 Allinea MAP • MAP – Application Profiling tool from Allinea • Graphical interface for profilling – Serial applications/codes – OpenMP applications/codes – MPI applications/codes
- 40. 40 Allinea MAP • Running your code with MAP – $ module load impi/5.1.2 – $ mpiicc -g -O0 -o wave_c wave_c.c – $ module load map/a.b.c – $ map mpiexec –n 4 ./wave_c 20
- 41. 41 Allinea MAP
- 43. GPU • GPUs – Graphic Processing Units were initially made to render better graphics performance • With the amount of research put on GPUs, it was identified that GPUs can perform better with Floating Point Operations as well • The term GPU changed to GPGPUs (General Purpose GPUs) • CUDAToolkit includes compiler, math libraries, tools, and debuggers 43
- 44. GPU in NSCC • GPU Configuration – Total 128 GPU nodes – Each server with 1 Tesla K40 GPU – 128 GB host memory per server – 12GB device memory – 2880 CUDA Cores • Connect to GPU server – To compile GPU application: • Submit interactive job requesting for GPU resource • Compile job using NVCC compiler – To submit GPU job • Flexible to among qsub for login nodes • OR login to compute manager 44
- 46. What is Environment modules • Environment modules helps to dynamically load/unload environment variables such as PATH, LD_LIBRARY_PATH, etc., • Environment modules are based on module files which are written in TCL language • Environment modules are shell independent • Helpful to maintain different version of same software • Flexibility to create module files by the users 46
- 47. Applications 47
- 48. Molecular Dynamics Computational Chemistry Compatible Applications 48
- 49. Compatible Applications Engineering Applications Quasiparticle calculationQuantum Chemistry Numerical Analysis Weather research 49
- 50. August 27, 2015 50 https://help.nscc.sg/software-list/
- 51. Managed Services offered 52 • Computational resources • Storage management Infrastructure Services • Hardware break fix • Software incident resolution Incident Resolution • Data management • Job management • Software installation etc., General Service Requests • Code Optimization • Special queue configuration, etc. Specialized Service Requests • Introductory class • Code optimization techniques • Parallel Profiling etc. Training Services • Portal/e-Mail/Phone • Request for a service via portal • Interactive Job submission portal Helpdesk
- 52. Where is NSCC • NSCC Petascale supercomputer in Connexis building • 40Gbps links extended to NUS, NTU and GIS • Login nodes are placed in NUS, NTU and GIS datacenters • Access to NSCC is just like your local HPC system 53 1 Fusionopolis Way, Level-17 Connexis South Tower, Singapore 138632
- 53. Supported Login methods • How do I login – SSH From aWindows PC use Putty or any standard SSH client software hostname is nscclogin.nus.edu.sg, use NSCC Credentials From Linux machine, use ssh username@login-astar.nscc.sg / ssh username@login-astar.nscc.sg From MAC, open terminal and ssh username@login-astar.nscc.sg / ssh username@login-astar.nscc.sg – File Transfer SCP or any other secure shell file transfer software fromWindows Use the command scp to transfer files from MAC/Linux – Compute Manager Open any standard web browser In the address bar, type https://loginweb-astar.nscc.sg Use NSCC credentials to login – Outside campus Connect to CampusVPN gain above mentioned services 54
- 54. NSCC HPC Support (Proposed to be available by 15th Mar) • Corporate Info – web portal http://nscc.sg • NSCC HPC web portal http://help.nscc.sg • NSCC support email help@nscc.sg • NSCC Workshop portal http://workshop.nscc.sg 55
- 55. 56 Help us improve.Take the online survey! Visit: http://workshop.nscc.sg >> Survey
- 56. Help portal 57 FAQs of NSCC Enroll to NSCC https://help.nscc.sg/
- 59. Web Site : http://nscc.sg Helpdesk : https://help.nscc.sg Email : help@nscc.sg Phone : +65 6645 3412 60
- 61. User Enrollment Instructions: • Open https://help.nscc.sg • Navigate User services -> Enrollment • Click on Login • Select your organization (NUS/NTU/A*Star) from the drop down • Input your credentials Ref: https://help.nscc.sg -> User Guides -> User Enrollment guide 62
- 62. Login to NSCC Login nodes • Download Putty form internet • Open Putty • Type login server name (login.nscc.sg) • Input your credentials to login 63
- 63. Compute manager • OpenWeb Browser (Firefox or IE) • Type https://nusweb.nscc.sg / https://ntuweb.nscc.sg / https://loginweb-astar.nscc.sg • Use your credentials to login • Submit a sample job 64
- 64. Transfer files • Use FileZilla to transfer files 65
- 65. Creating PBS Job submission script • Use the below sample script cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS -l place=scatter cd ${PBS_O_WORKDIR} sleep 30 qsub submit.pbs 66
- 66. Environment module • Open Putty • Type module avail • Type module load 67
- 67. Compiling simple C Program • Use putty to login • Create helloworld.c #include<stdio.h> void main() { printf("Helloworldn"); } • Use module load composerxe/2016.1.150 • Type icc heloworld.c -o helloworld.o 68
- 68. Submit job cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=1 cd ${PBS_O_WORKDIR} ./helloworld.o 69
- 69. Compiling mpi C Program • Use putty to login • Create helloworld.c #include <mpi.h> #include <stdio.h> #include <string.h> #include <mpi.h> #include <stdio.h> #include <unistd.h> int main(int argc, char **argv) { int rank; char hostname[256]; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); gethostname(hostname,255); printf("Hello world! I am process number: %d on host %sn", rank, hostname); MPI_Finalize(); return 0; } • Use module load composerxe/2016.1.150 • Type icc heloworld.c -o mpihello.o 70
- 70. Submit job cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS –l place=scatter cd ${PBS_O_WORKDIR} mpirun ./mpihello.o 71
- 71. Submit pre-compiled applicatin 72 cat submit.pbs #!/bin/bash #PBS -q dev #PBS -l select=1:ncpus=24:mpiprocs=24 #PBS –l place=scatter cd ${PBS_O_WORKDIR} mpirun ./mpihello.o
- 72. Using Scratch space #!/bin/bash #PBS -N My_Job # Name of the job #PBS -l select=1:ncpus=24:mpiprocs=24 # Setting number of nodes and CPUs to use #PBS -W sandbox=private # Get PBS to enter private sandbox #PBS -W stagein=file_io@wlm01:/home/adm/sup/fsg1/<my input directory> # Directory name where all the input files are alvailable # files in the input directory will be copied to scratch space creating a directory file_io #PBS -W stageout=*@wlm01:/home/adm/sup/fsg1/<myoutput directory> # Output directory path in my home directory # Once the job is finished, the files from file_io in scratch will be copied back to <myoutput directory> #PBS -q normal cd ${PBS_O_WORKDIR} echo " PBS_WORK_DIR is : $PBS_O_WORKDIR" echo "PBS JOB DIR is: $PBS_JOBDIR" #Notice that the output of pwd will be in lustre scratch space echo "PWD is : `pwd`" sleep 30 #mpirun ./a.out < input_file > output_file 73