HPC_Week_01.pdf

HIGH PERFORMANCE
COMPUTING
18 BS(CS)
DEPARTMENT OF COMPUTER SCIENCE
Engr. MIR MUHAMMAD
mir.juno@quest.edu.pk

Course Objective
 The goal of this course is to provide a survey of high
performance computing.
 The course will discuss topics related to accessing high
performance computing resources, developing applications
for those resources, and executing developed applications.
Wednesday, September 15,
2021
2

Learning Aspects..
 The students will be able to design, analysis, and
implementation, of high performance computational
science and engineering applications.
 The course focuses on advanced computer architectures,
parallel algorithms, parallel languages, and performance-
oriented computing
2021
3

Books
 TEXT BOOKS:
 ”Introduction to Parallel Processing”, Latest Edition,
By: M Sasikumar, Dinesh Shikhare, P.Ravi Prakash
 “Parallel Programming in C with MPI and OpenMP”
Latest Edition, By: Michael J. Quinn
 “Introduction to Parallel Computing”, Latest Edition,
By: Annanth Grama, Anshul Gupta, George karypis,
Vipin Kumar
2021
4

Books
 REFERENCE BOOKS:
 “Grid Computing: A practical Guide to Technology
and Applications”, Latest Edition,
By: Ahmer Abbas
2021
5

Course Contents
 INTRODUCTION TO HPC:
 Importance of HPC in Science and Engineering, Introduction to
Parallel Computers and Computational Overheads
 ARCHITECTURES AND BUILDING BLOCKS:
 Supercomputer Architecture, Vector Machines, Parallel
Processors, Data Parallel Processors, Single-Instruction-Multiple-
Data, Multiple-Instruction-Multiple-Data, Pipelining, Vectorization,
Parallelization, Comparison of Serial, Parallel and Vector
Architectures
2021
6

Course Contents
 PERFORMANCE MEASURES AND MODELS:
 Speed-up limitations, Theoretical Timings, Efficiencies, Amdahl's
Law and Extensions, Scaled Speed-up, Pipeline Speed-up, Data
Dependency Reduction. Data flow, Loop reordering, Loop
optimizations, Implementation Principal of Locality, Caches and
Buffers.
 PARALLEL ALGORITHMS:
 Purely Parallel Algorithms, Parallelization of Algorithms, Parallel
linear algebra routines, Cyclic Reduction; Iterative Algorithms
(Jacobi, Gauss-Seidel and Red-Black Orderings); Divide-and-Conquer
Algorithms, Adaptive Quadrature, Correct Termination, Block
Decomposition Methods, Massively Data Parallel Algorithms, Array
notation.
2021
7

Course Contents
 PARALLEL PROGRAMMING PACKAGES:
 Parallel Languages and Compilers, Fortran90 and HPC Fortran,
Parallel and Vector C Code, Massage passing interface (MPI),
Parallel Virtual Machine (PVM), Layout, Align, Replicate, Masking,
Shifting, Spreading, Broadcasting, Forall Loops, Levels of
Abstraction, Models of Computation and Parallel Overheads, Levels
of Abstraction, Multiple Program Counters in Hardware; Multi-
Thread Models, with Primary Sources of Overhead;; Task-Parallel
versus Data-Parallel Programming Models; Further Sources of
Overhead; Loop Transformations; Data Transformations;
Dependence Analysis; Compiler Strategies.
2021
8

Course Contents
 RESULTS AND BENCHMARKING:
 Memory Architecture and Memory Access Times and Associated
Sources of Overhead; Multi-Process Execution Model; Performance
Tuning via Overhead Reduction; Task Scheduling; Data Partitioning
and its Effect on Performance.
 Restructuring for Parallel Performance
2021
9

Let us begin …
2021
10

Introduction to High-Performance Computing
 What is High Performance Computing?
 High‐speed computing. Originally implemented only in
supercomputers for scientific research
 Used for scientific research and computational science
 Main area of discipline is developing parallel processing
algorithms and software so that programs can be divided into
small independent parts and can be executed simultaneously by
separate processors
 HPC systems have shifted from supercomputer to computing
clusters
2021
11

Introduction to High-Performance
Computing
2021
12

What is Cluster
Cluster is a group of machines interconnected
in a way that they work together as a single
system
 Terminology
 Node – individual machine in a cluster
 Head/Master node – connected to both
the private network of the cluster and a
public network and are used to access a
given cluster. Responsible for providing
user an environment to work and
distributing task among other nodes
 Compute nodes – connected to only the
private network of the cluster and are
generally used for running jobs assigned
to them by the head node(s)
2021
13

Types of Cluster
 Storage
 Storage clusters provide a consistent file system image
 Allowing simultaneous read and write to a single shared file system
 High‐availability (HA)
 ƒ
Provide continuous availability of services by eliminating single
points of failure
 Load‐balancing
 ƒ
Sends network service requests to multiple cluster nodes to balance
the requested load among the cluster nodes
 High‐performance
 ƒ
Use cluster nodes to perform concurrent calculations
 Allows applications to work in parallel to enhance the performance
of the applications
 Also referred to as computational clusters or grid computing
2021
14

Benefits of Cluster
 Reduced Cost
 The price of off‐the‐shelf consumer desktops has fallen in recent years,
and this drop in price has corresponded with a vast increase in their
processing power and performance. The average desktop PC today is
many times more powerful than the first mainframe computers.
 Processing Power
 The parallel processing power of a high‐performance cluster can, in
many cases, prove more cost effective than a mainframe with similar
power. This reduced price‐per‐unit of power enables enterprises to get
a greater ROI(Return On Investment).
 Scalability
 Perhaps the greatest advantage of computer clusters is the scalability
they offer. While mainframe computers have a fixed processing
capacity, computer clusters can be easily expanded as requirements
change by adding additional nodes to the network
2021
15

Benefits of Cluster
 Improved Network Technology
 In clusters, computers are typically connected via a single virtual
local area network (VLAN), and the network treats each
computer as a separate node. Information can be passed
throughout these networks with very little lag, ensuring that
data doesn’t bottleneck between nodes.
 Availability
 When a mainframe computer fails, the entire system fails.
However, if a node in a computer cluster fails, its operations can
be simply transferred to another node within the cluster,
ensuring that there is no interruption in service
2021
16

When We Need High Performance Computing?
 Case -1: Complete a time-consuming operation in less time
 I am an automotive engineer
 I need to design a new car that consumes less gasoline
 I’d rather have the design completed in 6 months than in 2 years
 I want to test my design using computer simulations rather than
building very expensive prototypes and crashing them
 Case 2: Complete an operation under a tight deadline
 I work for a weather prediction agency
 I am getting input from weather stations/sensors
 I’d like to predict tomorrow’s forecast today
2021
17

When We Need High Performance Computing?
 Case 3: Perform a high number of operations per seconds
 I am an engineer at Amazon.com
 My Web server gets 1,000 hits per seconds
 I’d like my web server and databases to handle 1,000 transactions
per seconds so that customers do not experience bad delays
2021
18

Where are we using HPC?
 Used to solve complex modeling problems in a spectrum of disciplines
 Topics include:
o Artificial intelligence o Physical oceanography
o Climate modeling o Plasma physics
o Automotive engineering o Quantum physics
o Cryptographic analysis o Quantum chemistry
o Geophysics o Nuclear physics
o Molecular biology o Solid state physics
o Molecular dynamics o Structural dynamics
 HPC is currently applied to business uses as well
 data warehouses
 transaction processing
2021
19

 From concept to engineering, from design to test and
manufacturing, from weather prediction to medical discoveries, our
day to day life depends more and more on HPC simulations
 Safer products, accurate predictions, research, etc.
 High-performance compute clusters provide the most efficient,
flexible, cost effective HPC environment for any HPC simulation
2021
20

Changing times
 From 1986 – 2002, microprocessors were speeding like a rocket,
increasing in performance an average of 50% per year
 Since then, it’s dropped to about 20% increase per year..
2021
21

The Problem
 Up to now, performance increases have been
attributed to increasing density of transistors
 But there are inherent problems
 A little Physics lesson:
 Smaller transistors = faster processors
 Faster processors = increased power consumption
 Increased power consumption = increased heat
 Increased heat = unreliable processors
2021
22

An intelligent solution
 Move away from single‐core systems to multicore processors
 “core” = processing unit
 Introduction of parallelism..!!
 But …
 Adding more processors doesn’t help much if programmers
aren’t aware of them..!!
 … or don’t know how to use them
 Serial programs don’t benefit from this approach (in most
cases)
2021
23

Parallel Computing
 Form of computation in which many calculations are carried out
simultaneously, operating on the principle that large problems can
often be divided into smaller ones, which are then solved
concurrently i.e. "in parallel“
 So, we need to rewrite serial programs so that they’re parallel.
 Write translation programs that automatically convert serial progra
ms into parallel programs.
 This is very difficult to do.
 Success has been limited
2021
24

Parallel Computing
 Example
 Compute n values and add them together.
 Serial solution:
2021
25

Parallel Computing
 Example
 We have p cores, p much smaller than n.
 Each core performs a partial sum of approximately n/p values.
2021
26

Parallel Computing
 Example
 After each core completes execution of the code, is a private
variable my_sum contains the sum of the values computed by
its calls to Compute_next_value.
 Ex., n = 200, then
• Serial – will take 200 addition
• Parallel (for 8 cores)
– Each core will perform n/p = 25 addition
– And master will perform 8 more addition + 8 receive
operation
– Total 41 operation.
2021
27

Parallel Computing
 Some coding constructs can be recognized by an automatic program
generator, and converted to a parallel construct.
 However, it’s likely that the result will be a very inefficient program.
 Sometimes the best parallel solution is to step back and devise an en
tirely new algorithm.
 Parallel computer programs are more difficult to write than
sequential programs
 Potential problems
 Race condition (output depending on sequence or timing of
other events)
 Communication and synchronization between the different
subtasks
2021
28

HPC_Week_01.pdf

More Related Content

Similar to HPC_Week_01.pdf

Recently uploaded

HPC_Week_01.pdf