Download to read offline
![Ten years ago, when GPUs were rst used to perform general-purpose computation, they were programmed using
low-level mechanism such as the interruption services of the BIOS, or by using graphic APIs such as OpenGL and
DirectX [16]. Later, the programs for GPU were developed in assembly language for each card model, and they had
very limited portability. So, high-level languages were developed to fully exploit the capabilities of the GPUs. In
2007, NVIDIA introduced CUDA [25], a software architecture for managing the GPU as a parallel computing device
without requiring to map the data and the computation into a graphic API. CUDA is based in an extension of the C
language, and it is available for graphic cards GeForce 8 Series and superior, using the 32 and 64 bits versions of
the Linux and Windows (XP and successors) operating systems. Three software layers are used in CUDA to
communicate with the GPU (see Figure 1): a lowlevel hardware driver that performs the data communications
between the CPU and the GPU, a high-level API, and a set of libraries that includes CUBLAS for linear algebra
calculations and CUFFT for Fourier transforms calculation. For the CUDA programmer, the GPU is a computing
device which is able to execute a large number of threads in parallel. A specic procedure to be executed many times
over dierent data can be isolated in a GPU-function using many execution threads. The function is compiled using a
specic set of instructions and the resulting program named kernel is loaded in the GPU. The GPU has its own DRAM,
and the data are copied from the DRAM of the GPU to the RAM of the host (and viceversa) using optimized calls to
the CUDA API. The CUDA architecture is built around a scalable array of multiprocessors, each one of them having
eight scalar processors, one multithreading unit, and a shared memory chip. The multiprocessors are able to create,
manage, and execute parallel threads, with reduced overhead. The threads are grouped in blocks (with up to 512
threads), which are executed in a single multiprocessor of the graphic card, and the blocks are grouped in grids.
Each time that a CUDA program calls a grid to be executed in the GPU, each one of the blocks in the grid is
numbered and distributed to an available multiprocessor.
When a multiprocessor receives one (or more) blocks to execute, it splits the threads in warps a set of 32
consecutive threads. Each warp executes a single instruction at a time, so the best eciency is achieved when the 32
threads in the warp executes the same instruction. Otherwise, the warp serializes the threads. Each time that a
block nishes its execution, a new block is assigned to the available multiprocessor. The threads are able to access
the data using three memory spaces: the shared memory of the block, which can be used by the threads in the
block; the local memory of the thread; and the global memory of the GPU. Minimizing the access to the slower
memory spaces (the local memory of the thread and the global memory of the GPU) is a very important feature to
achieve eciency in GPU programming. On the other side, the shared memory is placed within the GPU chip, thus it
provides a faster way to store the data.
79](https://image.slidesharecdn.com/introdefinitionsbasiclaws-230302205644-5747a598/85/intro-definitions-basic-laws-pptx-75-320.jpg)
intro, definitions, basic laws+.pptx
- 1. , WhenWe Need HighPerformance Computing? ► Case1 To perform time-consuming operations in less time/ before a tighter deadline. ►I am a bioinformatic engineer. ►I need to run computationally complex programs. ►I’d rather have the result in 5 minutes than in 5 days. ►Case 2 To do a high number of operations per seconds ►I am an engineer ofAmazon.com ►My Web server gets 1,000 hits per seconds ►I’d like my web server and my databases to handle 1,000 transactions per seconds so that customers do not experience bad delays Amazon does “process” several GBytes of data per seconds
- 2. , What Does HighPerformance Computing Study ► It includes following subjects Hardware - Computer Architecture - Network Connections Software - Programming paradigms - Languages - Middleware
- 3. 3 1 103 106109 1012 1015 KiloOPS MegaOPS GigaOPS TeraOPS PetaOPS One OPS 1951 Univac 1 1949 Edsac 1976 Cray 1 1982 Cray XMP 1988 Cray YMP 1964 CDC 6600 1996 T3E 1823 Babbage Difference Engine 1991 Intel Delta 1997 ASCI Red 2001 Earth Simulator 2003 Cray X1 1943 Harvard Mark 1 1959 IBM 7094 2006 BlueGene/L 2009 Cray XT5
- 4. , INTERNATIONAL COMPETITION FOR HPCLEADERSHIP Nations and global regions including China, the United States, Japan, and Russia, are racing ahead and have created national programs that are investing large sums of money to develop exascale supercomputers.
- 5. 5 Supercomputer: A computingsystem exhibiting high-end performance capabilities and resource capacities within practical constraints of technology, cost, power, and reliability. Thomas Sterling, 2007 Supercomputer: a large very fast mainframe used especially for scientific computations. Merriam-Webster Online Supercomputer: any of a class of extremely powerful computers. The term is commonly applied to the fastest high-performance systems available at any given time. Such computers are used primarily for scientific and engineering work requiring exceedingly high-speed computations. Encyclopedia Britannica Online
- 6. , Research Applications ofHPCs • Finance • Sports and Entertainment • Weather Forecasting • Space Research • Health-Care-Related Applications • to unravel the morphology of cancer cells, • to diagnose and treat cancers and improve the safety of cancer treatments • medical research, • biomedicine, • bioinformatics, • epidemiology, • Personalized medicine include ‘Big Data’ aspects
- 7. , THE GLOBAL HPCMARKET Company Share of Global HPC Revenues
- 8. THE GLOBAL HPCMARKET Countries Share of Global HPC Revenues
- 9. 9 Introduction -today’s lecture System Architectures (Single Instruction - Single Data, Single Instruction - Multiple Data, Multiple Instruction - Multiple Data, Shared Memory, Distributed Memory, Cluster, Multiple Instruction - Single Data) Performance Analysis of parallel calculations (the speedup, efficiency, time execution of algorithm…) Parallel numerical methods (Principles of Parallel Algorithm Design, Analytical Modeling of Parallel Programs, Matrices Operations, Matrix-Vector Operations, Graph Algorithms…) Software (Programming Using Compute Unified Device Architecture)
- 10. In the firstsection we will discuss the importance of parallel computing to high performance computing. We will show the basic concepts of parallel computing. The advantages and disadvantages of parallel computing will be discussed. We will present an overview of current and future trends in HPC hardware. The second section will provide an introduction to parallel GPU implementations of numerical methods, such as Matrices Operations, Matrix-Vector Operations, Graph Algorithms. Also second section will present the application of parallel computing techniques using Graphic Processing Unit (GPU) in order to improve the computational efficiency of numerical methods The third session will briefly discuss such important HPC topics like computing using graphic processing units (GPUs) with CUDA running relatively simple examples on this hardware. As tradition dictates, we will show how to write "Hello World" in CUDA. Some computational libraries available for HPC with GPU will be highlighted. 10
- 11. 11 What istraditional programming view? Why Use Parallel Computing? Motivation for parallelism (Moor’s law). An Overview of Parallel Processing Parallelism in Uniprocessor Systems Organization of Multiprocessor Flynn’s Classification System Topologies MIMD System Architectures Concepts and terminology.
- 12. Parallelism is amethod to improve computer system performance by executing two or more instructions simultaneously. The goals of parallel processing. One goal is to reduce the “wall-clock” time or the amount of real time that you need to wait for a problem to be solved. Another goal is to solve bigger problems that might not fit in the limited memory of a single CPU. 12
- 13. Consider your favoritecomputational application • One processor can give me results in N hours • Why not use N processors -- and get the results in just one hour? 13
- 14. Parallel computing: theuse of multiple computers or processors working together on a common task. Each processor works on its section of the problem. Processors are allowed to exchange information with other processors. 15 Grid of a Problem to be Solved
- 15. 16 Flynn’s Classification (Taxonomy) Was proposed by researcher Michael J. Flynn in 1966. It is the most commonly accepted taxonomy of computer organization. In this classification, computers are classified by whether it processes a single instruction at a time or multiple instructions simultaneously, and whether it operates on one or multiple data sets.
- 16. 17 4 categories ofFlynn’s classification of multiprocessor systems by their instruction and data streams Simple Diagrammatic Representation
- 17. 18 SISD machinesexecutes a single instruction on individual data values using a single processor. Based on traditional Von Neumann uniprocessor architecture, instructions are executed sequentially or serially, one step after the next. Until most recently, most computers are of SISD type.
- 18.
- 19. 20 An SIMDmachine executes a single instruction on multiple data values simultaneously using many processors. Since there is only one instruction, each processor does not have to fetch and decode each instruction. Instead, a single control unit does the fetch and decoding for all processors. SIMD architectures include array processors.
- 20.
- 21. 22 This categorydoes not actually exist. This category was included in the taxonomy for the sake of completeness.
- 22. 23 MIMD machinesare usually referred to as multiprocessors or multicomputers. It may execute multiple instructions simultaneously, contrary to SIMD machines. Each processor must include its own control unit that will assign to the processors parts of a task or a separate task. It has two subclasses: Shared memory and distributed memory
- 23. 24 Shared memory UMA(all processors have equal access to memory. Can talk via memory.) Distributed Memory Processors only Have access to their local memory “talk” to other processors over a network Hybrid Shared memory nodes connected by a network
- 24. Hybrid Machines •Add specialpurpose processors to normal processors •Not a new concept but, regaining traction Example: our Tesla Nvidia node, cuda 25
- 25. 26 Shared memoryand distributed memory machines becomes mainstream. Manycore architectures: GPUs used for computing (GPGPU) High performance computing almost equals to parallel computing
- 26. 27 Finally, thearchitecture of a MIMD system, contrast to its topology, refers to its connections to its system memory. A systems may also be classified by their architectures. Two of these are: Uniform memory access (UMA) Nonuniform memory access (NUMA)
- 27. 28 The UMAis a type of symmetric multiprocessor, or SMP, that has two or more processors that perform symmetric functions. UMA gives all CPUs equal (uniform) access to all memory locations in shared memory. They interact with shared memory by some communications mechanism like a simple bus or a complex multistage interconnection network.
- 28. 29 Shared Memory Processor 2 Processor 1 Processorn Communications mechanism
- 29. 30 NUMA architectures,unlike UMA architectures do not allow uniform access to all shared memory locations. This architecture still allows all processors to access all shared memory locations but in a nonuniform way, each processor can access its local shared memory more quickly than the other memory modules not next to it.
- 30. 31 Memory 1 Processor 1 Communicationsmechanism Memory 2 Processor 2 Memory n Processor n
- 31. 32 An analogy ofFlynn’s classification is the check-in desk at an airport SISD: a single desk SIMD: many desks and a supervisor with a megaphone giving instructions that every desk obeys MIMD: many desks working at their own pace, synchronized through a central database
- 32.
- 33. All parallel programscontain: • Parallel sections • Serial sections • Serial sections are when work is being duplicated or no useful work is being done, (waiting for others) • Serial sections limit the parallel effectiveness • If you have a lot of serial computation then you will not get good speedup • No serial work “allows” perfect speedup • Amdahl’s Law states this formally 35
- 34. Amdahl’s Lawplaces a strict limit on the speedup that can be realized by using multiple processors. Effect of multiple processors on run time • Effect of multiple processors on speed up • Where • fS = serial fraction of code • fp = parallel fraction of code • N = number of processors • Perfect speedup t=t1/n or S(n)=n 36
- 35.
- 36. Amdahl’s Lawprovides a theoretical upper limit on parallel speedup assuming that there are no costs for communications. In reality, communications will result in a further degradation of performance 38
- 37. Writing effective parallelapplication is difficult • Communication can limit parallel efficiency • Serial time can dominate • Load balance is important Is it worth your time to rewrite your application • Do the CPU requirements justify parallelization? • Will the code be used just once? 39
- 38. S(n) > n, maybe seen on occasion, but usually this is due to using a suboptimal sequential algorithm or some unique feature of the architecture that favors the parallel formation. One common reason for superlinear speedup is the extra cache in the multiprocessor system which can hold more of the problem data at any instant, it leads to less, relatively slow memory traffic. 42
- 39. Efficiency = Executiontime using one processor over the Execution time using a number of processors Its just the speedup divided by the number of processors 43
- 40. Used to indicatea hardware design that allows the system to be increased in size and in doing so to obtain increased performance - could be described as architecture or hardware scalability. Scalability is also used to indicate that a parallel algorithm can accommodate increased data items with a low and bounded increase in computational steps - could be described as algorithmic scalability. 44
- 41. Problem size: thenumber of basic steps in the best sequential algorithm for a given problem and data set size •Intuitively, we would think of the number of data elements being processed in the algorithm as a measure of size. •However, doubling the date set size would not necessarily double the number of computational steps. It will depend upon the problem. •For example, adding two matrices has this effect, but multiplying matrices quadruples operations. Note: bad sequential algorithms tend to scale well. 45
- 42. • Latency • Howlong to get between nodes in the network. • Bandwidth • How much data can be moved per unit time. • Bandwidth is limited by the number of wires and the rate at which each wire can accept data and choke points. 46
- 43. For ultimateperformance you may be concerned how your nodes are connected. Avoid communications between distant node. For some machines it might be difficult to control or know the placement of applications. 47
- 44. 48 Topologies A systemmay also be classified by its topology. A topology is the pattern of connections between processors. The cost-performance tradeoff determines which topologies to use for a multiprocessor system.
- 45. 49 A topology ischaracterized by its diameter, total bandwidth, and bisection bandwidth ◦ Diameter – the maximum distance between two processors in the computer system. ◦ Total bandwidth – the capacity of a communications link multiplied by the number of such links in the system. ◦ Bisection bandwidth – represents the maximum data transfer that could occur at the bottleneck in the topology.
- 46.
- 47. 51 Shared Bus Topology ◦Processors communicate with each other via a single bus that can only handle one data transmissions at a time. ◦ In most shared buses, processors directly communicate with their own local memory. M P M P M P Global memory Shared Bus Network Topologies
- 48. 52 Ring Topology ◦Uses direct connections between processors instead of a shared bus. ◦ Allows communication links to be active simultaneously but data may have to travel through several processors to reach its destination. P P P P P P
- 49. 53 Tree Topology ◦Uses direct connections between processors; each having three connections. ◦ There is only one unique path between any pair of processors. P P P P P P P
- 50. 54 Mesh Topology ◦In the mesh topology, every processor connects to the processors above and below it, and to its right and left. P P P P P P P P P
- 51. 55 Hypercube Topology ◦ Isa multiple mesh topology. ◦ Each processor connects to all other processors whose binary values differ by one bit. For example, processor 0(0000) connects to 1(0001) or 2(0010). P P P P P P P P P P P P P P P P
- 52. 56 Completely Connected Topology Every processor has n-1 connections, one to each of the other processors. There is an increase in complexity as the system grows but this offers maximum communication capabilities. P P P P P P P P
- 53. 57 Moore's Law describesa long- term trend in the history of computing hardware, in which the number of transistors that can be placed inexpensively on an integrated circuit has doubled approximately every two years.
- 54. Mechanical Computing ◦Babbage, Hollerith, Aiken Electronic Digital Calculating ◦ Atanasoff, Eckert, Mauchly von Neumann Architecture ◦ Turing, von Neumann, Eckert, Mauchly, Foster, Wilkes Semiconductor Technologies Birth of the Supercomputer ◦ Cray, Watanabe The Golden Age ◦ Batcher, Dennis, S. Chen, Hillis, Dally, Blank, B. Smith Common Era of Killer Micros ◦ Scott, Culler, Sterling/Becker, Goodhue, A. Chen, Tomkins Petaflops ◦ Messina, Sterling, Stevens, P. Smith, 58
- 55. 59 Historical Machines • LeibnizStepped Reckoner • Babbage Difference Engine • Hollerith Tabulator • Harvard Mark 1 • Un. of Pennsylvania Eniac • Cambridge Edsac • MIT Whirlwind • Cray 1 • TMC CM-2 • Intel Touchstone Delta • Beowulf • IBM Blue Gene/L
- 56. Eckert andMauchly, 1946. Vacuum tubes. Numerical solutions to problems in fields such as atomic energy and ballistic trajectories. 60
- 57. Maurice Wilkes,1949. Mercury delay lines for memory and vacuum tubes for logic. Used one of the first assemblers called Initial Orders. Calculation of prime numbers, solutions of algebraic equations, etc. 61
- 58. Jay Forrester,1949. Fastest computer. First computer to use magnetic core memory. Displayed real time text and graphics on a large oscilloscope screen. 62
- 59. Cray Research, 1976. Pipelined vector arithmetic units. Unique C-shape to help increase the signal speeds from one end to the other. 63
- 60. Thinking Machines Corporation,1987. Hypercube architecture with 65,536 processors. SIMD. Performance in the range of GFLOPS. 64
- 61. INTEL, 1990. MIMD hypercube. LINPACK rating of 13.9 GFLOPS . Enough computing power for applications like real- time processing of satellite images and molecular models for AIDS research. 65
- 62. Thomas Sterlingand Donald Becker, 1994. Cluster formed of one head node and one/more compute nodes. Nodes and network dedicated to the Beowulf. Compute nodes are mass produced commodities. Use open source software including Linux. 66
- 63. Japan, 1997. Fastest supercomputer from 2002-2004: 35.86 TFLOPS. 640 nodes with eight vector processors and 16 gigabytes of computer memory at each node. 67
- 64. IBM, 2004. First supercomputer ever to run over 100 TFLOPS sustained on a real world application, namely a three-dimensional molecular dynamics code (ddcMD). 68
- 65. 1975 –1992 Vector ◦ Cray-1&2, NEC SX, Fujitsu VPP SIMD ◦ Maspar, CM-2 Systolic ◦ Warp Dataflow ◦ Manchester, Sigma, Monsoon Multithreaded ◦ HEP, MTA Actor-based ◦ J-Machine 69 1976 Cray 1
- 66. 1992 topresent Killer Micro and mass market PCs High density DRAM High cost of fab lines CSP ◦ Message passing Economy of scale S-curve MPP Weak scaling ◦ Gustafson et al Beowulf, NOW Clusters MPI Ethernet, Myrinet Linux 70
- 67. Automated calculating ◦17th century Stored program digital electronic ◦ 1948 Vector ◦ 1975 SIMD ◦ 1980s MPPs ◦ 1991 Commodity Clusters ◦ 1993/4 Multicore ◦ 2006 71
- 68. Hybrid clustersolutions & services that fully leverage the performance of accelerators Demand for computing power is growing steadily, as scientists & engineers seek to tackle increasingly complex problems. The emergence of multi-core CPUs has allowed to keep pace with their demands, but energy consumption, space, & cooling have become major inhibitors to computing systems expansion. Hence the success of acceleration technologies such as GPGPUs (General-Purpose Graphics Processing Units), which offer both breakthrough performance & outstanding space & energy efficiency GPGPUs can accelerate processing by a factor of 1 to 100! 72
- 69. Starvation ◦ Notenough work to do due to insufficient parallelism or poor load balancing among distributed resources Latency ◦ Waiting for access to memory or other parts of the system Overhead ◦ Extra work that has to be done to manage program concurrency and parallel resources the real work you want to perform Waiting for Contention ◦ Delays due to fighting over what task gets to use a shared resource next. Network bandwidth is a major constraint. 73
- 70. Lecture 2-3/10 74 Programming ofGraphic Processors Olesia Barkovskaya , KHTURE, 2018 Electronic Computers Department
- 71. Simply saying, inarchitecture sense, CPU is composed of few huge Arithmetic Logic Unit (ALU) cores for general purpose processing with lots of cache memory and one huge control module that can handle a few software threads at a time. CPU is optimized for serial operations since its clock is very high. While GPU, on the other hand, has many small ALUs, small control modules and small cache. GPU is optimized for parallel operations. 75
- 72. A simple wayto understand the difference between a GPU and a CPU is to compare how they process tasks. A CPU consists of a few cores optimized for sequential serial processing while a GPU has a massively parallel architecture consisting of thousands of smaller, more efficient cores designed for handling multiple tasks simultaneously. GPUs have thousands of cores to process parallel workloads efficiently 76
- 73. Environments CUDA: Moremature, bigger ‘ecosystem’, NVIDIA only OpenCL: Vendor-independent, open industry standard Interfaces to C/C++, Fortran, Python, .NET, . . . Important: Hardware abstraction and ‘expressiveness’ are identical 77
- 74. Fourier Transforms CUFFT:NVIDIA, part of the CUDA toolkit APPML (formerly ACML-GPU): AMD Accelerated Parallel Processing Math Libraries Dense linear algebra CUBLAS: NVIDIA’s basic linear algebra subprograms APPML (formerly ACML-GPU): AMD Accelerated Parallel Processing Math Libraries CULA: Third-party LAPACK, matrix decompositions and eigenvalue problems MAGMA and PLASMA: BLAS/LAPACK for multicore and manycore (ICL, Tennessee) 78
- 75. Ten years ago,when GPUs were rst used to perform general-purpose computation, they were programmed using low-level mechanism such as the interruption services of the BIOS, or by using graphic APIs such as OpenGL and DirectX [16]. Later, the programs for GPU were developed in assembly language for each card model, and they had very limited portability. So, high-level languages were developed to fully exploit the capabilities of the GPUs. In 2007, NVIDIA introduced CUDA [25], a software architecture for managing the GPU as a parallel computing device without requiring to map the data and the computation into a graphic API. CUDA is based in an extension of the C language, and it is available for graphic cards GeForce 8 Series and superior, using the 32 and 64 bits versions of the Linux and Windows (XP and successors) operating systems. Three software layers are used in CUDA to communicate with the GPU (see Figure 1): a lowlevel hardware driver that performs the data communications between the CPU and the GPU, a high-level API, and a set of libraries that includes CUBLAS for linear algebra calculations and CUFFT for Fourier transforms calculation. For the CUDA programmer, the GPU is a computing device which is able to execute a large number of threads in parallel. A specic procedure to be executed many times over dierent data can be isolated in a GPU-function using many execution threads. The function is compiled using a specic set of instructions and the resulting program named kernel is loaded in the GPU. The GPU has its own DRAM, and the data are copied from the DRAM of the GPU to the RAM of the host (and viceversa) using optimized calls to the CUDA API. The CUDA architecture is built around a scalable array of multiprocessors, each one of them having eight scalar processors, one multithreading unit, and a shared memory chip. The multiprocessors are able to create, manage, and execute parallel threads, with reduced overhead. The threads are grouped in blocks (with up to 512 threads), which are executed in a single multiprocessor of the graphic card, and the blocks are grouped in grids. Each time that a CUDA program calls a grid to be executed in the GPU, each one of the blocks in the grid is numbered and distributed to an available multiprocessor. When a multiprocessor receives one (or more) blocks to execute, it splits the threads in warps a set of 32 consecutive threads. Each warp executes a single instruction at a time, so the best eciency is achieved when the 32 threads in the warp executes the same instruction. Otherwise, the warp serializes the threads. Each time that a block nishes its execution, a new block is assigned to the available multiprocessor. The threads are able to access the data using three memory spaces: the shared memory of the block, which can be used by the threads in the block; the local memory of the thread; and the global memory of the GPU. Minimizing the access to the slower memory spaces (the local memory of the thread and the global memory of the GPU) is a very important feature to achieve eciency in GPU programming. On the other side, the shared memory is placed within the GPU chip, thus it provides a faster way to store the data. 79