parallel computing, parallel processing


Published on

parallel computing brief.

Published in: Education, Technology
  • need permission from who post it
    Are you sure you want to  Yes  No
    Your message goes here
  • downloading is not possible since the author disabled the feature
    Are you sure you want to  Yes  No
    Your message goes here
  • how to download this ppt???
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

parallel computing, parallel processing

  1. 1. Parallel Computing
  2. 2. <ul><li>Traditionally, software has been written for serial computation: </li></ul><ul><ul><li>To be run on a single computer having a single Central Processing Unit (CPU) </li></ul></ul><ul><ul><li>A problem is broken into a discrete series of instructions. </li></ul></ul><ul><ul><li>Instructions are executed one after another. </li></ul></ul><ul><ul><li>Only one instruction may execute at any moment in time. </li></ul></ul>
  3. 3. <ul><li>Limits to serial computing - both physical and practical reasons pose significant </li></ul><ul><li>constraints to simply building ever faster serial computers: </li></ul><ul><li>Transmission speeds - the speed of a serial computer is directly dependent upon </li></ul><ul><li> how fast data can move through hardware. Absolute limits are the speed </li></ul><ul><li>of light (30 cm/nanosecond) and the transmission limit of copper wire </li></ul><ul><li>(9 cm/nanosecond). Increasing speeds necessitate increasing proximity </li></ul><ul><li>of processing elements. </li></ul><ul><li>Limits to miniaturization - processor technology is allowing an increasing number </li></ul><ul><ul><li>of transistors to be placed on a chip. However, even with molecular or </li></ul></ul><ul><ul><li>Atomic-level components, a limit will be reached on how small </li></ul></ul><ul><ul><li>components can be. </li></ul></ul><ul><li>Economic limitations - it is increasingly expensive to make single processor faster. </li></ul><ul><ul><li>Using a larger number of moderately fast commodity processors to </li></ul></ul><ul><ul><li>achieve the same (or better) performance is less expensive. </li></ul></ul>
  4. 4. <ul><li>What is Parallel Computing? </li></ul><ul><li>In the simplest sense, parallel computing is the simultaneous use of multiple compute </li></ul><ul><li>resources to solve a computational problem. </li></ul><ul><li>To be run using multiple CPUs </li></ul><ul><li>A problem is broken into discrete parts that can be solved concurrently </li></ul><ul><li>Each part is further broken down to a series of instructions </li></ul><ul><li>Instructions from each part execute simultaneously on different CPUs </li></ul>
  5. 5. Why Do Parallel Computing? <ul><li>Limits of single CPU computing </li></ul><ul><ul><li>Available memory </li></ul></ul><ul><ul><li>Performance </li></ul></ul><ul><li>Parallel computing allows: </li></ul><ul><ul><li>Solve problems that don’t fit on a single CPU’s memory space </li></ul></ul><ul><ul><li>Solve problems that can’t be solved in a reasonable time </li></ul></ul><ul><li>We can run… </li></ul><ul><ul><li>Larger problems </li></ul></ul><ul><ul><li>Faster </li></ul></ul><ul><ul><li>More cases </li></ul></ul>
  6. 6. <ul><li>Parallel computing: use of multiple computers or processors working together on a common task. </li></ul><ul><ul><li>Each processor works on its section of the problem </li></ul></ul><ul><ul><li>Processors can exchange information </li></ul></ul>Grid of Problem to be solved CPU #1 works on this area of the problem CPU #3 works on this area of the problem CPU #4 works on this area of the problem CPU #2 works on this area of the problem y x exchange exchange
  7. 8. Basic Components of a Parallel (or Serial) Computer CPU MEM C PU M EM CPU MEM CPU MEM CPU MEM CPU MEM CPU MEM CPU MEM CPU MEM
  8. 9. Classes of parallel computers Parallel computers can be roughly classified according to the level at which the hardware supports parallelism. This classification is broadly analogous to the distance between basic computing nodes. These are not mutually Exclusive.
  9. 10. Multicore computing A multicore processor is a processor that includes multiple execution units (&quot;cores&quot;). These processors differ from superscalar processors, which can issue multiple instructions per cycle from one instruction stream (thread); by contrast, a multicore processor can issue multiple instructions per cycle from multiple instruction streams. Each core in a multicore processor can potentially be superscalar as well—that is, on every cycle, each core can issue multiple instructions from one instruction stream. <ul><li>Symmetric multiprocessing </li></ul><ul><li>A symmetric multiprocessor (SMP) is a computer system with multiple identical processors that share memory and connect via a bus. Bus contention prevents bus architectures from scaling. As a result, SMPs generally do not comprise more than 32 processors.&quot; Because of the small size of the processors and the significant reduction in the requirements for bus bandwidth achieved by large caches, such symmetric multiprocessors are extremely cost-effective, provided that a sufficient amount of memory bandwidth exists.” </li></ul>
  10. 11. Distributed computing A distributed computer (also known as a distributed memory multiprocessor) is a distributed memory computer system in which the processing elements are connected by a network. Distributed computers are highly scalable. Cluster computing  A cluster is a group of loosely coupled computers that work together closely, so that in some respects they can be regarded as a single computer. Clusters are composed of multiple standalone machines connected by a network. While machines in a cluster do not have to be symmetric, load balancing is more difficult if they are not. The most common type of cluster is the Beowulf cluster, which is a cluster implemented on multiple identical commercial off-the-shelf computers connected with a TCP/IP Ethernet local area network.
  11. 12. Massive parallel processing A massively parallel processor (MPP) is a single computer with many networked processors. MPPs have many of the same characteristics as clusters, but they are usually larger, typically having &quot;far more&quot; than 100 processorsIn an MPP, “each CPU contains its own memory and copy of the operating system and application. Each subsystem communicates with the others via a high-speed interconnect. Grid computing Grid computing is the most distributed form of parallel computing. It makes use of computers communicating over the Internet to work on a given problem. Because of the low bandwidth and extremely high latency available on the Internet, grid computing typically deals only with embarrassingly parallel problems. Most grid computing applications use middleware, software that sits between the operating system and the application to manage network resources and standardize the software interface.
  12. 14. Types of Parallelism : Two Extremes <ul><li>Data parallel </li></ul><ul><ul><li>Each processor performs the same task on different data </li></ul></ul><ul><ul><li>Example - grid problems </li></ul></ul><ul><li>Task parallel </li></ul><ul><ul><li>Each processor performs a different task </li></ul></ul><ul><ul><li>Example - signal processing </li></ul></ul><ul><li>Most applications fall somewhere on the continuum between these two extremes </li></ul><ul><li>There are two other types of parallelism: </li></ul><ul><li>Bit-level parallelism </li></ul><ul><li>Instruction-level parallelism </li></ul>
  13. 15. <ul><li>Parallel Computers </li></ul><ul><li>􀂄 Massively Parallel Processors (MPPs) </li></ul><ul><ul><li>􀂅 Large number of processors connected to gigabytes of memory via proprietary hardware </li></ul></ul><ul><ul><li>􀂅 Enormous computing power ‘in a box’ </li></ul></ul><ul><ul><li>􀂅 Made by the likes of IBM, Cray, SGI, </li></ul></ul><ul><ul><li>Very costly! </li></ul></ul><ul><li>􀂄 Distributed / Cluster Computing </li></ul><ul><ul><li>􀂅 A number of computers connected by a network are used to solve a single large problem </li></ul></ul><ul><ul><li>􀂅 Could consist of ordinary workstations or even MPPs </li></ul></ul><ul><ul><li>􀂅 Considerably cheaper to set up </li></ul></ul>
  14. 16. <ul><li>Why Use Parallel Computing? </li></ul><ul><li>The primary reasons for using parallel computing: </li></ul><ul><ul><li>Save time - wall clock time </li></ul></ul><ul><ul><li>Solve larger problems </li></ul></ul><ul><ul><li>Provide concurrency (do multiple things at the same time) </li></ul></ul><ul><ul><li>Other reasons might include: </li></ul></ul><ul><ul><li>Taking advantage of non-local resources – </li></ul></ul><ul><ul><li>using available compute resources on a wide area network, or even the Internet when local compute resources are scarce. </li></ul></ul><ul><ul><li>Cost savings – </li></ul></ul><ul><ul><ul><ul><ul><li>using multiple &quot;cheap&quot; computing resources instead of paying for time on a supercomputer. </li></ul></ul></ul></ul></ul><ul><ul><li>Overcoming memory constraints – </li></ul></ul><ul><ul><li> single computers have very finite memory resources. For large problems, using the memories of multiple computers may overcome this obstacle. </li></ul></ul>
  15. 17. <ul><li>Currently the speed of off-the shelf micro processors is within one order of magnitude of the fastest serial computers. Micro processors cost many orders of magnitude less. </li></ul><ul><li>Connect only a few micro processors together to form a parallel computer with speed comparable to fastest serial computers. Cost of such parallel computer is less. </li></ul><ul><li>Connect a large number of processors into parallel computer overcomes the saturation rate of serial computers. </li></ul><ul><li>Parallel computers can provide much higher raw computation power than fastest serial computes. </li></ul><ul><li>Need to have actual applications that can take advantage of this high power. </li></ul>Cost Effectiveness of Parallel Processors
  16. 18. Parallel Computing Today Small class Beowulf cluster Japanese Earth Simulator machine