This document outlines the course policies and contents of an introduction to parallel computing course. The course will cover fundamentals of parallel platforms, parallel programming using message passing and threads, and parallel algorithms. It will introduce concepts like multicore processing, GPGPU computing, and parallel programming models. The course is divided into sections on fundamentals, programming, and algorithms. References for further reading on parallel and distributed computing are also provided.

1. Course Policies and outcomes

2. Chapter 1Chapter 1
Introduction to ParallelIntroduction to Parallel
ComputingComputing
Dr. Muhammad Hanif Durad
Department of Computer and Information Sciences
Pakistan Institute Engineering and Applied Sciences
hanif@pieas.edu.pk

3. Dr. Hanif Durad 3
Lecture Outline
 Motivating Parallelism
 Scope of Parallel Computing Applications
 Organization and Contents of the Course
 Texts and References

4. Parallel computing
 Parallel Processing is a term used to denote a large
class of techniques that are used to provide
simultaneous data processing tasks for the purpose
of
 Save time and/or money
 Solve larger problems
 Parallel computing is the simultaneous use of
multiple compute resources to solve a
computational problem 4
parallelcomputing-120214042541-phpapp02-130408023815-phpapp01.ppt

5. The Universe is Parallel
5
 Galaxy formation
 Planetary movement
 Weather and ocean patterns
 Tectonic plate drift
 Rush hour traffic
 Automobile assembly line
 Building a jet
 Ordering a hamburger
at the drive through.

6. Areas of Parallel Computing
 Physics – applied, nuclear, particle, condensed matter,
high pressure, fusion, photonics
 Bioscience, Biotechnology, Genetics
 Chemistry, Molecular Sciences
 Geology, Seismology
 Mechanical Engineering - from prosthetics to spacecraft
 Electrical Engineering, Circuit Design, Microelectronics
 Computer Science, Mathematics
Dr. Hanif Durad 6

7. Why Use Parallel Computing?
 Save time and/or money: In theory, throwing more
resources at a task will shorten its time to completion,
with potential cost savings. Parallel computers can be
built from cheap, commodity components.
 Solve larger problems: Many problems are so large
and/or complex that it is impractical or impossible to
solve them on a single computer, especially given limited
computer memory.
 Better response times: As the computing tasks are
engaged by a group of processors, the tasks are
completed in a smaller amount of time 7

8. Dr. Hanif Durad 8
Hardware Models
02Distributed_Computing.pptx

9. Ways to Classify Parallel
Computers
 One of the more widely used classifications, in use
since 1966, is called Flynn's Taxonomy
The 4 possible classifications according to Flynn’s
are :
 Single Instruction, Single Data (SISD)
 Single Instruction, Multiple Data (SIMD)
 Multiple Instruction, Single Data (MISD)
 Multiple Instruction, Multiple Data (MIMD)
Dr. Hanif Durad 9

10. Dr. Hanif Durad 10
Recent Trends in Computing

11. Technology Trends: Moore’s
Law
Dr. Hanif Durad 11
Slide from Maurice Herlihy
Clock speed
flattening
sharply
Transistor
count still
rising
6963_L1.ppt

12. Techology Trends: Power Issues
Dr. Hanif Durad 12
From www.electronics-cooling.com/.../jan00_a2f2.jpg6963_L1.ppt

13. Power Perspective
Dr. Hanif Durad 13
GigaFlop/s
MegaWatts
0.001
0.01
0.1
1
10
100
1000
10000
100000
1000000
10000000
100000000
1000000000
1960 1970 1980 1990 2000 2010 2020
Performance (Gflops)
Power
Slide source: Bob Lucas
6963_L1.ppt

14. Increase speed processor
 Greater no. of transistors
 Operation can be done in fewer clock cycles
 Increased clock speed
 More operations per unit time
 Example
 8088/8086 : 5 Mhz, 29000 transistors
 E6700 Core 2 Duo: 2.66 GHz, 291 million
transistor
lecture1-100131132656-phpapp02.pptx

15. The Multi-Core Paradigm Shift
 Key ideas:
 Movement away from increasingly complex
processor design and faster clocks
 Replicated functionality (i.e., parallel) is simpler to
design
 Resources more efficiently utilized
 Huge power management advantages
Dr. Hanif Durad 15
6963_L1.ppt

16. Multicore
 A multi-core processor is one processor that contains two
or more complete functional units. Such chips are now
the focus of Intel and AMD. A multi-core chip is a form
of SMP
lecture1-100131132656-phpapp02.pptx
parallelprogramming-130823023925-phpapp01.pptx

17. Concept of GPGPU (General-
Purpose Computing on GPUs)
 Idea:
 Potential for very high performance at low cost
 Architecture well suited for certain kinds of parallel applications (data
parallel)
 Demonstrations of 30-100X speedup over CPU
 Early challenges:
 Architectures very customized to graphics problems (e.g., vertex and
fragment processors)
 Programmed using graphics-specific programming models or libraries
 Recent trends:
 Some convergence between commodity and GPUs and their associated
parallel programming models
Dr. Hanif Durad 17
6963_L1.ppt

18. Parallel Programming Model
 Parallel programming models in common use:
 Shared Memory (without threads)
 Threads
 Distributed Memory / Message Passing
 Data Parallel
 Hybrid
 Single Program Multiple Data (SPMD)
 Multiple Program Multiple Data (MPMD)
 These models are NOT specific to a particular type of machine
or memory architecture
 Any of these models can be implemented on any underlying hardware
18
parallelprogramming-130823023925-phpapp01.pptx, p-60/114

19. Classes of Parallel Computers
 Cluster computing
 A cluster is a group of loosely coupled computers that
work together closely, so that in some respects they
can be regarded as a single computer
 Massive parallel processing
 A massively parallel processor (MPP) is a single
computer with many networked processors.
 MPPs have many of the same characteristics as
clusters, but MPPs have specialized interconnect
networks Dr. Hanif Durad 19
parallelprogramming-130823023925-phpapp01.pptx, p-99/114

20. Classes of Parallel Computers
 Grid computing
 Compared to clusters, grids tend to be more loosely
coupled, heterogeneous, and geographically dispersed
Dr. Hanif Durad 20

21. Cloud Computing
 A cloud provider has 100s of thousands of nodes (aka
servers).
 Cloud computing is massively-parallel computing with
multi-processors (i.e. many multi-core processors)
 In principle, your application may run on one, two, …
thousands of servers (i.e. processors)
 For your application to run on one, two, … thousands of
servers, your application code or data must be
parallelized.
 i.e. Split up into independent or relatively independent parts.
Dr. Hanif Durad 21
02Distributed_Computing.pptx

22. Dr. Hanif Durad 22
Organization and Contents of this
Course
1. Fundamentals: This part of the class covers basic parallel
platforms, principles of algorithm design, group
communication primitives, and analytical modeling
techniques.
2. Parallel Programming: This part of the class deals with
programming using message passing libraries and threads.
3. Parallel Algorithms: This part of the class covers basic
algorithms for matrix computations, graphs, sorting,
discrete optimization, and dynamic programming.
1+2=CIS-546 PC
3=CIS-645 PA

23. Summary
 Introduction to Parallel Computing
 Technology Trends
 Multicore
 GPGPU (General-Purpose Computing on GPUs)
 Parallel Programming Model
 Classes of Parallel Computers
 Cluster computing
 Massive parallel processing (MPP)
 Grid computing
 Cloud Computing
Dr. Hanif Durad 23

24. Dr. Hanif Durad 24
References
 Grama A., Gupta A., Karypis G., and Kumar V., Introduction
to Parallel Computing, 2nd ed., Addison-Wesley, 2003. (Main
Text Book)
 Thomas Rauber, Gudula Rünger, Parallel Programming: For
Multicore and Cluster Systems, Springer, 2010.
 Peter Pacheco, An Introduction to Parallel Programming,
Elsevier, 2011.
 Michael J. Quinn , Parallel Programming in C with MPI and
OpenMP, McGraw-Hill, 2004.

25. References
 Mostafa Abd-El-Barr and Hesham El-Rewini, Advanced
Computer Architecture and Parallel Processing, John Wiley
and Sons, 2005.
 Culler D. E., Singh J. P., and Gupta A., Parallel Computer
Architecture: A Hardware/Software Approach, Morgan
Kaufmann Publisher, 1999 .
 Foster I., Designing and Building Parallel Programs,
Addision-Wesley, 1995.
Dr. Hanif Durad 25