This document provides an introduction to parallel and distributed computing. It discusses traditional sequential programming and von Neumann architecture. It then introduces parallel computing as a way to solve larger problems faster by breaking them into discrete parts that can be solved simultaneously. The document outlines different parallel computing architectures including shared memory, distributed memory, and hybrid models. It provides examples of applications that benefit from parallel computing such as physics simulations, artificial intelligence, and medical imaging. Key challenges of parallel programming are also discussed.

1. 1
Introduction
Parallel and Distributed Computing
Lecture 1-2 / 18
High Performance Computing most
generally refers to the practice of
aggregating computing power in a
way that delivers much higher
performance than one could get out
of a typical desktop computer or
workstation in order to solve large
problems in science, engineering, or
business.

2. 2
Topics
 Introduction - today’s lecture
 System Architectures (Single Instruction - Single Data, Single Instruction -
Multiple Data, Multiple Instruction - Multiple Data, Shared Memory, Distributed
Memory, Cluster, Multiple Instruction - Single Data)
 Performance Analysis of parallel calculations (the speedup, efficiency, time
execution of algorithm…)
 Parallel numerical methods (Principles of Parallel Algorithm Design, Analytical
Modeling of Parallel Programs, Matrices Operations, Matrix-Vector Operations,
Graph Algorithms…)
 Software (Programming Using the Message-Passing Interface, OpenMP, CUDA,
Corba…)

3. 3
What will you learn today?
 Why Use Parallel Computing?
 Motivation for parallelism (Moor’s law)
 What is traditional programming view?
 What is parallel computing?
 What is distributed computing?
 Concepts and terminology
von Neumann Computer Architecture

4. 4
What is traditional programming view?
 Von Neumann View
- Program Counter + Registers = Thread/process
- Sequential change of Machine state
 Comprised of four main components:
 Memory
 Control Unit
 Arithmetic Logic Unit
 Input/Output

5. 5
Von Neumann Architecture
 Read/write, random access memory is used to store both program
instructions and data
 Program instructions tell the computer to do something
 Data is simply information to be used by the program
 Control unit fetches instructions/data from memory, decodes the
instructions and then sequentially coordinates operations to accomplish
the programmed task.
 Arithmetic Unit performs basic arithmetic operations.
 Input/Output is the interface to the human operator.

6. 6
All of the algorithms we’ve seen so far are
sequential:
• They have one “thread” of execution
• One step follows another in sequence
• One processor is all that is needed to run
the algorithm
Traditional (sequential) Processing View

7. 7
The Computational Power Argument
Moore's law states [1965]:
2X transistors/Chip Every 1.5 or 2 years
Microprocessors have become smaller,
denser, and more powerful.
Gordon Moore is a co-founder of Intel.

8. What problems are there ?
With the increased use of computers in every sphere of
human activity, computer scientists are faced with two
crucial issues today:
 Processing has to be done faster like never before
 Larger or complex computation problems need to
be solved
8

9. What problems are there ?
 Increasing the number of transistors as per Moore’s
Law isn’t a solution, as it also increases the power
consumption.
 Power consumption causes a problem of processor
heating…
The perfect solution is PARALLELISM - in
hardware as well as software.
9

10. What is PARALLELISM ?
PARALLELISM is a form of computation in
which many instructions are carried out
simultaneously, operating on the principle that
large problems can often be divided into smaller
ones, which are then solved concurrently (in
parallel).
10

11. 12
Why Use PARALLELISM ?
Save time and/or money
Expl: Parallel clusters can be built from cheap components
Solve larger problems
Expl: Many problems are so large and/or complex that it is impractical or
impossible to solve them on a single computer, especially given limited
computer memory
Provide concurrency
Expl: Multiple computing resources can do many things simultaneously.
Use of non-local resources
Limits to serial computing
Available memory
Performance
We can run…
 Larger problems
 Faster
 More cases

12. Parallel programming view
 Parallel computing is a form of computation in which many
calculations are carried out simultaneously.
 In the simplest sense, it is the simultaneous use of multiple
compute resources to solve a computational problem:
 1.To be run using multiple CPUs
 2.A problem is broken into discrete parts that can be solved
concurrently
 3.Each part is further broken down to a series of instructions
 4.Instructions from each part execute simultaneously on
different CPUs
13

13. Parallel / Distributed ( Cluster) / Grid
Computing
 Parallel computing: use of multiple computers or processors working
together on a common task.
 Each processor works on its section of the problem.
 Processors can exchange information
 Distributed (cluster) computing is where several different computers
(processing elements) work separately and they are connected by a network.
 Distributed computers are highly scalable.
 Grid Computing makes use of computers communicating over the
Internet to work on a given problem (so, that in some respects they can be
regarded as a single computer).
14

14. Parallel Computer Memory
Architectures
 Shared Memory
 Uniform Memory Access (UMA)
 Non-Uniform Memory Access (NUMA)
 Distributed Memory
 Hybrid Distributed-Shared Memory
18

15. Shared Memory
General Characteristics:
• Shared memory parallel computers vary widely, but generally have in common
the ability for all processors to access all memory as global address space.
• Multiple processors can operate independently but share the same memory
resources.
• Changes in a memory location effected by one processor are visible to all
other processors.
• Shared memory machines can be divided into two main classes based upon
memory access times: UMA and NUMA.
19

16. Shared Memory (UMA)
20
Uniform Memory Access (UMA):
 Identical processors, Symmetric Multiprocessor
(SMP)
 Equal access and access times to memory
 Sometimes called CC-UMA - Cache Coherent
UMA.
Cache coherent means if one processor updates
a location in shared memory, all the other
processors know about the update.

17. Shared Memory (NUMA)
Non-Uniform Memory Access (NUMA):
 Often made by physically linking two or more SMPs
 One SMP can directly access memory of another SMP
 Not all processors have equal access time to all memories
 Memory access across link is slower
 If cache coherency is maintained, then may also be called CC-
NUMA - Cache Coherent NUMA
21

18. Distributed Memory
Distributed memory systems require a communication network to connect inter-
processor memory.
 Processors have their own local memory.
 Because each processor has its own local memory, it operates independently.
Hence, the concept of cache coherency does not apply.
 When a processor needs access to data in another processor, it is usually the
task of the programmer to explicitly define how and when data is
communicated.
22

19. Hybrid Distributed-Shared Memory
 The largest and fastest computers in the world today employ both shared and
distributed memory architectures.
 The shared memory component is usually a cache coherent SMP machine.
Processors on a given SMP can address that machines memory as global.
 The distributed memory component is the networking of multiple SMPs. SMPs
know only about their own memory - not the memory on another SMP.
Therefore, network communications are required to move data from one SMP
to another.
23

20. Key Difference Between Data And Task
Parallelism
Data Parallelism
 It is the division of
threads(processes) or
instructions or tasks
internally into sub-parts for
execution.
 A task ‘A’ is divided into
sub-parts and then
processed.
24
Task Parallelism
 It is the divisions among
threads (processes) or
instructions or tasks
themselves for execution.
 A task ‘A’ and task ‘B’ are
processed separately by
different processors.

21. Implementation Of Parallel Computing
In Software
 When implemented in software(or rather algorithms), the
terminology calls it ‘parallel programming’.
 An algorithm is split into pieces and then executed, as seen
earlier.
 Important Points In Parallel Programming
 Dependencies - A typical scenario when line 6 of an algorithm is
dependent on lines 2,3,4 and 5
 Application Checkpoints - Just like saving the algorithm, or like creating
a backup point.
 Automatic Parallelisation - Identifying dependencies and parallelising
algorithms automatically. This has achieved limited success.
25

22. Implementation Of Parallel Computing
In Hardware
 When implemented in hardware, it is called as ‘parallel
processing’.
 Typically, when a chunk of load for execution is divided for
processing by units like cores, processors, CPUs, etc.
26

23. 27
Who is doing Parallel Computing?
What are they using it for?
Physics is parallel.
The Human World is parallel too!
Sequence is unusual
Computer programs = models, distributed processes, increasingly parallel

24. 28
Application Examples with
Massive Parallelism
Artificial Intelligence and Automation
AI is the intelligence exhibited by machines or software.
AI systems requires large amount of parallel computing
for which they are used.
1.Image processing
2.Expert Systems
3.Natural Language Processing(NLP)
4.Pattern Recognition

25. 29
Application Examples with
Massive Parallelism
Genetic Engineering
Several of these analysis produce huge amounts of
information which becomes difficult to handle using single
processing units because of which parallel processing
algorithms are used

26. 30
Application Examples with
Massive Parallelism
Medical Applications
 Parallel computing is used in medical image processing
 Used for scanning human body and scanning human brain
 Used in MRI reconstruction
 Used for vertebra detection and segmentation in X-ray
images
 Used for brain fiber tracking

27. 31
Impediments to Parallel Computing
 Algorithm development is harder
—complexity of specifying and coordinating concurrent activities
 Software development is much harder
—lack of standardized & effective development tools, programming models,
and environments
 Rapid changes in computer system architecture
—today’s hot parallel algorithm may not be suitable for tomorrow’s parallel
computer!

28. 32
Next lecture overview
 Some General Parallel Terminology
 Flynn's Taxonomy

29. 33
Test questions
 What is traditional programming view?
 Who is doing Parallel Computing?
 What are they using it for?
 Types of Parallel Computer Hardware.

30. 34
Textbooks
• Course Textbook:
1. Ananth Grama, George Karypis, Vipin Kumar, Anshul
Gupta “Introduction to Parallel Computing” (2nd Edition)
2. Marc Snir, William Gropp “MPI: The Complete
Reference (2-volume set)”
3. Victor Fijkhout with Edmond Chow, Robert Van De
Geijn “Introduction To High-Performance Scientific
Computing”
• Reserve Texts (recommended that you look at periodically)