The document discusses various parallel programming models. It describes data parallelism which emphasizes concurrent execution of the same task on different data elements, and task parallelism which emphasizes concurrent execution of different tasks. It also covers shared memory and distributed memory models, explicit and implicit parallelism, and common parallel programming tools like MPI. Message passing and data parallel models are explained in more detail.

1. UNIT 5UNIT 5
Parallel ProgrammingParallel Programming
ModelsModels

2. Parallel Programming Models
Parallel Programming Models:
 Data parallelism / Task parallelism
 Explicit parallelism / Implicit parallelism
 Shared memory / Distributed memory
 Other programming paradigms
• Object-oriented
• Functional and logic

3. Parallel Programming Models
 Parallel programming models exist as
an abstraction above hardware and
memory architectures.
 These models are NOT specific to a
particular type of machine or memory
architecture.
 Any of these models can be
implemented on any underlying
hardware

4. Parallel Programming Models
 Data Parallelism
Parallel programs that emphasize concurrent execution of the
same task on different data elements (data-parallel programs)
• Most programs for scalable parallel computers are data parallel in
nature.
 Task Parallelism
Parallel programs that emphasize the concurrent execution of
different tasks on the same or different data
• Used for modularity reasons.
• Parallel programs, structured as a task-parallel composition of data-
parallel components is common.

5. Parallel Programming Models
Data parallelism
Task Parallelism

6. Parallel Programming Models
 Explicit Parallelism
The programmer specifies directly the activities of the multiple
concurrent “threads of control” that form a parallel
computation.
• Provide the programmer with more control over program behavior
and hence can be used to achieve higher performance.
 Implicit Parallelism
The programmer provides high-level specification of program
behavior.
It is then the responsibility of the compiler or library to
implement this parallelism efficiently and correctly.

7. Parallel Programming Models
 Shared Memory
The programmer’s task is to specify the activities of a set of processes
that communicate by reading and writing shared memory.
• Advantage: the programmer need not be concerned with data-distribution
issues.
• Disadvantage: performance implementations may be difficult on computers
that lack hardware support for shared memory, and race conditions tend to
arise more easily
 Distributed Memory
Processes have only local memory and must use some other
mechanism (e.g., message passing or remote procedure call) to
exchange information.
• Advantage: programmers have explicit control over data distribution and
communication.

8. Shared vs Distributed Memory
 Shared memory
 Distributed memory
Memory
Bus
P P P P
P P P P
M M M M
Network

9. Parallel Programming Models
Parallel Programming Tools:
 Parallel Virtual Machine (PVM)
 Message-Passing Interface (MPI)
 High-Performance Fortran (HPF)
 Parallelizing Compilers

10. Parallel Programming Models
Message Passing Model
 Used on Distributed memory MIMD architectures
 Multiple processes execute in parallel
asynchronously
• Process creation may be static or dynamic
 Processes communicate by using send and
receive primitives

11. Parallel Programming Models
 Blocking send: waits until all data is received
 Non-blocking send: continues execution after
placing the data in the buffer
 Blocking receive: if data is not ready, waits until it
arrives
 Non-blocking receive: reserves buffer and
continue execution. In a later wait operation if data
is ready, copies it into the memory.

12. Distributed Memory / Message
Passing Model

13. Parallel Programming Models
 Synchronous message-passing: Sender and
receiver processes are synchronized
• Blocking-send / Blocking receive
 Asynchronous message-passing: no
synchronization between sender and receiver
processes
• Large buffers are required. As buffer size is finite, the
sender may eventually block.

14. Parallel Programming Models
Advantages of message-passing model
 Programs are highly portable
 Provides the programmer with explicit control over the
location of data in the memory
Disadvantage of message-passing model
 Programmer is required to pay attention to such details as
the placement of memory and the ordering of
communication.

15. Parallel Programming Models
Factors that influence the performance of message-passing
model
 Bandwidth
 Latency
 Ability to overlap communication with computation.

16. Parallel Programming Models
Shared Memory Model
 Used on Shared memory MIMD architectures
 Program consists of many independent threads
 Concurrently executing threads all share a single, common
address space.
 Threads can exchange information by reading and writing to
memory using normal variable assignment operations

17. Parallel Programming Models
Memory Coherence Problem
 To ensure that the latest value of a variable updated in one thread
is used when that same variable is accessed in another thread.
 Hardware support and compiler support are required
 Cache-coherency protocol
Thread 1 Thread 2
X

18. Parallel Programming Models
Synchronization operations in Shared Memory Model
 Monitors
 Locks
 Critical sections
 Condition variables
 Semaphores
 Barriers

19. Data Parallel Model

20. Data Parallel Model
 It require extension use of pre distributed
data sets.
 Inter connected data structure are also
needed to facilitate the data exchange
operations
 It emphasize on the local computations
and performs several data routing
operations like replication, reduction, etc

21. Data Parallel Model
 The data parallel model demonstrates the
following characteristics:
• Most of the parallel work focuses on performing
operations on a data set.
• The data set is typically organized into a
common structure, such as an array or cube.
• A set of tasks work collectively on the same data
structure, however, each task works on a
different partition of the same data structure.

22. Data Parallel Model
• Tasks perform the same operation on their
partition of work, for example, "add 4 to every
array element".
• On shared memory architectures, all tasks may
have access to the data structure through global
memory.
• On distributed memory architectures the data
structure is split up and resides as "chunks" in
the local memory of each task.

23. Data Parallel Model

27. Parallel language and compilers
 They are different as compared to the
sequential language.
 More focus is given on the hardware as
compared to the program parallelism
using high level of abstraction.

28. Parallel language and compilers
 Language features
• Algorithm design : the problem must be
decomposable so that it can be breakup in
the part and can work in parallel.
• Programming Language: language needs
specific statements for implementing the
parallel algorithm
• Parallel Computer : the computer or network
of computers needs hardware capabilities for
a real parallel running of the program.

29. Examples of the parallel
language
 FORTRAN
 JAVA
 C++
 C*: C extended for parallelism

30. Parallel Compilers
 It Automatically Transform the sequential
program into parallel program
 It identify the loops whose iteration can
be executed in parallel whenever
possible.
 It often uses stages and concepts of data
dependencies.

31. Steps involve in creating
parallel program
 Decomposition :
• breakup computation in tasks
 Assignment:
• involve mechanism to divide work among
process,
 Orchestration :
• Naming data, structure communication,
synchronization, organization data structure
and scheduling the task, data access ,
communication and synchronization
 Mapping : processes to processors

32. Language features for
parallelism