This document discusses parallel programming and algorithm design. It covers principles of parallel algorithms like decomposing computations into simultaneous tasks. It describes task decomposition techniques like data, recursive, exploratory and speculative decomposition. It also discusses mapping techniques to distribute tasks across processes, characteristics of tasks, and parallel algorithm models like data-parallel, task graph, work pool, master-slave, and producer-consumer. Finally, it briefly introduces GPU computing and references.

1. Parallel Programming
Prof. Shashikant V. Athawale
Assistant Professor | Computer Engineering
Department | AISSMS College of Engineering,
Kennedy Road, Pune , MH, India - 411001

2. Contents
❖ Principles of Parallel Algorithm Design
❖ Decomposition Techniques, Characteristics of
Tasks and Interactions
❖ Mapping Techniques for Load Balancing
❖ Methods for Containing Interaction Overheads
❖ Parallel Algorithm Models
❖ The Age of Parallel Processing
❖ The Rise of GPU Computing

3. Principles of Parallel Algorithm Design
❖ Parallel algorithm accepts inputs from user and
execute several instructions simultaneously on
different processing units.
❖ The output from the different units are combined to
form the overall output of the algorithm.

4. Age of Parallel Processing

5. Decomposition
❖ The overall computation that is to be computed can
be partitioned into number of small size computations
and these computations are executed in parallel.
❖ There are two types of computations :-
1. Fine-grained decomposition
2.Course-grained decomposition

6. ❖ Fine-grained decomposition is large number of small
tasks.
❖ Course-grained decomposition is small number of
large tasks.

7. Tasks
❖ Tasks is the basic unit of computation.
❖ It is controlled by operating system.
❖ Task dependency graph is a collection of nodes and
edges.
❖ Nodes represent tasks and edges represents
dependency between tasks.
Examples:

8. Decomposition Techniques
There are four types of decomposition techniques :-
❖Data decomposition
❖Recursive decomposition
❖Exploratory decomposition
❖Speculative decomposition

9. Data decomposition
❖ Identify the data on which computations are
performed.
❖ Partition this data across various tasks.
❖ This partitioning induces a decomposition of the
problem.
❖ Data can be partitioned in various ways – this
critically impacts performance of a parallel
algorithm.
❖ Example - Matrix multiplication.

10. Recursive decomposition
❖ Generally suited to problems that are solved using the
divide and-conquer strategy.
❖ A given problem is first decomposed into a set of sub-
problems.
❖ These sub-problems are recursively decomposed
further until a desired granularity is reached.
❖ Example - Merge sort

11. Exploratory decomposition
❖ In many cases, the decomposition of the problem goes
hand in-hand with its execution.
❖ These problems typically involve the exploration
(search) of a state space of solutions.
❖ Problems in this class include a variety of discrete
optimization problems (0/1 integer programming,
QAP, etc.), theorem proving, game playing, etc.
❖ Example - 15 puzzle problem

12. Speculative decomposition
❖ Used in situation when a program has many options
to take in terms of branches based upon the output of
other parts.
❖ The next task to be performed is /may be depend
upon the output of previous task.
❖ Example - Topological sorting.

13. Characteristics of tasks
❖ Generation of task
➢ Static task generation
➢ Dynamic task generation
❖ Size of tasks
❖ Knowledge of Task sizes
❖ Data size

14. Mapping techniques of load balancing
❖ Mapping techniques are used to map tasks into
processes so that parallel executions can be
performed.
❖ Techniques are :
➢ Static mapping
➢ Dynamic mapping

15. Static Mapping
❖ In this,mapping of tasks onto processes is performed
before execution.
❖ Means the tasks are distributed among available
processes prior to execution of algorithm.

16. Dynamic Mapping
❖ In this,the mapping of tasks onto processes is
performed during the execution of algorithms,
❖ This indicates the tasks are distributed among
available processes when algorithm actually executes.
❖ This technique is applied when large number of data
is associated with tasks.

17. Schemes of static mapping
❖ Array distribution schemes
❖ Block distributions
❖ Cyclic distributions
❖ Block-Cyclic distributions
❖ Randomized Block distributions
❖ Hierarchical mappings

18. Schemes of Dynamic mappings
❖ Dynamic mapping with Centralized Schemes
❖ Dynamic mapping with Distributed Schemes

19. Parallel Algorithm Models
❖ The Data-Parallel Model
❖ The Task Graph Model
❖ The Work Pool Model
❖ The Master Slave Model
❖ The producer Consumer Model
❖ Hybrid Models

20. The Data-Parallel Model

21. Task Graph Model

22. Work Pool Model

23. Master Slave Model

24. Producer Consumer Model

25. The Rise of GPU Computing

26. Nvidia Tesla GPU

27. References
❖ http://www.nvidia.in/object/tesla-high-performance-
computing-in.html
❖ https://www.tutorialspoint.com/parallel_algorithm/parall
el_algorithm_models.htm