Lecture-AI and Alogor Parallel Aglorithms.pptx

1. Parllel Prgramming Systems
Dr. Shahid Naseem
Assistant Professor (IT)
Division of Science & Technology
University of Education, Township Campus, Lahore
shahid.naseem@ue.edu.pk
Superior University, Gold Campus

2. Parallel Algorithms
Superior University, Gold Campus
• A parallel algorithm is a type of algorithm that divides
a larger problem into smaller independent sub-tasks,
which can be executed simultaneously on multiple
processing devices.
• After the sub-tasks are completed, their results are
combined to form a solution to the original problem.
• This approach enhances computational efficiency by
utilizing the capabilities of parallel computing, where
multiple processors perform tasks concurrently.

3. Parallel Algorithms
Superior University, Gold Campus
Parallel algorithms offer several advantages, particularly in
the context of parallel and distributed computing. Here
are some key benefits:
1. Increased Speed and Efficiency: By dividing tasks among
multiple processors, parallel algorithms can significantly reduce
the time required to solve complex problems, especially for
large datasets or computationally intensive tasks.
2. Scalability: Parallel algorithms can scale well as the number of
processors increases. This means that as more processing units
are added, the performance can continue to improve, allowing
for the handling of larger problems.

4. Parallel Algorithms
Superior University, Gold Campus
3. Resource Utilization: They make better use of available
hardware resources. Instead of relying on a single processor,
parallel algorithms distribute workloads across multiple
processors, which leads to more efficient use of memory and
processing power.
4. Handling Large Data Sets: Parallel algorithms are particularly
effective for applications involving large data sets, such as
scientific simulations, big data analytics, and image processing,
where data can be partitioned and processed concurrently.

5. Parallel Algorithms
Superior University, Gold Campus
5. Improved Performance with Tasks with Low Latency: When
tasks involve high latencies (like network communication),
parallel algorithms can improve performance by allowing other
processes to execute while waiting for operations to complete.
6. Improved Fault Tolerance: In distributed systems, parallel
algorithms can provide redundancy. If one processor fails,
others can continue processing, allowing the system to be
more resilient to failures.
7. Enhanced Problem Solving: Some problems, particularly in
fields like machine learning, optimization, and numerical
analysis, can only be practically solved using parallel algorithms
due to their complexity and size.

6. Concurrency and Synchronization
Superior University, Gold Campus
• Concurrency can be achieved by using multiple
processors, cores, or threads, depending on the level
of parallelism that you want to achieve.
• Synchronization is the coordination of your concurrent
tasks, to ensure that they do not interfere with each
other, or access shared resources in an inconsistent or
unsafe way.

7. Concurrency and Synchronization
Superior University, Gold Campus
• Concurrency and synchronization are essential
concepts in parallel and distributed computing.
• Concurrency allows multiple processes to
execute simultaneously, improving system
responsiveness and efficiency.
• In distributed systems, synchronization is crucial
for maintaining consistency and coordination
among different components, ensuring they can
cooperate effectively.

8. Concurrency and Synchronization
Superior University, Gold Campus
• This involves mechanisms that manage access
to shared resources and prevent conflicts,
which is vital for the integrity of distributed
computations.
• It's important to double-check specific
implementations and theories, as they can vary
widely depending on the context and system
architecture.

9. Techniques used for Synchronization
Superior University, Gold Campus
In parallel computing, various techniques are employed
for synchronization to ensure that processes coordinate
effectively and manage access to shared resources. Here
are some common synchronization techniques:
1. Mutexes (Mutual Exclusions): Mutexes are locks used to ensure
that only one thread can access a resource at a time. If one
thread holds a mutex, other threads are blocked until the mutex
is released.

10. Techniques used for Synchronization
Superior University, Gold Campus
2. Semaphores: Semaphores are signaling mechanisms that can
be used to control access to a common resource by multiple
processes. They can be binary (similar to mutexes) or counting,
allowing a certain number of threads to access resources
simultaneously.
3. Locks: Many languages and frameworks provide various types
of locks (e.g., read/write locks) that allow threads to request
permission to access a resource. Read locks allow multiple
threads to read simultaneously, while write locks ensure
exclusive access.

11. Techniques used for Synchronization
Superior University, Gold Campus
4. Barriers: Barriers are synchronization points where processes or
threads wait until all participants reach the barrier before
continuing. This is useful for ensuring that certain computations
are completed before proceeding.
5. Condition Variables: Condition variables allow threads to wait
for a certain condition to be met before continuing execution.
They are often used in conjunction with mutexes, allowing
threads to sleep until notified of a condition change.
6. Atomic Operations: Atomic operations are low-level operations
that complete in a single step relative to other threads. They
ensure that read-modify-write sequences can be performed
without interruption, avoiding race conditions.

12. Techniques used for Synchronization
Superior University, Gold Campus
7. Message Passing: In distributed environments or certain
parallel computing architectures, processes communicate and
synchronize through message passing, where they send and
receive messages to coordinate actions without shared
memory.
8. Fine-Grained Locking: Instead of locking a large section of code
or a resource, fine-grained locking involves locking smaller
sections, allowing more concurrency and reducing contention.

13. Granularity
Superior University, Gold Campus
• In parallel computing, granularity is a measure of the
ratio of computation to communication.
• Periods of computation are typically separated from
periods of communication by synchronization events.
• The grain of parallelism is constrained by the inherent
characteristics of the algorithms constituting the
application.

14. Granularity
Superior University, Gold Campus
• Granularity in parallel and distributed computing refers
to the size of tasks or units of work within a
computation.
• It measures the amount of computation relative to
communication, often expressed as the ratio of
computation time to communication time (G =
Tcomp / Tcomm).

15. Granularity
Superior University, Gold Campus
• A finer granularity involves smaller tasks that may
introduce more overhead due to communication,
while coarser granularity consists of larger tasks that
can be executed with less frequent communication but
may lead to underutilization of resources.

16. Types of Granularity
Superior University, Gold Campus
Granularity can be categorized into two main types:
1. Coarse Granularity:
• Involves large tasks that take a significant amount of time to
complete.
• Each task may involve a substantial amount of computation, and
there are relatively few tasks running concurrently.
• This can lead to lower overhead in communication between
tasks since there are fewer interactions, but it may not fully
utilize the available processing resources.

17. Types of Granularity
Superior University, Gold Campus
2. Fine Granularity:
• Involves breaking down tasks into smaller units of work that
can be executed in parallel.
• This allows for a high degree of concurrency, as many small
tasks can be processed simultaneously.
• However, fine granularity can also introduce overhead from
frequent communication and synchronization between tasks,
which can diminish the overall performance gains.

18. Load Balancing
Superior University, Gold Campus
• Load balancing in parallel and distributed computing
involves redistributing workloads among multiple
computing nodes, such as processors or servers, to
enhance performance and efficiency.
• This process helps ensure that no single node is
overwhelmed while others are underutilized, leading
to optimized resource use and faster processing times.

19. Load Balancing
Superior University, Gold Campus
Load balancing improves system performance in several
ways:
1. Optimized Resource Utilization: By distributing workloads
evenly across all available nodes, load balancing prevents any
individual node from becoming a bottleneck. This ensures that
resources such as CPU, memory, and network bandwidth are
used efficiently.
2. Reduced Response Time: When workloads are balanced, each
node can process requests more quickly, leading to faster
response times for users. This is particularly important in high-
traffic environments where timely responses are crucial.

20. Load Balancing
Superior University, Gold Campus
3. Increased Throughput: By maximizing the use of available
resources, load balancing can increase the overall throughput of
the system. More tasks can be completed in a given time
period, enhancing productivity.
4. Scalability: Load balancing facilitates scalability by allowing new
nodes to be added easily to the system. As demand increases,
additional nodes can take on more workload, maintaining
performance levels.
5. Fault Tolerance and Reliability: In the event that a node fails,
load balancers can redirect traffic to other operational nodes,
ensuring continued service availability. This redundancy
increases system reliability.

21. Load Balancing
Superior University, Gold Campus
6. Improved Latency Management: Load balancing can reduce
latency by routing requests to the nearest or least-busy server,
helping to enhance the user experience, especially in
distributed systems.