Parallel and distributed computing

1. PARALLEL AND DISTRIBUTED
COMPUTING

2. Types of Parallel Computer
A single computer with multiple processors.
Multiple computer interconnected to form high
performance computing platform.
 Shared Memory Multiprocessor system
 Distributed Memory Multicomputer
 Hybrid Memory Architecture

3. Shared Memory Multiprocessor system

4. Shared Memory Multiprocessor system
 At one time one CPU will use the
memory.
 In shared memory architecture every
CPU shares single memory.
 Processors work independently.
 Changes in memory location effected by
one processor are visible to all the
processors.

5. Shared Memory Model Categories
 Uniform Memory Access (UMA)
 Non-uniform Memory Access (NUMA)
 Cache-only Memory Access (COMA)
UMA
NUMA
COMA

6. Uniform Memory Access
MEMORY
CPU1 CPU2 CPU3 CPU4
BUS
 Uniform memory access (UMA) is a shared
memory architecture used in parallel
computers. All the processors in the UMA model
share the physical memory uniformly.

7. Uniform Memory Access
 All the processors have same access to
memory.
 Also known as symmetric multi processors
(SMPs) so no concept of master and slave.
 Less Complexity
There are 3 types of buses used in uniform
Memory Access which are: Single, Multiple and
Crossbar.

8. Non Uniform Memory Access(NUMA)
In a NUMA setup, the individual processors in a computing
system share local memory and can work together. Data
can flow smoothly and quickly.

9. Non Uniform Memory Access(NUMA)
 In non-uniform Memory Access, memory access
time is not equal.
 Processor can access its local memory faster
than non local memory.
Memory access across a link is slower.
 Data can move from home memory to other
cashes memory as needed.

10. Cache-only Memory Access (COMA)

11. Cache-only Memory Access (COMA)
 cache memory, also called cache,
supplementary memory system that temporarily
stores frequently used instructions and data for
quicker processing by the central processing unit
(CPU) of a computer
It is special type of NUMA.
 Local Memory is converted to cache.

12. Distributed Memory Multicomputer

13. Distributed Memory Multicomputer
 A distributed-memory multicomputer system is
modeled in the figure. The system includes
multiple computers known as nodes, related by a
message-passing network. Each node is an
independent computer including a processor,
local memory, and sometimes connected disks or
I/O peripherals.

14. Distributed Memory Multicomputer
 It consist of multicomputer and can be called
nodes or workstation.
 Communicate with each other through message
passing.
 The message passing network provides the
static connection among the nodes.
 Each node is autonomous , with its own
processor and local memory.
 It is loosely accompanied because every node
has its own memory.

15. Hybrid Memory Architecture
• A hybrid architecture is one that combines or adapts
one of the previously discussed systems. For
example, system manufacturers will connect multiple
SMP machines using a high-speed interconnect to
create a hybrid system with a communications model
involving two different levels of service.

16. Hybrid Memory Architecture
 Employ With both distributed and shared
memory architecture.
 Current trends seems to indicate that this type of
memory architecture will continue to prevail and
increase at the high end of computing.
 Performance of system increases.

17. Communication
• Computer communications describes a process
in which two or more computers or devices
transfer data, instructions, and information.
Communication
Synchronous Asynchronous

18. Synchronous Communication
• Synchronous communication is communication
that happens in real-time. The different parties
involved are all actively involved and exchanging
information with one another.
• All parties are online at the same time. When a
message or request is sent, there’s an immediate
response.
• Examples of synchronous communication include
video conferencing, instant messaging, and
telephone conversations.

19. Advantages of Synchronous Communication
• Responses and feedback can be quickly given
and received. The immediacy of synchronous
communication is ideal for timely and important
conversations.
• Communications are infused with a human
element. The emotional context in synchronous
communications gives deeper meaning to the
conversation, and it helps to avoid the risk of
feeling like you’re talking to a computer.

20. Asynchronous Communication
• Asynchronous communication does not
happen in real-time. It involved parties engage
with the conversation and participate in their own
time. With asynchronous communication, there is
no expectation for participants to immediately
respond.
• Asynchronous communication examples are
email conversations, digital workspaces, and
project management tools used by different
organizations.

21. Advantages of Asynchronous Communication
• Participants can respond proactively instead
of reactively. Because there is no pressure to
immediately respond, the different parties can
take the time to contribute more meaningful
content to the conversation.
• It is not dependent on anybody’s immediate
availability. This is valuable for remote teams
and larger group settings which may find it
difficult to find a common schedule for a
dedicated meeting.