This document discusses types of parallel and distributed computing systems. It describes shared memory multiprocessor systems which can have uniform memory access (UMA) with the same memory access time for all processors, or non-uniform memory access (NUMA) where memory access times differ. Distributed memory systems have independent nodes that communicate through message passing, while hybrid systems combine aspects of shared and distributed memory architectures. Synchronous communication occurs in real-time while asynchronous allows non-simultaneous engagement.

1. PARALLEL AND DISTRIBUTED
COMPUTING

2. Types of Parallel Computer
A single computer with multiple processors.
Multiple computer interconnected to form high
performance computing platform.
 Shared Memory Multiprocessor system
 Distributed Memory Multicomputer
 Hybrid Memory Architecture

3. Shared Memory Multiprocessor system

4. Shared Memory Multiprocessor system
 At one time one CPU will use the
memory.
 In shared memory architecture every
CPU shares single memory.
 Processors work independently.
 Changes in memory location effected by
one processor are visible to all the
processors.

5. Shared Memory Model Categories
 Uniform Memory Access (UMA)
 Non-uniform Memory Access (NUMA)
 Cache-only Memory Access (COMA)
UMA
NUMA
COMA

6. Uniform Memory Access
MEMORY
CPU1 CPU2 CPU3 CPU4
BUS
 Uniform memory access (UMA) is a shared
memory architecture used in parallel
computers. All the processors in the UMA model
share the physical memory uniformly.

7. Uniform Memory Access
 All the processors have same access to
memory.
 Also known as symmetric multi processors
(SMPs) so no concept of master and slave.
 Less Complexity
There are 3 types of buses used in uniform
Memory Access which are: Single, Multiple and
Crossbar.

8. Non Uniform Memory Access(NUMA)
In a NUMA setup, the individual processors in a computing
system share local memory and can work together. Data
can flow smoothly and quickly.

9. Non Uniform Memory Access(NUMA)
 In non-uniform Memory Access, memory access
time is not equal.
 Processor can access its local memory faster
than non local memory.
Memory access across a link is slower.
 Data can move from home memory to other
cashes memory as needed.

10. Cache-only Memory Access (COMA)

11. Cache-only Memory Access (COMA)
 cache memory, also called cache,
supplementary memory system that temporarily
stores frequently used instructions and data for
quicker processing by the central processing unit
(CPU) of a computer
It is special type of NUMA.
 Local Memory is converted to cache.

12. Distributed Memory Multicomputer

13. Distributed Memory Multicomputer
 A distributed-memory multicomputer system is
modeled in the figure. The system includes
multiple computers known as nodes, related by a
message-passing network. Each node is an
independent computer including a processor,
local memory, and sometimes connected disks or
I/O peripherals.

14. Distributed Memory Multicomputer
 It consist of multicomputer and can be called
nodes or workstation.
 Communicate with each other through message
passing.
 The message passing network provides the
static connection among the nodes.
 Each node is autonomous , with its own
processor and local memory.
 It is loosely accompanied because every node
has its own memory.

15. Hybrid Memory Architecture
• A hybrid architecture is one that combines or adapts
one of the previously discussed systems. For
example, system manufacturers will connect multiple
SMP machines using a high-speed interconnect to
create a hybrid system with a communications model
involving two different levels of service.

16. Hybrid Memory Architecture
 Employ With both distributed and shared
memory architecture.
 Current trends seems to indicate that this type of
memory architecture will continue to prevail and
increase at the high end of computing.
 Performance of system increases.

17. Communication
• Computer communications describes a process
in which two or more computers or devices
transfer data, instructions, and information.
Communication
Synchronous Asynchronous

18. Synchronous Communication
• Synchronous communication is communication
that happens in real-time. The different parties
involved are all actively involved and exchanging
information with one another.
• All parties are online at the same time. When a
message or request is sent, there’s an immediate
response.
• Examples of synchronous communication include
video conferencing, instant messaging, and
telephone conversations.

19. Advantages of Synchronous Communication
• Responses and feedback can be quickly given
and received. The immediacy of synchronous
communication is ideal for timely and important
conversations.
• Communications are infused with a human
element. The emotional context in synchronous
communications gives deeper meaning to the
conversation, and it helps to avoid the risk of
feeling like you’re talking to a computer.

20. Asynchronous Communication
• Asynchronous communication does not
happen in real-time. It involved parties engage
with the conversation and participate in their own
time. With asynchronous communication, there is
no expectation for participants to immediately
respond.
• Asynchronous communication examples are
email conversations, digital workspaces, and
project management tools used by different
organizations.

21. Advantages of Asynchronous Communication
• Participants can respond proactively instead
of reactively. Because there is no pressure to
immediately respond, the different parties can
take the time to contribute more meaningful
content to the conversation.
• It is not dependent on anybody’s immediate
availability. This is valuable for remote teams
and larger group settings which may find it
difficult to find a common schedule for a
dedicated meeting.