This document compares message passing and shared memory architectures for parallel computing. It defines message passing as processors communicating through sending and receiving messages without a global memory, while shared memory allows processors to communicate through a shared virtual address space. The key difference is that message passing uses explicit communication through messages, while shared memory uses implicit communication through memory operations. It also discusses how the programming model and hardware architecture can be separated, with message passing able to support shared memory and vice versa.

2. Group Members
Hamza Zahid (131391)
Fahad Nadeem khan
Abdual Hannan
AIR UNIVERSITY
MULTAN CAMPUS

3. Shared memory VS Message passing

4. Topics
• Message passing
• Shared memory
• Difference b/w message passing and shared memory

5. Message Passing

6. INTRODUCTION
• The architecture is used to communicate data among a
set of processors without the need for a global memory
• Each PE has its own local memory and communicates
with other PE using message

8. MP network
Two important factors must be considered;
 Link bandwidth –the number of bits that can be transmitted per unit
of times(bits/s)
 Message transfer through the network

9. Process communication
Processes running on a given processor use what is called
internal channels to exchange messages among themselves
Processes running on different processors use the external
channesls to exchange messages

10. Data exchanged
Data exchanged among processors cannot be shared; it
is rather copied (using send/ receive messages)
An important advantage of this form of data exchange is
the elimination of the need for synchronization constructs
, such as semaphores , which results in performance
improvement

11. Message-Passing Interface – MPI
Standardization
 MPI is the only message passing library which can be
considered a standard. It is supported on virtually all HPC
platforms. Practically, it has replaced all previous message
passing libraries.
Portability
There is no need to modify your source code when you port
your application to a different platform that supports the MPI
standard

12. Message-Passing Interface – MPI
Performance Opportunities
 Vendor implementations should be able to exploit native hardware
features to optimize performance.
Functionality
 Over 115 routines are defined.
Availability
 A variety of implementations are available, both vendor and public
domain.

13. MPI basics
Start Processes
Send Messages
Receive Messages
Synchronize
With these four capabilities, you can construct any program.
MPI offers over 125 functions.

14. Shared memory

15. Introduction
 Processors communicate with shared address space
 Easy on small-scale machines
 Shared memory allows multiple processes to share virtual memory
space.
 This is the fastest but not necessarily the easiest (synchronization-
wise) way for processes to communicate with one another.
 In general, one process creates or allocates the shared memory
segment.
 The size and access permissions for the segment are set when it is
created.

17. Uniform Memory Access (UMA)
 Most commonly represented today by Symmetric Multiprocessor
(SMP) machines
 Identical processors
 Equal access and access times to memory
 Sometimes called CC-UMA - Cache Coherent UMA. Cache
coherent means if one processor updates a location in shared
memory, all the other processors know about the update. Cache
coherency is accomplished at the hardware level.

18. Shared Memory (UMA)

19. Non-Uniform Memory Access (NUMA)
 Often made by physically linking two or more SMPs
 One SMP can directly access memory of another SMP
 Not all processors have equal access time to all memories
 Memory access across link is slower
 If cache coherency is maintained, then may also be called CC-
NUMA - Cache Coherent NUMA

20. Shared Memory (NUMA)

21. Advantages
 Global address space provides a user-friendly programming
perspective to memory
 Model of choice for uniprocessors, small-scale MPs
 Ease of programming
 Lower latency
 Easier to use hardware controlled caching
 Data sharing between tasks is both fast and uniform due to the
proximity of memory to CPUs

22. Disadvantages
 Primary disadvantage is the lack of scalability between memory
and CPUs. Adding more CPUs can geometrically increases traffic
on the shared memory-CPU path, and for cache coherent
systems, geometrically increase traffic associated with
cache/memory management.
 Programmer responsibility for synchronization constructs that
ensure "correct" access of global memory.
 Expense: it becomes increasingly difficult and expensive to design
and produce shared memory machines with ever increasing
numbers of processors.

23. Difference

24. Message Passing vs. Shared Memory
• Difference: how communication is achieved between tasks
 Message passing programming model
– Explicit communication via messages
– Loose coupling of program components
– Analogy: telephone call or letter, no shared location accessible to all
 Shared memory programming model
– Implicit communication via memory operations (load/store)
– Tight coupling of program components
– Analogy: bulletin board, post information at a shared space
• Suitability of the programming model depends on the problem to be solved.
Issues affected by the model include:
 Overhead, scalability, ease of programming

25. Message Passing vs. Shared Memory Hardware
• Difference: how task communication is supported in hardware
 Shared memory hardware (or machine model)
– All processors see a global shared address space
• Ability to access all memory from each processor
– A write to a location is visible to the reads of other processors
 Message passing hardware (machine model)
– No global shared address space
– Send and receive variants are the only method of communication between
processors (much like networks of workstations today, i.e. clusters)
• Suitability of the hardware depends on the problem to be solved as well as the
programming model.

26. Programming Model vs. Architecture
• Machine  Programming Model
– Join at network, so program with message passing model
– Join at memory, so program with shared memory model
– Join at processor, so program with SIMD or data parallel
• Programming Model  Machine
– Message-passing programs on message-passing machine
– Shared-memory programs on shared-memory machine
– SIMD/data-parallel programs on SIMD/data-parallel machine

27. Separation of Model and Architecture
 Shared Memory
– Single shared address space
– Communicate, synchronize using load / store
– Can support message passing
 Message Passing
– Send / Receive
– Communication + synchronization
– Can support shared memory