Parallel and distributed computing allows problems to be broken into discrete parts that can be solved simultaneously. This approach utilizes multiple processors that work concurrently on different parts of the problem. There are several types of parallel architectures depending on how instructions and data are distributed across processors. Shared memory systems give all processors access to a common memory space while distributed memory assigns private memory to each processor requiring explicit data transfer. Large-scale systems may combine these approaches into hybrid designs. Distributed systems extend parallelism across a network and provide users with a single, integrated view of geographically dispersed resources and computers. Key challenges for distributed systems include transparency, scalability, fault tolerance and concurrency.

1. UNDERLYING PRINCIPLES
OF PARALLELAND
DISTRIBUTED COMPUTING

2. WhatisParallelComputing?
 Traditionally, software has been written for serial
computation
To be run on a single computer having a single Central
Processing Unit
A problemis broken intoa discrete series of instructions
Instructions are executed one after another
Only one instruction may execute at any momentin time

3. WhatisParallelComputing?

4. WhatisParallelComputing?

5. UsesforParallelComputing
 ScienceandEngineering
 Historically, parallel computing has been used to model
difficult problems in many areas of science and engineering
Atmosphere, Earth, Environment, Physics, Bioscience,
Chemistry, Mechanical Engineering, Electrical
Engineering, Circuit Design, Microelectronics,
Defense, Weapons.

6. WhoisUsingParallelComputing?

8. ConceptsandTerminology

9.  Named after the Hungarian mathematician John von Neumann who first authored the general
requirements foran electroniccomputerinhis 1945papers
 Since then, virtuallyallcomputers have followed this
basic design
 Comprises offourmain components:
 RAM is used tostore both program instructions anddata
 Control unitfetches instructions or data frommemory,
 Decodes instructions and then sequentially coordinates
operations to accomplish the programmed task.
 Arithmetic Unitperforms basic arithmetic operations
 Input-Output is the interface tothe human operator
vonNeumannArchitecture

10. Flynn'sClassicalTaxonomy
 Widely used classifications is called Flynn's Taxonomy
 Based upon the number of concurrent instruction and data
streams available in thearchitecture

11. SingleInstruction,SingleData
 A sequential computer which exploits no parallelism in either the
instruction or datastreams
Single Instruction: Only one instruction stream is being acted on
by the CPU during any one clock cycle
Single Data: Only one data stream is being used as input during
any one clock cycle
Can have concurrent processing
characteristics: Pipelined execution

12. SingleInstruction,MultipleData
A computer which exploits multipledata streams against a single
instruction stream toperform operations.
Single Instruction: All processing units execute the same instruction
at any given clock cycle.
Multiple Data: Each processing unit can operate on a different
data element.

13. MultipleInstruction,SingleData
MultipleInstruction,SingleData
 Multiple Instruction: Each processing unit operates on the data
independently via separate instruction streams.
 Single Data: A single data stream is fed into multiple
processing units.
 Some conceivable uses might be:
Multiple cryptography algorithms attempting to crack a single
coded message.

14. MultipleInstruction,MultipleData
 Multiple autonomous processors simultaneously executing
different instructions on differentdata.
Multiple Instruction: Every processor may be executing a
different instruction stream.
Multiple Data: Every processor may be working with a
differentdata stream.

15. SomeGeneralParallelTerminology
 Parallelcomputing
Using parallel computer tosolve single problems faster
 Parallelcomputer
Multiple-processor or multi-core system supporting parallel
programming
 Parallelprogramming
Programming in a language that supports concurrency
explicitly
 SupercomputingorHighPerformanceComputing
Using the world's fastest and largest computers to solve large
problems

16. LimitsandCostsof Parallel
Programming
• Introducing the number of processors N performing the parallel
fraction of work P, the relationship can be modeledby:

17. ParallelComputerMemory
Architectures

18.  All processors access all memory as global address space
 Multiple processors can operate independently butshare the
same memory resources
 Changes in a memory location effected by one processor
are visible toall other processors
SharedMemory

19. SharedMemory Classification
 UMAandNUMA,based uponmemoryaccesstimes
 UniformMemoryAccess(UMA)
Most commonly represented today by Symmetric Multiprocessor
(SMP) machines
Identical processors
Equal access and access times to memory
Sometimes called CC-UMA - Cache Coherent UMA
Cache coherent means if one processor updates a location in
shared memory, all the other processors know about the update.
Cache coherency is accomplished at the hardware level

20. SharedMemory
 Non-UniformMemoryAccess(NUMA)
Often made by physically linking twoor more SMPs
One SMP can directly access memory of another SMP
Not all processors have equal access time toallmemories
Memory access across link is slower
If cache coherency is maintained, then may also be called CC-
NUMA - Cache Coherent NUMA

21. DistributedMemory
 Processors have theirownlocal memory
 Changes to processor’s local memory have no effect on
the memory of other processors.
 When a processor needs access to data in another processor,
it is usually the task of the programmer to explicitly define how
and when data is communicated.
 Synchronization between tasks is likewise the programmer's
responsibility.
 The network "fabric" used for data transfer varieswidely,
though itcan be as simple as Ethernet.

22. HybridDistributed-SharedMemory
 The largest and fastest computers in the world today employ both
shared and distributedmemory architectures
 The shared memory component can be a shared memory
machine or graphics processingunits.
 The distributed memory component is the networking of
multipleshared memory or GPU machines.

23. Distributed Systems

24. Distributed Systems
 A collection of independent computers that appear to the
users of the system as a single computer.
 A collection of autonomous computers, connected
through a network and distribution middleware which
enables computers to coordinate their activities and to
share the resources of the system, so that users perceive
the system as a single, integrated computing facility
Eg:Internet

25. Example-ATM

26. Example-Mobiledevicesina
DistributedSystem

27. Basic problems and challenges
Basic problem and challenges
 Transparency
 Scalability
 Fault tolerance
 Concurrency
 Openness
 These challenges can also be seen as the goals or desired
properties of a distributed system

28. REFERENCE
• Mastering Cloud computing foundation and
application programming rajkumar
buyya,christian vecchiola,s.Tamarai selvi

29. THANK YOU