Aca2 10 11

CSE539: Advanced Computer Architecture
PART-IV

{Chapter 10, 11}
Software for Parallel Programming
Book: “Advanced Computer Architecture – Parallelism, Scalability, Programmability”, Hwang & Jotwani

Sumit Mittu
Assistant Professor, CSE/IT
Lovely Professional University
sumit.12735@lpu.co.in

In this chapter…
• Chapter 10
o Parallel Programming Models
o Parallel Languages and Compilers

• Chapter 11
o Parallel Programming Environments

Sumit Mittu, Assistant Professor, CSE/IT, Lovely Professional University

2

PARALLEL PROGRAMING MODELS
• Programming Model
o A collection of program abstractions providing a simplified and transparent view of the computer
hardware/software system to the programmer.

• Parallel Programming models
o Specifically designed for:
• Multiprocessor, Multicomputer and Vector/SIMD Computer
o Basic Models
• Shared-Variable Model
• Message-Passing Model
• Data-Parallel Model
• Object-Oriented Model*
• Functional and Logic Models*


3

• Resources in Programming Systems
o ACTIVE RESOURCES – processors
o PASSIVE RESOURCES – memory and I/O devices

• Processes
o A processes are the basic computational units in parallel program
o A program is a collection of processes
o Parallelism depends on implementation of IPC (inter-process communication)

• Fundamentals issues around parallel programming
o Specification, Creation, Suspension, Reactivation, Migration, Termination and
Synchronization of concurrent processes
o Process address space may be shared or restricted by limiting scope and access rights

4

Shared-Variable Model
• Shared Variable Communication
o Shared Variable and mutual exclusion
o Main Issues:
• Protected access of critical sections
• Memory Consistency
• Atomicity of Memory Operations
• Fast synchronization
• Shared data structures
• Fast data movement techniques
o Critical Section (CS)
• Code segments accessing shared variable with atomic operation
• Requirements – Mutual Exclusion, No Deadlock in waiting, Non-preemption, Eventual Entry

5

• Shared Variable Communication
o Protected Access using CS
• CS boundary
o Too large…may limit parallelism
o Too small…may add unnecessary code complexity and software overhead
o Shorten a heavy-duty CS or use conditional CSs to maintain balanced performance
• Binary and Counting Semaphores
• Monitors

• Operational Modes used in programming multiprocessor systems
o Multiprogramming
o Level of

Multiprocessing

Multitasking


Multithreading

6

• Partitioning and Replication
o Program Partitioning
o Program Replication
o Role of programmer and compiler in partitioning and replication

• Scheduling and Synchronization
o Static Scheduling
o Dynamic Scheduling

• Cache Coherence and Protection
o Multiprocessor Cache Coherence
o Sequential Consistency Model
o Strong and Weak Consistency Model

7

Message-Passing Model
• Message may be:
o Instructions, Data, Synchronization Signals or Interrupt Signals

• Communication Delays caused by message passing is more than
that caused by accessing shared variables
• Message passing Models
o Synchronous Message Passing
o Asynchronous Message Passing

• Critical issue in programing this model
o How to distribute or duplicate program codes and data sets over processing modes?
• Distributing the computations!
o Trade-offs between computation time and communication overhead to be considered

8

Message-Passing Model
• Synchronous Message Passing
o
o
o
o
o

Sender and receiver processes synchronized in time and space
No need of mutual exclusion
No buffers required
Blocking communication scheme
Uniform Communication Delays, in general

• Asynchronous Message Passing
o
o
o
o

Sender and receiver processes do not require to be synchronized in time and space
Often uses buffers in channels
Non-blocking communication scheme
Non-uniform or arbitrary communication delays


9

Data-Parallel Model
•
•
•
•
•
•
•
•
•

SIMD Lockstep operations
Parallelism explicitly handled by hardware synchronization and flow control
Synchronization done at compile time rather than at run time
Data parallel programs require the use of pre-distributed data sets
Choice of parallel data structure is an important consideration
Emphasis on local computation and data routing operations
Can be applicable to fine-grain problems
Implemented either on SIMD or SPMD
Leads to high degree of parallelism involving thousands of data operations
concurrently

10

Data-Parallel Model
• Data Parallelism
o
o
o
o

Illiac-IV processing model
Connection Machine CM-2 processing model
Synchronous SIMD programming v/s Asynchronous MIMD programming
SIMD Characterization for data parallelism
• Scalar operations and scalar data operands
• Vector operations and vector data operands
• Constant data operands
• Masking operations
• Data-routing operations


11

Data-Parallel Model
• Array Language Extensions for data-parallel processing
o Languages
• Fortran 90 array notation
• CFD for Illiac IV
• DAP Fortran for AMT Distributed Array Processor (DAP)
• C* for TMC Connection Machine
• MPF for MasPar family of MPPs
• Actus for SIMD programming
o Expected Characteristics
• Global Address Space
• Explicit data routing among PEs
• No. of PEs be the function of problem size rather than machine size

12

Data-Parallel Model
• Compiler Support for data parallel programming
o Array language expressions and their optimizing compilers must be embedded in familiar standards
such as Fortran and C with an idea to:
• Unify program execution model
• Facilitate precise control of massively parallel hardware
• Enable incremental migration to data parallel execution
o Array Sectioning
• Allows a programmer to reference a section of a region of a multidimensional array
designated by specifying a:
o Start Index
o Bound (or upper limit)
o Stride (or step-size)
• Vector-valued subscripts are often used to construct arrays from arbitrary permutations of
another array

13

PARALLEL LANGUAGES AND COMPILERS
Language Features for Parallelism
• Chang and Smith (1990) classified language features for parallel
programming into 6 classes according to functionality:
o
o
o
o
o
o

Optimization features
Availability features
Synchronization/Communication features
Parallelism Control features
Data Parallelism features
Process Management features

• In practice, real languages might have some or no features of these
incorporated in them.
o The features act as guidelines for user-friendly and efficient programming environment
o Compiler support, OS assistance, integration with existing environment is required

14

• Optimization features
o Used for Program restructuring and compilation directives in coverting sequentially coded program
into parallel forms
o Purpose: to match software parallelism with hardware parallelism in target machine
o Automated parallelizer
• Express C, Alliant FX Fortran compiler
o Semi-automated parallelizer
• DINO
o Interactive restructure support
• MIMDizer from Pacific Sierra


15

• Availability features
o Purpose:
• Enhance user-friendliness
• Make the language portable to a large class of parallel computers
• Expand applicability of software libraries
o Scalability
• in terms of no. of processors available
o Independence
• From hardware topology
o Compatibility
• With an established sequential language
o Portability
• across shared-memory and message-passing computers

16

• Synchronization/Communication features
o
o
o
o
o
o
o

Single assignment languages
Shared variables (locks) for IPC
Logically shared memory (e.g. tuple space in Linda)
Send/receive primitives for message passing
Remote procedure call
Data flow languages (e.g. ID)
Support for Barriers, mailbox, semaphores and monitors


17

• Parallelism Control features
o
o
o
o
o
o
o
o
o

Coarse, Medium or Fine grain parallelism
Explicit or Implicit parallelism
Global parallelism
Loop parallelism
Task-split parallelism
Shared task queue
Divide-and-conquer paradigm
Shared abstract data types
Task dependency specification


18

• Data Parallelism features
o Purpose:
• How data are accessed and distributed in SIMD or MIMD computers
o Runtime automatic Decomposition
• Express C
o Mapping Specification
• DINO
o Virtual Processor Support
• PISCES 2 and DINO
o Direct Access to Shared Data
• Linda
o SPMD
• DINO and Hypertasking

19

• Process Management features
o Purpose:
• Efficient creation of parallel processes
• Implement multi-threading or multi-asking
• Program Partitioning and Replication
• Dynamic Load balancing at runtime
o Dynamic Process creation
o LWP (threads)
o Replicated Workers
o Partitioned networks
o Automatic Load Balancing


20

Parallel Language Constructs
• Fortran 90 Array Notations
o Lower Bound : Upper Bound : Stride
• E1 : E2 : E3
• E1: E2
• E1: * : E3
• E1 : *
• E1
• *

• Parallel Flow Control
o
o
o
o

Doall – Endall or Forall – Endall
Doacross – Endacross or ParDo – ParEnd
Cobegin – Coend or ParBegin – ParEnd
Fork – Join


21

Optimizing Compilers for Parallelism
• Major Phases of parallelizing compiler
o Phase I : Flow Analysis
• Data Dependence
• Control Dependence
• Reuse Analysis
o Phase II : Program Optimizations
• Vectorization
• Parallelizations
• Locality
• Pipelining
o Phase III : Parallel Code Generation
• Granularity
• Degree of Parallelism
• Code Scheduling

Flow Analyis

Program
Optimizations

Parallel Code
Generation
22

Optimizing Compilers for Parallelism
• Parallelizing Compilers
o Parafrase [1984]
• David Kuck, University of Illinois
• Vectorization of paralleiszation of Fortran 77 programs
o Parafrase 2
• David Kuck, University of Illinois
• Parallelization of C or Pascal programs
o KAP vectorizer
• Kuck and associatesBased on Parafrase 2
o PFC, PFC+ and Parascope
• Allen and Kennedy (1984)
o Other compilers
• PTRAN, Alliant FX/F, Convex Vectorizing compiler, Cray CFT compiler, VAST vectorizer, etc.

23

PARALLEL PROGRAMMING ENVIRONMENTS
• Constituents of Environment for parallel programming
o
o
o
o

Hardware Platforms or Machine models
Languages Supported
OS and Software Tools
Application Packages

• Software Tools and Environments
o Parallel Languages
o Integrated Environment Components
• Editor
• Debugger
• Performance Monitors
• Program Visualizers


24

• Fig 11.1


25

• Types of Integrated Environment
o Basic: Provides simple tools for
• Program tracing facility for debugging and performance monitoring
• Graphic mechanism for specifying dependence graphs
o Limited: Provides tools for
• Parallel Debugging
• Performance monitoring
• Program visualization beyond capability of basic environments
o Well-developed: Provides intensive tools for
• Debugging programs
• Interaction of textual/graphical representations of parallel program
• Visualization support for performance monitoring, program visualization, parallel I/O, etc.

26

• Environment Features
o Design Issues
• Compatibility
• Expressiveness
• Ease of use
• Efficiency
• Portability
o Parallel languages may developed as extension to existing sequential languages
o A new parallel programming language has the advantage of:
• Using high-level parallel concepts or constructs for parallelism instead of imperative
(algorithmic) languages which are inherently sequential
o Special optimizing compilers detect parallelism and transform sequential constructs into parallel.

27

• Environment Features
o Special optimizing compilers detect parallelism and transform sequential constructs into parallel.
o High Level parallel constructs can be:
• implicitly embedded in syntax, or
• explicitly specified by users
o Compiler Approaches
• Pre-processors:
o use compiler directives or macroprocessors
• Pre-compilers:
o include automated and semi-automated parallelizers
• Parallelizing Compilers


28

• Fig 11.2


29

• Summary of Important Environment Features

•

o Control flow graph generation
o Integrated textual/graphical map
o Parallel debugger at source code level
o Performance Monitoring by either software or hardware means
o Performance Prediction Model
o Program visualizer for displaying program structures and data flow patterns
o Parallel I/O for fast data movement
o Visualization support for program development and guidance for parallel computations
o OS support for parallelism in front-end or back-end environments
o Communication support in a network environment
Refer to Table 11.1 for attributes of Representative Parallel Programming Tools

30

Case Studies
• Cray Y-MP Software
• Intel Paragon XP/S Software
o Refer Table 11.2 for its characteristic attributes

• CM-5 Software
o Refer Figure 11.3 for several software layers of Connection Machine System


31

Case Studies


32

That’s All Folks!
Attempt your best in exams


33

Aca2 10 11

More Related Content

What's hot

Viewers also liked

Similar to Aca2 10 11

More from Sumit Mittu

Recently uploaded

Aca2 10 11