OpenHPI - Parallel Programming Concepts - Week 6

Parallel Programming Concepts
OpenHPI Course
Week 6 : Patterns and Best Practices
Unit 6.1: Parallel Programming Patterns
Dr. Peter Tröger + Teaching Team

Summary: Week 5
■  “Shared nothing” systems provide very good scalability
□  Adding new processing elements not limited by “walls”
□  Different options for interconnect technology
■  Task granularity is essential
□  Surface-to-volume effect
□  Task mapping problem
■  De-facto standard is MPI programming
■  High level abstractions with
□  Channels
□  Actors
□  MapReduce
2
OpenHPI | Parallel Programming Concepts | Dr. Peter Tröger
„What steps / strategy would you apply
to parallelize a given compute-intense program? “

The Parallel Programming Problem
3
Execution
Environment
Parallel Application Match ?
Configuration
Flexible
Type

Parallelization and Design Patterns
■  Parallel programming relies on experience
□  Identification of concurrency
□  Identification of feasible algorithmic structures
□  If done wrong, performance / correctness may suffer
■  Rule of thumb: Somebody else is smarter than you !!
■  Design Pattern
□  Best practices, formulated as a template
□  Focus on general applicability to common problems
□  Well-known in object-oriented programming (“gang of four”)
■  Parallel design patterns in literature
□  Structured parallelization methodologies (== pattern)
□  Algorithmic building blocks commonly found (== pattern)
4

Patterns for Parallel Programming
[Mattson et al.]
■  Phases in creating a parallel program
□  Finding Concurrency: Identify and
analyze exploitable concurrency
□  Algorithm Structure: Structure
the algorithm to take advantage
of potential concurrency
□  Supporting Structures: Define
program structures and data
structures needed for the code
□  Implementation Mechanisms:
Threads, processes, messages, …
■  Each phase is a design space
5

Finding Concurrency Design Space
■  Identify and analyze exploitable concurrency
■  Example: Data Decomposition Pattern
□  Context: Computation is organized around large data
manipulation, similar operations on different data parts
□  Solution: Array-based data access (row, block),
recursive data structure traversal
■  Example: Group Tasks Pattern
□  Context: Tasks shared temporal constraints (e.g. intermediate
data), work on shared data structure
□  Solution: Apply ordering constraints to groups of tasks, put
truly independent tasks in one group for better scheduling
6

Algorithm Structure Design Space
■  Structure the algorithm
■  Consider how the identified concurrency is organized
□  Organize algorithm by tasks
◊  Tasks are embarrassingly parallel, or organized linearly
-> Task Parallelism
◊  Tasks organized by recursive procedure
-> Divide and Conquer
□  Organize algorithm by by data dependencies
◊  Linear data dependencies -> Geometric Decomposition
◊  Recursive data dependencies -> Recursive Data
□  Organize algorithm by application data flow
◊  Regular data flow for computation -> Pipeline
◊  Irregular data flow -> Event-Based Coordination
7

Example: Parallelize Bubble Sort
8
■  Bubble sort
□  Compare pair-wise and swap,
if in wrong order
■  Finding concurrency demands data
dependency consideration
□  Compare-exchange approach
needs some operation order
□  Algorithm idea implies hidden
data dependency
□  Idea: Parallelize serial rounds
■  Odd-even sort –
Compare [odd|even]-indexed pairs
and swap, in case
□  Apply task parallelism pattern
1 24 18 12 77
<->
1 24 18 12 77
<->
1 18 24 12 77
<->
1 18 12 24 77
<->
1 18 12 24 77
<->
1 18 12 24 77
<->
1 12 18 24 77
<-> ...

Example: Parallelize Bubble Sort
9
1 24 18 12 77
<->
1 24 18 12 77
<->
1 18 24 12 77
<->
1 18 12 24 77
<->
1 18 12 24 77
<->
1 18 12 24 77
<->
1 12 18 24 77
<-> ...
1 24 18 12 77
<-> <->
1 24 12 18 77
<-> <->
1 12 24 18 77
<-> <->
1 12 18 24 77
<-> <->
1 12 18 24 77
<-> <->

Supporting Structures Design Space
■  Software structures that support the expression of parallelism
■  Program structuring patterns - Single Program Multiple Data
(SPMD), master / worker, loop parallelism, fork / join
■  Data structuring patterns - Shared data, shared queue,
distributed array
□  Example: Shared data pattern
◊  Define shared abstract data type with concurrency control
(read only, read / write, independent sub sets, …)
◊  Choose appropriate synchronization construct
■  Supporting structures map to algorithm structure
□  Example: SPMD works well with geometric decomposition
10

11
Design Space Parallelization Pattern
1. Finding Concurrency
Task Decomposition, Data Decomposition,
Group Tasks, Order Tasks, Data Sharing,
Design Evaluation
2. Algorithm Structure
Task Parallelism, Divide and Conquer,
Geometric Decomposition, Recursive Data,
Pipeline, Event-Based Coordination
3. Supporting Structures
SPMD, Master/Worker, Loop Parallelism,
Fork/Join, Shared Data, Shared Queue,
Distributed Array
4. Implementation Mechanisms
Thread & Process Creation and Destruction,
Memory Synchronization, Fences,
Barriers, Mutual Exclusion, Message Passing,
Collective Communication

Our Pattern Language (OPL)
■  Extended version of the Mattson el al. proposals
■  http://parlab.eecs.berkeley.edu/wiki/patterns/patterns
12
Structural Patterns
(map/reduce, ...)
Computational Patterns
(Monte Carlo, ...)
Algorithm Strategy Patterns
(Task / data parallelism, pipelining, decomposition, ...)
Implementation Strategy Patterns
(SPMD, fork/join, Actors, shared queue, BSP, ...)
Concurrent Execution Patterns
(SIMD, MIMD, task graph, message passing, mutex, ...)

Our Pattern Language (OPL)
■  Structural patterns
□  Describe overall computational goal of the application
□  “Boxes and arrows”
■  Computational patterns
□  Classes of computations (Berkeley dwarves)
□  “Computations occurring in the boxes”
■  Algorithm strategy patterns
□  High-level strategies to exploit concurrency and parallelism
■  Implementation strategy patterns
□  Structures realized in source code
□  Program organization and data structures
■  Concurrent execution patterns
□  Approaches to support the execution of parallel algorithms
□  Strategies that advance a program
□  Basic building blocks for coordination of concurrent tasks
13

14

Example: Discrete Event Pattern
■  Name: Discrete Event Pattern
■  Problem: Suppose a computational pattern can be decomposed
into groups of semi-independent tasks interacting in an
irregular fashion. The interaction is determined by the flow of
data between them which implies ordering constraints between
the tasks. How can these tasks and their interaction be
implemented so they can execute concurrently?
15

Example: Discrete Event Pattern
■  Solution: A good solution is based on expressing the data flow using
abstractions called events, with each event having a task that
generates it and a task that processes it. Because an event must be
generated before it can be processed, events also define ordering
constraints between the tasks. Computation within each task consists
of processing events.
Initialize!
while(not done)!
{!
receive event!
process event!
send events!
}!
finalize!
1 2
3
16

Patterns for Efficient Computation
[Cool et al.]
■  Nesting Patterns
■  Structured Serial Control Flow Patterns
(Selection, Iteration, Recursion, …)
■  Parallel Control Patterns
(Fork-Join, Stencil, Reduction, Scan, …)
■  Serial Data Management Patterns
(Closures, Objects, …)
■  Parallel Data Management Patterns
(Pack, Pipeline, Decomposition,
Gather, Scatter, …)
■  Other Parallel Patterns (Futures, Speculative Selection, Workpile,
Search, Segmentation, Category Reduction, …)
■  Non-Deterministic Patterns (Branch and Bound, Transactions, …)
■  Programming Model Support
1

OpenHPI Course
Unit 6.2: Foster’s Methodology

Designing Parallel Algorithms [Foster]
■  Map workload problem on an execution environment
□  Concurrency & locality for speedup, scalability
■  Four distinct stages of a methodological approach
■  A) Search for concurrency and scalability
□  Partitioning:
Decompose computation and data into small tasks
□  Communication:
Define necessary coordination of task execution
■  B) Search for locality and performance
□  Agglomeration:
Consider performance and implementation costs
□  Mapping:
Maximize processor utilization, minimize communication
19

Partitioning
■  Expose opportunities for parallel execution through
fine-grained decomposition
■  Good partition keeps computation and data together
□  Data partitioning leads to data parallelism
□  Computation partitioning leads to task parallelism
□  Complementary approaches, can lead to different algorithms
□  Reveal hidden structures of the algorithm that have potential
□  Investigate complementary views on the problem
■  Avoid replication of either computation or data,
can be revised later to reduce communication overhead
■  Activity results in multiple candidate solutions
20

Partitioning - Decomposition Types
■  Domain Decomposition
□  Define small data fragments
□  Specify computation for them
□  Different phases of computation
on the same data are handled separately
□  Rule of thumb:
First focus on large, or frequently used, data structures
■  Functional Decomposition
□  Split up computation into disjoint
tasks, ignore the data accessed
for the moment
□  With significant data overlap,
domain decomposition is more
appropriate
21
[Foster]
[Foster]

Partitioning - Checklist
■  Checklist for resulting partitioning scheme
□  Order of magnitude more tasks than processors ?
◊  Keeps flexibility for next steps
□  Avoidance of redundant computation and storage needs ?
◊  Scalability for large problem sizes
□  Tasks of comparable size ?
◊  Goal to allocate equal work to processors
□  Does number of tasks scale with the problem size ?
◊  Algorithm should be able to solve larger tasks with more
given resources
■  Identify bad partitioning by estimating performance behavior
■  In case, re-formulate the partitioning (backtracking)
□  May even happen in later steps
22

Communication
■  Specify links between data consumers and data producers
■  Specify kind and number of messages on these links
■  Domain decomposition problems might have tricky communication
infrastructures, due to data dependencies
■  Communication in functional decomposition problems can easily
be modeled from the data flow between the tasks
■  Categorization of communication patterns
□  Local communication (few neighbors) vs.
global communication
□  Structured communication (e.g. tree) vs.
unstructured communication
□  Static vs. dynamic communication structure
□  Synchronous vs. asynchronous communication
23

Communication - Hints
■  Distribute computation and communication,
don‘t centralize algorithm
□  Bad example: Central manager for parallel summation
■  Unstructured communication is hard to agglomerate,
better avoid it
■  Checklist for communication design
□  Do all tasks perform the same amount of communication ?
□  Does each task performs only local communication ?
□  Can communication happen concurrently ?
□  Can computation happen concurrently ?
■  Solve issues by distributing or replicating communication hot spots
24

Communication - Ghost Cells
■  Domain decomposition might lead to chunks that
demand data from each other
■  Solution 1: Copy necessary portion of data
(,ghost cells‘)
□  If no synchronization is needed after update
□  Data amount and frequency of update
influences resulting overhead and efficiency
□  Additional memory consumption
■  Solution 2: Access relevant data ,remotely‘
□  Delays thread coordination until the data is
really needed
□  Correctness („old“ data vs. „new“ data) must be
considered on parallel progress
25

Agglomeration
■  Algorithm so far is correct,
but not specialized for a particular execution environment
■  Check partitioning and communication decisions again
□  Agglomerate tasks for efficient execution on target hardware
□  Replicate data and / or computation for efficiency reasons
■  Resulting number of tasks can still be greater than the number of
processors
■  Three conflicting guiding decisions
□  Reduce communication costs by coarser granularity of
computation and communication
□  Preserve flexibility for later mapping by finer granularity
□  Reduce engineering costs for creating a parallel version
26

Agglomeration - Granularity
■  Since execution
environment is now
considered, the surface-
to-volume effect
becomes relevant
■  Late consideration keeps
core algorithm flexibility
27
[Foster]
Surface-to-volume effect

Agglomeration - Checklist
■  Communication costs reduced by increasing locality ?
■  Does replicated computation outweighs its costs in all cases ?
■  Does data replication restrict the problem size ?
■  Do the larger tasks still have similar
computation / communication costs ?
■  Do the larger tasks still act with sufficient concurrency ?
■  Does the number of tasks still scale with the problem size ?
■  How much can the task count decrease, without disturbing load
balancing, scalability, or engineering costs ?
■  Is the transition to parallel code worth the engineering costs ?
28

Mapping
■  Historically only relevant for shared-nothing systems
□  Shared memory systems have the operating system scheduler
□  With NUMA, this may become also relevant in shared memory
systems of the future (e.g. PGAS task placement)
■  Minimize execution time by …
□  … placing concurrent tasks on different nodes
□  … placing tasks with heavy communication on the same node
■  Conflicting strategies, additionally restricted by resource limits
□  Task mapping problem
□  Known to be compute-intense (bin packing)
■  Set of sophisticated (dynamic) heuristics for load balancing
□  Preference for local algorithms that do not need global
scheduling state
29

OpenHPI Course
Unit 6.3: Berkeley Dwarfs

Common Algorithmic Problems
■  Sources
□  Parallel
programming
courses
□  Parallel
Benchmarks
□  Development
guides
□  Parallel
Programming
books
□  User stories
31

A View From Berkeley
■  Technical report from
Berkeley (2006), defining
parallel computing research
questions and
recommendations
■  Definition of „13 dwarfs“
□  Common designs of
parallel computation and
communication
□  Allow better evaluation
of programming models
and architectures
32
The Landscape of Parallel Computing Research: A
View from Berkeley
Krste Asanovic
Ras Bodik
Bryan Christopher Catanzaro
Joseph James Gebis
Parry Husbands
Kurt Keutzer
David A. Patterson
William Lester Plishker
John Shalf
Samuel Webb Williams
Katherine A. Yelick
Electrical Engineering and Computer Sciences
University of California at Berkeley
Technical Report No. UCB/EECS-2006-183
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-183.html
December 18, 2006

A View From Berkeley
■  Sources
□  EEMBC benchmarks (embedded systems), SPEC benchmarks
□  Database and text mining technology
□  Algorithms in computer game design and graphics
□  Machine learning algorithms
□  Original „7 Dwarfs“ for supercomputing [Colella]
■  „Anti-benchmark“
□  Dwarfs are not tied to code or language artifacts
□  Can serve as understandable vocabulary across disciplines
□  Allow feasability study of hardware and software design
◊  No need to wait for applications being developed
33

13 Dwarfs
■  Dwarfs currently defined
□  Dense Linear Algebra
□  Sparse Linear Algebra
□  Spectral Methods
□  N-Body Methods
□  Structured Grids
□  Unstructured Grids
□  MapReduce
□  Combinational Logic
□  Graph Traversal
□  Dynamic Programming
□  Backtrack and Branch-and-
Bound
□  Graphical Models
□  Finite State Machines
34
■  One dwarf may be implemented based on another one
■  Increasing uptake in scientific publications
■  Several reference implementations for CPU / GPU

Dwarfs in Popular Applications
35
Hot →
Cold
[Patterson]

■  Classic vector and matrix operations on non-sparse data
(vector op vector, matrix op vector, matrix op matrix)
■  Data layout as continues array(s)
■  High degree of data dependencies
■  Computation on elements, rows, columns or matrix blocks
■  Issues with memory hierarchy, data distribution is critical
■  Demands overlapping of computation and communication
Dense Linear Algebra
36

Sparse Linear Algebra
■  Operations on a sparse matrix (with lots of zeros)
■  Typically compressed data structures, integer operations,
only non-zero entries + indices
□  Dense blocks to exploit caches
■  Complex dependency structure
■  Scatter-gather vector operations
are often helpful
37

N-Body Methods
■  Physics: Predicting individual motions of an object group
interacting gravitationally
□  Calculations on interactions between many discrete points
■  Hierarchical tree-based and mesh-based methods,
avoid computing all pair-wise interactions
■  Variations with particle-particle
methods (one point to all others)
■  Large number of independent
calculations in a time step,
followed by all-to-all
communication
■  Issues with load balancing and
missing fixed hierarchy
38

Structured Grid
■  Data as a regular multidimensional grid
□  Access is regular and statically determinable
■  Computation as sequence of grid updates
□  Points are updated concurrently using values
from a small neighborhood
■  Spatial locality to use long cache lines
■  Temporal locality to allow cache reuse
■  Parallel mapping with sub-grid per processor
□  Ghost cells, surface to volume ratio
■  Latency hiding
□  Increased number of ghost cells
□  Coarse-grained data exchange
39

Unstructured Grid
■  Elements update neighbors in irregular
mesh/grid - static or dynamic structure
■  Problematic data distribution and access
requirements, indirection through tables
■  Modeling domain (e.g. physics)
□  Mesh represents surface or volume
□  Entities are points, edges, faces, ...
□  Applying pressure, temperature, …
□  Computations involve numerical
solutions or differential equations
□  Sequence of mesh updates
■  Massively data parallel, but irregularly
distributed data and communication
40

MapReduce
■  Originally called “Monte Carlo” in dwarf concept
□  Repeated independent execution of a function
(e.g. random number generation, map function)
□  Results aggregated at the end
□  Nearly no communication between tasks,
embarrassingly parallel
■  Examples: Monte Carlo, BOINC project, protein structures
[http://climatesanity.wordpress.com]
41

■  Global optimization problem in large search space
■  Divide and Conquer principle
□  Branching into subdivisions
□  Optimize execution by ruling out regions
■  Examples: Integer linear programming,
boolean satisfiability, combinatorial optimization,
traveling salesman, constraint programming, …
■  Heuristics to guide search to productive regions
■  Parallel checking of sub-regions
□  Demands invariants about the search space
□  Demands dynamic load balancing, load prediction is hard
■  Example:
Place N queens on a chessboard so that no two attack each other
Backtrack / Branch-and-Bound
42

Branch-and-Bound
43
[http://docs.jboss.org/drools]

44
[http://docs.jboss.org/drools]

Berkeley Dwarfs
■  Relevance of single dwarfs
widely differs
■  No widely accepted single
benchmark implementation
■  Computational dwarfs on
different layers,
implementations may be
based on each other
■  OpenDwarfs project
□  Optimized code for
different platforms
■  Parallel Dwarfs project
□  In C++, C#, F# for
Visual Studio
45
The Landscape of Parallel Computing Research: A
View from Berkeley
Krste Asanovic
Ras Bodik
Bryan Christopher Catanzaro
Joseph James Gebis
Parry Husbands
Kurt Keutzer
David A. Patterson
William Lester Plishker
John Shalf
Samuel Webb Williams
Katherine A. Yelick
Electrical Engineering and Computer Sciences
University of California at Berkeley
Technical Report No. UCB/EECS-2006-183
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-183.html
December 18, 2006

OpenHPI Course
Unit 6.4: Some Future Trends

NUMA Impact Increases
47
Core
Core
Core
Core
Q
P
I
Core
Core
Core
Core
Q
P
I
Core
Core
Core
Core
Q
P
I
Core
Core
Core
Core
Q
P
I
L3Cache
L3Cache
L3Cache
MemoryController
MemoryController
MemoryController
L3Cache
MemoryController
I/O
I/O
I/O
I/O
Memory
Memory
Memory
Memory

Innovation in Memory Technology
■  3D NAND
■  Hybrid Memory Cube
□  Intel, Micron, …
□  3D array of DDR-alike
memory cells
□  Early samples
available, 160GB/s
□  Through-silicon via
(TSV) approach with
embedded controllers,
attached to CPU
■  RRAM / ReRAM
□  Non-volatile memory
48
[computerworld.com][extremetech.com]

Power Wall 2.0 = Dark Silicon
“Dark Silicon and the End
of Multicore Scaling”
by Hadi Esmaeilzadeh, Emily
Blem, Renée St. Amant,
Karthikeyan Sankaralingam,
Doug Burger
49

Hardware / Software Co-Design
■  Increasing number of cores by Moore‘s law
■  Power wall / dark silicon problem will become worse
□  In addition, battery-powered devices become more relevant
■  Idea: Use additional transistors for specialization
□  Design hardware for a software problem
□  Make it part of the processor („compile into hardware“)
□  More efficiency, less flexibility
□  Partially known from ILP SIMD support
□  Examples: Cryptography, regular expressions
■  Example: Cell processor (Playstation 3)
□  64-bit Power core
□  8 specialized co-processors
50

Software at Scale [Dongarra]
■  Effective utilization of many-core and hybrid hardware
□  Break fork-join parallelism
□  Dynamic data driven execution, consider block layout
□  Exploiting mixed precision (GPU vs. CPU, power consumption)
■  Aim for self-adapting software and auto-tuning support
□  Manual optimization is too hard
□  Let software optimize the software
■  Consider fault-tolerant software
□  With 1.000.000's of cores, things break all the time
■  Focus on algorithm classes that reduce communication
□  Special problem in dense computation
□  Aim for asynchronous iterations
51

OpenMP 4.0
■  SIMD extensions
□  Portable primitives to describe SIMD parallelization
□  Loop vectorization with simd construct
□  Several arguments for guiding the compiler (e.g. alignment)
■  Targeting extensions
□  Thread with the OpenMP program executes on the host device
□  Implementation may support multiple target devices
□  Control off-loading of loops and code regions on such devices
■  New API for device data environment
□  OpenMP - managed data items can be moved to the device
□  New primitives for better cancellation support
□  User-defined reduction operations
52

OpenACC
■  „OpenMP for accelerators“ (GPU, FPGAs, ...)
□  Partners: Cray, supercomputing centers, NVIDIA, PGI
□  Annotation in C, C++, and Fortran source code
□  OpenACC code can also be started on the accelerator
■  Features
□  Specification of data locality and asynchronous execution
□  Abstract specification of data movement, loop parallelization
□  Caching and synchronization support
□  Management of data movement by compiler and runtime
□  Implementations available, e.g. for Xeon Phi
53

Autotuners
■  Optimize parallel code by generating many variants
□  Try many or all optimization switches
◊  Loop unrolling, utilization of processor registers, …
□  Rely on parallelization variations defined in the application
■  Automatically tested on target platform
■  Research shows promising results
□  Can be better than manually optimized code
□  Optimization can fit to multiple execution environments
□  Known examples for sparse and dense linear algebra libraries
◊  ATLAS (Automatically Tuned Linear Algebra Software)
54

Intel Math Kernel Library (MKL)
■  Intel library with heavily optimized functionality, for C & Fortran
□  Linear algebra
◊  Basic Linear Algebra Subprograms (BLAS) API
◊  Follows standards in high-performance computing
◊  Vector-vector, matrix-vector, matrix-matrix operations
□  Fast Fourier Transforms (FFT)
◊  Single precision, double precision, complex, real, ...
□  Vector math and statistics functions
◊  Random number generators and probability distributions
◊  Spline-based data fitting
■  High-level abstraction of functionality,
parallelization completely transparent for the developer
55

Future Trends
■  Active research on next-generation hardware
□  Driven by exa-scale efforts in supercomputing
□  Driven by combined power wall and memory wall
□  Driven by shift in computer markets (desktop -> mobile)
■  Impact on software development will get more visible
□  Hybrid computing is the future default
□  Heterogeneous mixture of CPU + specialized accelerators
□  Old assumptions are broken (flat memory, constant access
time, homogeneous processing elements)
□  Old programming models no longer match
□  Extending the existing programming paradigms seems to work
□  High-level specialized libraries get more relevance
56

OpenHPI Course
Summary

Course Organization
■  Week 1: Terminology and fundamental concepts
□  Moore’s law, power wall, memory wall, ILP wall
□  Speedup vs. scaleup, Amdahl’s law, Flynn’s taxonomy, …
■  Week 2: Shared memory parallelism – The basics
□  Concurrency, race condition, semaphore, deadlock, monitor, …
■  Week 3: Shared memory parallelism – Programming
□  Threads, OpenMP, Cilk, Scala, …
■  Week 4: Accelerators
□  Hardware today, GPU Computing, OpenCL, …
■  Week 5: Distributed memory parallelism
□  CSP, Actor model, clusters, HPC, MPI, MapReduce, …
■  Week 6: Patterns and best practices
□  Foster’s methodology, Berkeley dwarfs, OPL collection, …
58

Week 1:
The Free Lunch Is Over
■  Clock speed curve
flattened in 2003
□  Heat, power,
leakage
■  Speeding up the serial
instruction execution
through clock speed
improvements no
longer works
■  Additional issues
□  ILP wall
□  Memory wall
[HerbSutter,2009]
59

Three Ways Of Doing Anything Faster
[Pfister]
■  Work harder
(clock speed)
Ø  Power wall problem
Ø  Memory wall problem
■  Work smarter
(optimization, caching)
Ø  ILP wall problem
■  Get help
(parallelization)
□  More cores per single CPU
□  Software needs to exploit
them in the right way
Problem
CPU
Core
Core
Core
Core
Core
60

Parallelism on Different Levels
ProgramProgramProgram
ProcessProcessProcessProcessTask
PE
PE
PE
PE
Memory
Node
Network
PE
PE
PE
Memory
PE
PE
PE
Memory
PE
PE
PE
Memory
PE
PE
PE
Memory
61

The Parallel Programming Problem
■  Execution environment has a particular type
(SIMD, MIMD, UMA, NUMA, …)
■  Execution environment maybe configurable (number of resources)
■  Parallel application must be mapped to available resources
Execution EnvironmentParallel Application Match ?
Configuration
Flexible
Type
62

Amdahl’s Law
63

Gustafson-Barsis’ Law (1988)
■  Gustafson and Barsis: People are typically not interested in the
shortest execution time
□  Rather solve a bigger problem in reasonable time
■  Problem size could then scale with the number of processors
□  Typical in simulation and farmer / worker problems
□  Leads to larger parallel fraction with increasing N
□  Serial part is usually fixed or grows slower
■  Maximum scaled speedup by N processors:
■  Linear speedup now becomes possible
■  Software needs to ensure that serial parts remain constant
■  Other models exist (e.g. Work-Span model, Karp-Flatt metric)
64
S =
TSER + N · TP AR
TSER + TP AR

Week 2:
Concurrency vs. Parallelism
■  Concurrency means dealing with several things at once
□  Programming concept for the developer
□  In shared-memory systems, implemented by time sharing
■  Parallelism means doing several things at once
□  Demands parallel hardware
■  Parallel programming is a misnomer
□  Concurrent programming aiming at parallel execution
■  Any parallel software is concurrent software
□  Note: Some researchers disagree, most practitioners agree
■  Concurrent software is not always parallel software
□  Many server applications achieve scalability
by optimizing concurrency only (web server)
65
Concurrency
Parallelism

Parallelism [Mattson et al.]
■  Task
□  Parallel program breaks a problem into tasks
■  Execution unit
□  Representation of a concurrently running task (e.g. thread)
□  Tasks are mapped to execution units
■  Processing element (PE)
□  Hardware element running one execution unit
□  Depends on scenario - logical processor vs. core vs. machine
□  Execution units run simultaneously on processing elements,
controlled by some scheduler
■  Synchronization - Mechanism to order activities of parallel tasks
■  Race condition - Program result depends on the scheduling order
66

Concurrency Issues
■  Mutual Exclusion
□  The requirement that when one concurrent task is using a
shared resource, no other shall be allowed to do that
■  Deadlock
□  Two or more concurrent tasks are unable to proceed
□  Each is waiting for one of the others to do something
■  Starvation
□  A runnable task is overlooked indefinitely
□  Although it is able to proceed, it is never chosen to run
■  Livelock
□  Two or more concurrent tasks continuously change their states
in response to changes in the other activities
□  No global progress for the application
6OpenHPI | Parallel Programming Concepts | Dr. Peter Tröger
67

Week 3:
Parallel Programming for Shared Memory
68

Process
Explicitly Shared Memory
■  Different programming models for
concurrency with shared memory
■  Processes and threads mapped to
processing elements (cores)
■  Task model supports more
fine-grained parallelization than
with native threads
Memory

Process
Memory
Thread
Thread
Task
Task
Task
Task
Concurrent Processes Concurrent Threads
Concurrent Tasks
Main Thread

Process
Memory
Main Thread

Process
Memory
Main Thread
Thread
Thread

Input Data
Parallel
Processing
Result Data
Task Parallelism and Data Parallelism
69

OpenMP
■  Programming with the fork-join model
□  Master thread forks into declared tasks
□  Runtime environment may run them in parallel,
based on dynamic mapping to threads from a pool
□  Worker task barrier before finalization (join)
70
[Wikipedia]

High-Level Concurrency
71
Microsoft Parallel Patterns Library java.util.concurrent

Partitioned Global Address Space
72
Place-shifting operations
• at(p) S
… …… …
Activities
Local
Heap
Place 0
……
…
Activities
Local
Heap
Place N
…
Global Reference
Distributed heap
• GlobalRef[T]
APGAS in X10: Places and Tasks
Task parallelism
• async S
• finish S
Concurrency control within a place
• when(c) S
• atomic S
■  Parallel tasks, each operating in one place of the PGAS
□  Direct variable access only in local place
■  Implementation strategy is flexible
□  One operating system process per place, manages thread pool
□  Work-stealing scheduler
[IBM]

Week 4:
Cheap Performance with Accelerators
■  Performance
■  Energy / Price
□  Cheap to buy and to maintain
□  GFLOPS per watt: Fermi 1,5 / Kepler 5 / Maxwell 15 (2014)
73
0
200
400
600
800
1000
1200
1400
0 10000 20000 30000 40000 50000
ExecutionTimein
Milliseconds
Problem Size (Number of Sudoku Places)
Intel E8500 CPU
AMD R800 GPU
NVIDIA GT200 GPU
lower means faster
GPU: Graphics Processing Unit
(CPU of a graphics card)

CPU vs. GPU Architecture
□  Some huge threads
□  Branch prediction
□  1000+ light-weight threads
□  Memory latency hiding
74
Control
PE
PE
PE
PE
Cache
DRAM DRAM
CPU GPU „many-core“„multi-core“

OpenCL Platform Model
□  OpenCL exposes CPUs, GPUs, and other Accelerators as “devices”
□  Each “device” contains one or more “compute units”, i.e. cores, SMs,...
□  Each “compute unit” contains one or more SIMD “processing elements”
75

Best Practices for Performance Tuning
• Asynchronous, Recompute, SimpleAlgorithm Design
• Chaining, Overlap Transfer & ComputeMemory Transfer
• Divergent Branching, PredicationControl Flow
• Local Memory as Cache, rare resourceMemory Types
• Coalescing, Bank ConflictsMemory Access
• Execution Size, EvaluationSizing
• Shifting, Fused Multiply, Vector TypesInstructions
• Native Math Functions, Build OptionsPrecision
76

Week 5:
Shared Nothing
■  Clusters: Stand-alone machines connected by a local network
□  Cost-effective technique for a large-scale parallel computer
□  Users are builders, have control over their system
□  Synchronization much slower than in shared memory
□  Task granularity becomes an issue
77

Processing
Element
Task

Processing
Element
Task
Message
Message
Message
Message
Local
Memory
Local
Memory

Shared Nothing
■  Supercomputers / Massively Parallel Processing (MPP) systems
□  (Hierarchical) cluster with a lot of processors
□  Still standard hardware, but specialized setup
□  High-performance interconnection network
□  For massive data-parallel applications, mostly simulations
(weapons, climate, earthquakes, airplanes, car crashes, ...)
■  Examples (Nov 2013)
□  BlueGene/Q, 1.5 million cores, 1.5 PB memory, 17.1 TFlops
□  Tianhe-2, 3.1 million cores,
1 PB memory, 17.808 kW power,
33.86 PFlops (quadrillions
calculations per second)
■  Annual ranking with the TOP500 list
(www.top500.org)
78

Surface-To-Volume Effect
79
[nicerweb.com]
■  Fine-grained decomposition for
using all processing elements ?
■  Coarse-grained decomposition
to reduce communication
overhead ?
■  A tradeoff question !

Message Passing
■  Parallel programming paradigm for “shared nothing” environments
□  Implementations for shared memory available,
but typically not the best approach
■  Users submit their message passing program & data as job
■  Cluster management system creates program instances
Instance
0
Instance
1
Instance
2 Instance
3
Execution Hosts
80
Cluster Management Software
Submission
Host
Job
Appli-
cation

Single Program Multiple Data (SPMD)
81
// … (determine rank and comm_size) …
int token;
if (rank != 0) {
// Receive from your ‘left’ neighbor if you are not rank
0
MPI_Recv(&token, 1, MPI_INT, rank - 1, 0,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Process %d received token %d from process %dn",
rank, token, rank - 1);
} else {
// Set the token's value if you are rank 0
token = -1;
}
// Send your local token value to your ‘right’ neighbor
MPI_Send(&token, 1, MPI_INT, (rank + 1) % comm_size,
0, MPI_COMM_WORLD);
// Now rank 0 can receive from the last rank.
if (rank == 0) {
MPI_Recv(&token, 1, MPI_INT, comm_size - 1, 0,
rank, token, comm_size - 1);
}
Input data
SPMD program
int token;
if (rank != 0) {
// Receive from your ‘left’ neighbor if you are not rank 0
} else {
token = -1;
}
0, MPI_COMM_WORLD);
if (rank == 0) {
}
int token;
if (rank != 0) {
} else {
token = -1;
}
0, MPI_COMM_WORLD);
if (rank == 0) {
}
int token;
if (rank != 0) {
} else {
token = -1;
}
0, MPI_COMM_WORLD);
if (rank == 0) {
}
int token;
if (rank != 0) {
} else {
token = -1;
}
0, MPI_COMM_WORLD);
if (rank == 0) {
}
int token;
if (rank != 0) {
} else {
token = -1;
}
0, MPI_COMM_WORLD);
if (rank == 0) {
}
Instance 0
Instance 1
Instance 2
Instance 3
Instance 4

Actor Model
■  Carl Hewitt, Peter Bishop and Richard Steiger. A Universal Modular
Actor Formalism for Artificial Intelligence IJCAI 1973.
□  Mathematical model for concurrent computation
□  Actor as computational primitive
◊  Local decisions, concurrently sends / receives messages
◊  Has a mailbox for incoming messages
◊  Concurrently creates more actors
□  Asynchronous one-way message sending
□  Changing topology allowed, typically no order guarantees
◊  Recipient is identified by mailing address
◊  Actors can send their own identity to other actors
■  Available as programming language extension or library
in many environments
82

Week 6:
■  Phases in creating a parallel program
□  Finding Concurrency: Identify and
analyze exploitable concurrency
□  Algorithm Structure: Structure the
algorithm to take advantage of
potential concurrency
□  Supporting Structures: Define
program structures and data
structures needed for the code
□  Implementation Mechanisms:
Threads, processes, messages, …
■  Each phase is a design space
83

Popular Applications vs. Dwarfs
Hot →
Cold
84

Designing Parallel Algorithms [Foster]
■  Map workload problem on an execution environment
□  Concurrency & locality for speedup, scalability
■  Four distinct stages of a methodological approach
■  A) Search for concurrency and scalability
□  Partitioning –
Decompose computation and data into small tasks
□  Communication –
Define necessary coordination of task execution
■  B) Search for locality and performance
□  Agglomeration –
Consider performance and implementation costs
□  Mapping –
Maximize processor utilization, minimize communication
85

86
And that’s it …

The End
■  Parallel programming is exciting again!
□  From massively parallel hardware to complex software
□  From abstract design patterns to specific languages
□  From deadlock freedom to extreme performance tuning
■  Some general concepts are established
□  Take this course as starting point
□  Learn from the high-performance computing community
■  Thanks for your participation
□  Lively discussion, directly and in the forums, we learned a lot
□  Sorry for technical flaws and content errors
■  Please use the feedback link
87

Lecturer Contact
■  Operating Systems and Middleware Group at HPI
http://www.dcl.hpi.uni-potsdam.de
■  Dr. Peter Tröger
http://www.troeger.eu
http://twitter.com/ptroeger
http://www.linkedin.com/in/ptroeger
peter.troeger@hpi.uni-potsdam.de
■  M.Sc. Frank Feinbube
http://www.feinbube.de
http://www.linkedin.com/in/feinbube
frank.feinbube@hpi.uni-potsdam.de
88

OpenHPI - Parallel Programming Concepts - Week 6

Recommended

Recommended

More Related Content

Similar to OpenHPI - Parallel Programming Concepts - Week 6

Similar to OpenHPI - Parallel Programming Concepts - Week 6 (20)

More from Peter Tröger

More from Peter Tröger (20)

Recently uploaded

Recently uploaded (20)

OpenHPI - Parallel Programming Concepts - Week 6