The goals and reason of choosing parallel programming is much easier while seeing the architecture of parallel programming. By giving a example of loop dependency easily understand for beginners as same as loop parallelization

1. PARALLEL PROGRAMMING CONCEPT
DEPENDENCY AND LOOP
PARALLELIZATION
•Parallel programming concept and
their examples
•Dependency and their two types
with examples
•Loop parallelism and their types
with examples
R.AISHWARYA

2. Parallel programming???

4. • Simultaneous use of multiple compute
resources to solve a computational problem
• Compute resources
• Primary reasons
• Best practices
• Goals
• Steps

7. LOOP DEPENDENCY

8. Statement 2: b=a+2; “5” Statement 1: a=5;

9. •Find dependencies within iterations of a loop
•Goal of determining different relationships
between statements.
•To allow multiple processors to work on different
portions of the loop in parallel
•First analyze the dependencies within individual
loops.
•It help determine which statements in the loop
need to be completed before other statements
can start.
•Two general categories of dependencies: Data and
Control dependency

10. function Dep(a,b)
c:=a.b
d:=2.c
end function
(flow dependency)
function Nodep(a,b)
c:=a.b
d:=2.b
e:=a+b
end function
(no dependency)

11. DATA DEPENDENCY
TYPE NOTATION DESCRIPTION
True (Flow) Dependence
S1 ->T S2
A true dependence between S1
and S2 means that S1 writes
to a location later read from
by S2
Anti Dependence S1 ->A S2
An anti-dependence between
S1 and S2 means that S1 reads
from a location later written
to by S2.(before)
Output Dependence S1 ->I S2
An input dependence between
S1 and S2 means that S1 and
S2 read from the same
location.

12. EXAMPLES
True dependence
S0: int a, b;
S1: a = 2;
S2: b = a + 40;
S1 ->T S2, meaning that S1 has a true dependence on S2 because
S1writes to the variable a, which S2 reads from.
Anti-dependence
S0: int a, b = 40;
S1: a = b - 38;
S2: b = -1;
S1 ->A S2, meaning that S1 has an anti-dependence on S2
because S1reads from the variable b before S2 writes to it.
Output-dependence
S0: int a, b = 40;
S1: a = b - 38;
S2: a = 2;
S1 ->O S2, meaning that S1 has an output dependence on S2
because both write to the variable a.

13. CONTROL DEPENDENCY
if(a == b)
then
{
c = “controlled”;
}
d=“not
controlled”;
if(a == b)
then
{
}
c = “controlled”;
d=“not
controlled”;
if(a == b)
then
{
c = “controlled”;
d=“not
controlled”;
}

14. DEPENDENCY IN LOOP
Loops can have two types of dependence:
•Loop-carried
dependency
•Loop-independent
dependency

15. LOOP CARRIED DEPENDENCY
• In loop-carried dependence, statements in an
iteration of a loop depend on statements in
another iteration of the loop.
for(i=0;i<4;i++)
{
S1: b[i]=8;
S2: a[i]=b[i-1] + 10;
}

16. LOOP INDEPENDENT DEPENDENCY
• In loop-independent dependence, loops have
inter-iteration dependence, but do not have
dependence between iterations.
• Each iteration may be treated as a block and
performed in parallel without other
synchronization efforts.
for (i=0;i<4;i++)
{
S1: b[i] = 8;
S2: a[i] =b[i] + 10;
}

17. for (i=1; i<4; i++)
for (j=1; j<4; j++)
S3: a[i][j] = a[i][j-1] + 1;
Node : Point in the iteration space
Directed Edge: Dependency
Node: Point in the iteration space
Directed Edge: next point that will
be encountered
after the current point is
traversed

18. LOOP PARALLELIZATION

19. •Extraction parallel tasks from loops
•Data is stored in random access data structures
•A program exploiting loop-level parallelism will use
multiple threads or processes which operate on same time
•It provides speedup
•Amdhal’s law

20. Examples of Loop
parallelization
for (int i = 0; i < n; i++)
{
S1: L[i] = L[i] + 10;
}
for (int i = 1; i < n; i++)
{
S1: L[i] = L[i - 1] + 10;
}

21. Can the following Loop be made
Parallel?
for (i=1;i<=100;i=i+1)
{
A[i+1] = A[i] + C[i]; /*S1*/
B[i+1] = B[i] + A[i+1]; /*S2*/
}

22. METHODOLOGIES FOR
PARALLELIZING LOOPS
• DISTRIBUTED Loop
• DOALL Parallelism
• DOACROSS Parallelism
• HELIX
• DOPIPE Parallelism

23. DISTRIBUTED LOOP
for (int i = 1; i < n; i ++)
{
S1: a[i] = a[i -1] + b[i];
S2: c[i] = c[i] + d[i];
}
loop1: for (int i = 1; i < n; i ++)
{
S1: a[i] = a[i -1] + b[i];
}
loop2: for (int i = 1; i < n; i ++)
{
S2: c[i] = c[i] + d[i];
}

24. DO ALL PARALLELISM
for (int i = 0; i < n; i++)
{
S1: a[i] = b[i] + c[i];
}
begin_parallelism();
for (int i = 0; i < n; i++)
{
S1: a[i] = b[i] + c[i];
end_parallelism();
}
block();

25. DO ACROSS PARALLELISM
for (int i = 1; i < n; i++)
{
a[i] = a[i - 1] + b[i] + 1;
}
S1: int tmp = b[i] + 1;
S2: a[i] = a[i - 1] + tmp;
post(0);
for (int i = 1; i < n; i++)
{
S1: int tmp = b[i] + 1;
wait(i - 1);
S2: a[i] = a[i - 1] + tmp;
post(i);
}

26. DO PIPE PARALLELISM
for (int i = 1; i < n; i++)
{
S1: a[i] = a[i - 1] + b[i];
S2: c[i] = c[i] + a[i];
}
for (int i = 1; i < n; i++)
{
S1: a[i] = a[i - 1] + b[i];
post(i);
}
for (int i = 1; i < n; i++)
{
wait(i);
S2: c[i] = c[i] + a[i];
}