This talk was based on my Master's thesis which I had completed earlier that year. It gives an overview on how certain parallel dynamic programming can be computed in parallel efficiently, and what we want that to mean here. The plots in "Performance Examples" show speedup S on the left and efficiency E on the right, both against input size. Read more over here: http://reitzig.github.io/publications/Reitzig2012

1. Parallelising Dynamic Programming
Raphael Reitzig
University of Kaiserslautern
Department of Computer Science
Algorithms and Complexity Group
September 27th, 2012

2. Vision
Compile dynamic programming recurrences into efficient parallel
code.

3. Goal 1
Understand what efficiency means in parallel algorithms.

4. Goal 1
Understand what efficiency means in parallel algorithms.
Goal 2
Characterise dynamic programming recurrences in a suitable way.

5. Goal 1
Understand what efficiency means in parallel algorithms.
Goal 2
Characterise dynamic programming recurrences in a suitable way.
Goal 3
Find and implement efficient parallel algorithms for DP.

6. Analysing Parallelism

7. Complexity theory
Classifies problems

8. Complexity theory
Classifies problems
Focuses on inherent parallelism

9. Complexity theory
Classifies problems
Focuses on inherent parallelism
Answers: How many processors do you need to be really fast
on inputs of a given size?

10. Complexity theory
Classifies problems
Focuses on inherent parallelism
Answers: How many processors do you need to be really fast
on inputs of a given size?
But...
...p grows with n – no statement about constant p and growing n!

11. Amdahl’s law
Parallel speedup ≤ 1
1−γ+γ
p
.

12. Amdahl’s law
Parallel speedup ≤ 1
1−γ+γ
p
.
Answers: How many processors can you utilise on given inputs?

13. Amdahl’s law
Parallel speedup ≤ 1
1−γ+γ
p
.
Answers: How many processors can you utilise on given inputs?
But...
...does not capture growth of n!

14. Work and depth
Work W = TA
1 and depth D = TA
∞

15. Work and depth
Work W = TA
1 and depth D = TA
∞
Brent’s Law: A with W
p ≤ TA
p < W
p + D is possible in a certain
setting.

16. Work and depth
Work W = TA
1 and depth D = TA
∞
Brent’s Law: A with W
p ≤ TA
p < W
p + D is possible in a certain
setting.
But...
...has limited applicability and D can be slippery!

17. Relative runtimes
Speedup SA
p :=
TA
1
TA
p

18. Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p

19. Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p
But...
...what are good values?

20. Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p
But...
...what are good values?
Clear: SA
p ∈ [0, p] and EA
p ∈ [0, 1]

21. Relative runtimes
Speedup SA
p :=
TA
1
TA
p
Efficiency EA
p := TB
p·TA
p
But...
...what are good values?
Clear: SA
p ∈ [0, p] and EA
p ∈ [0, 1] – but we can certainly not always
hit the optima!

22. Proposal: Asymptotic relative runtimes
Definition
SA
p(∞) := lim inf
n→∞
SA
p(n)
?
= p
EA
p (∞) := lim inf
n→∞
EA
p (n)
?
= 1

23. Proposal: Asymptotic relative runtimes
Definition
SA
p(∞) := lim inf
n→∞
SA
p(n)
?
= p
EA
p (∞) := lim inf
n→∞
EA
p (n)
?
= 1
Goal
Find parallel algorithms that are asymptotically as scalable and
efficient as possible for all p.

24. Disclaimer
This means:
A good parallel algorithm can utilise any number of processors if
the inputs are large enough.

25. Disclaimer
This means:
A good parallel algorithm can utilise any number of processors if
the inputs are large enough.
Not:
More processors are always better.

26. Disclaimer
This means:
A good parallel algorithm can utilise any number of processors if
the inputs are large enough.
Not:
More processors are always better.
Just as in sequential algorithmics.

27. Afterthoughts
Machine model
Keep it simple: (P)RAM with p processors and spawn/join.

28. Afterthoughts
Machine model
Keep it simple: (P)RAM with p processors and spawn/join.
Which quantities to analyse?
Elementary operations, memory accesses, inter-thread
communication, ...

29. Afterthoughts
Machine model
Keep it simple: (P)RAM with p processors and spawn/join.
Which quantities to analyse?
Elementary operations, memory accesses, inter-thread
communication, ...
Implicit interaction – blocking, communication via memory, ... – is
invisible in code!

30. Attacking Dynamic Programming

31. Disclaimer
Only two dimensions
Only finite domains
Only rectangular domains
Memoisation-table point-of-view

32. Reducing to dependencies
e(i, j) :=



0 i = j = 0
j i = 0 ∧ j > 0
i i > 0 ∧ j = 0
min



e(i − 1, j) + 1
e(i, j − 1) + 1
e(i − 1, j − 1) + [ vi = wj ]
else

33. Reducing to dependencies
e(i, j) :=



0 i = j = 0
j i = 0 ∧ j > 0
i i > 0 ∧ j = 0
min



e(i − 1, j) + 1
e(i, j − 1) + 1
e(i − 1, j − 1) + [ vi = wj ]
else

34. Gold standard

37. ?

38. ?

39. ?

43. ?

48. Simplification
DL D DR
UL U UR
L R

49. Three cases
Impossible

50. Three cases
Impossible
Possible

51. Three cases
Assuming dependencies are area-complete and uniform, there are
only three cases up to symmetry:

52. Facing Reality

53. Challenges
Contention

54. Challenges
Contention
Method of synchronisation

55. Challenges
Contention
Method of synchronisation
Metal issues (moving threads, cache sync)

56. Performance Examples
Edit distance on two-core shared memory machine:
0 0.2 0.4 0.6 0.8 1 1.2 1.4
·105
0
0.5
1
1.5
2
2.5
0 0.2 0.4 0.6 0.8 1 1.2 1.4
·105
0
0.5
1
1.5
2
2.5

57. Performance Examples
Edit distance on four-core NUMA machine:
0 1 2 3 4
·105
0
1
2
3
4
0 1 2 3 4
·105
0
1
2
3
4

58. Performance Examples
Pseudo-Bellman-Ford on two-core shared memory machine:
0 0.2 0.4 0.6 0.8 1 1.2 1.4
·105
0
0.5
1
1.5
2
2.5
0 0.2 0.4 0.6 0.8 1 1.2 1.4
·105
0
1
2
3
4

59. Performance Examples
Pseudo-Bellman-Ford on four-core NUMA machine:
0 1 2 3 4
·105
0
1
2
3
4
0 1 2 3 4
·105
0
2
4
6
8

60. Future Work
Fill gaps in theory (caching and communication).

61. Future Work
Fill gaps in theory (caching and communication).
Generalise theory to more dimensions and interleaved DPs.

62. Future Work
Fill gaps in theory (caching and communication).
Generalise theory to more dimensions and interleaved DPs.
Improve and extend implementations.

63. Future Work
Fill gaps in theory (caching and communication).
Generalise theory to more dimensions and interleaved DPs.
Improve and extend implementations.
More experiments (different problems, more diverse machines).

64. Future Work
Fill gaps in theory (caching and communication).
Generalise theory to more dimensions and interleaved DPs.
Improve and extend implementations.
More experiments (different problems, more diverse machines).
Improve compiler integration (detection, backtracing, result
functions).

65. Future Work
Fill gaps in theory (caching and communication).
Generalise theory to more dimensions and interleaved DPs.
Improve and extend implementations.
More experiments (different problems, more diverse machines).
Improve compiler integration (detection, backtracing, result
functions).
Integrate with other tools.