An introduction to parallel programming; from basic problems and parallel primitives to patterns and lock free programming.

1. Beyond The Critical Section

5. Moore’s Law

7. Console trends

11. Race Condition Example x++ x++ x=0 x=? Thread A Thread B

12. Race Condition Example R1 = 0 x=0 Thread A Thread B

13. Race Condition Example R1 = 0+1 x=0 Thread A Thread B

14. Race Condition Example R1 = 1 R1 = 0 x=0 Thread A Thread B

15. Race Condition Example R1 = 1 R1 = 0+1 x=1 Thread A Thread B

19. Race Condition Solution A AtomicInc(x) AtomicInc(x) x=0 Thread A Thread B

20. Race Condition Solution A AtomicInc(x) AtomicInc(x) x=0 Thread A Thread B x=1

21. Race Condition Solution A AtomicInc(x) AtomicInc(x) x=0 Thread A Thread B x=1 x=2

23. Race Condition Solution B x=0 Thread A Thread B Lock A x++ Unl0ck A Lock A x++ Unl0ck A

24. Race Condition Solution B x=0 Thread A Thread B Lock A x++ Unl0ck A Lock A x++ Unl0ck A

25. Race Condition Solution B x=0 Thread A Thread B x=1 Lock A x++ Unl0ck A Lock A x++ Unl0ck A

26. Race Condition Solution B x=0 Thread A Thread B x=1 Lock A x++ Unl0ck A Lock A x++ Unl0ck A

27. Race Condition Solution B x=0 Thread A Thread B x=1 Lock A x++ Unl0ck A Lock A x++ Unl0ck A

28. Race Condition Solution B x=0 Thread A Thread B x=1 x=2 Lock A x++ Unl0ck A Lock A x++ Unl0ck A

29. Race Condition Solution B x=0 Thread A Thread B x=1 x=2 Lock A x++ Unl0ck A Lock A x++ Unl0ck A

39. Consider a list and a thread pool… head a c b … ..

40. Thread A about to CAS head from a to b head a c b … .. CAS(&head->next,a,b);

41. Threads B: deq a & b head c … .. a b A & B are released into thread local pools

42. Thread B enq A - reused head a c … .. b A is added back

43. Thread A executes CAS head a c … .. b CAS(&head->next,a,b);

44. Thread A executes CAS successfully! head a c … .. b CAS(&head->next,a,b);

52. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(3) Use results Do stuff

53. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(3) Use results Do stuff Calculating

54. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(3) Use results Do stuff Done

55. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(3) Use results Do stuff Signal

56. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(2) Use results Do stuff Do other stuff

57. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(2) Use results Do stuff Calculating

58. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(2) Use results Do stuff Done Calculating

59. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(2) Use results Do stuff Signal Done

60. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(1) Use results Do stuff More code Signal

61. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(0) Use results Do stuff Calc pi

62. Barrier example Thread 1 Thread 2 Thread 3 Thread 4 Barrier(0) Use results Do stuff

67. Problem Decomposition Problem From “Patterns for Parallel Programming”

68. Problem Decomposition Problem Organise By Tasks Organise By Data Decomposition Organise By Data Flow From “Patterns for Parallel Programming”

69. Problem Decomposition Problem Organise By Tasks Organise By Data Decomposition Organise By Data Flow Linear Recursive Linear Recursive Linear Recursive From “Patterns for Parallel Programming”

70. Problem Decomposition Problem Organise By Tasks Organise By Data Decomposition Organise By Data Flow Linear Recursive Linear Recursive Linear Recursive Task Parallelism Divide and Conquer Geometric Decomposition Recursive Data Pipeline Event-Based Coordination From “Patterns for Parallel Programming”

77. Supporting Structures SPMD Master/Worker Loop Parallelism Fork/Join Program Structures Data Structures Shared Data Distributed Array Shared Queue

78. Program Structures SPMD Master/Worker Loop Parallelism Fork/Join Program Structures Data Structures Shared Data Distributed Array Shared Queue

83. Supporting Data Structures SPMD Master/Worker Loop Parallelism Fork/Join Program Structures Data Structures Shared Data Distributed Array Shared Queue

89. Adding a node to a list head a c tail b

90. Adding a node: Step 1 head a c tail b Find where to insert

91. Adding a node: Step 2 head a c tail b newNode->Next = prev->Next;

92. Adding a node: Step 3 head a c tail b prev->Next = newNode;

94. Add ‘b’ and ‘c’ concurretly head a d tail b c Find where to insert

95. Add ‘b’ and ‘c’ concurretly head a d tail b c newNode->Next = prev->Next;

96. Add ‘b’ and ‘c’ concurrently head a d tail b c prev->Next = newNode;

97. Add ‘b’ and ‘c’ concurrently head a d tail b c

101. Coarse Grain head a c tail b

102. Step 1: Lock list b head a c tail

103. Step 2 & 3:Find then Insert b head a c tail

104. Step 4:Unlock head a c tail b

107. Fine Grained Locking head a c tail b

108. Fine Grained Locking a c tail b head

109. Fine Grained Locking c tail b head a

110. Fine Grained Locking head tail b a c

111. Fine Grained Locking head tail b a c

112. Fine Grained Locking head a c tail b

115. Optimistic: Add(“g”) head a c d tail f k m g

116. Step 1: Search head a c d tail f k m g

117. Step 1: Search head a c d tail f k m g

118. Step 1: Search head a c d tail f k m g

119. Step 1: Search head a c d tail f k m g

120. Step 1: Search head a c d tail f k m g

121. Step 1: Search head a c d tail f k m g

122. Step 2: Lock head a c d tail m g f k

123. Step 3: Validate head a c d tail m g f k

124. Step 3: Validate head a c d tail m g f k

125. Step 3: Validate - FAIL head a tail m g d f k

126. Step 3a: Validate (retry) head a e tail m g d f k

127. Step 3a: Validate (retry) head a e tail m g d f k

128. Step 3a: Validate (retry) head a e tail m g d f k

129. Step 3a: Validate (retry) head a e tail m g d f k

130. Step 3a: Validate (success) head a e tail m g d f k

131. Step 4: Add head a e tail m g d f k

132. Step 5: Unlock head a e tail f k m g d

134. Delete Caveat: Validate head a e tail m g d f k

135. Delete Caveat: Validate head a e tail m g d f k

136. Delete Caveat: delete ‘d’ head a e tail m g f k d

137. Delete Caveat: Validate head a e tail m g f k d

138. Delete Caveat: Validate head a e tail m g f k d

139. Delete Caveat: Valid! head a e tail m g f k d

142. Lazy: Add(“g”) head a c d tail f k m g

143. Step 1: Search head a c d tail f k m g

144. Step 1: Search head a c d tail f k m g

145. Step 1: Search head a c d tail f k m g

146. Step 1a: Search (delete c) head a c d tail f k m g

147. Step 1a: Search (delete c) head a c d tail f k m g

148. Step 1a: Search (delete c) head a c d tail f k m g

149. Step 1a: Search (delete c) head a c d tail f k m g

150. Step 1b: Search (lock) head d tail f k m g a c

151. Step 1c: Search (mark) head d tail f k m g a c

152. Step 2d: lock (skip/unlock) head a c d tail m g f k

153. Step 3: Add/Validate head a d tail m g c f k

154. Step 4: Unlock head a d tail f k m g c

156. Lazy Synchronisation ~330ms

158. Delete ‘a’ and add ‘b’ concurrently head a c tail b prev->next=curr->next; | prev->next=b;

159. Delete ‘a’ and add ‘b’ concurrently head a c tail b prev->next=curr->next; | prev->next=b;

160. Delete ‘a’ and add ‘b’ concurrently head a c tail b head->next=a->next; | prev->next=b;

161. Delete ‘a’ and add ‘b’ concurrently head a c tail b Effectively deletes ‘a’ and ‘b’.

164. Lock Free: Remove ‘d’ head a c d tail f k m Start loop:

165. Step 1: Find ‘d’ head a c tail f k m pred curr succ d if(!InternalFind(‘d’)) continue;

166. Step 2: Mark ‘d’ head a c tail f k m pred curr succ d if(!curr->next->CAS(succ,succ,false,true)) continue;

167. Step 3: Skip ‘d’ head a c tail f k m pred curr succ d pred->next->CAS(curr,succ,false,false);

169. Second InternalFind() head a c tail f k m pred curr succ pred curr succ d

170. If succ is marked… head a c tail f k m pred curr succ pred curr succ d

171. … Skip it head a c tail f k m pred curr succ pred curr succ d

174. Performance comparison