1. Patterns of Parallel
Programming
Prepared by Yan Drugalya
ydrugalya@gmail.com
@ydrugalya

2. Agenda
• Why parallel?
• Terms and measures
• Building Blocks
• Patterns overview
– Pipeline and data flow
– Producer-Consumer
– Map-Reduce
– Other

3. Why Moore's law is not working
anymore
• Power consumption
• Wire delays
• DRAM access latency
• Diminishing returns of more instruction-level
parallelism

4. Power consumption
Sun’s Surface
10,000
1,000 Rocket Nozzle
Power Density (W/cm2)
100 Nuclear Reactor
10 Pentium® processors
Hot Plate
1
8080
‘70 ‘80 ’90 ’00 ‘10

5. Wire delays

6. Diminishing returns
• 80’s
– 10 CPI  1 CPI
• 90
– 1 CPI  0.5CPI
• 00’s: multicore

7. No matter how fast processors get, software
consistently finds new ways to eat up the extra
speed.
Herb Sutter

8. Survival
 To scale performance, put many processing cores on the
microprocessor chip
 New Moore’s law edition is about doubling of cores.

9. Terms & Measures
• Work = T1
• Span = T∞
• Work Law: Tp>=T1/P
• Span Law: Tp>=T∞
• Speedup: Tp/T1
– Linear: θ(P)
– Perfect: P
• Parallelism: T1/T∞
• Tp<=(T1-T∞)/P + T∞

10. Definitions
• Concurrent
- Several things happenings at the same time
• Multithreaded
– Multiple execution contexts
• Parallel
– Multiple simultaneous computations
• Asynchronous
– Not having to wait

11. Dangers
• Race Conditions
• Starvations
• Deadlocks
• Livelock
• Optimizing compilers
• …

12. Data parallelism
Parallel.ForEach(letters, ch => Capitalize(ch));

13. Task parallelism
Parallel.Invoke(() => Average(), () => Minimum() …);

14. Fork-Join
• Additional work may be started only when specific subsets of
the original elements have completed processing
• All elements should be given the chance to run even if one
invocation fails (Ping)
Parallel.Invoke(
() => ComputeMean(),
Fork () => ComputeMedian(),
() => ComputeMode());
Compute Compute Compute static void MyParallelInvoke(params Action[] actions)
Median Mean Mode {
var tasks = new Task[actions.Length];
for (int i = 0; i < actions.Length; i++)
tasks[i] = Task.Factory.StartNew(actions[i]);
Join Task.WaitAll(tasks);
}

15. Pipeline pattern
Task<int> T1 = Task.Factory.StartNew(() =>
Task 1 { return result1(); });
Task<double> T2 = T1.ContinueWith((antecedent) =>
Task 2 { return result2(antecedent.Result); });
Task<double> T3 = T2.ContinueWith((antecedent) =>
Task 3 { return result3(antecedent.Result); });

16. Producer/Consumer
Disk/Net
Read 1 Read 2 Read 3
BlockingCollection<T> Process Process Process

18. Other patterns
• Speculative Execution
• APM (IAsyncResult, Begin/end pairs)
• EAP(Operation/Callback pairs)

19. References
• Patterns for Parallel Programming: Understanding and Applying
Parallel Patterns with the .NET Framework 4
• Pluralsight:
– Introduction to Async and Parallel Programming in .NET 4
– Async and Parallel Programming: Application Design
• The Free Lunch Is Over: A Fundamental Turn Toward
Concurrency in Software
• Chapter 27 Multithreaded Algorithms from Introduction to
algorithms 3rd edition