Introduction to Concurrent Programming

Introduction to
Concurrent Programming
CS4532 Concurrent Programming
Dilum Bandara
Dilum.Bandara@uom.lk
Slides adapted from “The Art of Multiprocessor Programming”
by Maurice Herlihy & Nir Shavit Slightly, & Dr. Srinath Perera

Moore’s Law
2
No of transistors on a
chip tends to double
about every 2 years
Transistor
count still
rising
Clock speed
flattening
sharply
www.extremetech.com/wp-
content/uploads/2012/02/CPU-Scaling.jpg

Migration
Memory
CPU
3
Uniprocessor Shared Memory
Multiprocessor (SMP)
Cache
Bus
Shared memory
Cache
Cache

Why Do We Care?
 Time no longer cures software bloat
 The “free ride” is over
 When you double your program’s path length
 You can’t just wait 6 months
 Your software must somehow exploit twice as much
concurrency
4

Traditional Scaling Process
5
User code
Traditional
Uniprocessor
Speedup
1.8x
7x
3.6x
Time: Moore’s law

Multi-Core Scaling Process
6
User code
Multicore
Speedup
1.8x
7x
3.6x
Unfortunately, not so simple…

Real-World Scaling Process
7
1.8x 2x 2.9x
User code
Multicore
Speedup
Parallelization & Synchronization
require great care…

Parallel & Concurrent Computing?
 “is a form of computation in which many
calculations are carried out simultaneously by
many thread of executions”
 We are concerned with
 Theory around parallelism
 Tools & tricks
 Use cases & potential applications
 Common solution patterns
8

Sequential Computation
9
memory
object object
thread

Concurrent Computation
10
memory
object object

Concurrent vs. Parallel
 Concurrent computing – when multiple tasks can be in progress at any
instance in a program
 Parallel computing – multiple tasks cooperate closely to solve a
problem within a program
 Distributed computing – when a program need to cooperate with other
programs to solve a problem 11

Asynchrony
Sudden unpredictable delays
 Cache misses (short)
 Page faults (long)
 Scheduling quantum used up (really long)
12
 Threads are scheduled by OS & thread library
 Execution can be stopped at anytime, & another thread may be started
 No timing guarantees are given, algorithms must not make any
assumptions on speed of executions

Why Parallel/Concurrent Computing?
1. Only way to solve some problems
 Rendering animations
 Weather prediction
 Big bang
2. Most real-world problems are parallel
 Weather, Galaxy
 Bee/Ant Colony, Traffic
13
http://movies.disney.com/movies/ratatouille
1532 CPU-years to render

Why Parallel/Concurrent Computing?
3. To fully utilize your computer
 Computer often spend time waiting on I/O, which
takes much more time than CPU processing
 Unless application is number crunching, which is
often not the case, parallelism can give lots of gains
4. To make use of the cloud
 Cloud going to place 100’s to 1000’s of machines on
normal people’s finger tips
 Do you know how to use them?
 Solution is parallel computing
14

Flynn’s Taxonomy
 SISD
 Normal sequential programs
 Uniprocessor
 SIMD
 Data parallelism
 GPUs & vector processors
 MISD
 Fault tolerant schemes
 MIMD
 Most parallel programs
 Multi-core
I – instruction, D – data, S – Single, M - Multiple 15

Road Map
 We are going to focus on principles first, then
practice
 Start with idealized models
 Look at simplistic problems
 Emphasize correctness over pragmatism
 “Correctness may be theoretical, but incorrectness
has practical impact”
16

Example – Parallel Primality Testing
 Challenge
 Print primes from 1 to 1010
 Given
 10-processor multiprocessor
 1 thread per processor
 Goal
 Get 10-fold speedup (or close)
17

void primePrint {
int i = ThreadID.get(); // IDs in {0..9}
for (j = i*109+1, j<(i+1)*109; j++) {
if (isPrime(j))
print(j);
}
}
Load Balancing
 Split the work evenly
 Each thread tests range of 109
18
…
…
109 1010
2·109
1
P0 P1 P9

Issues
 Higher ranges have fewer primes
 Yet larger numbers are harder to test
 Thread workloads
 Uneven
 Hard to predict
 Need dynamic load balancing
19

Shared Counter
20
17
18
19
Each thread takes
a number

Procedure for Thread i
21
int counter = new Counter(1);
void primePrint {
long j = 0;
while (j < 1010) {
j = counter.getAndIncrement();
if (isPrime(j))
print(j);
}
}

Procedure for Thread i
22
Counter counter = new Counter(1);
void primePrint {
long j = 0;
while (j < 1010) {
j = counter.getAndIncrement();
if (isPrime(j))
print(j);
}
}
Shared counter
object
Stop when every value
taken
Increment & return
each new value

What It Means
23
public class Counter {
private long value;
public long getAndIncrement() {
return value++;
}
}
temp = value;
value = temp + 1;
return temp;

Not So Good…
24
time
Value… 1
read
1
read
1
write
2
read
2
write
3
write
2
2 3 2

Is The Problem Inherent?
25
E.g., Walking on a narrow passage, trying to
catch a ball in Cricket
read
write read
write

Challenge
26
private long value;
temp = value;
value = temp + 1;
return temp;
}
}
Make these steps
atomic (indivisible)

Hardware Solution
27
private long value;
temp = value;
value = temp + 1;
return temp;
}
} ReadModifyWrite()
instruction

An Aside: Java™
28
private long value;
synchronized {
temp = value;
value = temp + 1;
}
return temp;
}
}
Synchronized block
Mutual exclusion

Mutual Exclusion or “Alice & Bob
share a pond”
29
A B

The Problem
32
A B
The pets don’t
get along

Formalizing the Problem
 2 types of formal properties in asynchronous
computation
 Safety Properties
 Nothing bad happens ever
 Consistency
 Liveness Properties
 Something good happens eventually
 No deadlock, starvation
 Efficiency is also important
 You have to look at your solution for those
properties, & you will be graded on that
33

Formalizing our Problem
 Mutual Exclusion
 Both pets never in pond simultaneously
 This is a safety property
 No Deadlock
 If only 1 wants in, it gets in
 If both want in, 1 gets in
 This is a liveness property
 How can we allow 2 pets to use pond without
getting into trouble???
34

Simple Protocol
 Idea
 Just look at the pond
 Gotcha
 Trees obscure the view
 Interpretation
 Threads can’t “see” what other threads are doing
 Explicit communication required for coordination
35

Cell Phone Protocol
 Idea
 Bob calls Alice (or vice-versa)
 Gotcha
 Bob takes shower
 Alice recharges battery
 Bob out shopping for pet food…
 Interpretation
 Message-passing doesn’t work
 Recipient might not be listening or there at all
 Communication must be
 Persistent (like writing)
 Not transient (like speaking) 36

Can Protocol
 Idea
 Cans on Alice’s window sill, Strings lead to Bob’s house
 Bob pulls strings, knocks over cans
 Alice can look at cans when she want
 Gotcha
 Cans can’t be reused
 Bob runs out of cans – windows still has limited space
 Interpretation
 Can’t solve mutual exclusion with interrupts
 Sender sets fixed bit in receiver’s space
 Receiver resets bit when ready
 Requires unbounded number of interrupt bits 40

Alice’s Protocol (sort of)
42
A B

Bob’s Protocol (sort of)
43
A B

Protocol
Alice
 Raise flag
 Wait until Bob’s flag is
down
 Unleash pet
 Lower flag when pet
returns
Bob
 Raise flag
 Wait until Alice’s flag is
down
 Unleash pet
returns
44

Protocol – 2nd Try
Alice
 Raise flag
 Wait until Bob’s flag is
down
 Unleash pet
returns
Bob
 Raise flag
 While Alice’s flag is up
 Lower flag
 Wait for Alice’s flag to go
down
 Raise flag
 Unleash pet
returns
45
Bob defers to
Alice

Flag Principle
 Raise the flag
 Look at other’s flag
 Flag Principle
 If each raises & looks, then
 Last to look must see both flags up
 Can prove
 Satisfies mutual exclusion
 By deriving a contradiction
 No deadlocks
 If Bob sees Alice’s flag, he gives her priority
46

Remarks
 Protocol is unfair
 Bob’s pet might never get in
 Protocol uses waiting
 If Bob is eaten by his pet, Alice’s pet might never get in
47

Waiting
 Both solutions use waiting
 Alice & Bob need time to decide “is the flag really up?”
 Flag can be in infinitely many positions
 You need to time to think & take a decision
 while(mumble){}
 Waiting is problematic
 If one participant is delayed
 So is everyone else
 But delays are common & unpredictable
48

Moral of Story
 Mutual Exclusion can’t be solved by
 Message passing – e.g., cell phones
 Transient communication
 Interrupts – e.g., tin cans
 Can be postponed
 It can be solved by
 One-bit shared variables that can be read or written,
i.e., a flag
49

The Fable Continues
 Alice & Bob fall in love & marry
 Then they fall out of love & divorce
 She gets the pets
 He has to feed them
 Leading to a new coordination problem
 Producer-Consumer
50

Bob Puts Food in the Pond
51
A

Alice Releases Her Pets to Feed
52
mmm… B
mmm…

Producer/Consumer
 Alice & Bob can’t meet
 Each has restraining order on other
 So he puts food in pond
 And later, she releases pets
 Avoid
 Releasing pets when there’s no food
 Putting out food, if uneaten food remains
 Need a mechanism so that
 Bob lets Alice know when food has been put out
 Alice lets Bob know when to put out more food
53

Bob Puts Food in Pond
55
A B
cola

Alice Releases Pets
57
A B
yum… B
yum…

Alice Resets Can When Pets are Fed
58
A B
cola

Pseudocode
59
while (true) {
while (can.isUp()){};
pet.release();
pet.recapture();
can.reset();
}
Alice’s code
while (true) {
while (can.isDown()){};
pond.stockWithFood();
can.knockOver();
}
Bob’s code

Correctness
 Mutual Exclusion
 Pets & Bob never together in pond
 Safety property
 No Starvation
 If Bob always willing to feed, & pets always famished,
then pets eat infinitely often
 Liveness property
 Producer/Consumer
 Pets never enter pond unless there is food, & Bob
never provides food if there is unconsumed food
 Safety property
60

But
 Didn’t we say mutual exclusion can’t be solved
with Interrupts (tin cans)???
 Well, this problem is somewhat different
 There’s a need for Alice & Bob to cooperate
 If not, pets will die & Bob will be in Jail
 Hence, resetting the tin can (i.e., interrupt-bit) can’t be
postponed forever
 Producer & consumer take turns
 Like setting & resetting an interrupt bit
61

The Fable Drags On …
 Bob & Alice still have issues
 So they need to communicate
 So they agree to use billboards …
62
E1
D2
C3
B3
A1
Letter
Tiles
From Scrabble™ box

Billboards are Large – Write 1 Letter
at a Time …
63
E1
D2
C3
B3
A1
W4
A1
S1
H4

To Post a Message
64
W4
A1
S1
H4
A1
C3
R1
T1
H4
E1
whe
w

Let’s Send Another Message
65
S1
S1
E1
L1
L1
L1
V4
L1 A1
M3
A1
A1
P3

Uh-Oh
66
A1
C3
R1
T1
H4
E1
S1
E1
L1
L1
L1
OK

Readers/Writers
 Devise a protocol so that
 Writer writes one letter at a time
 Reader reads one letter at a time
 Reader sees
 Old message or new message
 No mixed messages
67

Readers/Writers (Cont.)
 Easy with mutual exclusion
 But mutual exclusion requires waiting
 One waits for the other
 Everyone executes sequentially
 Remarkably
 We can solve R/W without mutual exclusion
68

Summary
 Why concurrent programming?
 Mutual exclusion & cooperation
 Requirements
 Safety, Liveliness, & Efficiency
 What to avoid
 Race Conditions, Deadlocks, & Starvation
69

Introduction to Concurrent Programming

More Related Content

Similar to Introduction to Concurrent Programming

More from Dilum Bandara

Recently uploaded

Introduction to Concurrent Programming