Vern Paxson’s Paper"
“End-to-End"
Internet Packet
Dynamics”, 1997/99
22 August 2008
NUS CS5229 Semester 1 1
2008/09

How often are packets
dropped?"
How often are packets
reordered?"
:
22 August 2008
NUS CS5229 Semester 1 2
2008/09

Why these
questions?
22 August 2008
NUS CS5229 Semester 1 3
2008/09

1. Understand the
Internet
22 August 2008
NUS CS5229 Semester 1 4
2008/09

“
when you can measure what you are
speaking about, and express it in
numbers, you know something about
it; but when you cannot measure it,
when you cannot express it in
”
numbers, your knowledge is of a
meagre and unsatisfactory kind;
- Lord Kelvin
22 August 2008
NUS CS5229 Semester 1 5
2008/09

2. Model the Internet
22 August 2008
NUS CS5229 Semester 1 6
2008/09

3. Enable more
accurate evaluation
through simulations
22 August 2008
NUS CS5229 Semester 1 7
2008/09

4. Lead to a better
application/systems
design
22 August 2008
NUS CS5229 Semester 1 8
2008/09

How often are packets
dropped?"
How often are packets
reordered?"
:
22 August 2008
NUS CS5229 Semester 1 9
2008/09

How to answer these
questions?
22 August 2008
NUS CS5229 Semester 1 10
2008/09

Collect lots of packet
traces"
Analyze the traces
22 August 2008
NUS CS5229 Semester 1 11
2008/09

Trace collection:"
large number of flows"
a variety of sites"
many packets per
flow"
use TCP
22 August 2008
NUS CS5229 Semester 1 12
2008/09

Why TCP:"
real-world traffic"
will not overload the
network
22 August 2008
NUS CS5229 Semester 1 13
2008/09

Time between measurement is
Poisson distributed."
PASTA Theorem: Intuitively, if we
make n observations and k
observations is in some state S and
n-k in other states, then we can
assume prob of observing S is
approximately k/n.
22 August 2008
NUS CS5229 Semester 1 14
2008/09

Two traces:"
N1: Dec94 "
N2: Nov-Dec95"
use tcpdump at sender +
receiver"
22 August 2008
NUS CS5229 Semester 1 15
2008/09

100 kB
Size of file transfered
22 August 2008
NUS CS5229 Semester 1 16
2008/09

21
Number of sites
22 August 2008
NUS CS5229 Semester 1 17
2008/09

20800
Number of trace pairs
22 August 2008
NUS CS5229 Semester 1 18
2008/09

Part 1:"
The Unexpected
22 August 2008
NUS CS5229 Semester 1 19
2008/09

Packet
Reordering
22 August 2008
NUS CS5229 Semester 1 20
2008/09

1
2
5
2 reorderings
3
4
22 August 2008
NUS CS5229 Semester 1 21
2008/09

N1
N2
36%
12%
Percentage of connections with
at least one out-of-order delivery
22 August 2008
NUS CS5229 Semester 1 22
2008/09

N1
N2
2%
.3%
Percentage of data packets out-of-order
22 August 2008
NUS CS5229 Semester 1 23
2008/09

N1
N2
.6%
.1%
Percentage of ACK packets out-of-order
22 August 2008
NUS CS5229 Semester 1 24
2008/09

Data packets are
usually sent
closer together.
22 August 2008
NUS CS5229 Semester 1 25
2008/09

From
To
15%
.2%
Percentage of packets out-of-order
to and from U of Colorado in N1.
22 August 2008
NUS CS5229 Semester 1 26
2008/09

Route fluttering:
alternate packets
can take different
route to dest.
22 August 2008
NUS CS5229 Semester 1 27
2008/09

Taken from Paxson’s PhD Thesis: Alternate routes
are taken for packets from WUSTL to U Mannheim
22 August 2008
NUS CS5229 Semester 1 28
2008/09

Fig 1 from the paper, showing large gap and two slopes.
22 August 2008
NUS CS5229 Semester 1 29
2008/09

T1
(new arrival)
Ethernet
(buffered packets)
Fig 1 from the paper, showing large gap and two slopes.
22 August 2008
NUS CS5229 Semester 1 30
2008/09

Impact of Packet
Reordering
22 August 2008
NUS CS5229 Semester 1 31
2008/09

Recap: TCP’s fast
retransmit
22 August 2008
NUS CS5229 Semester 1 32
2008/09

S = 1000
S = 2000
S = 3000
S = 4000
A = 2000
S = 5000
A = 2000
A = 2000
A = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 33
2008/09

S = 1000
S = 2000
S = 3000
S = 4000
A = 2000
S = 5000
A = 2000
A = 2000
A = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 34
2008/09

Nd = 3 is a
conservative
choice.
22 August 2008
NUS CS5229 Semester 1 35
2008/09

What if receiver
wait longer before
sending dup ack?
22 August 2008
NUS CS5229 Semester 1 36
2008/09

S = 2000
W
S = 2000
22 August 2008
NUS CS5229 Semester 1 37
2008/09

1
2
5
Delivery Gap: "
time between receiving "
3
an out-of-order packet and "
the packet sent before it.
4
22 August 2008
NUS CS5229 Semester 1 38
2008/09

Taken from Paxson’s PhD Thesis: CDF for delivery gap between reordered
packets.
22 August 2008
NUS CS5229 Semester 1 39
2008/09

N2
N1
higher BW
22 August 2008
NUS CS5229 Semester 1 40
2008/09

N1
N2
20ms
8ms
Waiting time with which 70% of "
out-of-order delivery would be identified.
22 August 2008
NUS CS5229 Semester 1 41
2008/09

Is needless
retransmission a
problem?
22 August 2008
NUS CS5229 Semester 1 42
2008/09

Good
S = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 43
2008/09

Bad
S = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 44
2008/09

N1
N2
22
300
Number of good retransmissions
for every bad retransmission.
Nd = 3, W = 0
22 August 2008
NUS CS5229 Semester 1 45
2008/09

N1
N2
~ 7
100
Number of good retransmissions
for every bad retransmission.
Nd = 2, W = 0
22 August 2008
NUS CS5229 Semester 1 46
2008/09

N1
N2
15
300
Number of good retransmissions
for every bad retransmission.
Nd = 2, W = 20ms
22 August 2008
NUS CS5229 Semester 1 47
2008/09

Packet
Corruption
22 August 2008
NUS CS5229 Semester 1 48
2008/09

1 in 5000
packet is corrupted
22 August 2008
NUS CS5229 Semester 1 49
2008/09

1 in 65536
corrupted packet goes undetected
using TCP checksum
(assuming each possible checksum is equally likely)
22 August 2008
NUS CS5229 Semester 1 50
2008/09

1 in 300million
Internet packet is corrupted "
and is undetected.
22 August 2008
NUS CS5229 Semester 1 51
2008/09

Part 2:"
Bottleneck
Bandwidth
22 August 2008
NUS CS5229 Semester 1 52
2008/09

Packet Pair
22 August 2008
NUS CS5229 Semester 1 53
2008/09

B bps
b bytes
22 August 2008
NUS CS5229 Semester 1 54
2008/09

Q x B = b
Q s
22 August 2008
NUS CS5229 Semester 1 55
2008/09

Q s
22 August 2008
NUS CS5229 Semester 1 56
2008/09

Q s
22 August 2008
NUS CS5229 Semester 1 57
2008/09

Q s
22 August 2008
NUS CS5229 Semester 1 58
2008/09

Problems with
Packet Pair
22 August 2008
NUS CS5229 Semester 1 59
2008/09

1. Asymmetric
Link
22 August 2008
NUS CS5229 Semester 1 60
2008/09

22 August 2008
NUS CS5229 Semester 1 61
2008/09

2. ACK
Compression
22 August 2008
NUS CS5229 Semester 1 62
2008/09

22 August 2008
NUS CS5229 Semester 1 63
2008/09

3. Out of order
delivery
22 August 2008
NUS CS5229 Semester 1 64
2008/09

22 August 2008
NUS CS5229 Semester 1 65
2008/09

4. Clock
resolution
22 August 2008
NUS CS5229 Semester 1 66
2008/09

Suppose
B = 1000 kBps
b = 1 kB
Q = ?
22 August 2008
NUS CS5229 Semester 1 67
2008/09

5. Changing
bottleneck
bandwidth
22 August 2008
NUS CS5229 Semester 1 68
2008/09

Fig 2 from the paper, showing changing
bandwidth.
22 August 2008
NUS CS5229 Semester 1 69
2008/09

Fig 3 from the paper, showing multi-channel
links.
22 August 2008
NUS CS5229 Semester 1 70
2008/09

6. Multi-channel
Links
22 August 2008
NUS CS5229 Semester 1 71
2008/09

Asymmetric links"
ACK compression"
Out-of-order delivery"
Clock resolution"
Changes in bottleneck
bandwidth"
Multi-channel links"
22 August 2008
NUS CS5229 Semester 1 72
2008/09

Measure at receiver:"
Asymmetric links"
ACK compression"
Packet bunch:"
Out-of-order delivery"
Clock resolution"
Changes in bottleneck
bandwidth"
Multi-channel links
22 August 2008
NUS CS5229 Semester 1 73
2008/09

2Q
22 August 2008
NUS CS5229 Semester 1 74
2008/09

Collect multiple estimates,
take the most freq
occurrence (modes) as the
bottleneck bandwidth.
22 August 2008
NUS CS5229 Semester 1 75
2008/09

22 August 2008
NUS CS5229 Semester 1 76
2008/09

Part 3:"
Packet Loss
22 August 2008
NUS CS5229 Semester 1 77
2008/09

N1
N2
2.7%
5.2%
Percentage of packets that were lost.
22 August 2008
NUS CS5229 Semester 1 78
2008/09

N1
N2
50%
50%
Percentage of loss free connections
22 August 2008
NUS CS5229 Semester 1 79
2008/09

N1
N2
5.7%
9.2%
Loss rate on lossy connections
22 August 2008
NUS CS5229 Semester 1 80
2008/09

17%
Loss rate on connections from EU to US
22 August 2008
NUS CS5229 Semester 1 81
2008/09

Are packet losses
independent?
22 August 2008
NUS CS5229 Semester 1 82
2008/09

Compute:"
Pu = Pr [ p lost ]"
Pc = Pr [ p lost | prev pkt lost ]
22 August 2008
NUS CS5229 Semester 1 83
2008/09

Pu
Pc
2.8%
49%
Loss rate for “queued data pkt” on N1
22 August 2008
NUS CS5229 Semester 1 84
2008/09

Fig 6 from the paper, showing outage
duration.
22 August 2008
NUS CS5229 Semester 1 85
2008/09

Are
retransmission
redundant?
22 August 2008
NUS CS5229 Semester 1 86
2008/09

Unavoidable
X
ACK
22 August 2008
NUS CS5229 Semester 1 87
2008/09

Coarse Feedback
22 August 2008
NUS CS5229 Semester 1 88
2008/09

Bad RTO
22 August 2008
NUS CS5229 Semester 1 89
2008/09

N1
N2
26%
28%
Percentage of retransmissions that"
are redundant
22 August 2008
NUS CS5229 Semester 1 90
2008/09

4
44 Unavoidable
Coarse Feedback
Bad RTO
51
Type of redundant retransmission in N1.
22 August 2008
NUS CS5229 Semester 1 91
2008/09

Part 4:"
Packet Delay
22 August 2008
NUS CS5229 Semester 1 92
2008/09

OTT is not well
approximated
using RTT/2
22 August 2008
NUS CS5229 Semester 1 93
2008/09

ACK
Compression
22 August 2008
NUS CS5229 Semester 1 94
2008/09

(might affect TCP self-clocking)
22 August 2008
NUS CS5229 Semester 1 95
2008/09

sent
recv
22 August 2008
NUS CS5229 Semester 1 96
2008/09

Sending interval
Receiving interval
22 August 2008
NUS CS5229 Semester 1 97
2008/09

22 August 2008
NUS CS5229 Semester 1 98
2008/09

22 August 2008
NUS CS5229 Semester 1 99
2008/09

Compression
event if ξ < .75
22 August 2008
NUS CS5229 Semester 1 100
2008/09

N1
N2
50%
60%
Percentage of connection that experiences"
at least one compression event.
22 August 2008
NUS CS5229 Semester 1 101
2008/09

N1
N2
50%
60%
Percentage of connection that experiences"
at least one compression event.
22 August 2008
NUS CS5229 Semester 1 102
2008/09

2
Average number of events per connection.
22 August 2008
NUS CS5229 Semester 1 103
2008/09

Estimating
Available
Bandwidth
22 August 2008
NUS CS5229 Semester 1 104
2008/09

Qb: time to transit the bottleneck"
ψi: expected time spent queuing behind
predecessor (derived from sending time)"
γi: diff between packet OTT and min OTT
22 August 2008
NUS CS5229 Semester 1 105
2008/09

time packet i is sent
22 August 2008
NUS CS5229 Semester 1 106
2008/09

22 August 2008
NUS CS5229 Semester 1 107
2008/09

β = 1 means all bandwidth
is available."
β = 0 means none of the
bandwidth is available.
22 August 2008
NUS CS5229 Semester 1 108
2008/09

Fig 10 from the paper, showing distribution
of available bandwidth.
22 August 2008
NUS CS5229 Semester 1 109
2008/09

Conclusion
22 August 2008
NUS CS5229 Semester 1 110
2008/09

The numbers in the
paper are not important. "
(the Internet has changed)
22 August 2008
NUS CS5229 Semester 1 111
2008/09

Measurement is
difficult but useful
22 August 2008
NUS CS5229 Semester 1 112
2008/09

Many new techniques
needed (e.g to
measure bottleneck
bandwidth)
22 August 2008
NUS CS5229 Semester 1 113
2008/09

We can improve
current design (e.g.
TCP if we know more
about reordering)
22 August 2008
NUS CS5229 Semester 1 114
2008/09

We can identify
problem (e.g. packet
corruption)
22 August 2008
NUS CS5229 Semester 1 115
2008/09

We can better model
the behavior (e.g.
bursty packet loss)
22 August 2008
NUS CS5229 Semester 1 116
2008/09

We can infer much
info from just a
packet trace (e.g.
available bandwidth)
22 August 2008
NUS CS5229 Semester 1 117
2008/09