This document summarizes Vern Paxson's 1997/1999 paper "End-to-End Internet Packet Dynamics". The paper studied packet reordering, loss, and bottleneck bandwidth on the Internet through analyzing packet traces collected in 1994 and 1995. It found significant levels of packet reordering due to route changes. Packet loss rates were around 2-5% but most connections experienced no loss. The paper introduced new techniques to measure bottleneck bandwidth and addressed challenges in accurate measurement.

1. Vern Paxson’s Paper"
“End-to-End"
Internet Packet
Dynamics”, 1997/99
22 August 2008
NUS CS5229 Semester 1 1
2008/09

2. How often are packets
dropped?"
How often are packets
reordered?"
:
22 August 2008
NUS CS5229 Semester 1 2
2008/09

3. Why these
questions?
22 August 2008
NUS CS5229 Semester 1 3
2008/09

4. 1. Understand the
Internet
22 August 2008
NUS CS5229 Semester 1 4
2008/09

5. “
when you can measure what you are
speaking about, and express it in
numbers, you know something about
it; but when you cannot measure it,
when you cannot express it in
”
numbers, your knowledge is of a
meagre and unsatisfactory kind;
- Lord Kelvin
22 August 2008
NUS CS5229 Semester 1 5
2008/09

6. 2. Model the Internet
22 August 2008
NUS CS5229 Semester 1 6
2008/09

7. 3. Enable more
accurate evaluation
through simulations
22 August 2008
NUS CS5229 Semester 1 7
2008/09

8. 4. Lead to a better
application/systems
design
22 August 2008
NUS CS5229 Semester 1 8
2008/09

9. How often are packets
dropped?"
How often are packets
reordered?"
:
22 August 2008
NUS CS5229 Semester 1 9
2008/09

10. How to answer these
questions?
22 August 2008
NUS CS5229 Semester 1 10
2008/09

11. Collect lots of packet
traces"
Analyze the traces
22 August 2008
NUS CS5229 Semester 1 11
2008/09

12. Trace collection:"
large number of flows"
a variety of sites"
many packets per
flow"
use TCP
22 August 2008
NUS CS5229 Semester 1 12
2008/09

13. Why TCP:"
real-world traffic"
will not overload the
network
22 August 2008
NUS CS5229 Semester 1 13
2008/09

14. Time between measurement is
Poisson distributed."
PASTA Theorem: Intuitively, if we
make n observations and k
observations is in some state S and
n-k in other states, then we can
assume prob of observing S is
approximately k/n.
22 August 2008
NUS CS5229 Semester 1 14
2008/09

15. Two traces:"
N1: Dec94 "
N2: Nov-Dec95"
use tcpdump at sender +
receiver"
22 August 2008
NUS CS5229 Semester 1 15
2008/09

16. 100 kB
Size of file transfered
22 August 2008
NUS CS5229 Semester 1 16
2008/09

17. 21
Number of sites
22 August 2008
NUS CS5229 Semester 1 17
2008/09

18. 20800
Number of trace pairs
22 August 2008
NUS CS5229 Semester 1 18
2008/09

19. Part 1:"
The Unexpected
22 August 2008
NUS CS5229 Semester 1 19
2008/09

20. Packet
Reordering
22 August 2008
NUS CS5229 Semester 1 20
2008/09

21. 1
2
5
2 reorderings
3
4
22 August 2008
NUS CS5229 Semester 1 21
2008/09

22. N1
N2
36%
12%
Percentage of connections with
at least one out-of-order delivery
22 August 2008
NUS CS5229 Semester 1 22
2008/09

23. N1
N2
2%
.3%
Percentage of data packets out-of-order
22 August 2008
NUS CS5229 Semester 1 23
2008/09

24. N1
N2
.6%
.1%
Percentage of ACK packets out-of-order
22 August 2008
NUS CS5229 Semester 1 24
2008/09

25. Data packets are
usually sent
closer together.
22 August 2008
NUS CS5229 Semester 1 25
2008/09

26. From
To
15%
.2%
Percentage of packets out-of-order
to and from U of Colorado in N1.
22 August 2008
NUS CS5229 Semester 1 26
2008/09

27. Route fluttering:
alternate packets
can take different
route to dest.
22 August 2008
NUS CS5229 Semester 1 27
2008/09

28. Taken from Paxson’s PhD Thesis: Alternate routes
are taken for packets from WUSTL to U Mannheim
22 August 2008
NUS CS5229 Semester 1 28
2008/09

29. Fig 1 from the paper, showing large gap and two slopes.
22 August 2008
NUS CS5229 Semester 1 29
2008/09

30. T1
(new arrival)
Ethernet
(buffered packets)
Fig 1 from the paper, showing large gap and two slopes.
22 August 2008
NUS CS5229 Semester 1 30
2008/09

31. Impact of Packet
Reordering
22 August 2008
NUS CS5229 Semester 1 31
2008/09

32. Recap: TCP’s fast
retransmit
22 August 2008
NUS CS5229 Semester 1 32
2008/09

33. S = 1000
S = 2000
S = 3000
S = 4000
A = 2000
S = 5000
A = 2000
A = 2000
A = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 33
2008/09

34. S = 1000
S = 2000
S = 3000
S = 4000
A = 2000
S = 5000
A = 2000
A = 2000
A = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 34
2008/09

35. Nd = 3 is a
conservative
choice.
22 August 2008
NUS CS5229 Semester 1 35
2008/09

36. What if receiver
wait longer before
sending dup ack?
22 August 2008
NUS CS5229 Semester 1 36
2008/09

37. S = 2000
W
S = 2000
22 August 2008
NUS CS5229 Semester 1 37
2008/09

38. 1
2
5
Delivery Gap: "
time between receiving "
3
an out-of-order packet and "
the packet sent before it.
4
22 August 2008
NUS CS5229 Semester 1 38
2008/09

39. Taken from Paxson’s PhD Thesis: CDF for delivery gap between reordered
packets.
22 August 2008
NUS CS5229 Semester 1 39
2008/09

40. N2
N1
higher BW
22 August 2008
NUS CS5229 Semester 1 40
2008/09

41. N1
N2
20ms
8ms
Waiting time with which 70% of "
out-of-order delivery would be identified.
22 August 2008
NUS CS5229 Semester 1 41
2008/09

42. Is needless
retransmission a
problem?
22 August 2008
NUS CS5229 Semester 1 42
2008/09

43. Good
S = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 43
2008/09

44. Bad
S = 2000
S = 2000
22 August 2008
NUS CS5229 Semester 1 44
2008/09

45. N1
N2
22
300
Number of good retransmissions
for every bad retransmission.
Nd = 3, W = 0
22 August 2008
NUS CS5229 Semester 1 45
2008/09

46. N1
N2
~ 7
100
Number of good retransmissions
for every bad retransmission.
Nd = 2, W = 0
22 August 2008
NUS CS5229 Semester 1 46
2008/09

47. N1
N2
15
300
Number of good retransmissions
for every bad retransmission.
Nd = 2, W = 20ms
22 August 2008
NUS CS5229 Semester 1 47
2008/09

48. Packet
Corruption
22 August 2008
NUS CS5229 Semester 1 48
2008/09

49. 1 in 5000
packet is corrupted
22 August 2008
NUS CS5229 Semester 1 49
2008/09

50. 1 in 65536
corrupted packet goes undetected
using TCP checksum
(assuming each possible checksum is equally likely)
22 August 2008
NUS CS5229 Semester 1 50
2008/09

51. 1 in 300million
Internet packet is corrupted "
and is undetected.
22 August 2008
NUS CS5229 Semester 1 51
2008/09

52. Part 2:"
Bottleneck
Bandwidth
22 August 2008
NUS CS5229 Semester 1 52
2008/09

53. Packet Pair
22 August 2008
NUS CS5229 Semester 1 53
2008/09

54. B bps
b bytes
22 August 2008
NUS CS5229 Semester 1 54
2008/09

55. Q x B = b
Q s
22 August 2008
NUS CS5229 Semester 1 55
2008/09

56. Q s
22 August 2008
NUS CS5229 Semester 1 56
2008/09

57. Q s
22 August 2008
NUS CS5229 Semester 1 57
2008/09

58. Q s
22 August 2008
NUS CS5229 Semester 1 58
2008/09

59. Problems with
Packet Pair
22 August 2008
NUS CS5229 Semester 1 59
2008/09

60. 1. Asymmetric
Link
22 August 2008
NUS CS5229 Semester 1 60
2008/09

61. 22 August 2008
NUS CS5229 Semester 1 61
2008/09

62. 2. ACK
Compression
22 August 2008
NUS CS5229 Semester 1 62
2008/09

63. 22 August 2008
NUS CS5229 Semester 1 63
2008/09

64. 3. Out of order
delivery
22 August 2008
NUS CS5229 Semester 1 64
2008/09

65. 22 August 2008
NUS CS5229 Semester 1 65
2008/09

66. 4. Clock
resolution
22 August 2008
NUS CS5229 Semester 1 66
2008/09

67. Suppose
B = 1000 kBps
b = 1 kB
Q = ?
22 August 2008
NUS CS5229 Semester 1 67
2008/09

68. 5. Changing
bottleneck
bandwidth
22 August 2008
NUS CS5229 Semester 1 68
2008/09

69. Fig 2 from the paper, showing changing
bandwidth.
22 August 2008
NUS CS5229 Semester 1 69
2008/09

70. Fig 3 from the paper, showing multi-channel
links.
22 August 2008
NUS CS5229 Semester 1 70
2008/09

71. 6. Multi-channel
Links
22 August 2008
NUS CS5229 Semester 1 71
2008/09

72. Asymmetric links"
ACK compression"
Out-of-order delivery"
Clock resolution"
Changes in bottleneck
bandwidth"
Multi-channel links"
22 August 2008
NUS CS5229 Semester 1 72
2008/09

73. Measure at receiver:"
Asymmetric links"
ACK compression"
Packet bunch:"
Out-of-order delivery"
Clock resolution"
Changes in bottleneck
bandwidth"
Multi-channel links
22 August 2008
NUS CS5229 Semester 1 73
2008/09

74. 2Q
22 August 2008
NUS CS5229 Semester 1 74
2008/09

75. Collect multiple estimates,
take the most freq
occurrence (modes) as the
bottleneck bandwidth.
22 August 2008
NUS CS5229 Semester 1 75
2008/09

76. 22 August 2008
NUS CS5229 Semester 1 76
2008/09

77. Part 3:"
Packet Loss
22 August 2008
NUS CS5229 Semester 1 77
2008/09

78. N1
N2
2.7%
5.2%
Percentage of packets that were lost.
22 August 2008
NUS CS5229 Semester 1 78
2008/09

79. N1
N2
50%
50%
Percentage of loss free connections
22 August 2008
NUS CS5229 Semester 1 79
2008/09

80. N1
N2
5.7%
9.2%
Loss rate on lossy connections
22 August 2008
NUS CS5229 Semester 1 80
2008/09

81. 17%
Loss rate on connections from EU to US
22 August 2008
NUS CS5229 Semester 1 81
2008/09

82. Are packet losses
independent?
22 August 2008
NUS CS5229 Semester 1 82
2008/09

83. Compute:"
Pu = Pr [ p lost ]"
Pc = Pr [ p lost | prev pkt lost ]
22 August 2008
NUS CS5229 Semester 1 83
2008/09

84. Pu
Pc
2.8%
49%
Loss rate for “queued data pkt” on N1
22 August 2008
NUS CS5229 Semester 1 84
2008/09

85. Fig 6 from the paper, showing outage
duration.
22 August 2008
NUS CS5229 Semester 1 85
2008/09

86. Are
retransmission
redundant?
22 August 2008
NUS CS5229 Semester 1 86
2008/09

87. Unavoidable
X
ACK
22 August 2008
NUS CS5229 Semester 1 87
2008/09

88. Coarse Feedback
22 August 2008
NUS CS5229 Semester 1 88
2008/09

89. Bad RTO
22 August 2008
NUS CS5229 Semester 1 89
2008/09

90. N1
N2
26%
28%
Percentage of retransmissions that"
are redundant
22 August 2008
NUS CS5229 Semester 1 90
2008/09

91. 4
44 Unavoidable
Coarse Feedback
Bad RTO
51
Type of redundant retransmission in N1.
22 August 2008
NUS CS5229 Semester 1 91
2008/09

92. Part 4:"
Packet Delay
22 August 2008
NUS CS5229 Semester 1 92
2008/09

93. OTT is not well
approximated
using RTT/2
22 August 2008
NUS CS5229 Semester 1 93
2008/09

94. ACK
Compression
22 August 2008
NUS CS5229 Semester 1 94
2008/09

95. (might affect TCP self-clocking)
22 August 2008
NUS CS5229 Semester 1 95
2008/09

96. sent
recv
22 August 2008
NUS CS5229 Semester 1 96
2008/09

97. Sending interval
Receiving interval
22 August 2008
NUS CS5229 Semester 1 97
2008/09

98. 22 August 2008
NUS CS5229 Semester 1 98
2008/09

99. 22 August 2008
NUS CS5229 Semester 1 99
2008/09

100. Compression
event if ξ < .75
22 August 2008
NUS CS5229 Semester 1 100
2008/09

101. N1
N2
50%
60%
Percentage of connection that experiences"
at least one compression event.
22 August 2008
NUS CS5229 Semester 1 101
2008/09

102. N1
N2
50%
60%
Percentage of connection that experiences"
at least one compression event.
22 August 2008
NUS CS5229 Semester 1 102
2008/09

103. 2
Average number of events per connection.
22 August 2008
NUS CS5229 Semester 1 103
2008/09

104. Estimating
Available
Bandwidth
22 August 2008
NUS CS5229 Semester 1 104
2008/09

105. Qb: time to transit the bottleneck"
ψi: expected time spent queuing behind
predecessor (derived from sending time)"
γi: diff between packet OTT and min OTT
22 August 2008
NUS CS5229 Semester 1 105
2008/09

106. time packet i is sent
22 August 2008
NUS CS5229 Semester 1 106
2008/09

107. 22 August 2008
NUS CS5229 Semester 1 107
2008/09

108. β = 1 means all bandwidth
is available."
β = 0 means none of the
bandwidth is available.
22 August 2008
NUS CS5229 Semester 1 108
2008/09

109. Fig 10 from the paper, showing distribution
of available bandwidth.
22 August 2008
NUS CS5229 Semester 1 109
2008/09

110. Conclusion
22 August 2008
NUS CS5229 Semester 1 110
2008/09

111. The numbers in the
paper are not important. "
(the Internet has changed)
22 August 2008
NUS CS5229 Semester 1 111
2008/09

112. Measurement is
difficult but useful
22 August 2008
NUS CS5229 Semester 1 112
2008/09

113. Many new techniques
needed (e.g to
measure bottleneck
bandwidth)
22 August 2008
NUS CS5229 Semester 1 113
2008/09

114. We can improve
current design (e.g.
TCP if we know more
about reordering)
22 August 2008
NUS CS5229 Semester 1 114
2008/09

115. We can identify
problem (e.g. packet
corruption)
22 August 2008
NUS CS5229 Semester 1 115
2008/09

116. We can better model
the behavior (e.g.
bursty packet loss)
22 August 2008
NUS CS5229 Semester 1 116
2008/09

117. We can infer much
info from just a
packet trace (e.g.
available bandwidth)
22 August 2008
NUS CS5229 Semester 1 117
2008/09