On the Distribution of Traffic Volumes in the Internet and its Implications infocom 2019

1. Internet Traffic Analysis:
Coseners, 2019
Mohammed Alasmar
https://ieeexplore.ieee.org/document/8737483
‘19

2. 2
 Reliable traffic modelling is important for network
planning, deployment and management; e.g.
(1) network dimensioning,
(2) traffic billing.
 Historically, network traffic has been widely assumed to
follow a Gaussian distribution.
 Deciding whether Internet flows could be heavy-tailed
became important as this implies significant departures
from Gaussianity.
Motivations

3. Traffic volumes at different T
 𝑋𝑖 : the amount of traffic seen in the time period [𝑖𝑇, (𝑖 + 1)𝑇)
 Aggregation at different sampling times (T)
Internet trace.pacp

4. 4
Traffic volumes at different T
0 300 600 900
Time (seconds)
0
10
20
30
40
50
Datarate(Mbps)
0 300 600 900
Time (seconds)
0
10
20
30
40
50
Datarate(Mbps)
T = 5 sec
T = 1 sec
T = 10 msec

5. Goal
5
0 10 20 30 40
Data rate (Mbps)
0
0.05
0.1
PDF
T = 10 ms
T = 1 sec
T = 5 sec
Goal
 Investigating the distribution of the amount of traffic
per unit time using a robust statistical approach.

6. 6
 Investigating the distribution of the
amount of traffic per unit time using
a robust statistical approach.
Goal

7. 7
Goal
T = 10 ms
T = 1 sec
T = 5 sec

8. 8
Dataset #Traces
Twente1 40
MAWI2 107
Auckland3 25
Waikato4 30
Caida5 27
[1] https://www.simpleweb.org/wiki/index.php/Traces , 2009.
[2] http://mawi.wide.ad.jp/mawi/ , 2016-2018.
[3] https://wand.net.nz/wits/auck/9/ , 2009.
[4] https://wand.net.nz/wits/waikato/8/ , 2010-2011.
[5] http://www.caida.org/data/overview/ , 2016.
 We study a large number of traffic traces (230) from many
different networks: 2009  2018
Datasets

9. 9
 Our analysis is based on the framework proposed in:
 The framework combines maximum-likelihood fitting
methods with goodness-of-fit tests based on the
Kolmogorov–Smirnov statistic and likelihood ratios.
Power-law test

10. 10
Power-law test
𝑝 𝑥 =
𝛼 − 1
xmin
𝑥
xmin
−𝛼
0 20 40 60 80 100 120
data rate (bps) 106
0
500
1000
1500
2000
2500
3000PDF
xmin
Power-law distribution:
𝑝 𝑥 = 𝑥 −𝛼 introduce
new variables
𝛼: scaling exponent

11. 11
Power-law test

12. 12
𝑹, 𝑝 = 𝑓𝑖𝑡. 𝒅𝒊𝒔𝒕𝒓𝒊𝒃𝒖𝒕𝒊𝒐𝒏𝑪𝒐𝒎𝒑𝒂𝒓𝒆(𝑝𝑜𝑤𝑒𝑟𝑙𝑎𝑤, 𝑎𝑙𝑡𝑒𝑟𝑛𝑎𝑡𝑖𝑣𝑒)
Likelihood Ratio: 𝑹
• If 𝑹 > 0, then the power-law is favoured.
• If 𝑹 < 0, then the alternative is favoured.
• If 𝑝 < 0.1 , then the value of 𝑹 can be trusted.
• Weibull
• Lognormal
• Exponential
Likelihood ratio:
𝑹 =
𝐿1
𝐿2
= 𝑖=1
𝑛
𝑝1 𝑥
𝑖=1
𝑛
𝑝2(𝑥)
power-law likelihood function
alternative likelihood function
 𝑳og−Likelihood ratio: 𝑹

13. 13
Normalised Log-Likelihood Ratio (LLR)
T=100 msec
5 10 15 20 25
Rank of trace
-50
-40
-30
-20
-10
0
10
NormalisedLLR
5 10 15 20 25
Rank of trace
-50
-40
-30
-20
-10
0
10
NormalisedLLR
5 10 15 20 25
Rank of trace
-50
-40
-30
-20
-10
0
10
NormalisedLLR
5 10 15 20 25
Rank of trace
0
0
0
0
Weibull
Lognormal
ExponentialWeibull
Lognormal
Exponential
Circled points p > 0.1
The log-
normal is the
best fit for the
vast majority
of traces.
(𝑹)

14. 10 20 30 40
Rank of trace
-60
-40
-20
0
20
NormalisedLLR
Weibull
Lognormal
Exponential
14
5 10 15 20 25 30
Rank of trace
-30
-20
-10
0
10
20
30
NormalisedLLR
Weibull
Lognormal
Exponential
Waikato traces
60 70 80 90 100
Rank of trace
-5
0
5
10
NormalisedLLR
Weibull
Lognormal
Exponential
MAWI traces
Twente traces
5 10 15 20 25
Rank of trace
-50
-40
-30
-20
-10
0
10
NormalisedLLR
Weibull
Lognormal
Exponential
Auckland traces
The log-normal distribution is not
the best fit for …
Anomalous traces
The log-normal
distribution is the
best fit for the vast
majority of traces.
o 1 out of 25 Auckland traces
o 9 out of 107 MAWI traces
o 1 out of 27 CAIDA traces
o 2 out of 30 Waikato traces
o 5 out of 40 Twente traces

15. 15
Anomalous traces
 Anomalous traces are a poor fit for all distributions
tried.
 This is often due to traffic outages or links that hit
maximum capacity.
0 500 1000
Data rate (Mbps)
0
0.005
0.01
0.015
0.02
PDF
0 500 1000
Data rate (Mbps)
0
0.01
0.02
0.03
PDF
Log-normal
trace
Anomalous
trace

16. 16
Normalised Log-Likelihood Ratio (LLR) test results for all studied
traces and log-normal distribution at different timescales
5 10 15 20 25
Rank of trace
-20
-15
-10
-5
0
NormalisedLLR T = 5 sec
T = 1 sec
T = 100 msec
T = 5 msecCAIDA traces
𝑹 < 0, i.e.,
log-normal
is favoured.
𝑹
At different sampling times: T

17. 17
The correlation coefficient test
5 10 15 20 25
Rank of Traces
0.7
0.8
0.9
T=5sec
T=1sec
T=100msec
T=5msec
5 10 15 20 25
Rank of Traces
0.9
0.95
1
T=5sec
T=1sec
T=100msec
T=5msecCAIDA traces CAIDA traces
GaussianLog-normal
 Strong goodness-of-fit (GOF) is assumed to exist when the
value of 𝛾 is greater than 0.95.

18. 18
 Bandwidth provisioning approach
provides the link by the essential
bandwidth that guarantees the
required performance.
 Overprovisioning. In the conventional
methods the bandwidth is allocated by
up-grading the link bandwidth to 30%
of the average traffic value.
Use case 1: Bandwidth provisioning

19. 19
𝑃
𝐴 𝑇
𝑇
≥ 𝐶 ≤ 𝜀
 The following inequality (the ‘link transparency
formula’) has been used for bandwidth provisioning:
i.e., the probability that the captured traffic A T over a
specific aggregation timescale T is larger than the link
capacity C has to be smaller than the value of a
performance criterion ε.
 𝛆 has to be chosen carefully by the network
provider in order to meet the specified SLA.
Use case 1: Bandwidth provisioning

20. MAWI traces
Expected link capacity
Gaussian
Weibull
Log-normal
𝑬𝒙𝒂𝒎𝒑𝒍𝒆: 𝛆 = 𝟎. 𝟎𝟏Use case 1: Bandwidth provisioning
Performance criterion ε
𝑷
𝑨 𝑻
𝑻
≥ 𝑪 ≤ 𝜺

21. 21
M T C W A
0
0.2
0.4
0.6 T=0.1s T=0.5s T= 1s
Target: ε = 0.5
Log-normal
M T C W A
0
0.2
0.4
0.6 T=0.1s T=0.5s T= 1s
M T C W A
0
0.2
0.4
0.6
T=0.1 s T=0.5 s T= 1 s
Target: ε = 0.5
Weibull Gaussian
Target: ε = 0.5
M: MAWI, T: Twente, C: CAIDA, W: Waikato, A: Auckland
Bandwidth provisioning: Results
0.5

22. 22
Use case 2: 95th percentile pricing
0 100 200 300
Time (sec)
0
200
400
600
800
Datarate(Mbps)
0 50 100
Time (sec)
0
200
400
600
800
Datarate(Mbps)
 Customers are not billed for brief spikes in
network traffic.
[5 minutes]
Percentile

23. 23
• Log-normal model provides much more accurate predictions of the 95th percentile.
95th percentile pricing: Results
The red reference
line to show where
perfect predictions
would be located.
0 500 1000 1500
Actual value (Mbps)
0
500
1000
1500
Predictedvalue(Mbps)
0 500 1000 1500
Actual value (Mbps)
0
500
1000
1500
Predictedvalue(Mbps)
0 500 1000 1500
Actual value (Mbps)
0
500
1000
1500
Predictedvalue(Mbps)
0 500 1000 1500
Actual value (Mbps)
0
500
1000
1500
Predictedvalue(Mbps)
MAWI traces

24. 24
Thanks! Questions?
More details ….

25. 25
 The distribution of traffic on Internet links is an important problem that has received relatively little
attention.
 We use a well-known, state-of-the-art statistical framework to investigate the problem using a large corpus
of traces.
 We investigated the distribution of the amount of traffic observed on a link in a given (small) aggregation
period which we varied from 5 msec to 5 sec.
 The vast majority of traces fitted the lognormal assumption best and this remained true all timescales tried.
 We investigate the impact of the distribution on two sample traffic engineering problems.
1. Firstly, we looked at predicting the proportion of time a link will exceed a given capacity.
2. Secondly, we looked at predicting the 95th percentile transit bill that ISP might be given.
 For both of these problems the log-normal distribution gave a more accurate result than heavy-tailed
distribution or a Gaussian distribution.
Summary

26. 26
Backup ……

27. Estimating: (𝛼 , xmin , ntail )
using MLE & KS test
Uncertainty in the fitted
parameters (Bootstrapping)
Goodness-of-fit
p-value
ℛℛ < 0
Alternative is
favoured
p > 0.1 p < 0.1
Ho: Power-law is favoured
fail to
reject Ho
reject Ho
ℛ > 0
Noneis
favoured
p-value
for ℛ
Power-law is
favoured
Noneis
favoured
p > 0.1 p < 0.1 p > 0.1 p < 0.1
ℛℛ > 0 ℛ < 0
Noneis
favoured
p-value
for ℛ
Alternative is
favoured
Noneis
favoured
p > 0.1 p < 0.1
1
2
3
4
5
p-value
for ℛ
Power-law Test
Power-law distribution:
𝑝 𝑥 =
𝛼−1
xmin
𝑥
xmin
−𝛼
Power-law test
Log-Likelihood ratio (ℛ)
[Ref] A. Clauset, C. S. Rohilla, and M. Newman, “Power-law
distributions in empirical data,” arXiv:0706.1062v2, 2009.

28. 28
50 100 150
Actual value (Mbps)
0
50
100
150
200
250
Predictedvalue(Mbps)
Log-normal
Weibull
Gaussian
Auckland traces
2 3 4 5
Actual value (Gbps)
2
3
4
5
6
Predictedvalue(Gbps)
Log-normal
Weibull
Gaussian
0 10 20 30
Actual value (Mbps)
0
10
20
30
40
50
Predictedvalue(Mbps)
Log-normal
Weibull
Gaussian
Twente traces
0 50 100
Actual value (Mbps)
0
50
100
Predictedvalue(Mbps)
Log-normal
Weibull
Gaussian
Waikato traces
CAIDA traces

29. 29
Log-normalWeibullMeent