The last ten to fifteen years have seen a pervasive growth of the Internet both in terms of its depth of penetration into user population as well the breadth of areas into which Internet is now present. As Internet access becomes faster and applications move to the cloud the profile of Internet traffic continues to change. Peer to Peer traffic, video sharing and OTT (over the top) services coupled with almost ubiquitous access to high speed internet poses new challenges to service providers (how to better utilize bandwidth) as well OEMs (how to increase bits per second and packets per second through the equipment). A key to understanding and solving these challenges is to understand what constitutes Internet traffic and how the internet traffic will look like in the coming years and then based on that optimize networks and infrastructure to better utilize available resources. This is what this project aims to address i.e. understanding internet traffic from various perspectives (application, protocol, packet size and others) such that this understanding can then feed into network and infrastructure design. A data product named SITAPT (Study of Internet Traffic to Analyze and Predict Traffic) is built which addresses the aims of this project.

1. SITAPT
STUDY OF INTERNET TRAFFIC TO ANALYZE
AND PREDICT TRAFFIC
Amit Arora
aa1603@georgetown.edu
https://www.linkedin.com/in/amit-arora-539120a

2. WHAT DOES INTERNET TRAFFIC LOOK
LIKE?
Word cloud of mean percentage packets contributed by various
applications from 2008 to 2015

3. BACKGROUND
 Pervasive growth of the Internet.
 Internet access becomes faster
and applications move to the
cloud the profile of Internet
traffic continues to change.
 Peer to Peer traffic, video
sharing and OTT services
coupled with almost ubiquitous
access to high speed internet
poses new challenges
 Service providers: how to
better utilize bandwidth ?
 OEMs: how to increase bits
per second and packets per
second through the
equipment ?
 A key to understanding and solving
these challenges is to
 understand what constitutes Internet
traffic
 how the internet traffic will look like
in the coming years
 optimize networks and infrastructure
to better utilize available resources.
 This is what this project aims to
address i.e. understanding internet
traffic from various perspectives
(application, protocol, packet size
and others)
 This understanding can then feed
into network and infrastructure
design.
A data product named SITAPT (Study of Internet Traffic to Analyze and
Predict Traffic) is built which addresses the above requirements.

4. TARGET AUDIENCE
Network OEM Service Provider

5. SCOPE OF SITAPT
Visualization of Traffic
Data
• Hundreds of
applications (web,
secure web, file
transfer etc)
• Tens of protocols (TCP,
UDP, ESP, GRE etc).
• Various packet sizes
Time Series Analysis for
Traffic Prediction
• Multivariate timeseries
analysis to predict
traffic for various
applications and
protocols in the next 12
months.
• Identify trends in key
and upcoming
applications/protocols
Clustering to Explore
Similarity
• Use machine learning
algorithms to identify
possible clusters of
similarity between
traffic patterns across
multiple years.
Relationship betwen
traffic types
• Identify and model
relationship between
key protcols

6. DATA SCIENCE PIPELINE
SITAPT itself is implemented completely in Python (version 2.7.11), although it relies heavily on other
python packages such as numpy etc. that might be written in other programming languages for speed.
The entire code for SITAPT is available on Github SITAPT repo.
Obtain anonymized internet traces from CAIDA.
The traces are available in pcap format and
contain IP and Transport layer headers only.
Read the packet trace and convert
information from each packet into a JSON
which is then stored in MongoDB.
Remove all data that is not IP.
Find out the traffic mix by
protocols, application, packet
sizes and other criteria. Convert
the results in a pandas
dataframe which is again stored
into Mongo.
· Model traffic as a multivariate time series. Use time
series analysis techniques to forecast traffic for various
applications and protocols.
· Use unsupervised machine learning (clustering) to
identify similarity in traffic across the dataset.
· Identify and model relationship between key protocols
Create visualizations for
understanding the data.
Representing time series and
describing trends.
Explore clustering and regression

7. ARCHITECTURE
Application
Layer
Application
Layer
AnonymizedInternettraces
Internet trace from CAIDA
Internet trace from a
Service Provider
Ingestion module
(BeautifulSoup4, wget,
gzip, pycapfile)
Ingestion
WORM
ETL
Offline (non-realtime),
Authenticated ingestion
NOSQL
- MongoDB
Immutable Data store
ETL = Extract only IP packets, Transform
to JSON, Load in data store
Computation & Modeling
Models
- AR(I)MA
- PCA, KMeans
- Linear Regression.
Packages
- Statsmodels
- SciKit Learn
Train model on available data sets
Wallofinterpretation
Data from models
made available in
various formats

8. A WORD ABOUT DATA USED IN SITAPT
CAIDA (Center for Applied Internet Data Analysis, http://www.caida.org/) maintains a lot of data
that can be used as analyzing and understanding Internet traffic.
Anonymized internet traces from the year 2008 to 2015 are available upon request from CAIDA
(see http://www.caida.org/data/passive/passive_2015_dataset.xml ), these traces form the
dataset used by SITAPT.
All traces in this dataset are anonymized (by CAIDA itself) with the same key. In addition, the
payload has been removed from all packets (again by CAIDA itself).

9. A WORD ABOUT DATA USED IN SITAPT
(CONTD..)
How much data does SITAPT use?
CAIDA provides around 13,000 files (in compressed format) for the period from 2008 to 2015.
The combined size of the uncompressed version of the files stored into a database would run
into several tens of terabytes.
The current version of SITAPT is not a *big data* product.
Clearly, analyzing this amount of data requires horizontal scaling which is outside the scope of
the current project.
To reduce the problem to a more manageable level, SITAPT works with one file for every month
of every year from 2008 to 2015 (for 2014 and 2015 CAIDA provides one file per quarter).
In total, SITAPT analyses 73 packet trace files from 2008 to 2015. Each trace contains millions
of packets.
The size of the database that stores the JSON representation of the trace files is more than
1TB.

10. DATA TRANSFORMATION FOR
COMPUTATION
 Ingested data is stored in Mongo collections (one for each year).
 This data needs to be transformed into matrix form to make it amenable for
computation.
 Once created, the three collections (applications, protocols and packet size
distribution) are also stored as CSV files such that the modeling phase do
not have to interact with the database at all and can read data on which
they need to work on directly from the CSV file.

11. DATA TRANSFORMATION FOR
COMPUTATION (CONTD..)
These files represent time series data such that each row is a parameterized
representation of traffic expressed either as combination of applications or
protocols or packet size distribution.
Date sun-sr-
iiop
transmit-
port
ieee-
mms-
ssl
passgo joaJewelSuite ovhpas sdo-tls interwise lm-
instmgr
3/19/200
8
4.36E-06 3.49E-05 0 8.72E-
06
0 0.00079
8
0.00095
9
0.00105
1
0.00060
6
4/30/200
8
9.00E-06 9.00E-06 9.00E
-06
9.00E-
06
9.00E-06 9.00E-06 9.00E-06 9.00E-06 9.00E-06
5/15/200
8
0 0 0 0 0 3.96E-05 0.00118
7
0.00441
3
0
6/19/200
8
0 0 0 0 0 0.00036
8
0.00040
3
0.00026
6
0
7/17/200
8
0.00074
4
6.20E-05 0 0 0 0.00329
9
0.00282
8
1.24E-05 2.48E-05
Time
Applications
Read row wise: composition of Internet traffic by packets percentage
contributed by each application in a trace captured at a particular day.
Read column wise: packets percentage of each application as a time
series.

12. VISUALIZATIONS
Three different types of visualizations are explored
 Word cloud (for applications and protocols)
 Stacked chart (for applications, protocols and packet size
distribution)
 Heat map (for packet size distribution)
 Parallel Coordinates (protocols)

13. VISUALIZATIONS: STACKED CHARTS
Observations:
 Almost exponential increase in HTTPS traffic
 Decrease in unclassified (unknown) traffic
 Logarithmic decay in applications that contribute less than 0.5% of packets individually.

14. VISUALIZATIONS: HEAT MAP
Observations:
Packet size distribution is almost entirely bimodal, with only the 1 to 100 bytes range
and the 1400 to 1500 bytes range showing packet percentages of any significance.
Only two rows show any dark colors (which represents a significant packet percentage)
and these are the 1 to 100 packet size row and the 1400 to 1500 packet size row..

15. VISUALIZATIONS: PARALLEL
COORDINATES
The protocols parallel coordinates for protocols contributing more than 0.01% to overall traffic.
This chart clearly shows a negative correlation between TCP and UDP protocol traffic.

16.  Each time series is studied and analyzed individually. The
following operations are done on each time series.
 Plot the first difference series to identify trends.
 Evaluate ACF and PCF to identify dependencies of the series
upon previous time samples.
 Seasonal decomposition to identify trend and seasonality and
residuals.
 ARMA and ARIMA modeling of the time series
 Modeling is done using “statsmodels” package
 The output of each of the above steps is available as part of
SITAPT analysis.
TIME SERIES ANALYSIS

17. TIME SERIES MODELING FOR TCP
PROTOCOL

18. TIME SERIES MODELING FOR TCP
PROTOCOL (CONTD..)

19. TIME SERIES MODELING FOR TCP
PROTOCOL (CONTD..)

20. TIME SERIES MODELING FOR MULTIPLE
PROTOCOLS AND APPLICATIONS
Forecasted traffic mix

21. TRENDS IN SOME IMPORTANT
APPLICATIONS
Almost exponential increase in HTTPS traffic
HTTP traffic is decreasing but still contributes a significant percentage

22. CLUSTERING
 To explore if there are any patterns hidden in the internet traffic data
a clustering technique is employed.
 Each protocol or application or packet size interval is treated as a
feature and each trace is treated as an instance.
 Clustering is done in two steps:
 Dimensionality reduction via PCA (Principal Component Analysis)
 For applications, PCA reduces 5000+ dimensions to 10.
 Clustering via KMeans
 K = 4
 PCA and KMeans are both done using the scikit-learn API.

23. CLUSTERING (CONTD..)
Some clustering present in applications and
protocols data, not so much in packet size
distribution data (needs higher K maybe).

24. CLUSTERING
Date Year Half Quarter Fort
night
DayOf
TheWeek
cluster TCP ESP UDP
5/17/2012 2012 1 2 2 Thursday 2 91.01197292 0.190882127 8.219679951
7/19/2012 2012 2 3 2 Thursday 2 91.75760765 0.084797929 7.588290657
9/20/2012 2012 2 3 2 Thursday 2 93.10399212 0.024899575 6.125210312
10/18/2012 2012 2 4 2 Thursday 2 90.43387514 0.492682745 8.564568542
11/15/2012 2012 2 4 1 Thursday 2 95.2341703 0.262422627 3.843731031
12/20/2012 2012 2 4 2 Thursday 2 91.12739062 0.035976644 8.60410183
3/21/2013 2013 1 1 2 Thursday 2 94.65616355 0.019147528 5.206672769
6/20/2013 2013 1 2 2 Thursday 2 90.5349465 0.061648601 9.195006053
9/18/2014 2014 2 3 2 Thursday 2 90.60646827 0.020378309 8.885648934
 It is a matter of further analysis to figure out what event or phenomenon was happening which caused the
Internet traffic at during different times between 2008 to 2015 to be similar.
 If this study was being done on traffic from a closed network (such as from a single ISP) then it would be
much easier to attribute this clustering to real world events (such as the OS update for mobile phones for
example).
 The following is an excerpt from the generated CSV file for protocols showing the additional fields added,
including the label field provided by the clustering algorithm.
 The table is filtered on cluster (label) type 2 and it is seen that traces which has higher than usual TCP
traffic % (90 to 95%) are clustered together.

25. LINEAR REGRESSION
Parallel coordinates showed negative correlation between the
percentage of TCP traffic and the percentage of UDP traffic.
Creating a scatter plot of TCP Vs UDP and then creating a linear
regression model to fit a straight line through it.
The coefficients vector is [ -1.00805723] and the variance score
is 0.96.

26. WHAT WORKED?
 The fact that all the packet traces are now available in a
document database means that the data is now available in
a consumable format and this really opens up avenues for
further analysis, asking different types of questions off the
data.
 The time series analysis revealed interesting trends about the
data, such as an almost exponential increase in secure HTTP
traffic which was expected but at the same time there is not
a huge decrease in non-secure HTTP traffic which was
somewhat unexpected.
 Various types of visualization techniques (like parallel
coordinates) and tools like Bokeh provide a really good
insight into the data.

27. WHAT DID NOT WORK?
 With the amount of data involved, this is clearly a Big Data project,
since that was not something that could be done completed in a
short time so the alternative was use to trace file for each month and
that reduced the number of data points available for analysis (only 73
data points). This limited the prediction ability of the time series
models, not all applications and protocols could be modeled within
the 95% confidence interval and a MAPE of < 5%.
 This data would provide much more insights if it corresponded to
traffic from a closed network rather than the Internet. For example,
such as an ISP’s network limited to certain geographical areas because
then the data would have less variability and would be easier to
explain the clustering.
 For the time series model, only the MAPE was considered while
choosing between the AR(I)MA models. There are other criteria as well
such as Durbin-Watson statistic, the BIC and HQIC etc. which should
have been explored but were not.

28. CONCLUSION
SITAPT provides valuable insights into network traffic
composition and trends.
 In terms of applications there is an exponential growth trend
in HTTPS traffic, a trend that is visible even at a macro level
(generic internet packet trace).
 The time series analysis is able to provide predictions for
applications and protocols.
 In terms of packet sizes there is a bi-modal distribution.
 Clustering reveals patterns in terms of both application and
protocols