This document proposes a novel system for monitoring and analyzing large-scale network traffic data using Hadoop. It describes a three-step algorithm for identifying communities in network traffic graphs and a Jaccard-based learning method for building a cellular device model database. Experimental results revealed network traffic and user behavior phenomena not previously shown.

1. Jun Liu ; Feng Liu and Ansari, N.
Beijing Univ. of Posts & Telecommun., Beijing, China
IEEE Network • July/August 2014
Advisor : Dr. Jenq-Shiou Leu
Student : Chia-Yun Chan
Date : 2014/12/09

2. Introduction
System Architecture
Traffic Analysis Algorithms
Experimental Results
Conclusions

3. Network traffic monitoring and analysis is significance
for optimizing network resource and improving user
experience
Existing solutions usually rely on a high-performance
server with large storage capacity, are not scalable for
detailed analysis of big traffic data

4. The features of Hadoop
Distributed parallel computing
Low-cost scale-out capability
High fault tolerance
But some important issues in large-scale commercial
telecommunication networks have not been solved

6. Application-layer analysis
Web service provider analysis
User behavior analysis

9. d
30 30
80 54
a
e
90
b
c
120
60
80
100
200
64
20 10

10. We develop a three-step algorithm:
1. Measuring affinity
2. Sparsifying a graph
3. Identifying communities

11. Mobile operators want to know the user behaviors of
cellular devices including models, prices, and features
We design a novel Jaccard-based learning method to
build a cellular device model database
1. Extract all keywords of a device model
2. Filter candidate keywords
3. Calculate the Jaccard coefficient index using statistical
information, and select the keyword with the highest
Jaccard index to represent the device model

17. A novel system for monitoring and analyzing large-scale
network traffic data
Designed algorithms and implemented MapReduce
programs for network traffic analysis from different
perspectives
Revealed a number of network traffic and user
behavior phenomena not shown before