This document summarizes a research project that develops a tool to cluster and visualize Internet outage events from log data using time-series analysis. The tool uses a MapReduce algorithm that runs in O(n log n) time to cluster outage blocks based on their start and end times. The data is partitioned into bins and processed in parallel on Hadoop. An evaluation shows that smaller bin sizes help reduce the maximum number of blocks in each bin, improving the tool's ability to handle bursts of outage events.

1. Fast clustering algorithm to visualize outage events
Nguyen Tran
University of Southern California
1. Introduction
As the Internet becomes more common in
everyday life among people around the world,
an Internet outage nowadays could cause big
troubles toward finance, education, government,
and corporations. In many cases, scientists and
researchers want to review and study past outage
events to understand the root cause and make
changes to network and infrastructure for better
stability. There is a critical need for a tool to
help visualize outage events based on raw logs.
We develop a tool to help cluster outages event
together from log of timeseries of different
vantage points. Our clustering algorithm runs in
Big-O (n log n) time and the big data set is
processed in parallel using Map/Reduce and
Hadoop. We then graph the outage events to
better visualize what blocks of IP addresses are
out and at what time. Thereby, we can recognize
what happened to the world at that point.
2. Why does the research matter?
This research project is interesting because it
helps turn a large chunk of network timeseries
data collected through different vantage points
into meaningful events that happened in the past.
Thereby, it greatly helps the investigation and
the studying of some important events such as
an Internet outage or a hacking attempt. The tool
could be a complement of an intrusion detection
system and helps network administrators easily
recognize and visualize suspecting events in
their network.
I’m very interested in learning new technologies
that the industry is using and love to write codes
to polish my programming skills. After this
directed research, I gained significant
knowledge in Hadoop infrastructure and how to
utilize it for parallel computation. I have become
more proficient in writing MapReduce
application using Python scripting language. I
also learned a lot about self-independent and
self-motivated while coping with new and
difficult problems
3. Related Work
This research is a continuation of previous work
where a Big-O (n2
) algorithm was devised to
cluster similar blocks of an outage events. The
similarity was based on the blocks that was
down at the same time which indicated the same
outage. Full previous research content could be
found in [1].
4. Design and Implementation
4.1 Design
The data we are working with is huge in size
since it contains a lot of information such as
block (in hex), start, duration, and status of the
block at probing time. Each block is probed
every 11 minutes and if its status has changed
(from up to down, or vice versa), a new block
entry will be added to the data set. We have
roughly hundred thousands of block /24
addresses. Therefore, it will be very slow to
process such data input in sequential order. We
need to find a way to process our input in
parallel in order to increase speed and
performance.
We decide to process the input data set in
parallel using Hadoop infrastructure to reduce
the number of inputs for our clustering
algorithm. We accomplish the reduction task by
partitioning the data set into bins. Outage blocks
which start time is approximately within bin
start time will be in 1 bin and we will process
each bin in parallel to scale up to about a factor
of 100. We perform experiment 5.1 to find
optimal bin size in order to avoid cases where
number of blocks in a bin explodes.
Our clustering algorithm in each bin has a run
time of Big-O (n log n). The algorithm clusters

2. blocks which might belong to the same outage
events together.
4.1.2 Clustering Algorithm Design:
The goal of the algorithm is to cluster similar
events together. Similarity is defined based on
start and end times of events. For instance, all
blocks that have start and end times within a
delta value (default is 1800 which corresponds
to 30 minutes) will be clustered together. This
clustering algorithm design is much faster than
the one we had previously where the run time
has Big-O (n2
) for 2 reasons [1]:
a. The input is processed in parallel by
being partitioned into bins using
Hadoop. This will speed up a constant
factor of roughly 100
b. The algorithm only needs to go through
the input of each bin once from top to
bottom; hence it has run time Big-O (n).
However, it requires that the input is
sorted by duration and therefore, sorting
run time which has Big-O (n log n) is
the overall run time.
Our algorithm design is as of follow:
**The code above only perform matching and
does not do any optimal ordering. Instead, we
utilize Hadoop secondary sort to help sort the
input based on event’s duration.
In order to use Hadoop, we need to implement a
mapper process in which we make duplicates of
entries where their start time is within a delta
pre-defined in our program. The current delta
value is 1800 which corresponds to 30 minutes
meaning that any entries that have start time
within 30 minutes of next bin will be made
available in next bin. We are making duplicates
in bins in order to handle edge cases where
outage event happens between 2 bins.
After the clustering step, we need to go through
an intermediate step where we find the original
cluster id of duplicate entries and then go
through post-processing step to combine clusters
that belong to the same events together. The
reason we have to post process data after
clustering step since we make duplicates which
might be in different clusters than their original
blocks. Hence, we need to combine the clusters
which contain the original and duplicates
together and remove duplicates for data integrity
4.1.3 Intermediate step to process output of
clustering
In order for the post-processing step to know
which clusters to be put together, we need to
find out the original cluster id in which a
duplicate belong to. This intermediate step is
necessary to provide the original cluster id to
which the duplicates belong to. The information
is critical since it will help put clusters that
belong to the same event together in the post-
processing step. For example, after the
clustering step, we might have clusters of blocks
ABC and CDE (C is a duplicate in second
cluster). We want to combine these two clusters
into a cluster ABCDE.
The design of intermediate step is to output
block as key in mapper. This will help gather all
entries of same block in one reducer and it is
quite simple for a reducer to figure out the
Algorithm 1: Clustering of blocks

3. duplicates. Once a duplicate is found, it will be
linked to its original cluster by put the original
cluster id as an extra column at the end of its
entry.
4.1.4 Post Process Design
Since we are making duplicates to deal with
edge case in the clustering step, we will need to
filter out those duplicates in this post-processing
step as well as combine clusters into one cluster
when applicable.
The design of post-processing step is to output
cluster id as key. Then we will have all entries
that belong to the same cluster id end up in a
reducer. Inside each reducer, we will change the
cluster id of non-duplicate entries to the original
cluster id (this information is obtained in
intermediate step) of duplicate entries. This step
is basically a combined step of two clusters into
a cluster. For example, assuming cluster ABC
has cluster id of 1. In a reducer where C is a
duplicate, we have
cluster_id block original_cluster_id
2 C 1
2 D 2
2 E 2
Here we don’t actually output C since it’s a
duplicate entry, but we use C’s
original_cluster_id to change cluster_id of D and
E to 1. Note that the original _cluster_id column
is obtained from the intermediate step. Now
ABC and DE all have cluster id of 1, and can be
easily combined in the output with a simple sort.
4.2 Implementation
We write MapReduce application using Python
scripting language to process the input data set
on Hadoop in parallel. Mapper’s responsibility is
to read in data input, check if a row contains
good data, and output key as bin number and
value as block of address in hex, start time, and
duration of this block. Bin number is calculated
as follow:
Bin_num = int (block_start_time / 1800) * 1800
Based on the bin_num calculation, all blocks
that have start time within 30 minutes will end
up in the same reducer.
Reducer’s responsibility is to cluster blocks that
are in the same outage event based on end times.
Initially we sort blocks in each reducer using
Hadoop secondary sort based on duration and
start time. Then we apply Algorithm 1 above to
cluster blocks of same outage event and output
each outage event one by one in a reducer.
5. Evaluation
The experiment 5.1 is designed to help us reason
why we choose our bin number as 30 minutes.
Experiment 5.2 is designed to prove the
correctness of our algorithm and to check if our
algorithm could handle big burst of outage in
some bins.
5.1 Experiment 1: Bin Size Analysis
The main purpose of the experiment is to check
and avoid bin size numbers that put abnormally
large number of entries in a bin. We try to avoid
cases where n explodes in a particular bin since
our algorithm might not handle it well. This
experiment also gives us a clear observation on
what approximately max size of number of
blocks that our algorithm has to be able to
handle.
This experiment is designed to explore the
number of blocks in each bin with different bin
sizes. We start with bin size of 24 hours which
indicates that all outages which start within a
day will be grouped into a bin. We explore
different bin sizes of 16 hours, 8 hours, and 1
hour.
5.1.1 Hypothesis
Smaller bin size will help reduce number of
blocks in each bin; hence, reduce the burst of
outages if it happens in a bin.
5.1.2 Methodology

4. We use three different sets of data input from
a16, a17, and a18 for our analysis. For each data
set, we compute number of blocks in each bin
with different bin sizes of 24 hours, 16 hours, 8
hours, and 1 hour. Then we make graphs from
those output files to visualize and understand the
end results
5.1.3 Results
The results are very interesting. For bin size of
24h and 16h outputs depicted in Figure 1 and 2,
we see outage bursts in some months of about
500K blocks and large outage burst at the last
bin with about 1M blocks. This same behavior
happens to all three data sets a16, a17, and a18.
It might indicate that we have blocks that are not
responding to our vantage point probing. These
blocks does hurt since the algorithm will cluster
them and think that it is real outage. We need to
filter out those large bursts.
For bin size of 8h and 1h outputs depicted in
Figure 3 and 4, we see the same behavior as in
bin size of 24h and 16h in data sets a16 and a17.
However, data set a18 shows a more interesting
results. With bin size of 8h and 1h, it seems we
have eliminated the large outage burst at the last
bin although we still have a lot of outage bursts
of about 500K blocks in some bins.

5. Some further experiment into the last bin of data
set a17 exposes the reason why there are many
outage blocks. Many blocks are seen as down
with status column is just “j” within 5PM-6PM
on Oct 1st
, 2014. They also have very short and
consistent downtime duration (please see Figure
Figure 1: Bin size of 24h on a16, a17, a18 data sets
Figure 3: Bin size of 8h on a16, a17, a18 data sets
Figure 4: Bin size of 1h on a16, a17, a18 data sets
Figure 2: Bin size of 16h on a16, a17, a18 data sets

6. 5 for more details). This suggests that the outage
might be due to some network activities when
we ended the data set a17 and get ready for new
data set a18.
5.1.5 Conclusions
My current conclusion is that it does not matter
what bin size we choose to use since it will still
produce outage bursts in some bins that our
algorithm need to be able to handle. In addition,
I propose to ignore part of the last bin since it is
not really an outage event in real life. I filter out
and remove all blocks that has duration less than
500, and end up having around 500K blocks for
last bin which is normal bursts as in other bins.
5.2 Experiment 2: Correctness of new
algorithm
The main purpose of this experiment is to prove
that our new algorithm runs properly by
accurately identifying known outage event in the
data set.
Our data set contains probing information from
our vantage points from July 1st
, 2014 to
October 1st
, 2014. During this 3-month period,
there is one large outage that we would like to
take a closer look at in order to see if our
algorithm can correctly identify it.
The outage event that we are looking at is the
nationwide Time Warner (TW) outage happened
in the morning of August 27th
, 2014. This outage
causes many TW customers to not having
Internet connection for almost 2 hours. The
exact outage time was around 09:31 GMT to
11:21 GMT
5.2.1 Hypothesis
Our algorithm outputs the correct TW outage
event for different bin size of 2 hours, 1 hour,
and 30 minutes.
5.2.2 Methodology
We apply our Algorithm 1 mentioned above to
the entire data set and then specifically check the
bin number that has start time close to TW
outage start time. We then investigate its output
to check if the outage event found there match
the start and end time of the real TW outage
event and if the blocks in that outage event
actually belongs to TW by performing reversed-
DNS IP lookup given the block /24 addresses in
hex. We used different bin number of 2 hours, 1
hour, and 30 minutes to ensure the correctness of
our algorithm
5.2.3 Results
The result is what we expect. For bin number of
2 hours, we find a big outage event starting
around [08:47:47 GMT – 09:59:07 GMT] and
ends around [10:15:47 GMT - 11:27:07 GMT]
which matches the real TW outage event. The
TW outage event times are confirmed through
our web monitoring tool [2]. Similarly, we find
the same outcome for bin number of 1 hour and
30 minutes. For bin number of 1 hour, we find
the big outage event starting around [9:15:11
GMT – 09:58:55 GMT] and ending around
[11:00:41 GMT – 11:44:25 GMT]. For bin
number of 30 minutes, we find a big outage
event starting around [09:30:00 GMT – 09:55:19
GMT] and ending around [11:04:30 GMT –
11:28:36 GMT]. Both also closely matches the
real TW outage.
Figure 5: Small sample from last bin of 24h
bins of data set a17

7. We prove that the clustering algorithm is
working properly by plotting all blocks in our
TW cluster and see that they all have common
outage on Aug 27th
from 9:30 GMT to 11:30
GMT which matches with TW outage event.
Based on our reversed-DNS IP lookup result,
among 38K outage blocks in our cluster, there
are 94.36% of them belonging to TW and about
99% of these blocks are from the United States
which confirms that it is the real TW outage.
There are a few false positives in the above
result, but it’s too small comparing to the whole
cluster and can be ignored. The false positives
come from the fact that we have no way to
distinguish if a block is a TW block if it has start
and end times are within 30 minutes of the TW
start and end times. We believe that there might
be a few false negatives since if a TW block
somehow started late (after 30 minutes of TW
event start) or ended early (before 30 minutes of
TW event end), then it won’t be in the TW
cluster
5.2.4 Conclusions
Our algorithm properly identifies and clusters
big outage events given different bin number
sizes.
5.3 Experiment 3: Discovering unknown
outages
In experiment 2, we have proved that our new
algorithm properly identify known outage event
such as TW. The main purpose of this
experiment is to discover unknown outage
events by using our new algorithm. Then we do
reverse-DNS IP lookup and dig in more
information such as the country and city of the
origin, and compare to the result of our previous
study to check if it is a real outage [1].
5.3.1 Hypothesis
Our new algorithm can identify unknown
outages properly.
5.3.2 Methodology
Figure 6: Time Warner outage on Aug 27th
from 9:30 GMT – 11:30 GMT

8. We use the same input data set as in experiment
2 with our new algorithm and then explore
different clusters in the end result. We have
found many small outage events which include a
few hundred blocks and a few large outage
events which includes thousands of blocks.
5.3.3 Results
The result is interesting since we have
discovered some new unknown outages in our
input data set. Below are the 3 clusters that we
want to showcase.
a. Free SAS ISP outage
We discovered that there was an outage event
happened on Sep 8th
from 7:30 GMT to 8:30
GMT. It includes totally 14K block /24
addresses.
Figure 7 shows the plot of all 14K blocks in our
cluster and we can see that they all have
common outage on the date and time of the Free
ISP outage in France.
Based on our reversed-DNS IP lookup result,
among 14K outage blocks in our cluster, there
are 90.54% of them belonging to Free SAS ISP
and about 90% of these blocks are from France.
The main outage happened in Paris and some
other cities such as Marseille, Toulouse, and
Lyon.
b. Telefonica de Argentina outage
Besides outage found in France, we also found
another outage happened in Argentina on Sep 4th
from 4:45 GMT to 6:25 GMT. It includes totally
7K block /24 addresses.
Figure 8 shows the plot of all 7K blocks in this
cluster. Based on our reversed-DNS IP lookup
result, among 14K outage blocks in our cluster,
there are 89.99% of them belonging to
Telefonica de Argentina and Telefonía Pública y
Privada S.A. ISP and about 91% of these blocks
are from Argentina. The main city of the outage
is the Argentina capital Buenos Aires.
c. Uninet S. A. de C. V. outage
The last outage we want to show is the outage
event happened in Mexico on Sep 18th
from
20:00 GMT to 21:50 GMT. It includes totally
4K block /24 addresses.
Figure 9 shows the plot of all 4K blocks in this
cluster. Based on our reversed-DNS IP lookup
result, among 14K outage blocks in our cluster,
Figure 7: Free ISP outage event in France on Sep 8th
from 07:50 GMT to 8:45 GMT

9. there are 89.73% of them belonging to Uninet S.
A. de C. V. ISP and about 91% of these blocks
are from Mexico. The main city of this outage
event is Mexico city
Similarly to TW outage cluster, these 3 outage
clusters also contain some false positives. The
reason is the same. We have no way to know if a
block actually belongs to the real outage if it has
start and end time within our a delta value (1800
or 30 minutes used as default) of the real outage
event start and end times. There might also be a
few false negatives since if an outage block
somehow started or ended out of the delta value
range of the outage event start and end times, it
won’t be clustered.
Figure 8: Telefonica de Argentina outage event in Argentina on Sep 4th
from 4:45 GMT to 5:45 GMT
Figure 9: Uninet S. A. de C. V. outage event in Mexico on Sep 18th
from 20:00 GMT to 21:45 GMT

10. 5.4 Experiment 4: New algorithm
performance
The main purpose of this experiment is to show
the performance (or how fast) our new algorithm
runs for each bin number of 2 hours, 1 hour, and
30 minutes.
In the previous work, we already had a working
algorithm to cluster blocks based on XORing the
block bitmap [1]. However, its total run time is
Big-O (n2
). The new algorithm we proposed in
this research is much faster than the previous
one where its total run time is Big-O (n log n).
5.4.1 Hypothesis
Our new algorithm runs fast and will not take
more than couple minutes to complete given any
bin number.
5.4.2 Methodology
We use the same input data set to be processed
on Hadoop through MapReduce with different
bin number of 2 hours, 1 hour and 30 minutes.
We then investigate how long it would take the
MapReduce process in each case to complete.
We would expect the new algorithm to run
significantly faster than the algorithm in
previous work where it could take hours to a few
days to complete [1].
5.4.3 Results
The result is fascinating as we successfully
reduce the processing time from hours and days
to minutes. Table 1 shows the total runtime from
start to finish of 2-hour, 1-hour, and 30-minute
bin number respectively.
In all three bin number cases, the algorithm has
average total run time is roughly 2 minutes
given the input data set size is about 4.7GB.
5.4.4 Conclusions
We conclude that our new algorithm indeed
achieve very good performance as it
significantly reduces processing time to just
minutes.
6. Acknowledgements
I would like to thank Dr. Heidemann and Dr.
Pradkin for all the help and support in the entire
semester in order to make this project possible. I
have learned a lot from this project about
Hadoop & Map/Reduce, Python and analytical
skills and hope to continue the research with
both professors in the future
7. References
[1] Lin Quan, John Heidemann, and Yuri
Pradkin. Visualizing Sparse Internet Events:
Network Outages and Route Changes.
Computing, Springer. January 2013.
[2] John Heidemann. ANT Research News.
February 17, 2015. Web:
https://ant.isi.edu/blog/?p=638
Appendix A
This section includes a deeper technical
information on how to run the clustering
algorithm, intermediate and post-processing
steps on Hadoop in order to obtain final output.
All of Map/Reduce codes can be checked out
here:
“/nfs/jade/vint/ANT_SVN/trunk/lander_code/ou
tage_detection/clustering”
i. Clustering Algorithm
In this step, the mapper and reducer codes
used are: updown_mapper_edge.py and
updown_reducer_edge.py
Bin size Elapsed time
30 mins 2mins, 19sec
1 hour 2mins, 25sec
2 hours 2mins, 26sec
Table 1: Runtime of clustering
algorithm in different bin size cases

11. As mentioned in the script’s name, this
Map/Reduce pair will cluster blocks by
event (based on start and end times) and also
take care of the edge condition.
Full Hadoop command:
bin/hadoop jar hadoop-streaming.jar –D
mapreduce.job.output.key.comparator.class
=org.apache.hadoop.mapred.lib.KeyFieldB
asedPartitioner –D
mapreduce.partition.keypartitioner.options=
'-k1,1' –D
mapreduce.job.output.key.comparator.class
=org.apache.hadoop.mapred.lib.KeyFieldB
asedComparator –D
mapreduce.partition.keycomparator.options
='-k1,1 -k4,4n -k3,3n' –D
stream.num.map.output.key.fields=4 –input
$hadoop_dir/data/a17_all_updown_clean.fs
db -output $hadoop_dir/data/data_test1 –
output –mapper
$dir/updown_mapper_edge.py –reducer
$dir/updown_reducer_edge.py –partitioner
org.apache.hadoop.mapred.lib.KeyFieldBas
edPartitioner
The input file “a17_all_updown_clean.fsdb”
is post-processed using these two scripts
“outages_to_outagedownup.py” and
“outages_consolidate.py” from our
trinocular data set “a17all.outages.fsdb.bz2”.
Input file could be found here:
“/nfs/lander/working/trannk/clustering_algor
ithm/updownList_testing/a17_all_updown_c
lean.fsdb”
The two scripts can be checked out from this
svn repository:
“/nfs/jade/vint/ANT_SVN/trunk/lander_cod
e/outage_detection/vantage_agreement/filter
”
ii. Intermediate steps:
In this step, the mapper and reducer codes
used are: updown_intermediate_mapper.py
and updown_intermediate_reducer.py
Full Hadoop command:
bin/hadoop jar hadoop-streaming.jar –input
[output of clustering_step] -output [output to
a dir] –mapper
updown_intermediate_mapper.py –reducer
updown_intermediate_reducer.py
iii. Post-processing step
In this step, the mapper and reducer codes
used are: updown_post_process_mapper.py
and updown_post_process_reducer.py
Full Hadoop command:
bin/hadoop jar hadoop-streaming.jar –input
[output of intermediate step] -output [output
to a dir] –mapper
updown_post_process_mapper.py –reducer
updown_post_process_reducer.py