BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS: A NETWORK TELESCOPE PERSPECTIVE

CONTACT: PRAVEEN KUMAR. L (,+91 – 9791938249)
MAIL ID: , praveen@nexgenproject.com
Web: www.nexgenproject.com,
BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS:
A NETWORK TELESCOPE PERSPECTIVE
ABSTRACT
This paper addresses the problems of data sanitization and cyber situational awareness by
analyzing 910 GB of real Internet-scale traffic, which has been passively collected by monitoring
close to 16.5 million dark net IP addresses from a /8 and a /13network telescopes. First, the paper
offers a novel probabilistic dark net preprocessing model, which aims at sanitizing dark net data
to prepare it for effective use in the task of cyber threat intelligence generation. Such model has
been engineered using a distributed multithreaded approach, rendering it operational and highly
effective on dark net big data. Second, the paper further contributes by presenting an innovative
approach to infer large-scale orchestrated probing campaigns by leveraging dark net data, for
Internet cyber situational awareness. The approach uniquely reduces the dimensionality of such
big data by utilizing its artifacts, instead of processing the actual raw data. This is accomplished
by extracting and analyzing probing time series using formal methods rooted in Fourier
transform and Kaman filtering. Thorough empirical evaluations indeed validate the accuracy and
the performance of the proposed methods and techniques. We assert that the dark net sanitization
model and the probing orchestration inference approach are of significant value, given their
postulated highly applicable nature to the field of Internet measurements for cyber security in the
era of big data. EXISTING SYSTEM:
RELATED WORK
In this section, we review the related work on various concerned topics and show how the
proposed work is unique. In the area of extracting probing events, Li et al. [31]considered large
spikes of unique source counts as probing events. The authors extracted those events from dark

net(i.e., network telescope) traffic using time series analysis; they first automatically identified
and extracted the rough boundaries of events and then manually refined the event starting and
ending times. At this point, they used manual analysis and visualization techniques to extract the
event. In alternate work, Jin et al. [32] considered any incoming flow that touches any temporary
dark (grey) IP address as potentially suspicious. The authors narrowed down the flow swath
sustained suspicious activities and then investigated whether certain source or destination ports
are repeatedly used in those activities. Using these ports, the authors separated the probing
activities of an outside host from other traffic that is generated from the same host. In this work,
we not only extract probing activities, but further correlate their activities to infer orchestrated
probing campaigns. In the topic of analyzing probing events, the authors of[32] [33] studied
probing activities towards a large campus network using endow data. Their goal was to infer the
probing strategies of scanners and thereby assess the harmfulness of their actions. They
introduced the notion of gray IP space, developed techniques to identify potential scanners, and
subsequently studied their scanning behaviors. In another work, the authors of [31] [34]
presented an analysis that drew upon extensive honey net data to explore the prevalence of
different types of scanning. Additionally, they designed mathematical and observational schemes
to extrapolate the global properties of scanning events including total population and target
scope. In contrary, we aim at inferring large-scale probing campaigns rather than focusing on
analyzing specific probing events.
PROPOSED SYSTEM:
To tackle the aforementioned obstacles, this paper initially proposes a novel and a formal
preprocessing model to sanitize dark net data prior to its utilization for CTI. The model has been
specifically designed and engineered to execute in a distributed fashion for processing dark net
big data and currently possesses a collective throughput closet 12 GB/s when running on a 5-
node cluster. Subsequently, the paper contributes by offering an innovative approach to infer
orchestrated probing campaigns for cyber situational awareness. The approach uniquely operates

by processing “light” artifacts of dark net data using formal techniques, instead of processing the
raw data, thus significantly reducing the data’s dimensionality, hence minimizing the approach’s
complexity. Specifically, we frame this paper ‘score contributions as follows: Proposing a
probabilistic preprocessing model that aims at sanitizing misconfiguration traffic that is
embedded in dark net data to prepare it for effective CT use. The model is novel as it does not
rely on arbitrary cut-off thresholds, provides different likelihood models to distinguish between
misconfiguration and other dark net traffic, and is independent from the nature of the source of
the traffic. Further, the proposed model neatly captures the natural behavior of misconfiguration
traffic as it targets the darken .To the best of our knowledge, the presented model presents a first
attempt ever to systematically finger print and thus filter-out dark net miss configuration traffic.
Additionally, the model has been designed and implemented to execute on a distributed cluster
using a multithreaded approach to provide high throughput processing of dark net big data._
Designing an innovative approach to infer Internet scale or chest rated probing campaigns to
provide prompt cyber situational awareness. The novelty of the approach arises from the use of
dark net data’s artifacts, namely, its probing time series, rather than processing the actual raw
data. To this end, the probing time series are analyzed to detect orchestrated probing activities
and filter out non-coordinated ones by leveraging formal techniques rooted in Fourier transform
and Kaman filtering. To the best of our knowledge, (1) the cyber security capability to infer such
orchestrated campaigns do not exist and (2) the approach’s methodology, related to time series
interpolation and prediction for orchestration inference, has never been attempted before._
Empirically evaluating the dark net sanitization model using 670 GB of real dark net data and
validating its output against two other approaches. Moreover, we empirically evaluate the
orchestrated probing campaigns’ inference approach using 330 GB of dark net data, compare its
outcome against our own previous work [19] and validate its inferences using third party
publicly available threat repositories.
CONCLUDING REMARKS

This paper exploited passive monitoring of Internet-scaletraffic to investigate methods and
techniques that aim atcontributing to the task of cyber threat intelligence generationin the era of
big data. In particular, the paper offered anovel sanitization model that aims at cleansing darknet
datato prepare it for effective use. Additionally, the paper addressedthe problem of cyber
situational awareness throughtackling the challenging problem of inferring orchestratedprobing
campaigns by solely observing traffic destined tonetwork telescopes. Both approaches were
specifically designedand implemented to operate on darknet big data;while the sanitization
model was engineered to operate in adistributed, multithreaded manner, the proposed
inferenceapproach scrutinized the artifacts of such data to reduce itsdimensionality and thus the
processing complexity. Thoroughempirical evaluations using two large darknet datasetswere
conducted, which validated the generated results andinferences through comparisons with other
literature approachesand by relying on publicly available threat repositories.Given the plethora of
research and industrial effortsthat exploit darknet big data for cyber threat intelligencegeneration,
we concur that the devised models, approachesand techniques posses a significant impact on the
field ofInternet measurements for cyber security, especially in theera of big data.As for future
work, other than addressing the limitationsas mentioned in Section 7, we aim at correlating the
inferredorchestrated probing campaigns with malware data in anattempt to infer Internet-scale
infected hosts. Such informationis postulated to be distributed to concerned networkoperators for
effective remediation. Additionally, we aimat leveraging the darknet sanitization model to
provideeffective CTI tailored towards evolving paradigms such asthe Internet-of-Things and
Cyber-Physical Systems.
REFERENCES
[1] C. Rossow, “Amplification Hell: Revisiting Network Protocols forDDoS Abuse,” in Network
and Distributed System Security (NDSS)Symposium, 2014.

[2] Z. Durumeric, J. Kasten, D. Adrian, J. A. Halderman, M. Bailey,F. Li, N. Weaver, J. Amann,
J. Beekman, M. Payer et al., “The matterof heartbleed,” in Proceedings of the 2014 Conference
on InternetMeasurement Conference. ACM, 2014, pp. 475–488.
[3] N. Falliere, L. O. Murchu, and E. Chien, “W32. stuxnet dossier,”White paper, Symantec
Corp., Security Response, vol. 5, p. 6, 2011.
[4] N. Andronio, S. Zanero, and F. Maggi, “HelDroid: dissecting anddetecting mobile
ransomware,” in International Workshop on RecentAdvances in Intrusion Detection. Springer,
2015, pp. 382–404.
[5] “Massive DDoS attack harnesses 145,000 hacked IoTdevices,”
http://www.healthcareitnews.com/news/massive-ddos-attack-harnesses-145000-hacked-iot-
devices.
[6] T. Mahmood and U. Afzal, “Security analytics: Big data analyticsfor cybersecurity: A review
of trends, techniques and tools,” inInformation assurance (ncia), 2013 2nd national conference
on. IEEE,2013, pp. 129–134.
[7] J. M. Tien, “Big data: Unleashing information,” Journal of SystemsScience and Systems
Engineering, vol. 22, no. 2, pp. 127–151, 2013.
[8] C. Fachkha and M. Debbabi, “Darknet as a Source of CyberIntelligence: Survey, Taxonomy,
and Characterization,” IEEE CommunicationsSurveys & Tutorials, vol. 18, no. 2, pp. 1197–
1227, 2016.
[9] U. Goel, M. P. Wittie, K. C. Claffy, and A. Le, “Survey of End-toEndMobile Network
Measurement Testbeds, Tools, and Services,”IEEE Communications Surveys & Tutorials, vol.
18, no. 1, pp. 105–123, 2016.

[10] C. V. Zhou, C. Leckie, and S. Karunasekera, “A survey of coordinatedattacks and
collaborative intrusion detection,” Computers &Security, vol. 29, no. 1, pp. 124–140, 2010.
[11] H.-J. Liao, C.-H. R. Lin, Y.-C. Lin, and K.-Y. Tung, “Intrusiondetection system: A
comprehensive review,” Journal of Network andComputer Applications, vol. 36, no. 1, pp. 16–
24, 2013.

BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS: A NETWORK TELESCOPE PERSPECTIVE

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS: A NETWORK TELESCOPE PERSPECTIVE

Similar to BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS: A NETWORK TELESCOPE PERSPECTIVE (20)

More from Nexgen Technology

More from Nexgen Technology (20)

Recently uploaded

Recently uploaded (20)

BIG DATA SANITIZATION AND CYBER SITUATIONALAWARENESS: A NETWORK TELESCOPE PERSPECTIVE