1. Universidade Federal de Pernambuco
Centro de Informática
Pós-Graduação em Ciência da Computação
Daniel Araújo Melo
ARCA – Alerts Root Cause Analysis Framework
Dissertação de Mestrado
Recife
2014

2. Universidade Federal de Pernambuco
Centro de Informática
Daniel Araújo Melo
ARCA - Alerts Root Cause Analysis Framework
This dissertation has been submitted to the Informat-
ics Center of the Federal University of Pernambuco as
a partial requirement to obtain the degree of Master
in Computer Science.
Orientador: Djamel F. H. Sadok
Recife
2014

3. Catalogação na fonte
Bibliotecária Jane Souto Maior, CRB4-571
M528a Melo, Daniel Araújo
ARCA - Alerts root cause analysis framework / Daniel Araújo
Melo. – Recife: O Autor, 2014.
122 f.: il., fig., tab.
Orientador: Djamel Fawzi Hadj Sadok.
Dissertação (Mestrado) – Universidade Federal de Pernam-
buco. CIn, Ciência da computação, 2014.
Inclui referências.
1. Redes de computadores. 2. Segurança da informação. I.
Sadok, Djamel Fawzi Hadj (orientador). II. Título.
004.6 CDD (23. ed.) UFPE- MEI 2015-42

4. Daniel Araújo Melo
ARCA - Alerts Root Cause Analysis
Dissertação apresentada ao Programa de
Pós-Graduação em Ciência da Computação
da Universidade Federal de Pernambuco,
como requisito parcial para a obtenção do tí-
tulo de Mestre em Ciência da Computação.
Aprovado em: 08/09/2014
BANCA EXAMINADORA
__________________________________________
Prof. Dr. Stênio Flávio de Lacerda Fernandes
Centro de Informática / UFPE
__________________________________________
Prof. Dr. Arthur de Castro Callado
Mestrado e Doutorado em Ciências da Computação / UFC
___________________________________________
Prof. Dr. Djamel Fawzi Hadj Sadok (Orientador)
Centro de Informática / UFPE

5. A minha família, esposa e filhos.

6. Acknowledgments
Initially, I would like to thank my family, especially my mother, Carmem Dolores,
my wife Juliana, my son Enos Daniel and my grandmothers, Olga and Inez. They have
always stood by my side even when I was absent working in this research.
I would like to gratefully acknowledge the supervision of Professor Djamel
Sadok. He provided me important suggestions and encouragement during the course
of this work and offered the opportunity to join GPRT research team
My sincere thanks also goes to Professor Judith Kelner for pulling my ears when
needed and helping me when I lost the matriculation. I would not complete the aca-
demic requirements without her help.
I´d like to thank to my examination committee, Stenio Fernandes e Arthur Cal-
lado, for suggestions that enriched this work.
I cordially thank to my colleagues from GPRT for the help and revision of my
presentation, and colleagues from SERPRO, especially those that always believed that
this moment would come.
I want to express my gratitude to Andre Tio, Lalá, Tadeu, Noemi, Iuri, Nacho,
Suana, Amanda, Maíra, for the good vibrations.
And finally, thanks Universe!

7. “If you know the enemy and know yourself you need not fear the results of hundred
battles.”
- Sun Tzu

8. Abstract
Modern virtual plagues, or malwares, have focused on internal host infection and em-
ploy evasive techniques to conceal itself from antivirus systems and users. Traditional
network security mechanisms, such as Firewalls, IDS (Intrusion Detection Systems)
and Antivirus Systems, have lost efficiency when fighting malware propagation. Recent
researches present alternatives to detect malicious traffic and malware propagation
through traffic analysis, however, the presented results are based on experiments with
biased artificial traffic or traffic too specific to generalize, do not consider the existence
of background traffic related with local network services or demands previous
knowledge of networks infrastructure. Specifically don’t consider a well-known intru-
sion detection systems problem, the high false positive rate which may be responsible
for 99% of total alerts. This dissertation proposes a framework (ARCA – Alerts Root
Cause Analysis) capable of guide a security engineer, or system administrator, to iden-
tify alerts root causes, malicious or not, and allow the identification of malicious traffic
and false positives. Moreover, describes modern malwares propagation mechanisms,
presents methods to detect malwares through analysis of IDS alerts and false positives
reduction.
ARCA combines an aggregation method based on Relative Uncertainty with
Apriori, a frequent itemset mining algorithm. Tests with 2 real datasets show an 88%
reduction in the amount of alerts to be analyzed without previous knowledge of network
infrastructure.
Palavras-chave: Intrusion detection. Malwares. Alerts correlation. Advanced persis-
tent threats.

9. Resumo
As pragas virtuais modernas focam na contaminação de estações em redes internas,
e empregam técnicas evasivas para se ocultarem dos sistemas antivírus e dos usuá-
rios dos sistemas. Mecanismos tradicionais de segurança de rede, como firewalls, sis-
temas de detecção de intrusão (IDS – Intrusion Detection Systems) e sistemas antiví-
rus, perdem eficiência no combate a propagação de malwares. Pesquisas apresentam
alternativas para detectar de tráfego malicioso e propagação de malwares através da
análise de tráfego, mas apresentam resultados baseados em conjuntos de dados ar-
tificiais enviesados ou reais específicos demais para serem generalizados, não consi-
deram a existência de tráfego de background relacionado com serviços de rede local
ou exigem conhecimento prévio da infraestrutura de rede. Especificamente não con-
sideram um problema bem conhecido dos IDS: a alta taxa de falsos positivos, que
podem chegar a 99% do total de alertas. Esta dissertação propõe um framework
(ARCA – Alerts Root Cause Analysis) capaz de auxiliar um engenheiro de segurança
a identificar causas-raiz de alertas, maliciosos ou não, permitindo a identificação de
tráfego malicioso e falsos positivos. Adicionalmente, descreve os mecanismos de pro-
pagação de malwares modernos, propostas de detecção de malwares através da aná-
lise de alertas emitidos por IDS e propostas de redução de falsos positivos.
ARCA combina um mecanismo de agregação de alertas baseado na Incerteza
Relativa com o algoritmo de análise de itens frequentes Apriori. Testes realizados com
dados reais demonstraram uma redução em até 88% a quantidade de alertas a serem
analisados sem conhecimento prévio da infraestrutura de rede
Palavras-Chaves: Intrusion detection. Malware. Alerts correlation. Advanced persis-
tent threats.

10. Lista de Figuras
Figure 1 Worm propagation model (ZOU et al., 2005)..............................................24
Figure 2 Typical bonet´s elements (SILVA et al., 2013) ...........................................26
Figure 4 Typical botnet life-cycle proposed in (FEILY; SHAHRESTANI; RAMADASS,
2009).........................................................................................................................29
Figure 5 Botnet life cycle proposed in (RODRÍGUEZ-GÓMEZ; MACIÁ-FERNÁNDEZ;
GARCÍA-TEODORO, 2013) ......................................................................................31
Figure 6 IRC-based botnet DDOS Attack (COOKE; JAHANIAN; MCPHERSON, 2005)
..................................................................................................................................33
Figure 7 Hybrid P2P network....................................................................................36
Figure 10 Gameover Zeus network topology. Dotted line indicates information flow.
..................................................................................................................................41
Figure 11 Organizations Categories (MCAFEE, 2010).............................................43
Figure 12 Victim´s Country of Origin (MCAFEE, 2010) ............................................44
Figure 13 Model for APT stages proposed by (GIURA; WANG, 2012).....................44
Figure 14 A targeted attack in action (SOOD; ENBODY, 2013) ...............................45
Figure 15 Infected Hosts according Wan IP (FALLIERE; MURCHU; CHIEN, 2011) 48
Figure 16 Overview of Stuxnet Malware Operation ..................................................49
Figure 17 Countries affected by Flame according to McAfee (GOSTEV, 2012b)....51
Figure 18 Countries affected by Flame according Symantec (SYMANTEC, 2012b) 52
Figure 19 Flame C&C Platform(ZHIOUA, 2013).......................................................54
Figure 20 An example of (a) bipartite graph and (b) one-mode projection. ..............55
Figure 21 BotHunter System by (PORRAS, 2009) ..................................................56
Figure 22 Vulnerabilities reported do NVD (NIST, 2014)..........................................59
Figure 23 Incidents reported to Cert.br (CERT.BR, 2014)........................................60
Figure 24 Layout of the proposed classification system in (PARIKH; CHEN, 2008).
..................................................................................................................................68

11. Figure 25 A sample multi-step-attack (SOLEIMANI; GHORBANI, 2008) .................70
Figure 26 Generic view of alarm correlation according (HUBBALLI;
SURYANARAYANAN, 2014).....................................................................................71
Figure 27 Generic view of graph ordering (PAO et al., 2012)...................................74
Figure 28 ATLANTIDES architecture (BOLZONI; CRISPO; ETALLE, 2007)............75
Figure 29 Proposed Architecture (HUBBALLI; BISWAS; NANDI, 2011). .................76
Figure 30 Normalized SrcIp and DstIp quantities per significant class (SID).
[Max(SrcIp), Min(SrcIp)]=[309,1] and [Max(DstIp), Min(DstIp)]=[542,2]. ...................85
Figure 31 ARCA Architecture ...................................................................................86
Figure 32 ARCA Workflow........................................................................................87
Figure 33 - Atable and Ctable..................................................................................89
Figure 34 Job1 collects the alerts and runs RUA and FIM .......................................90
Figure 35 Job2 imports one or more RCARs and removes the selected alerts ........91
Figure 36 Histogram of Class Counter from SERPRO’s dataset..............................93
Figure 37 Histogram of SrcIP Counter from SERPRO’s dataset ..............................94
Figure 38 Histogram of DstIP Counter from SERPRO’s dataset ..............................94
Figure 39 Normalized alert quantities per significant alert class (SID)......................96
Figure 40 Normalized SrcIp and DstIp quantities per significant class (SID)............96
Figure 41 Alert Reduction in 12 hours interval........................................................101
Figure 42 Total Alerts versus Final Alerts in 12 hours interval................................101
Figure 43 Histogram of Class Counter from MACCDC’s dataset ...........................102
Figure 44 Histogram of SrcIP Counter from MACCDC’s dataset............................103
Figure 45 Histogram of DstIP Counter from MACCDC’s dataset............................103

12. Lista de Tabelas
Comparison of life-cycle models................................................................................28
APT’s model comparison...........................................................................................47
Methods comparison.................................................................................................77
Apriori parameters.....................................................................................................90
Results from RU Algorithm. Class clustering from 8:00 am to 8:00 pm.....................95
Results from RU Algorithm. SrcIP clustering from 8:00 am to 8:00 pm.....................95
Results from RU Algorithm. DstIp clustering from 8:00 am to 8:00 pm. ....................95
Root Cause Association Rules from Serpro’s dataset, between 8:00 am and 9:00 am.
..................................................................................................................................97
Apriori’s Association Rules for Rule 1 .......................................................................98
Apriori’s Association Rules for Rule2 ........................................................................99
Apriori’s Association Rules for Rule3 ........................................................................99
Apriori’s Association Rules for Rule4 ......................................................................100
New RCARs created from new alerts detected between 15 and 17 pm..................102
RCAR Rules From MACCDC 2012 dataset ............................................................104
Alerts triggered by Rule 1........................................................................................104
Destinations from alerts triggered by Rule 2............................................................104

13. Lista de Algoritmos
Algorithm 1 Simplified significant cluster extraction algorithm .................................82

14. Lista de Siglas
IDS Intrusion Detection System
ARCA Alerts Root Cause Analysis
MLP Multilayer Perceptron
TP True Positive
FP False Positive
FQDN Fully Qualified Domain Name
RR Resource Record
NIDS Network-based Intrusion Detection
HIDS Host-based Intrusion Detection
IPS Intrusion Prevention
RCAR Root Cause Association Rule

15. Sumário
CHAPTER 1 INTRODUCTION ............................................................................................................. 17
1.1 MOTIVATION.................................................................................................................................. 18
1.2 OBJECTIVES.................................................................................................................................. 20
1.3 DOCUMENT ORGANIZATION............................................................................................................ 20
CHAPTER 2 MALICIOUS SOFTWARE ............................................................................................... 21
2.1 MALWARE TYPES........................................................................................................................... 22
2.1.1 WORMS........................................................................................................................... 22
2.1.1.1 Propagation Model....................................................................................................................... 22
2.1.1.2 P2P worms .................................................................................................................................. 24
2.1.2 BOTS AND BOTNETS .................................................................................................... 25
2.1.2.1 Botnet Life-Cycle ......................................................................................................................... 27
2.1.2.2 C&C Architectural Designs .......................................................................................................... 31
2.1.2.3 Fast-Flux...................................................................................................................................... 37
2.1.2.4 Domain-flux.................................................................................................................................. 38
2.2 MODERN MALWARES ..................................................................................................................... 38
2.2.1 MARIPOSA...................................................................................................................... 38
2.2.2 TDL4 ................................................................................................................................ 39
2.2.3 GAMEOVER ZEUS ......................................................................................................... 40
2.3 ADVANCED PERSISTENT THREATS.................................................................................................. 42
2.3.1 APT MODEL.................................................................................................................... 44
2.3.2 STUXNET ........................................................................................................................ 47
2.3.3 FLAME............................................................................................................................. 50
2.4 FIGHTING MALWARE PROPAGATION................................................................................................ 54
2.5 CHAPTER SUMMARY ...................................................................................................................... 57
CHAPTER 3 INTRUSION DETECTION AND FALSE ALARM REDUCTION..................................... 58
3.1 IDS CLASSIFICATION ..................................................................................................................... 61
3.2 PROBLEMS WITH DARPA DATASET................................................................................................ 62
3.3 FALSE ALARM GENERATION ........................................................................................................... 63
3.3.1 SIGNATURE ENHANCEMENT....................................................................................... 65
3.3.2 STATEFUL SIGNATURES.............................................................................................. 65
3.3.3 VULNERABILITY SIGNATURES .................................................................................... 66
3.3.4 ALARM MINING .............................................................................................................. 66
3.3.4.1 Clustering .................................................................................................................................... 67
3.3.4.2 Classification................................................................................................................................ 67
3.3.4.3 Neural network approach............................................................................................................. 69
3.3.4.4 Frequent pattern mining............................................................................................................... 69
3.3.5 ALARM CORRELATION ................................................................................................. 70
3.3.5.1 Multi-step correlation ................................................................................................................... 72
3.3.5.2 Causal relation based correlation ................................................................................................ 72
3.3.5.3 Attack graphs based correlation .................................................................................................. 73
3.3.6 ALARM VERIFICATION .................................................................................................. 74
3.3.7 HYBRID METHODS ........................................................................................................ 75
3.4 CHAPTER SUMMARY ...................................................................................................................... 77
CHAPTER 4 ARCA FRAMEWORK ..................................................................................................... 79

16. 4.1 FUNDAMENTAL CONCEPTS............................................................................................................. 80
4.1.1 ROOT CAUSES............................................................................................................... 80
4.1.2 RELATIVE UNCERTAINTY CLUSTERING .................................................................... 80
4.1.2.1 Extracting Significant Cluster....................................................................................................... 82
4.1.3 FREQUENT ITEMSET MINING ...................................................................................... 82
4.2 ARCA ARCHITECTURAL DESIGN .................................................................................................... 84
4.3 IMPLEMENTATION........................................................................................................................... 87
4.3.1 RUA – RELATIVE UNCERTAINTY AGGREGATOR ...................................................... 87
4.3.2 FIM – FREQUENT ITEMSET MINER ............................................................................. 89
4.3.3 ALERTS AGGREGATION............................................................................................... 90
4.4 EXPERIMENTS ............................................................................................................................... 91
4.4.1 ALERTS PREPROCESSING .......................................................................................... 92
4.4.2 EXPERIMENT WITH THE SERPRO DATASET............................................................. 92
4.4.2.1 Results evaluation ....................................................................................................................... 98
4.4.3 EXPERIMENT WITH THE MACCDC´S DATASET....................................................... 102
CHAPTER 5 CONCLUSIONS ........................................................................................................... 106
5.1 CONTRIBUTIONS .......................................................................................................................... 107
5.2 DIFFICULTIES FOUND................................................................................................................... 107
5.3 LEARNED LESSONS...................................................................................................................... 108
5.4 FUTURE WORK ........................................................................................................................... 108
REFERENCES.................................................................................................................................... 109

17. 17
Chapter 1
Introduction
Incident report statistics and ongoing researches at specialized centers such as
Cert.br (CERT.BR, 2014), Enisa (ENISA, 2014) and Cert/cc (CERT, 2014), show an
alarming increase of threats directed to end users and hosts. Many works from the
industry also describe techniques adopted by malicious software (malwares), with the
objective to steal private data and use infected computers to perpetrate network at-
tacks (KAMLUK, 2009) (GONCHAROV, 2012).
Furthermore, recent researches show that malwares have evolved from self-
propagating programs, a.k.a. ‘worms’, (ZHOU, CHENFENG VINCENT; LECKIE;
KARUNASEKERA, 2010), to controlled machines via Command and Control (C&C)
servers, a.k.a., ‘bots’ (TSAI et al., 2011; YU et al., 2014). Moreover, the security com-
munity has devoted efforts to research the rising of Advanced Persistent Threats (APT)
and Remote Administration Tools (RAT), potentially harmful malwares with political or
industrial espionage motivation (BAIZE; CORP, 2012; BRADBURY, 2010; GIURA;
WANG, 2012; SOOD; ENBODY, 2013; TANKARD, 2011).
Given the malware’s code obfuscation techniques, each infection may produce
a new code and circumvent traditional signature-based antivirus systems
(OUELLETTE; PFEFFER; LAKHOTIA, 2013; SZÖR; FERRIE, 2001; WONG; STAMP,
2006). As a consequence, malware signatures may be outdated when distributed to
antivirus clients. The problem is amplified by traditional network security countermeas-
ures limitations when fighting malware propagation or internal attacks (BAIZE; CORP,
2012; PORRAS, 2009). Therefore, academia and industry have directed efforts on re-
search network techniques to track malware traffic (PORRAS, 2009).
Along this document we will discuss malware evolution, how to improve Intru-
sion Detection Systems (IDS) to detect malware traffic, drawbacks that may influence

18. 18
IDS in a negative way and a proposed framework, named ARCA (Alerts Root Cause
Analysis), whose main objective is to group alerts and allow security engineers to an-
alyze alerts root cause.
The remainder of this chapter describes the focus of this dissertation and starts
by presenting its motivation in Section 1.1 and a clear definition of the objectives in
Section 1.2. Section 1.3 describes how this dissertation is organized.
1.1 Motivation
Traditional network security countermeasures lose efficiency when fighting mal-
ware propagation, or internal attacks (BAIZE; CORP, 2012; PORRAS, 2009). Firewalls
are generally deployed to protect local networks from outsiders and cannot avoid in-
ternal attacks or attacks between workstations - unless a security policy demands fire-
wall deployment in workstations and local servers. Intrusion Detection Systems (IDS)
have been well utilized to spot inbound attacks or malicious outbound traffic, but in-
fected hosts and internal attackers may direct attacks to other workstations and local
network services while avoiding firewalls. Moreover, communication channels between
infected machines and control servers may use encryption. Anti-Virus Systems cannot
follow malware polymorphic capabilities and a malware signature may be outdated
when distributed (OUELLETTE; PFEFFER; LAKHOTIA, 2013; PORRAS, 2009; SZÖR;
FERRIE, 2001; WONG; STAMP, 2006).
In last years, a great deal of work was dedicated to developing methods that
classify and extract malicious from normal traffic, as in (GU et al., 2007, 2009;
MANIKOPOULOS; PAPAVASSILIOU, 2002a; SHAHRESTANI et al., 2009; XU;
WANG; GU, 2011a; YU et al., 2014). According to (SAAD et al., 2011) detection though
network traffic behavior is advantageous because it´s possible to detect malwares ma-
licious activities during any phase of its life cycle and has a lower cost than deep packet
inspection. On the other hand, (PORRAS, 2009) has presented the challenges faced
by such methods: malwares can be stealthy, irregular and deceptive, therefore, gen-
erate few anomalies in network traffic.
Modern malwares are in constant evolution. Each new version or variant imple-
ments more deceptive techniques, to conceal itself from traffic analysis and system

19. 19
administrators, as presented in Chapter 2. However, it is possible to observe a partic-
ular characteristic that, to this date, remains unchanged and common to modern mal-
wares: the majority of exploits used to infect new hosts are directed to known patchable
vulnerabilities, the same was observed by McHugh et al. (MCHUGH; FITHEN;
ARBAUGH, 2000) more than 10 years ago.
Contemporary open source NIDS, such as Snort and Suricata, have active com-
munities and industry initiatives developing signatures to detect exploitation of known
vulnerabilities, network protocols anomalies and policy violations (EMERGING
THREATS, 2013; SOURCEFIRE, 2013; SURICATA, 2014). Most of vulnerabilities ex-
ploited by malwares presented in Chapter 2 have correspondent signatures; moreover,
there are specific signature subsets with the objective to detect tools and protocols
related with potential leaks, such as P2P protocols, binary downloads through HTTP,
internet anonymizes, instant message, and others. Therefore, a NIDS may provide
useful information to detect malicious traffic related with malware propagation.
However, IDS have well-known drawbacks. The work presented in (HUBBALLI;
SURYANARAYANAN, 2014) provides a survey on several schemes with a major con-
cern, namely, how to minimize the false alarm rate in IDS. It also argues that hybrid
approaches, mixing data mining schemes and filtering based schemes, are better
suited to dynamic environments like an internal network perimeter. The survey’s con-
clusion addresses questions to the research community with gaps to motivate future
efforts, like incremental learning, testing with common datasets and real time capabil-
ity.
Given the IDS’s important role against potential malware propagation and the
reduction of False Positive (FP) rate, the research community must consider the exist-
ence of false positives and its influence on experimental results. So far, it seems to
handle malicious behavior identification and false alerts reduction as separate prob-
lems. Moreover, schemes have been tested with private datasets from traffic too par-
ticular to generalize or biased artificially generated datasets (BRUGGER; CHOW,
2005; HUBBALLI; SURYANARAYANAN, 2014; MAHONEY; CHAN, 2003; MCHUGH,
2000; TJHAI et al., 2008).

20. 20
1.2 Objectives
The main goal of this dissertation is to investigate and propose a method to fight
malware propagation in internal networks, through the enhancement of contemporary
signature-based NIDS.
As secondary goals, it’s important to:
 Evaluate how the alert aggregation method proposed in (FEITOSA,
EDUARDO LUZEIRO, 2010) will behave when facing alerts from two
real distinct traffic samples;
 Evaluate if malicious activities generate regular statistical significant
alerts;
 Evaluate if the proposed method is useful to detect malware spreading
and reduce alerts volume.
 Survey modern malwares behavior and spread techniques;
 Survey relevant strategies leading to false alerts reduction;
1.3 Document Organization
This dissertation is organized as follows:
 Chapter 2 - Malware Evolution - describes malware evolution, the rise
of APT (Advanced Persistent Threats) and proposals to fight malware
propagation;
 Chapter 3 – Intrusion Detection Systems – describes the evolution of
Intrusion Detection and the research to minimize the false alarm rate
problem;
 Chapter 4 – ARCA Framework – ARCA’s theoretical basis is explained,
implementation details are described and the tests results are pre-
sented;
 Chapter 5 – Conclusions – final conclusions and discussion about con-
tributions and future work are made.

21. 21
Chapter 2
Malicious Software
In this Chapter modern malwares are discussed, its fundamental concepts are
presented and examples of the most relevant malwares are discussed. Moreover,
methods to detect malicious traffic related with malwares are also presented.
Malicious software, or software with malicious purposes, namely, malware, is a
source of significant amount of unwanted traffic on the Internet (FEITOSA, EDUARDO
LUZEIRO, 2010). First malwares were created in the early 1980´s and since them mal-
wares have evolved with the objective to circumvent traditional security countermeas-
ures, from simple code that infected boot sectors to complex software with multiple
propagation vectors (AYCOCK, 2006; OUELLETTE; PFEFFER; LAKHOTIA, 2013).
Modern malwares explore technical and social weaknesses to propagate. Non-
solicited e-mails (SPAM) use social engineering to persuade users to execute mali-
cious code and explore system vulnerabilities, or even take advantage of users per-
missions. After the successful infection, if the infected station is part of a local network,
attacks may be triggered to infect other stations or compromise internal servers (YU et
al., 2014).
There is no consensus of the financial impact of malware on the global econ-
omy, but the participation of organized crime on malware development is well known,
and estimations from Industry about cybercrime are alarming. McAfee estimates the
financial global impact between $300 billion and $1 trillion (CENTER OF STRATEGIC
AND INTERNATIONAL STUDIES, 2013), and Symantec estimates that cybercrime
has a cost of $388 billion to online adults from 24 countries (SYMANTEC, 2013).
In the following sections the terms virus and malware are used interchangeably.

22. 22
2.1 Malware Types
(AYCOCK, 2006) had classified malware according to its operational method.
Three characteristics were used in the classification scheme:
 Self-replication – When malwares actively attempt to autonomously
spread by creating new copies, without user interference;
 Population growth – The rate of a malware’s population growth due to
self-replication;
 Parasitic behavior – If a malware requires another executable, or any
computer component like a boot block code on a disk or binary code, to
exist.
2.1.1 Worms
A worm is a self-replicating program that spreads by exploiting vulnerabilities
found in other machines (ANDROULIDAKIS; CHATZIGIANNAKIS; PAPAVASSILIOU,
2009). While a virus propagates infecting other code, a worm searches for vulnerabili-
ties across a network or dispatches emails with infected attachments, seeking to trick
users or explore e-mail clients vulnerabilities. It also employs obfuscation techniques
like encryption, oligomorphism, polymorphism or metamorphism
2.1.1.1 Propagation Model
Worms generally use multiple techniques, or propagation vectors, to spread.
(ZOU; TOWSLEY; GONG, 2006) proposed two major classes of worms, according to
the way it spreads:
 Email worms – propagate through e-mails and infect hosts when users
read the email content or open attachments. Human interference is re-
quired to propagate and thus propagation speed is relatively slow;
 Scan-based worms – scan IP addresses prefixes and directly exploit vul-
nerabilities on target hosts. As no human interference is required, they
are faster than email worms;

23. 23
According to (ZOU; TOWSLEY; GONG, 2006; ZOU et al., 2005), the epidemic
model is adequate to model a scan-based worm, or “uniform scan worm”, which uni-
formly picks IP addresses and scans for vulnerable targets.
The epidemic model assumes that each subject resides in two states, has a
single transition, from susceptible to infected state, and once infected, remains in the
infectious state forever. Moreover, the model assumes all subjects can directly contact
each other and don´t collaborate with their infection efforts.
The model for a finite population is
𝑑𝐼𝑡
𝑑𝑡
= 𝛽𝐼𝑡[𝑁 − 𝐼𝑡]
(1)
Where 𝐼𝑡 is the number of infected subjects at time 𝑡 and 𝑁 is the size of vul-
nerable population before any infection take place. 𝛽 is called pairwise rate of infec-
tion, it represents “infection intensity” from infected to susceptible subjects and corre-
sponds to
𝛽 =
𝜂
Ω
(2)
Where 𝜂 is average number of scans an infected host starts per unit time and
Ω is number of available IP addresses. Therefore, every scan has a probability of 1 Ω⁄
to hit any IP address from this scanning space. At 𝑡 = 0, 𝐼0 subjects are initially in-
fected while the remaining 𝑁 − 𝐼0 subjects are susceptible.
(ZOU et al., 2005) also argues that it is possible to roughly partition the propa-
gation in three phases, as may be seen in Figure 1:
 Slow start phase – Since 𝐼𝑡 ≪ 𝑁 the number of infected hosts grows ex-
ponentially;
 Fast spread phase – Many hosts are infected and start to infect others in
a linear speed;
 Slow finish phase – The infection rate decreases because fewer suscep-
tible vulnerable computers are left.

24. 24
Figure 1 Worm propagation model (ZOU et al., 2005)
The infection rate is the average number of vulnerable hosts that can be infected
per unit of time by one infected host during the early stage of a worm’s propagation.
It should be noted that model (1), for the sake of simplicity, does not consider
two major factors affecting a worm’s spreading: human counteraction and network con-
gestion. The former has to be considered to model a slow spreading worm, such as e-
mail worm, while the later has to be considered to model fast spreading worm, such as
uniform scan worm.
2.1.1.2 P2P worms
Peer-to-peer attacks are an increasingly popular technique for worm propaga-
tion due to its simplicity (SZOR, 2005). After a succeeded infection, a worm searches
for P2P download folders and makes a copy of itself to the folders found. Anything
available in a download folder is shared in a P2P network and worms may overwrite or
infect legitimate binary files.

25. 25
2.1.2 Bots and Botnets
Bots are compromised computers controlled by one or more human operators,
commonly known as botmasters, with the intent to perform malicious activities, and
part of a network of infected computers, is known as botnet (RODRÍGUEZ-GÓMEZ;
MACIÁ-FERNÁNDEZ; GARCÍA-TEODORO, 2013; SILVA et al., 2013). According to
the survey in (ZHU et al., 2008) a botnet is “a collection of software robots, or bots,
which run autonomously and automatically”. The infection methods used to compro-
mise systems are similar to other classes of malwares, by exploiting vulnerabilities,
code insertion and social engineering that leads users to download malicious code.
According to (SILVA et al., 2013): “The primary purpose of botnets is for the
controlling criminal, group of criminals or organized crime syndicate to use hijacked
computers for fraudulent online activity”.
Industry reports have called attention to the severity of botnet problems (SILVA
et al., 2013). Botnets are responsible for 80% of all SPAM circulating in the Internet
and some botnets had infected millions of hosts. It was claimed that the Mariposa bot-
net had infected 12 million hosts in 190 countries (SINHA et al., 2010). Moreover,
academic research had alerted to the outgrowing number of botnets (COOKE;
JAHANIAN; MCPHERSON, 2005; RODRÍGUEZ-GÓMEZ; MACIÁ-FERNÁNDEZ;
GARCÍA-TEODORO, 2013; ZHUGE et al., 2007).
The major characteristic of a botnet is the control channel which allows the bot-
master, or botnetmaster, to send commands and updates to the infected system. The
updates include new exploits or code update to bypass signature-based antivirus. This
command and control (C&C) channel can operate in different network topologies and
use different network protocols. The general components of a botnet are illustrated in
Figure 2 and in Section 2.1.2.2 the architectural design will be discussed in details.

26. 26
Figure 2 Typical bonet´s elements (SILVA et al., 2013)
The communication between a botmaster and bots in a P2P network can be
push-based or pull-based, depending on whether the first a bot waits for commands
from the botmaster or asks the botmaster for commands (WANG, PING et al., 2009).
Apart from botnets elements already illustrated, (RODRÍGUEZ-GÓMEZ;
MACIÁ-FERNÁNDEZ; GARCÍA-TEODORO, 2013) extend the model and includes
roles to represent the related social context :
 Developer – A person, or group, who designs and implements the botnet.
Not necessarily the botmaster, because development work may be subcon-
tracted. There are development kits, commonly named Do-it-Yourself (DIY),
that provide tools to assist botnets development and maintenance.
 Client – Those that rent botnet services from a botmaster or seek to control
a botnet and used it for their own purposes.
 Victim – A system, person, network or organization which is the attack tar-
get.
 Passive Participant – the owner of the host infected.

27. 27
2.1.2.1 Botnet Life-Cycle
Three botnets life-cycle models were proposed in literature, each one covers
states observed in dissection of bots and botnets reported by security practitioners and
researchers. Although they differ in how the life-cycle is detailed and the number of
possible states, each draws attention for two common states: how the infection initi-
ated, i.e. it is focused on initial infection or recruitment, and how the communication is
established between C&C servers and bots, i.e. the C&C protocol and how the C&C
servers are reached.
Sinha et al. (SINHA et al., 2010) have observed that new generation botnets
tends to employ automated strategies to spread, as worms. Several researchers have
identified worms, such as Conficker(BURTON, 2010) and Sdbot(TREND MICRO,
[S.d.]), as the main recruiting strategy of botnets. (SINHA et al., 2010) have observed
that botnets combine capabilities of worms, viruses and Trojan horses.
A new strategy has been identified in P2P botnets: propagation through existing
P2P networks, such as VBS.Gnutella(SYMANTEC, 2007); however, the number of
possible targets is limited by the P2P network size.
Wang et al. (WANG, PING et al., 2009) had observed the rise of botnets with
multiple spread mediums like e-mail, instant messages and file exchange. In
(POLYCHRONAKIS; MAVROMMATIS; PROVOS, 2008) and (COVA; KRUEGEL;
VIGNA, 2010) a new method called drive-by download attack is discussed. According
to Polychronakis et al. (POLYCHRONAKIS; MAVROMMATIS; PROVOS, 2008): “In a
drive-by download attack, a malicious web page exploits a vulnerability in a web
browser, media player, or other client software to install and run malware on the un-
suspecting visitor’s compute”.
Once infected, a bot has to communicate with its C&C servers; otherwise it will
be an isolated infected host. Each C&C architecture has particularities and will be dis-
cussed in subsection 2.1.2.2.Table 2.1 presents a comparison of the proposed models
and shows their common steps.

28. 28
Table 2.1 Comparison of life-cycle models
Ramadass et al.
(FEILY; SHAHRESTANI;
RAMADASS, 2009)
Wang et al.
(WANG, PING et al., 2009)
Rodríguez-Gómez et. al.
(RODRÍGUEZ-GÓMEZ;
MACIÁ-FERNÁNDEZ;
GARCÍA-TEODORO, 2013)
Conception
Initial infection Recruiting Bot members Recruitment
Secondary injection
Connection Forming the botnet
Interaction
Malicious command and
control Stand by for instructions
Update and maintenance
Marketing
Attack Execution
Attack Sucess
Ramadass et al. depicted a lifecycle with five phases (FEILY; SHAHRESTANI;
RAMADASS, 2009), as may be seen in Figure 3:
1. Initial infection – The attacker scans a network for known vulnerability and
exploits it to gain control of attacked system;
2. Secondary injection – A shell-code is executed and downloads via FTP,
HTTP, or P2P, the actual bot binary to install itself on infected system, which
become a “zombie”, full controlled by botnetmaster. The bot code is automat-
ically executed each system boot;
3. Connection – the bot establishes the C&C connection with the C&C server ;
4. Malicious command and control – bot programs receive and execute com-
mand sent by botmaster;
5. Update and maintenance – Bot code may be updated to evade detection,
correct bugs or change C&C server;

29. 29
Figure 3 Typical botnet life-cycle proposed in (FEILY; SHAHRESTANI; RAMADASS,
2009)
In (WANG, PING et al., 2009) a new life-cycle model with three stages was
proposed for P2P Botnets:
1. Recruiting Bot members – Similar to initial infection, as proposed in
(FEILY; SHAHRESTANI; RAMADASS, 2009).
2. Forming the botnet – After infection, a host has to join the P2P network,
otherwise it will be an isolated infected one. The initial procedure to join a
P2P network is called “bootstrap” and according to (WANG, PING et al.,
2009) two methods are well known:
a. An initial list is hardcoded in each P2P client, and the bot tries
to contact the nodes in this list to update its neighbor list.
b. A shared web cache stores the initial host list and each bot has
its address hardcoded.
3. Stand by for instructions – After a successful join, the bot keeps waiting for
a command from the botmaster. The communication model may be push,

30. 30
pull or a combination of both. More details about the communication model
in P2P botnets are found in Section 2.1.2.2.
Rodríguez-Gómez et. al. (RODRÍGUEZ-GÓMEZ; MACIÁ-FERNÁNDEZ;
GARCÍA-TEODORO, 2013) extended the botnet life-cycle model, covering from its
conception to the achievement of the desired (malicious) purpose. The life-cycle pro-
posed is a linear sequence of stages and the failure of any intermediate stage thwarts
the botnet aim. The proposed model is composed of six stages, depicted in Figure 4:
1. Conception – The main characteristics and botnet purposes are de-
fined in this first stage;
2. Recruitment – After conceived and created, the botnet needs to re-
cruit/infect hosts;
3. Interaction – The communication between an infected machine and
a botnet server is established. The information exchanged is com-
posed of commands and maintenance operations;
4. Marketing – the developer needs to make the botnet and its capabil-
ities public, in order to attract clients and profit from it;
5. Attack Execution – The infected hosts may offer rentable private in-
formation to the attacker, like financial data, and launch attacks, like
DDOS attacks or phishing dissemination, according client’s interests;
6. Attack Success – when the botnet objective is fulfilled.

31. 31
Figure 4 Botnet life cycle proposed in (RODRÍGUEZ-GÓMEZ; MACIÁ-FERNÁNDEZ;
GARCÍA-TEODORO, 2013)
2.1.2.2 C&C Architectural Designs
According to (ZHU et al., 2008), the C&C architecture may be classified as:
 IRC bot – The first, and most prevalent, botnets used Internet Relay Chat
(IRC) protocol, with a centralized C&C mechanism, due to the flexibility
and scalability of this protocol.
 HTTP bot – The C&C channel uses the Hyper Text Transfer Protocol
(HTTP) due to its encryption capabilities and firewall policies that allow
internet access through TCP ports 80 and 443;
 P2P bot – A P2P architecture offers a more stable architecture to a C&C
channel than a centralized point of failure;

32. 32
 Fast-flux (FF) networks - An advanced technique, first presented in
(HONEYNET PROJECT, 2008), and also surveyed in (SHENG YU;
SHIJIE ZHOU; SHA WANG, 2010) and (ZHANG et al., 2011), used to
avoid the C&C channel detection. The idea is to rapidly change the map-
ping between multiple IP addresses and one single domain. More details
are presented in section 2.1.2.3.
The survey in (SILVA et al., 2013) classifies C&C channels according to their
specific architecture and operational modes, whether it is: centralized, decentralized,
hybrid or random architectures, and has persistent or periodic (sporadic) modes.
Centralized C&C
This architecture implements the traditional client-server model where all bots
establish connection with one or more C&C servers. The main advantage of a central-
ized architecture is the fast information exchange between server and clients, and
whether the major drawback is the C&C server as central point of failure.
Earlier centralized botnets, such as Agobot, Phatbot and IRCbot, used IRC as
their communication protocol in a push-base model, where the botmaster pushes com-
mands to a bot, which then responds accordingly (FEDYNYSHYN; CHUAH; TAN,
2011). The advantages of using IRC as C&C channel protocol are:
 Flexibility – botmasters can split the bots in groups and send different
commands to each one, moreover, IRC servers can forward messages
to bots at different servers ;
 Open source – There are several open source servers available on the
Internet;
 Redundancy – Bots can connect to backup servers if the primary server
is down and IRC servers can be part of an IRC network – group of inter-
connected IRC servers;
 Scalability – Tests comparing IRC servers performance demonstrated
capacity to millions of users(PITCOCK, 2010). Moreover, IRC servers
may be part of an IRC servers network and distribute bots load between
these servers.
 Versatility – Beyond message exchanges, IRC servers can transfer files.

33. 33
In Figure 5, the elements of an IRC-based botnet are presented as proposed
in (COOKE; JAHANIAN; MCPHERSON, 2005). The botmaster (commander)
sends commands through an IRC network, which servers may be public or hid-
den from the general public. The commands may be directed to all bots, or a
group. A bot, or zombie, starts a malicious activity immediately after receiving a
message from the botmaster, e.g. a DDOS attack.
Figure 5 IRC-based botnet DDOS Attack (COOKE; JAHANIAN; MCPHERSON,
2005)
Contemporary IRC botnets have evolved to obfuscate IRC messages and
evade signature-based detection, but IRC C&C channel remains possible to detect
because IRC traffic is not common in corporate networks. Therefore, a network admin-
istrator can prevent botnet activity by blocking IRC traffic in firewalls. Due to this limi-
tation, HTTP became popular in botnets, such as Storm and Bobax, as a C&C protocol,
because HTTP has considerable advantages over IRC: it’s generally allowed between
organizations, the bots poll the C&C server in a pull-based model, this means that C&C
traffic behaves like normal HTTP traffic, and has cryptographic capabilities using TLS
(Transport Layer Security).
Though advantageous, HTTP has the main disadvantage of a centralized archi-
tecture, the central point of failure. In (WANG, PING; SPARKS; ZOU, 2010) C&C

34. 34
servers are evidenced as having the following fundamental weak points in contempo-
rary botnets, which are:
 Limited number of IP addresses facilitates the C&C server detection;
 If a C&C server is shutdown, the botmaster will lose control over infected
hosts;
 If a C&C server is hijacked by authorities or security researches, the en-
tire botnet can be exposed;
Wang et al. (WANG, PING; SPARKS; ZOU, 2010) also argues that as security
practitioners develop means to disrupt botnets, cybercriminal practitioners will develop
more resilient and evasive C&C architectures.
Decentralized C&C
Given the limitations in a centralized architecture, security researches and law
enforcement have succeeded in taking down attempts to disrupt botnets (BARFORD;
YEGNESWARAN, 2007; FEDYNYSHYN; CHUAH; TAN, 2011; RODRÍGUEZ-
GÓMEZ; MACIÁ-FERNÁNDEZ; GARCÍA-TEODORO, 2013; STONE-GROSS et al.,
2011; WANG, PING; SPARKS; ZOU, 2010). The cybercrime answer was the develop-
ment of botnets with a decentralized and more resilient architecture, organized as P2P
networks, such as Waledac, Mariposa and Torpig (ROSSOW et al., 2013). The re-
search in (ROSSOW et al., 2013) argues that even after being analyzed and disrupted,
some P2P botnets keep in execution and their exact size is unknown, even a size
estimation is a complex task.
Jelasity et. al. (JELASITY; BILICKI, 2009) proposed that P2P botnets are based
on a structured P2P overlay, such as Kademlia (CROWCROFT et al., 2005). Thus,
this improves the botnet resiliency because failure of peers won’t cause network-wide
failure and data is replicated across multiple peers.
In (WANG, PING et al., 2009) P2P botnets are classified in three terms, accord-
ing to the way a P2P botnet subverts, or not, an existent P2P network:
 Parasite – all the bots are selected from vulnerable hosts within an exist-
ing P2P network, and it uses this available P2P network for command
and control.
 Leeching – members join an existing P2P network and depend on this
P2P network for C&C communication, but the bots could be vulnerable

35. 35
hosts that were either inside or outside of the existing P2P network, e.g.
early version of Storm botnet;
 Bot-only – the P2P botnet builds its own P2P network, in which all mem-
bers are bots, e.g. Stormnet and Nugache.
A parasite botnet uses available P2P protocols to allow bots to locate and com-
municate with each other, no design is required from the botmaster and the bootstrap
method is already implemented by the P2P client. In leeching and bot-only botnets the
botmaster must design bootstrap modules, in order to add an infected host which is
not a member of the P2P network.
The C&C mechanism in P2P networks was evaluated in (WANG, PING et al.,
2009) and the way push and pull methods can be applied were discussed. For leeching
and parasites P2P botnets the same mechanism that existent P2P protocols use for
file search is adapted to command asking: In a pull-based method bots send requests
for commands and botmasters answers with commands instead of files. Implementa-
tion of a push method is more complex, but feasible in structured P2P networks. For
bot-only P2P networks a new P2P communication protocol may be developed, or an
existing P2P protocol may be extendedHybrid C&C
This architecture employs characteristics from centralized and decentralized ar-
chitectures. Wang et al. (WANG, PING; SPARKS; ZOU, 2010) argues that even with
advanced designs, such as the absence of a bootstrap process in the Slapper Worm
and Sinit, the public key cryptography to authenticate users in Sinit, or the encrypted
control channel in Nugache, the P2P botnets have weaknesses and are not mature. A
single captured bot can expose all the network and the complicated communication
mechanisms facilitate detection through network flow analysis.

36. 36
Figure 6 Hybrid P2P network
Given the weaknesses found in centralized and decentralized architectures
(WANG, PING; SPARKS; ZOU, 2010) proposed a hybrid model, depicted in Figure 6,
with the following features:
 A bootstrap procedure is required, because the methods to detect boot-
strap are well known;
 Each bot has a limited list of peers, and if a bot is captured just a partial
list of nodes will be exposed;
 A botmaster can send report commands to a group of bots and the an-
swer is redirected to a different node, called sensor node, every time a
command is issued. This avoids the detection and blocking of sensor
nodes;
 A botmaster can update nodes list in each bot with a single update com-
mand;
 The bots with static IP addresses that are accessible from the Internet
are candidates for being servant bots. In P2P terminology servant nodes
acts like servers and clients simultaneously.

37. 37
 Each servant bot listens for incoming connections and uses symmetric
cryptography to ensure confidentiality, command and node authentica-
tion, and to evade network analysis.
Random C&C
According to (COOKE; JAHANIAN; MCPHERSON, 2005), in random botnets
no single bot knows about any more than another bot. In addition, when a botmaster
wants to send a message to bots, it starts a random scan in the Internet and when a
bot is found, a connection is established to the exchange encrypted messages and
finished immediately. Despite the protocol simplicity and obscurity, a single bot cannot
compromise the whole network and the message latency and the lack of delivery guar-
antees are a major drawback. Even the random behavior is detectable.
2.1.2.3 Fast-Flux
Fast-Flux is a mechanism used in botnets to evade C&C channel detection, first
introduced in (HONEYNET PROJECT, 2008). The main idea is to associate a fully
qualified domain name (FQDN) to multiple, even thousands, IP addresses, using a
very short Time-to-Live (TTL) for any given particular DNS Resource Record (RR)
(IETF, 1987). Therefore, a bot may establish a new connection to a different C&C
server, or botnet node, every 3-10 minutes. In addition, the bots don’t connect directly
to C&C servers, but to blind proxies that forward content to backend servers.
Two different types of fast-flux networks were categorized in (HONEYNET
PROJECT, 2008): Single-flux and Double-flux. In a Single-flux network, every 3-10
minutes the DNS record is changed and the bot starts a new DNS resolution, which
will deliver a new IP address from a fast-flux redirector, responsible for content for-
warding between bot and the backend server, named “mothership”. These redirectors
are generally infected hosts and if a redirector is shut down, another redirector on
stand-by will take its place in IP address pool. In a Double-flux network, DNS A and
NS records are continually changed in a round robin manner and advertised into the
fast-flux network.

38. 38
2.1.2.4 Domain-flux
Fast-flux networks have a single point of failure, the DNS resolution. A bot, or
fast-flux agent, needs to resolve the FQDN and several techniques were proposed to
detect botnet’s DNS resolutions (ZHANG et al., 2011).
In (STONE-GROSS et al., 2011) a new evasion technique was presented,
namely Domain-flux, in which each bot independently uses a domain generation algo-
rithm (DGA) to compute a list of domains names. For each round, instead of a new
DNS resolution with the same FQDN, the bot generates a new FQDN previously reg-
istered by attackers, asks for this FQDN resolution and if the IP address provides a
valid response, it is considered valid until the next round. In (ZHANG et al., 2011),
several techniques to detect fluxing domains are also presented.
2.2 Modern Malwares
2.2.1 Mariposa
It was claimed that Mariposa botnet had infected around 12.7 million hosts in
190 countries until its disruption(GOODIN, 2010). Sinha et al. (SINHA et al., 2010)
stated that Mariposa was extremely harmful because it could:
 Download and execute binary code on the fly, using Direct Code Injection (DCI)
to inject malicious code inside the address space of the explorer.exe program;
 Infect machines already infected with different bots;
Moreover, Mariposa had implemented a proprietary UDP-based C&C protocol,
named the Iserdo Transport Protocol.
Three main spreading techniques were detected in Mariposa Analysis:
 USB Spreading: the bot copies itself to USB when a device is connected
to the infected host;
 MSN Spreading: if the infected host has the MSN messenger installed,
malicious crafted messages are sent to recipients found in the infected
host;

39. 39
 P2P Spreading: If the infected host has a P2P application, such as: Ares,
BearShare, iMesh, Shareaza, Kazaa, DC++, eMule and LimeWire, the
bot copies itself to the shared folder.
A successful infection occurs if the binary code is executed whatever user’s
permissions are, because the code is injected into the explorer.exe address space and
can download other modules with new functionalities, including from other bots like
Zeus, using HTTPS, HTTP, FTP or Butterfly Network Protocol. In addition, the modules
can turn the infected host into a DDOS participant or a reverse proxy server.
Sinha et al. (SINHA et al., 2010) summarized Mariposa C&C architecture, as:
 Bot client - the infected host with spread functionalities already pre-
sented;
 Bot Server – A mediator with 2 functions: anonymizes the master and
acts as a load balancer;
 Bot Master – The core of operations, acts as a manager to multiple serv-
ers. It has the ability to enable and disable servers and clients.
Actually there is no consensus about the exact number of Servers, but several
domains were identified, three hard-coded (SINHA et al., 2010) and the rest observed
during analysis (DEFENCE INTELLIGENCE, 2010; ICS-CERT, 2010). It sends an en-
crypted message to a server candidate and waits for the reply. If the server does not
respond, it tries another one until a successful connection is achieved.
2.2.2 TDL4
TDL4, detected on June, 2011, is the fourth generation of a previously detected
bot TDSS, which have evolved to version 4 as the most sophisticated contemporary
bot, and according to the Kaspersky team (GOLOVANOV; SOUMENKOV; IGOR,
2011) had infected over 4.5 million hosts. Bots from the TDSS family spread using
multiple techniques (SYMANTEC, 2008):
 Drive-by-download infections, discussed in Section 2.1.2.1, through fake
blogs, forum comments, legitimate hacked, forged websites and affiliate
programs;
 Fake torrent files and P2P downloads;
 Cracks in Warez websites;

40. 40
On infection, TDL4 installs an advanced rootkit in the Master Boot Record
(MBR), in order to load before the operating system. The code in MBR is encrypted
and capable to evade most of signature-based antivirus software; moreover, TDL4 re-
moves approximately 20 others malicious programs.
The main purpose of TDL4 is to generate revenue to cybercriminals by redirect-
ing internet access from infected hosts to affiliated sites.
The C&C architecture is hybrid, TDL4 may use a centralized architecture with
approximately 60 HTTP C&C servers or embed its C&C protocol in the Kad network
P2P protocol. Hence, TDL4 uses centralized servers or a public P2P network in order
to transmit commands to infected hosts; moreover, the communication is encrypted
with an unknown algorithm, probably developed by the attackers.
It is worth to notice that TDL4 exploits the MS10-061 vulnerability, patched by
Microsoft since 2010.
2.2.3 Gameover Zeus
Gameover Zeus, also called P2P Zeus is, to this date, the newer variant of Zeus
malware (ALAZAB et al., 2013; ANDRIESSE et al., 2013), a credential-stealing Trojan
first discovered in 2007. This new variant introduced a P2P decentralized C&C proto-
col, which network is divided in several virtual sub-botnets independently controlled by
several botmasters.
According to the Dell SecureWorks Counter Threat Unit (STONE-GROSS,
2012), P2P Zeus uses Cutwail (TREND MICRO, 2009), another SPAM botnet, to send
massive amounts of email that impersonates well-known online retailers, cellular
phone companies, social networking sites, and financial institutions. The e-mails con-
tains links to fake webpages which use Blackhole (SURI, 2011), a commercial exploit
kit which targets vulnerabilities in web browsers and plugins such as Adobe Reader,
Flash and Java.
According to (ANDRIESSE et al., 2013) Gameover Zeus network topology is
organized in three disjoint layers, as depicted in Figure 7:

41. 41
Figure 7 Gameover Zeus network topology. Dotted line indicates information flow.
 P2P Layer - Formed by infected hosts, which can play 2 roles: harvester bot
and proxy bot. The first steals information located in the infected host and it
sends to proxy bots and waits for commands from proxy bots, while the latter
forward commands from C&C proxy servers and also sends the information
stolen from harvester bots. Moreover, proxy bots also act as harvester bots
and are elected manually by botmasters;
 C&C Proxy Layer - Proxy bots interact with the C&C proxy layer to update
their command repository and to forward the stolen data collected from the
bots to the C&C server in the upper layer;
 C&C Layer – The C&C server manages C&C proxy servers and its bots.
The communication between bots is usually UDP-based, except for the C&C
communication between harvester bots and proxy bots, and binary/configuration up-
date exchanges, both of which are TCP-based. Moreover, critical messages are en-
crypted with RSA-2048.
Bootstrapping onto the network is achieved through a hardcoded bootstrap peer
list. This list contains the IP addresses, ports and unique identifiers of up to 50 Zeus

42. 42
bots. Zeus port numbers range from 1024 to 10000 in versions after June 2013, and
from 10000 to 30000 in older versions. Unique identifiers are 20 bytes long and are
generated at infection time by taking a SHA-1 hash over the Windows ComputerName
and the Volume ID of the first hard-drive. These unique identifiers are used to keep
contact information for bots with dynamic IPs up-to-date. Moreover, bots check the
responsiveness of their neighbors every 30 minutes. Each neighbor is contacted in
turn, and given 5 opportunities to reply. If a neighbor does not reply within 5 retries, it
is discarded from the peer list.
A Domain Generation Algorithm (DGA) is used to generate 1000 unique domains
per week, which are the addresses of C&C proxy servers
2.3 Advanced Persistent Threats
While worms and bots usually attack broadly, without a specific target, several
academic researches and industry reports have alerted to the growing number of tar-
geted attacks, where the attacker has a monetary or political motivation to attack a
specific organization (SOOD; ENBODY, 2013), (TANKARD, 2011), (LI, FRANKIE; LAI;
DDL, 2011), (DE VRIES et al., 2012), (BAIZE; CORP, 2012), (THOMSON,
2011),(MANDIANT, 2010),(MCAFEE, 2010),(ISACA, 2013).
The industry called such targeted attacks as Advanced Persistent Threats, or
APT (MANDIANT, 2010; MCAFEE, 2010), because the attackers are professionals,
more insidious, stealthy and persistent. The motivation isn’t the immediate gain pur-
sued by cybercriminals, but trade secrets, intellectual property or governments classi-
fied information. According to (TANKARD, 2011) ‘persistent’ refers to: “the fact that
the goal of an APT is to gain access to targeted information and to maintain a presence
on the targeted system for long-term control and data collection”. Moreover, according
(SOOD; ENBODY, 2013): “Persistence is a characteristic of targeted attacks because
they persist in the face of adversity instead of moving on to weaker targets”. Giura et
al. (GIURA; WANG, 2012) have explained APT as follows: Advanced means that at-
tackers are well trained, well-funded and with a wide spectrum of intrusion technolo-
gies; Persistent means it is persistent over time; Threat means the attackers´ intention
is to inflict damage or steal proprietary data.

43. 43
The first industry report to address APTs is the report “Revealed: Operation
Shady RAT” (MCAFEE, 2010), which describes how McAfee´s team had detected mal-
ware variants with heuristic signatures which indicated an encrypted C&C HTML chan-
nel. After they successfully gained access to one C&C server, they were able to identify
a victim population since mid-2006 when the log collection began. It must be noticed
that the malicious activity may have initiated before 2006, but the earlier evidence
shows 2006. Most alarming were the number of organizations evidenced as victims:
71 organizations from 14 countries. The organizations were classified in 32 unique
categories, as seen in Figure 8, and the 14 countries are depicted in Figure 9. The
term RAT means Remote Access Trojan, defined by (AYCOCK, 2006) as programs
that allow a computer to be monitored and controlled remotely.
Figure 8 Organizations Categories (MCAFEE, 2010)

44. 44
Figure 9 Victim´s Country of Origin (MCAFEE, 2010)
Following (ZHIOUA, 2013), given the amount of effort required to build sophisti-
cated malware like APTs, and the consequences of the attacks, it´s possible to con-
clude that the developers, or attackers, are not typical cybercriminals or hacktivists,
and moreover, these malwares are using state-of-art hacking techniques.
2.3.1 APT Model
Giura and Wang (GIURA; WANG, 2012) analyzed industry reports and con-
cluded that each APT is customized for each target. However, the stages of APT have
similarities and differ mostly in the methods they use at each stage. Therefore, Giura
and Wang proposed a model to APT´s stages, as shown in Figure 10:
Figure 10 Model for APT stages proposed by (GIURA; WANG, 2012).
 Reconnaissance
Attackers gather public information about the target, identify IP address range
used by an organization and scan the targeted network seeking for vulnerable servers.
Information about the employees gathered from social networks is used to build pro-
files, which will provide information to social engineering attacks.
 Delivery
Information gathered in the Reconnaissance initial stage will be used to craft a
spear-phishing email, which is a phishing specially crafted to the targeted employees.

45. 45
The e-mail might contain attached malicious files or a link to a malicious URL that the
user is guided to trust. Emails are the main infection technique, but other infection
channels may be used, such as USB based malware and time activated Trojan.
 Exploitation
Once the successful infection of a host in the targeted network is achieved,
the APT establishes a connection with a C&C server and uploads information gathered
in the infected host, including passwords, e-mails, network usernames and network
shared resources.
 Operation
Attackers maintain the persistent presence and scans internal network seeking
potential targets which store sensitive information.
 Data Collection
Attackers use privilege credential harvested in previous stages to collect sensitive
data, compress and encrypt it before uploading.
 Exfiltration
The data organized in previous stage is uploaded to multiple servers, in order to
prevent investigators to find the final data destination.
Figure 11 A targeted attack in action (SOOD; ENBODY, 2013)

46. 46
Sood and Enbody (SOOD; ENBODY, 2013) developed a model of targeted attacks
depicted in three phases, as show in Figure 11:
 Intelligence Gathering
To perform reconnaissance, attackers collect target´s information from public
available resources, such as DNS queries and WHOIS lookups, and organizational
webpages. Useful information regarding employees, vendors and daily operations, can
also be collected in social networks, such as Facebook or Twitter, or personal
webpages.
With this information attackers start to scan the target network looking for vul-
nerabilities, opened ports, address range, outdated systems, virtualized platforms, and
all available information about the target network infrastructure. Moreover, organiza-
tion webpages are scanned for known vulnerabilities, such as SQL Injection (SQLI)
and Cross-site Scripting (XSS).
 Threat Modeling
The attackers create a profile of the target and its environment; even a replica
of the target is constructed so that attackers may test penetrations and no suspicion is
raised at the target.
 Attacking and Exploiting Targets
In general, the attack aims to load a malware onto a target´s host and use it as
a platform to analyze internal infrastructure and compromise other hosts. Attacks can
vary but exhibit common patterns:
 Drive-by-download and spear phishing;
 Exploiting web infrastructure;
 Exploiting communication protocols;
 Exploiting co-location services;
 Physical attacks.
Several Elements are used frequently in targeted attacks:
 Malware Infection Frameworks;
 RATs and Rootkits;
 Morphing and Obfuscation Toolkits;

47. 47
 Interface with underground market.
In Table 2.2 a comparison of the two proposed models is presented. The model pro-
posed by Giura and Wang (GIURA; WANG, 2012) is more detailed; the Reconnaissance
step is equivalent to Information Gathering and Threat Modeling in the model proposed
by Sood and Enbody (SOOD; ENBODY, 2013). However, the latter offers more details
about tools and techniques than the former.
Table 2.2 APT’s model comparison
Giura and Wang
(GIURA; WANG, 2012)
Sood and Enbody
(SOOD; ENBODY, 2013)
Reconnaissance Information Gathering
Threat Modeling
Delivery
Attacking and
Exploiting Targets
Exploitation
Operation
Data Collection
Exfiltration
2.3.2 Stuxnet
Stuxnet is considered the first cyberwarfare weapon in the history of security
(LANGNER, 2011) and, according to Symantec (MCDONALD et al., 2013), in the wild
since early November 2007, first noticed by the industry in 2008 and in development
as early as November 2005, and with 4 different versions: 0.500, 1.001, 1.100 and
1.101. Contrary to initial belief, Stuxnet’s objective was not industrial espionage, but to
physically destroy an industrial controller, specific from one manufacturer (Siemens),
attached to a SCADA system (GALLOWAY; HANCKE, 2013).
An industrial control network is a system of interconnected equipment used to
monitor and control physical equipment in industrial environments (GALLOWAY;
HANCKE, 2013). It is composed of specialized components and applications, such as
Programmable Logic Controllers (PLCs), Supervisory Control and Data Acquisition
(SCADA) systems and Distributed Control Systems (DCSc). SCADA is a software

48. 48
layer whose objective is to provide an interface between PLC and user level software,
it captures signals from devices and sends high level control commands, e.g. the in-
struction to start an engine or change control parameters, such as rotation speed.
Stuxnet had taken a longer time in the slow start phase then conventional
worms, mainly because its main spreading technique relied on local exploitation,
through USB sticks and/or local networks. Moreover, the infection process included a
fingerprinting procedure to deploy the payload only if the controller model identified
was a model used by Iran´s Government to enrich uranium (LANGNER, 2011). Figure
12 presents the origin countries of hosts infected, according to Symantec (FALLIERE;
MURCHU; CHIEN, 2011).
Figure 12 Infected Hosts according Wan IP (FALLIERE; MURCHU; CHIEN, 2011)
According to (ZHIOUA, 2013), the Stuxnet attack operates at three levels:
(1)Windows OS, (2) Step 7 Software, and (3) PLC. Figure 13 gives an overview of
how Stuxnet operates. Its main goal is to compromise the PLC through the infection of
the Windows host connected to the PLC.

49. 49
Figure 13 Overview of Stuxnet Malware Operation
Stuxnet’s main infection technique is the LNK exploit (MS10-046) delivered in a
USB drive (MICROSOFT, 2010a). The vulnerability allows the execution of a malicious
code inserted in shortcuts (.LNK files) when the shortcut icon is displayed. A Windows
host is compromised when Windows Explorer is used to open the USB drive containing
the malicious LNK file. During the infection process Stuxnet uses rootkit techniques to
hide files and inject code into processes.
If the host has the Step 7 installed (SIEMENS, [S.d.]), Stuxnet will hook specific
APIs used to open Step 7 projects and execute each time a project is loaded, this
allows Stuxnet to propagate using the infected files and infect the host again in case
of SO update or replacement.
After a successful infection Stuxnet initiates local network propagation
(MCDONALD et al., 2013; ZHIOUA, 2013) through the exploitation of:
 Print spooler service vulnerability (MS10-061) (MICROSOFT, 2010b),
as it allows remote code execution through a Printer Service, if a printer
is shared on the local network .
 Windows Server service vulnerability (MS08-067) (MICROSOFT, 2008),
allows remote code execution through Remote Procedure Call (RPC).
It is worth to notice that these vulnerabilities were discovered during Stuxnet
analysis which was unpatched then.

50. 50
Stuxnet tries to communicate with a C&C servers and, if the connection is es-
tablished can get updates, as well as more binary codes to execute in the infected
machine, and upload infected host information, including installed Industrial Control
Systems software. The control connection is not a mandatory procedure (MCDONALD
et al., 2013), Stuxnet was developed to be autonomous with a behavior similar to a
worm; therefore, the C&C protocol is simple, HTTP-based with 2 domains, where en-
cryption is used only when uploading host information, and 4 servers in 4 countries
were identified until Stuxnet disruption. Moreover, compromised hosts within the same
local network established a P2P network, and the host capable to communicate with
the C&C server acts as a proxy, and distributes information through the local P2P net-
work.
The payload is dropped and executed only if the PLC uses a Profibus commu-
nication processor (TEXAS INSTRUMENTS, [S.d.]). The malicious code monitors the
Profibus messaging bus and modifies the spinning frequency of the attached equip-
ment, to 1410Hz then to 2Hz then to 1064Hz, with the objective to stress and destroy
the equipment.
2.3.3 Flame
Flame was an APT discovered in 2012 by (IRAN NATIONAL CERT, 2008) and
initially mistaken as related with Stuxnet. At a first glance Flame has evaded 43 antivi-
ruses, demonstrated multiple spread and obfuscation techniques, and related with a
mass data loss in Iran.
The first in-depth study of flame was conducted at Budapest University of Tech-
nology and Economics by the Laboratory of Cryptography and System Security –
CrySyS Lab (CRYSYS, 2012). Flame was characterized as an info-stealer malware
and with a modular structure which allows it to incorporate multiple techniques to prop-
agate and to obfuscate, such as 5 different encryption methods, 3 different compres-
sion techniques and 5 different file formats.
According to Symantec (SYMANTEC, 2012f) Flame’s main characteristic is not
to spread until asked to. After the initial infection process, no spread action is taken by
the infected host until the C&C connection is established and a command to spread
arrives. Moreover, Flame is maybe the first malware with a “suicide” routine

51. 51
(SYMANTEC, 2012c, d): after the Flame details came to public, a new module was
distributed by C&C servers to infected hosts and few weeks later a command to exe-
cute this module and completely remove Flame was sent. The Flame activity gradually
ceased since them.
There is no consensus about the geographical information where Flame has
attacked and what is its main spread technique.
Kaspersky (GOSTEV, 2012b) stated that Flame had attacked middle-east
countries, mostly in Iran and Israel, as seen in Figure 14, but Symantec (SYMANTEC,
2012b) said that the primary targets of this threat are located in the Palestinian West
Bank, Hungary, Iran, and Lebanon; however, additional reports indicated infections in
Austria, Russia, Hong Kong, and the United Arab Emirates, as seen in Figure 15. A
possible explanation for this discrepancy is because each company handles infections
from different constituencies.
Figure 14 Countries affected by Flame according to McAfee (GOSTEV, 2012b)

52. 52
Figure 15 Countries affected by Flame according Symantec (SYMANTEC, 2012b)
Flame has multiple spreading techniques, including exploits to vulnerabilities
already exploited by Stuxnet, and patched by Microsoft since 2010 at least: Windows
Print Spooler Service vulnerability (MS10-061), Microsoft Windows Shortcut ‘LNK/PIF’
Files Automatic File Execution vulnerability (MS10-046) and Print Spooler Service vul-
nerability (MS10-061). Some confusion about Flame being an evolution of Stuxnet has
been considered by researchers, but this idea was discarded when a more in-depth
analysis evolved.
Unsuccessful efforts have been made to identify Flame´s main spread tech-
nique, i.e. no one has identified how the infection initiated. The Kaspersky team
(GOSTEV, 2012a) reported that no zero-day vulnerability was found and fully patched
Windows 7 was infected. However, one of the spread techniques found may indicate
how: attackers had forged Microsoft’s digital certificates (SYMANTEC, 2012g), actually
revoked, and intercepted Microsoft Update Service requests to execute code in the
target host as Microsoft´s (GOSTEV, 2012a). A module found in flame allows an in-
fected host to act as a proxy for Windows updates requests, i.e. an infected host de-
tects network clients configured to automatic proxy detection, announces itself as a
proxy server, intercepts update requests and introduces malicious code signed with
Microsoft’s forged digital certificates. There’s no evidence of this attack or interception
on Internet Service Providers (ISP), but it may be applied into ISP´s infrastructure as
well.

53. 53
Analysis from the CrySyS laboratory (CRYSYS, 2012) and Symantec
(SYMANTEC, 2012a) had drawn attention to a particular Flame’s module able to enu-
merate devices around the infected host, to announce the host as a discoverable de-
vice and encode the status of the malware in device information using base64 encod-
ing. Symantec (SYMANTEC, 2012a) argues how an attacker can do with this func-
tionality;
 Identification of victim social networks – Monitoring devices within Blue-
tooth range, attacker may catalog the devices encountered and maps the
victim’s social and professional circles;
 Identification of victim physical locations – By measuring the strength of
Bluetooth’s radio waves it is possible to calculate the distance between
hosts and attackers can identify other near devices, including those
owned by organization’s employees; moreover, attackers can deploy
Bluetooth monitoring devices in public places in order to track them;
 Enhanced information gathering – Attackers can steal contacts from mo-
bile devices, SMS messages and any data. Attackers may even turn on
the microphone of mobile devices and record a conversation.
Flame infection installs a Lua interpreter (LUA, 1993) which allows attackers to
deploy new functionalities through multiple scripts. Following Symantec (SYMANTEC,
2012e) the attackers have something equivalent to an “app store” where new modules
can be retrieved. The scripts provide functionalities to extract data form infected hosts,
capture users credentials – if the user has administrative clearance, the credentials
are used to access domain servers and add user accounts with default passwords,
distribute malicious code through network shares, and more, as found in (CRYSYS,
2012).
After a successful infection, the infected host establishes a connection with a
C&C Server, sends initial data collected and waits for instruction. Figure 16 presents
Flame’s C&C architecture: 80 domains were used to obfuscate 22 C&C servers. The
protocol used to communication between servers and infected hosts was HTTPS and
attackers had accessed the Servers through SSH, to perform system administrative
tasks, or HTTPS, to access a web application used to control the infected hosts
(SYMANTEC, 2012d).

54. 54
Figure 16 Flame C&C Platform(ZHIOUA, 2013)
2.4 Fighting Malware Propagation
(SAAD et al., 2011) shows that malware detection though network traffic behav-
ior has the following advantages:
 It is possible to detect bots during any phase of their life-cycle, and as a
consequence, also detect worms network behavior;
 Has a lower cost than deep packet inspection or honeypot behavior anal-
ysis;
 A bot may be detected during formation phase or through C&C connec-
tion.
On the other hand, (PORRAS, 2009) has presented the challenges faced by
such methods:

55. 55
 Malware can be stealthy and embed its communication protocol on ex-
istent protocols already present in the network, such as HTTPS.
 The communication with a C&C server may take irregular intervals and
with a low rate enough to does not generate significant anomalies on
network traffic;
Several researches have dedicated efforts to detect malware propagation
through traffic analysis (GU et al., 2007, 2009; MANIKOPOULOS; PAPAVASSILIOU,
2002a; SHAHRESTANI et al., 2009; XU; WANG; GU, 2011a; YU et al., 2014).
Gu et al. (XU; WANG; GU, 2011b) proposed a method to cluster end hosts with
similar behavior within the same network prefixes. Bipartite graphs are used to model
the social behavior of end hosts, i. e. with whom a host communicates. A one-mode
projection of the bipartite graph is used to capture social behavior similarity: edges are
used to connect hosts with a same destination or source. Subsequently, a spectral
clustering algorithm discovers inherent behavior within the same network prefix. Fig-
ure 17 presents an example of bipartite graph and the projection with edges connect-
ing nodes with the same source or destination, e.g. a1 and a4 have b4 as destination,
and hence an edge connects them.
Figure 17 An example of (a) bipartite graph and (b) one-mode projection.
Tests were conducted with network traffic available at the Cooperative Associ-
ation for Internet Data Analysis (CAIDA). Scanning activities and a DDOS attack was
detected in the Internet backbone traffic, a worm was also detected in its earlier stage

56. 56
of propagation in a sample with Witty Worm; however, no evidence of performance
was presented considering background traffic in an internal network.
The BotHunter System was proposed by (PORRAS, 2009). Its main objective is
to detect inside hosts trying to propagate infections out. An infection dialog correlation
strategy was modeled as a set of loosely ordered communication flows that are ex-
changed between an internal host and one or more external entities, i.e. bots are mod-
eled as sharing a common set of underlying actions that occur during the infection life
cycle: target scanning, infection exploit, binary egg download and execution, command
and control channel establishment, and outbound scanning. The model is depicted in
Figure 18.
Figure 18 BotHunter System by (PORRAS, 2009)
Experiments were conducted, using Snort rules to detect evidence of direct ex-
ploit detection (E2), binary download (E3) and C&C communication (E4). The rule-set
was specially customized to malware detection, and two preprocessors were added to

57. 57
a Snort configuration, Slade and Spade, in order to detect anomalies such as inbound
scanning (E1). The results presented demonstrated a significant performance in a con-
trolled environment with honeypots, 95,1% of true positive rate and a 4,9% false neg-
ative rate; The experiments in a university campus network were inconclusive, mali-
cious traffic was injected in real background traffic and the detection rate was 100%
for 10 malicious patterns; however, after 4 months 98 malicious patterns were detected
and approximately 61% of these were false positives; Experiments in a production
internal network during 10 days were also inconclusive, a single detection was a false
positive.
2.5 Chapter Summary
In this Chapter the most relevant malware threats, bots and worms, were de-
picted, and their spreading techniques were presented. The modern malwares pre-
sented have demonstrated a continuous evolution in order to evade local host and
traffic detection, the latter using techniques to obfuscate the C&C communication with
botmasters. Moreover, botnets have absorbed autonomous spread techniques from
Trojans and worms, and rootkit capabilities to conceal themselves. However, the tech-
niques to exploit vulnerabilities are common to most of them, and the vulnerabilities
are generally already patched.
Solutions to detect malware through traffic analysis were also presented; how-
ever, they mostly presented positive results when tested in traffic without background
noise which is generated by regular services and network protocol.

58. 58
Chapter 3
Intrusion Detection
and False Alarm
Reduction
This Chapter presents the most relevant methods to reduce false alerts in Intru-
sion Detection.
Due the common flaws and vulnerabilities found in computer systems, even se-
curity mechanisms such as access control and firewalls cannot avoid security
breaches. According to (DENNING, 1987), most existing systems have security flaws
and developing a system absolutely secure is generally impossible. The number of
vulnerabilities reported in the last few years demonstrate that Denning´s statements
still contemporary. Figure 19 presents the number of vulnerabilities with software flaws
reported to the NVD - National Vulnerability Database (NIST, 2014), since 1998. More-
over, 8,495 high severity vulnerabilities were reported since 2010, representing
36.83% among all vulnerabilities reported, and modern malwares take advantage of
such flaws, as discussed in Chapter 2.

59. 59
Figure 19 Vulnerabilities reported do NVD (NIST, 2014).
The discussion from (MCHUGH; FITHEN; ARBAUGH, 2000) and malwares pre-
sented in Chapter 2, shows attackers exploiting most systems through widely known
security vulnerabilities. There are several reasons why administrators may fail to install
software patches:
• Disruption: if a patch installation requires a system reboot, and the service
uptime is crucial, the system administrator may postpone it.
• Unreliability: Software patches are typically released as soon as possible, af-
ter vulnerability is disclosed. The patch may have not been tested enough and causes
severe disruption or even damage to the host systems to which they are applied.
Therefore, the system administrator may choose not to install it and accepts the risk of
a compromise.
• Irreversibility: Most patches are not designed to be easily reversible due to the
ordering of changes that have been made to the system. Once applied, there is often
no easy way of reverse to the original state. This factor increases the risk associated
with applying a patch.
• Unawareness: An administrator may simply miss a patch announcement for
some reason, and therefore be unaware of it, or may have neglected acting on a re-
ceived announcement.
0
1000
2000
3000
4000
5000
6000
7000
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014

60. 60
The number of reported security incidents has grown as well. Figure 20 pre-
sents the number of incidents reported to Cert.br (CERT.BR, 2014) since its creation
in 1999.
Figure 20 Incidents reported to Cert.br (CERT.BR, 2014)
Given this scenario, Intrusion Detection Systems (IDS) have risen as counter-
measures, implemented as hardware or software, able to monitor and report attacks
or attempts to exploit possible flaws (FEITOSA, EDUARDO LUZEIRO, 2010;
HUBBALLI; SURYANARAYANAN, 2014). An intrusion, or malicious activity, is any ac-
tivity that aims to compromise the confidentiality, integrity or availability of computer
systems (MUKHERJEE; HEBERLEIN; LEVITT, 1994)
The idea to monitor user activities with the objective to detect malicious behavior
was first introduced by (DENNING, 1987) and (ANDERSON, 1980) and, since then,
several methods were proposed by security researchers (HUBBALLI; BISWAS;
NANDI, 2011; HUBBALLI; SURYANARAYANAN, 2014; KUMAR, 1995;
MANIKOPOULOS; PAPAVASSILIOU, 2002b; MUKHERJEE; HEBERLEIN; LEVITT,
1994).
IDS are composed of sensors that generate and send events and security alerts
to management stations whenever a malicious activity is detected. Each alert consists
of information describing the attack, such as type of attack, source address and desti-
nation address. Along this chapter, the terms alert and alarm will be used interchange-
ably.

61. 61
The remaining of this Chapter presents the types and classifications of IDS, a
discussion about the major drawback of IDS, regarding the alarm volume and false
alarm rate, and the state-of-the-art of alarm reduction and false alarm minimization.
3.1 IDS Classification
An IDS may be classified following the method used to detect an intrusion and
the data source monitored.
According to the method used (AXELSSON, 2000; FEITOSA, EDUARDO
LUZEIRO, 2010), traditionally IDS can be classified as:
 Signature-based (or misuse-based) – known attacks are described as
signatures, or rules;
 Anomaly-based – deviations from what is considered normal behavior
are classified as malicious;
The former approach considers everything that is known, described in rules, as
malicious, while the later considers the unknown as malicious. Moreover, signatures
describe known attacks but new attacks can be unnoticed, while anomalies may indi-
cate new attacks but new normal behavior can be mistaken as being malicious.
According to the data source, (MUKHERJEE; HEBERLEIN; LEVITT, 1994) de-
fined IDS as:
 Host-based IDS (HIDS) – Monitors the host’s operational system param-
eters and audits trails to detect malicious behavior. Log files, processes
behavior and file system changes may also be monitored.
 Network-based IDS (NIDS) – Monitors network traffic to detect malicious
behavior. A NIDS may be deployed as a passive monitor, collecting traffic
from a switch mirror port or a network tap, or deployed as a bridge with
the capacity to block malicious traffic. According (CHRUN; CUKIER;
SNEERINGER, 2008), when a NIDS is able to block traffic, it’s called
Intrusion Prevention System (IPS).
An HIDS can identify a malicious process or binary file, even evidence of a net-
work attack found in audit trails, but if the host is successfully compromised an attacker
can shut the HIDS process down and/or can use rootkit techniques to conceal itself.

62. 62
An NIDS can detect the host where the malicious traffic came from, but cannot identify
the malicious process; however, if a host is compromised, the NIDS is not affected.
In (VIGNA et al., 2003) a new classification is proposed, the application-based
intrusion detection, which is tightly coupled with an application server, or web server,
and where requests are analyzed before processed.
This dissertation is focused on Signature-based NIDS because it has a lower
false positive rate than anomaly-based (MUKHERJEE; HEBERLEIN; LEVITT, 1994)
and malware detection throughout traffic analysis is discussed as a possible solution
to the problem of malware detection in Chapter 2.
3.2 Problems with DARPA Dataset
Given the research effort to minimize the false positive rate in IDS, as discussed
in Section 3.3, research efforts also have been conducted to evaluate the performance
of IDS, in terms of its detection rate and false positive rate (TJHAI et al., 2008). In 1998
DARPA recognized the need to provide a common dataset to allow comparisons be-
tween different IDS methods. Thus, MIT’s Lincoln Labs was contracted to work with
the Air Force Research Laboratory in Rome, NY to build an evaluation dataset and
perform an evaluation of the then current IDS research being funded by DARPA
(BRUGGER; CHOW, 2005). Since then, DARPA dataset kept the status of default da-
taset to compare the performance of a new IDS strategy with previous researches.
However, several criticisms have raised indicating flaws in the way the dataset
was created, and statistical problems which might make the obtained results by exper-
iments with DARPA dataset unrealistic:
 Statistics used to describe the real traffic and the measures used to es-
tablish similarity are not given (MCHUGH, 2000);
 The taxonomy used in the Lincoln Lab evaluation offers very little support
for developing an understanding of intrusions and their detec-
tion(MCHUGH, 2000);
 Hostile IP packets have a TTL value which is lower by 1 than the back-
ground traffic (MAHONEY; CHAN, 2003)

63. 63
 Several attacks can be detected by anomalies in the TCP window size
field, without a reasonable explanation for why these anomalies should
occur (MAHONEY; CHAN, 2003).
 Only 9 of the possible 256 TTL values were observed in DARPA while
177 different values were observed in real traffic. For TOS, 4 values were
observed in DARPA while 44 values were observed in real traffic
(MAHONEY; CHAN, 2003).
 No fragmented traffic were found in DARPA dataset, the DF (Don’t Frag-
ment) flag was set in all traffic (MAHONEY; CHAN, 2003).
 Only HTTP GET requests were observed in the DARPA dataset
(MAHONEY; CHAN, 2003).
 The majority of malicious connections in the DARPA dataset come from
denial of service attacks and probing activity(BRUGGER; CHOW, 2005);
3.3 False Alarm Generation
The major drawbacks identified in IDS research are the alert volume and the
false alarm rate (JULISCH, KLAUS, 2003; PIETRASZEK; TANNER, 2005b). In fact, it
has been estimated that 99% of alerts are not related to security issues (AXELSSON,
2000). According to (AXELSSON, 2000), the research in process automation indicates
that a human operator will completely lose faith in a device which false alert rate
reaches 50%.
(AXELSSON, 2000) also proposed that the effectiveness of an IDS is affected
by the Bayesian base-rate fallacy. Let 𝐼 and ¬𝐼 denote intrusive and nonintrusive be-
havior, respectively, and 𝐴 and ¬𝐴 denote the presence or absence of an intrusion
alert. Given the conditional probability:
𝑃(𝐴|𝐵) =
𝑃(𝐴) ∙ 𝑃(𝐵|𝐴)
𝑃(𝐵)
(3)
The four possible cases are:
 True positive rate, or detection rate, is the probability 𝑃(𝐴|𝐼);
 False positive rate, or false alarm rate, is the probability 𝑃(𝐴|¬𝐼);
 True negative rate is the probability 𝑃(¬𝐴|¬𝐼);
 False negative rate is the probability 𝑃(¬𝐴|𝐼);

64. 64
Assuming that 1,000,000 packets were analyzed, and only 20 were intrusions,
even with a perfect detection rate of 1.0 and a very low false positive rate on the order
of 10-5 , 33% of alerts will be false positives. With a more realistic detection rate of 0.7,
42% of alerts will be false positives. This shows that building an IDS with a low false
positive rate is, according to (PIETRASZEK; TANNER, 2005b), extremely difficult.
(HUBBALLI; SURYANARAYANAN, 2014) presented general reasons for false
positives generation:
 Intrusion activity sometimes deviates slightly from normal and some
cases are difficult to differentiate.
 A context in which a particular event has happened often decides the
usefulness of the alert. For example, ‘‘Microsoft Distributed Transaction
(MDT)’’ service was vulnerable to the intrusion of large packets, which
was generating a buffer overflow. This triggers a denial of service for the
MDT service. However, this vulnerability was exploitable only in the Win-
dows 2000 operating system which was not patched with latest patches.
 Certain actions which are normal may be malicious under different pre-
vailing circumstances. For example, network scan is normal if done by a
security administrator.
 Many IDS not only detect intrusions but also the number of attempts of
intrusions. An attempt may not necessarily lead to a compromised sys-
tem if the vulnerability does not exist or was corrected.
 An alarm may represent a stage in a multistage attack which may even-
tually fail due to various other reasons.
With regard to signature-based IDS, (HUBBALLI; SURYANARAYANAN, 2014) also
presented the following reasons for false positives:
 Good quality signatures are often difficult to write and their presence is highly
dependent on expert knowledge. An attack may have several variations and if
a signature fails to match a specific attack it is a false negative. If it matches
non-intrusive behavior it is a false positive. As the discovery of new flaws and
vulnerabilities occurs, an expert has to understand the flaw behavior provided
by sufficient data to analyze. Moreover, two conditions may affect the signature
quality:

65. 65
o Analyzing the irrelevant portion of related traffic;
o Analyzing the wrong application data for finding a match.
 The default signatures supplied with most IDS are not customized to the local
network, and a signature which does not threaten the organization, such as an
attack aiming to exploit unavailable services or operational systems, has to be
disabled. This demands expert and infrastructure knowledge.
 Latency in deployment of newly created signatures. The signature database has
to be updated regularly and if this is not the case, poor quality signatures won´t
be replaced by better ones.
Several false alert minimization techniques were surveyed in (HUBBALLI;
SURYANARAYANAN, 2014) and according to the proposed taxonomy, the most rele-
vant and recent, related with this dissertation research, are presented in the following
subsections.
3.3.1 Signature enhancement
Signature enhancement methods enhance regular signatures with context in-
formation. (SOMMER; PAXSON, 2003) and (MASSICOTTE et al., 2007) proposed
signature models with context information, such as the type of host’s operating system
stack. Both obtained satisfactory results with low false positive rate, however, signature
modification is error prone, needs knowledge and experience, and the experiments
were realized with traffic from academic internet links.
3.3.2 Stateful signatures
A stateful IDS stores the state of the network, or previous packet information,
while evaluating a new arriving packet, in other words, a stateful signature is applied
to a full stream of packets instead of a in single packet.
In (ECKMANN; VIGNA; KEMMERER, 2002) an attack language STATL with a
high level specification allows to model multistep attacks and scenarios, using a state
transition model which represents the evolution of an attack’s steps. Experiments have
demonstrated effectiveness using the DARPA dataset, but DARPA has several statis-
tical problems as discussed in Section 3.2.