Survey on Intrusion Detection System based on Entropy MEthods IEEE Papers Raj Kamal IIT Guwahati June 8, 2012
Table 1: Entropy Based IEEE PapersTittle Author Year Abstract ThemeAn Eﬃcient and Giseop No† 2009 In this paper, we pro- Uses fast entrpoy andReliable DDoS and Ilkyeun pose a fast entropy scheme moving average to cal-Attack Detection Ra. De- that can overcome the is- cualte entropy.If networkUsing a Fast En- partment of sue of false negatives and traﬃc changes from nor-tropy Computation Computer will not increase the com- mal to abnormal statusMethod Science and putational time. Our sim- such as when the DDoS Engineering. ulation shows that the attacker sends a bulk of University fast entropy computing packets with the same of Colorado method not only reduced port number to saturate a Denver USA. computational time by certain port, the entropy more than 90 % compared of this port number will be to conventional entropy, decreased. By contrast, but also increased the under normal conditions, detection accuracy com- the entropy of the port pared to conventional and number will be increased. compression entropy ap- This phenomenon can be proaches. applied to various net- work information such as source IP address, desti- nation IP address, source port, destination port, to- tal number of packets, and even in the data cluster- ing schemes. our Fast Entropy scheme reduced computational time by 90 /of conventional entropy scheme while maintaining detection accuracy. Fast Entropy is even faster than compression entropy scheme in computing en- tropy values with same or better detection accu- racy. For our future work, we have been developing an adaptive fast entropy algorithm that will fur- ther reduce the false posi- tives as well as false nega- tives without adding over- head by introducing dy- namic moving average and detection threshold value with respect to behavior of attacks. 1
Table 2: Entropy Based IEEE PapersTittle Author Year Abstract ThemeEﬀective Discovery Chan- 2009 This IDS is based on the We implemented theof Attacks using Kyu Han notion of packet dynam- proposed algorithm usingEntropy of Packet Hyoung- ics, rather than packet perl and ran it on realDynamics Kee Choi content, as a way to traﬃc traces available on Sungkyunkwan cope with the increasing the Internet. We used University complexity of attacks. four traces containing ﬁve We employ a concept of malicious attacks: they entropy to measure time- are Code Red Worm, variant packet dynamics Witty Worm, Slammer and, further, to extrapo- Worm, DoS and DDOS late this entropy to detect attacks.Here thermody- network attacks. The namic approach is used entropy of network traﬃc with moving average . It should vary abruptly once further uses ROC curve the distinct patterns of to ﬁnd out thershold. packet dynamics embed- ded in attacks appear. The proposed classiﬁer is evaluated by comparing independent statistics de- rived from ﬁve well-known attacks. Our classiﬁer detects those ﬁve attacks with high accuracy1 and does so in a timely man- ner For instance, a Denial of Service (DoS) attack and ﬂash crowds cause destination hosts to con- centrate the distribution of traﬃc on the victim. Network scanning has a dispersed distribution for destination hosts and a bottleneck distribution for destination services. This bottleneck distribution is concentrated on the vulnerable ports. Con- centration and dispersion are, respectively, two pat- terns of packet dynamics frequently perceived in a DoS attack and network scanning. The key idea is that once abnormal traﬃc contaminates long-term behavior, the entropy value of the system should 2 immediately reﬂect this contamination.This de- tection method takes advantage of ﬂuctua- tions in the entropy values of ﬂow-related metrics.Bogus requests do not generate immediate
Table 3: Entropy Based IEEE PapersTittle Author Year Abstract ThemeEntropy-Based Tsern-Huei 2009 we present an entropy- In this paper, we pro-Proﬁling of Net- Lee , Jyun- based network traﬃc posed a novel, two-stagework Traﬃc for De He proﬁling scheme for de- approach for detectingDetection of Secu- Department tecting security attacks. network attacks. Inrity Attack of Com- The proposed scheme the ﬁrst stage, normal munication consists of two stages. behavior proﬁles are Engineering The purpose of the ﬁrst constructed based on National stage is to systematically Relative Uncertainty. In Chiao Tung construct the probability the second stage, the Chi- University distribution of Relative Square Goodness-of-Fit ,Taiwan Uncertainty for normal Test is performed for the network traﬃc behavior. distributions obtained In the second stage, from behavior proﬁling we use the Chi-Square and network activities Goodness-of-Fit Test, a collected online. We calculation that measures demonstrated the eﬀec- the level of diﬀerence of tiveness of our proposed two probability distribu- scheme with the KDD tions, to detect abnormal 1999 dataset for DoS at- network activities. The tacks. Simulation results probability distribution of show that our proposed the Relative Uncertainty scheme achieves lower for short-term network complexity and higher behavior is compared accuracy than previous with that of the long- schemes. Based on the term proﬁle constructed experimental results, we in the ﬁrst stage. We believe that the proposed demonstrate the perfor- scheme could be a good mance of our proposed choice for network behav- scheme for DoS attacks ior proﬁling and attack with the dataset derived detection. from KDD CUP 1999. Experimental results show that our proposed scheme achieves high accuracy if the features are selected appropriately. The top six features ranked by the accuracy are srcbytes,dstbytes, srvdiﬀhos- trate,dsthostcount,dsthostsamesrcportrate and dsthostsrvdiﬀhos- trate.These features can be used to detect DoS attacks eﬀectively. 3
Table 4: Entropy Based IEEE PapersTittle Author Year Abstract ThemeEntropy-Based Shui Yu 2008 A community network we focus on detection ofCollaborative De- and Wanlei often operates with the DDoS attacks in commu-tection of DDOS Zhou School same Internet Service nity networks. Our mo-Attacks on Com- of Engi- Provider domain or the tivation comes from dis-munity Networks neering and virtual network of dif- criminate the DDoS at- Information ferent entities who are tacks from surge legiti- Technol- cooperating with each mate accessing, and iden- ogy Deakin other. In such a federated tify attacks at the early University, network environment, stage, even before the at- Burwood, routers can work closely tack packages reaching the VIC 3125, to raise early warning target server. The en- Australia of DDoS attacks to void tropy of ﬂows at a router, catastrophic damages. router entropy, is calcu- However, the attackers lated, if the router entropy simulate the normal is less than a given thresh- network behaviors, e.g. old, then a attack alarm pumping the attack is raised; the routers on packages as Poisson the path of the suspected distribution, to disable ﬂow will calculate the en- detection algorithms. tropy rate of the suspected We noticed that the ﬂow. If the entropy rates attackers use the same are the same or the diﬀer- mathematical functions ence is less than a given to control the speed of value, then we can conﬁrm attack package pumping that it is an attack, other- to the victim. Based wise, it is a surge of legit- on this observation, the imate accessing. diﬀerent attack ﬂows of a DDoS attack share the same regularities, which is diﬀerent from the real surging accessing in a short time period. We apply information theory parameter, entropy rate, to discriminate the DDoS attack from the surge legitimate accessing. We proved the eﬀectiveness of our method in theory, Here number of packets to diﬀerent destinations are used. 4
Table 5: Entropy Based IEEE PapersTittle Author Year Abstract ThemeLow-Rate DDoS Yang Xiang, 2011 A low-rate distributed de- we propose two new andAttacks Detection Member, nial of service (DDoS) at- eﬀective information met-and Traceback by IEEE, Ke Li, tack has signiﬁcant ability rics for low-rate DDoS at-Using New Infor- and Wanlei of concealing its traﬃc be- tacks detection: general-mation Metrics Zhou, Senior cause it is very much like ized en- tropy and in- Member, normal traﬃc. An infor- formation distance met- IEEE mation metric can quan- ric. The experimental re- tify the diﬀerences of net- sults show that these met- work traﬃc with various rics work eﬀectively and probability distributions. stably. They out- per- In this paper, we innova- form the traditional Shan- tively propose using two non entropy and Kull- new information metrics back–Leibler distance ap- such as the generalized en- proaches, respectively, in tropy metric and the in- detecting anomaly traﬃc. formation distance metric In particular, these met- to detect low-rate DDoS rics can improve (or match attacks by measuring the the various re- quirements diﬀerence between legit- of) the systems’ detection imate traﬃc and attack sensitivity by eﬀectively traﬃc. The proposed adjusting the value of or- generalized entropy met- der of the generalized en- ric can detect attacks sev- tropy and information dis- eral hops earlier than the tance metrics. As the traditional Shannon met- proposed metrics can in- ric. The proposed in- crease the information dis- formation distance met- tance (gap) between at- ric outperforms the pop- tack traﬃc and legitimate ular Kullback–Leibler di- traﬃc, they can eﬀectively vergence approach as it detect low-rate DDoS at- can clearly enlarge the tacks early and reduce the adjudication distance and false positive rate clearly. then obtain the op- timal The pro- posed informa- detection sensitivity. The tion distance metric over- experimental results show comes the properties of that the proposed infor- asymmetric of both Kull- mation metrics can ef- back–Leibler and informa- fectively detect low-rate tion diver- gences. Fur- DDoS attacks and clearly thermore, the proposed IP reduce the false positive traceback scheme based rate. Furthermore, the on information metrics proposed IP traceback al- can eﬀectively trace all gorithm can ﬁnd all at- attacks until their own tacks as well as at- tackers LANs (zombies). In from their own local area conclusion, our proposed networks (LANs) and dis- infor- mation metrics can card attack packet substantially improve the performance of low-rate DDoS attacks detection 5 and IP traceback over the tra- ditional approaches.
Table 6: Entropy Based IEEE PapersTittle Author Year Abstract ThemeJoint Entropy Hamza 2009 Network traﬃc charac- In this paper, we haveAnalysis Model Rahmani, terization with behaviour proposed statistical ap-for DDoS Attack Nabil Sahli, modelling could be a proach for DDoS attacksDetection Farouk good indication of attack detection. Our experi- Kammoun detection witch can be ences were made on a real CRISTAL performed via abnormal traﬃc ﬂow issued from Lab., Na- behaviour identiﬁcation. a “CAIDA data collec- tional School Moreover, it is hard to tion” collected in 2007. for Com- distinguish the diﬀerence Our proposed approach is puter Sci- of an unusual high volume based on the evaluation ences of of traﬃc which is caused of the degree of coherence Tunis Uni- by the attack or occurs between the received traf- versity when a huge number of ﬁc volume and the num- campus users occasionally ac- ber of connections per Manouba cess the target machine time interval with the aim Manouba, at the same time. We of thresholding calculated Tunisia observe that the time distances between a cur- series of IP-ﬂow number rent observation window and aggregate traﬃc size and a given reference. The are strongly statistically main contribution of this dependant. The occur- paper is that our proposal rence of attack aﬀects this model allows us to identify dependence and causes DDoS attacks regardless a rupture in time series of the traﬃc volume size. of joint entropy values. A legitimate augmenta- Experiment results show tion at large scale will not that this method could be detected through this lead to more accurate method which minimising and eﬀective DDoS de- false alarms. In addition, tection.We propose a our proposal does need to measurement method inspect few ﬁelds for each which focuses on quan- packet. This makes it sim- tifying the information pler and more practical for expressed by the joint real-time implementation. system of two random variables in traﬃc-based network. By measuring the degree of coherence between the number of packets and the number of IP-ﬂow ﬁrst obtained in regular traﬃc, then in traﬃcs presenting a large variety of anoma- lies including mainly legitimate anomalies, we can diﬀerentiate traﬃc changes caused by ﬂash crowd (FC) or by DoS 6 attack. This method allows reducing signiﬁ- cantly the false positives alarms. To study the network characteristics by generating the histogram of the size of IP-ﬂow during a timeinterval T.
Table 7: Entropy Based IEEE PapersTittle Author Year Abstract ThemeA Network Ya-ling 2009 A new network anomaly The RETAD sets upAnomaly De- Zhang, detection method has SVLNM by training thetection Method Zhao-guo been proposed in this normal network traﬃc.Based on Relative Han, Jiao- paper. The main idea of The network anomalyEntropy Theory xia Ren the method is network detection system based School of traﬃc is analyzed and es- on RET is achieved by Computer timated by using Relative comparing SVLD with Science and Entropy Theory (RET), SVLNM. The test results Engineering and a network anomaly show that the detection Xi’an Uni- detection model based on rate of RETAD is higher versity of RET is designed as well. than the EMERALD, Technology The numerical value of PHAD, ALAD, NETAD Xi’an, China relative entropy is used and FAD. The RETAD to alleviate the inherent has three advantages. contradictions between Firstly, algorithm compu- improving detection rate tation is so easy that it and reducing false alarm can be used to the high rate, which is more pre- speed network. Secondly, cise and can eﬀectively the method has a strong reduce the error of es- detection capability, es- timation. On the 1999 pecially for the detection DARPA/Lincoln Labo- of intermittent anomalies. ratory IDS evaluation In addition, the RETAD data set, the detection has a good adaptability. results showed that the Based on RET, the packet method can reach a length has been chose higher detection rate at as measures to detect the premise of low false anomaly. Furthermore, alarm rate.These mea- the detection models sures have three features: using other measures need compose a full-probability to be further studied. event and cover all gath- ered information;be able to comprehensively reﬂect a variety of abnormal that cause the abnor- mal network traﬃc;does not contain sensitive information, such as IP address, port number or packet content informa- tion. Packet Lengths are taken into account to calculate relative entropy and drawing conclusions. 7
Table 8: Entropy Based IEEE PapersTittle Author Year Abstract ThemeAn Approach on Zhiwen 2011 In this paper we propose The test data set withDetecting Network Wang, Qin an approach on detecting more alerts is used to eval-Attack Based on Xia De- network attack based on uate our method. ThereEntropy partment of entropy from millions of are 166,326 alerts in the Computer alerts. Shannon entropy test data. 9.83them are Science and is developed ﬁrstly to ana- generated by 86 network Technol- lyze the distribution char- attack occurs within 430 ogy Xi’an acteristics of alert with seconds. We successfully Jiaotong ﬁve key attributes includ- detect all the attacks with University ing source IP address, 2 false detections.In this Xi’an, China destination IP address, paper, we proposed a new source threat, destina- network attack detection tion threat and datagram method base on entropy. length. Then, the Renyi Five features of IDS alerts cross entropy is employed are selected from tens of to fuse the Shannon en- Snort alert attributions. tropy vector and detect The Shannon entropy is the anomalies. The IDS used to analyze the alerts used in our experiment is to measure the regularity Snort, and the experimen- of current network status. tal results based on actual The Renyi cross entropy network data show that is employed to detect net- our approach can detect work attack. The Renyi network attack quickly cross entropy value is near and accurately. In this 0 when the network runs paper, Snort is used to in normal, otherwise the monitor the network and value will change abruptly ﬁve statistical features of when attack occurs. The the Snort alert are se- experimental results un- lected: source IP address, der actual data show that destination IP address, the framework in our work source threat, destina- can detect network attack tion threat and datagram quickly and accurately. In length. The Shannon en- next step, more alerts tropy is used to analyze from diﬀerent time seg- the distribution character- ments will be collected to istics of alert that reﬂect test our method and an at- the regularity of network tack classiﬁcation method status. When the moni- will be considered. tored network runs in nor- mal way, the entropy val- ues are relatively smooth. Otherwise, the entropy value of one or more fea- tures would change. The Renyi cross entropy of these features is calculated to measure the network status and detect network 8 attacks. Time series is cal- culated based on shannon entropy and which is used to calculate renny entropy and compared with previ- ous and alarm is generated based on thereshod.
Table 9: Entropy Based IEEE PapersTittle Author Year Abstract ThemeDetecting DDoS Yun Liu 2010 After analyzing the The results demonstrateAttacks Using Con- ,Jieren characteristics of DDoS that TFCE is more ro-ditional Entropy Cheng,Jianping attacks and the existing bust of the interference of Yin,Boyun approaches to detect background traﬃc. The Zhang DDoS attacks, a novel reason lies in the fact School of detection method based that the corresponding re- Computer, on conditional entropy lations between traﬃc fea- National is proposed in this pa- tures are considered here. University per. First, a group of TFCE compute the rele- of Defense statistical features based tive distribution between Technology on conditional entropy is traﬃc features and include Changsha, deﬁned, which is named the information of joint China Traﬃc Feature Condi- probilities of traﬃc fea- tional Entropy (TFCE), tures, so has stronger abil- to depict the basic charac- ity to uncover the diﬀer- teristics of DDoS attacks, ence of attack traﬃc and such as high traﬃc vol- normal traﬃc. ume and Multiple-to-one relationships. Then, a trained support vector machine (SVM) classiﬁer is applied to identify the DDoS attacks. We experiment with the MIT Data Set in order to evaluate our approach. The results show that the proposed method not only can distinguish between attack traﬃc and normal traﬃc accurately, but also is more robustness to resist disturbance of back- ground traﬃc compared with its counterparts. Sr- cIP,DestIP,DestPort are taken into account.Then use three conditional entropy and sip sip dip H( ), H( )H( ) dip dport dport to characterize three kinds of multiple-to-one rela- tion in DDoS attacks, namely, called Traﬃc Fea- ture Conditional Entropy (TFCE).This measure the 9 diversity of sip to dip,sip to dport, dport to dip,or their uncertainity. After we include SVM into pic- ture ,train it with same set of factors and used it to detect real time anamoly.
Table 10: Entropy Based IEEE PapersTittle Author Year Abstract ThemeA New Relative Jin 2010 Distributed Denial of Ser- This paper analyzes theEntropy Based Wang,Xiaolong vice (abbreviated DDoS) application layer DDoSApp-DDoS Detec- Yang Keping attack is a serious problem and proposes a new rel-tion Method Long Re- to the network services. ative entropy based app- search This paper analyzed some DDoS detection method. Center for solutions to the appli- We validate our method Optical cation layer DDoS (ab- by simulation, and the Internet breviated app-DDoS) at- results suggest that our Mobile In- tack, and proposed a rel- method can be used to fonnation ative entropy based app- detect app-DDoS attacks. Network, DDoS detection method. This paper validates the University Our scheme includes two usefulness of the relative of Electronic stages: learning stage and entropy based app-DDoS Science detection stage. Firstly at detection method. Our Technology the learning stage, it ex- future work will focus on of China, tracts main click features how to handle false detec- Chengdu of web objects with the tion. Sichuan cluster methods. Then 610056,China. at the detection stages, it Network computes the relative en- Center of tropy for each session ac- Chengdu cording to the learning re- University, sult. The greater the ses- Chengdu sion’s relative entropy, the Sichuan more suspicious the ses- 610106, sion is. At last, simula- China tion results suggest that this method can diﬀerenti- ate the attack session with high detection rate and low false alarm. 10
Table 11: Entropy Based IEEE PapersTittle Author Year Abstract ThemeEntropy-based Suratose 2010 The most common type of In summary, an entropy-Input-Output Tritilanunt, DoS attack occurs when based technique providesTraﬃc Mode De- Suphannee adversaries ﬂood a large more accurately denial-of-tection Scheme for Sivakorn, amount of bogus data service detection than aDoS/DDoS Attacks Choochern to interfere or disrupt volume-based technique. Juengjin- the service on the server. Moreover, the detecting charoen, Au- By using a volume- time to discover both sanee Siri- based scheme ,packe- long- term and short- pornpisan trate,bandwidth,packetsize term denial-of-service Computer to detect such attacks, attacks in our scheme Engineering this technique would not is another key strength Department, be able to inspect short- over a feature-based de- Faculty of term denial-of- service tection approach. These Engineering, attacks, as well as cannot two major advantages Mahidol distinguish between heavy are supported by the University, load from legitimate users experimental results as Thailand and huge number of bogus demonstrated in this sec- 25/25, messages from attackers. tion.Short term and long Salaya, As a result, this paper term attacks are detected. Phutta- provides a detection monthol, mechanism based on a Nakorn- technique of entropy- pathom, based input-output traﬃc Thailand, mode detection scheme. 73170 The experimental re- sults demonstrate that our approach is able to detect several kinds of denial-of-service attacks, even small spike of such attacks. This paper uses entropy of packet size to detect attacks. 11
Table 12: Entropy Based IEEE PapersTittle Author Year Abstract ThemeEntropy Based Laleh Ar- 2011 In this paper we present a The point is that asSYN Flooding shadi Amir novel approach for detect- the arrival rate decreasesDetection Hossein ing SYN ﬂooding attacks the packets become less Jahangir by investigating the en- dependent and the en- Computer tropy of SYN packet inter- tropy increases as a re- Engineering arrival times as a mea- sult whereas an increase Department sure of randomness. We in the arrival rate re- Sharif Uni- argue that normal SYN sults in more dependency versity of packets are almost inde- between the packets and Iran Tehran, pendent leading to higher a decrease in the en- Iran values of entropy while tropy consequently. There SYN ﬂooding attacks con- are two major challenges sist of a high volume of faced by the anomaly de- related SYN packets and tection techniques. First so the entropy of their is the problem of deﬁn- inter-arrival times would ing a general rule for be less than normal. We the distinction of normal apply this entropy-based and anomalous traﬃc and method on diﬀerent data the second is the high sets of network traﬃc both volume of the processing in oﬀ-line and real-time data. We see that our modes. In this paper we entropy based detection examine the changes in technique can easily over- the entropy of inter-arrival come both challenges by times of TCP SYN pack- investigating the random- ets to detect SYN ﬂood- ness of TCP SYN packets’ ing attacks. Our ex- inter-arrival times. While periments are based upon deriving the SYN pack- this argument that nor- ets, extracting their inter- mal SYN packets are al- arrival times and comput- most independent leading ing the entropy is not com- to higher values of en- putationally intensive and tropy while SYN ﬂooding can easily be performed attacks consist of many in real-time As for fu- related SYN packets sent ture work it may be use- from either the same ori- ful to observe the entropy gin to various destinations of other ﬂow inter-arrival or from multiple sources times, e.g. TCP-SYN- to a single destination and ACK, TCP- ACK, TCP- consequently the entropy RST, UDP or ICMP pack- of their inter-arrival times ets. In case the entropy would be less than normal. changes as an anomaly oc- curs, it would be possible to identify the anomalous portions of the traﬃc in the same way we detect the SYN ﬂooding attacks 12