Building a Cloud Computing Analysis System for Intrusion Detection System DATE:4/14/09 Wei-Yu Chen, Yao-Tsung Wang National Center for High-Performance Computing, Taiwan {waue,jazz}@nchc.org.tw

Taiwan Introduction

NCHC Introduction The NCHC is responsible for Taiwan’s cyberinfrasructure, R&D in HPC and networking applications 3 Business Units located in science parks to support local high tech industry The NCHC integrates information, engineering and scientific disciplines The NCHC provides computing, networking and storage services National Center for High-performance Computing

Outline Motivation The IDEA Architecture Procedure Results Pros and Cons Conclusions

Motivation

The IDEA

Architecture

Procedure

Results

Pros and Cons

Conclusions

Current Situation of IDS Intrusion Detection System IDS (IDS) Detecting unwanted attempts at accessing, manipulating or disabling of computer systems through Internet. IDS Detect Rate “ false positive” “ false negative” Accuracy = top mission of IDS ?

Intrusion Detection System IDS (IDS)

Detecting unwanted attempts at accessing, manipulating or disabling of computer systems through Internet.

IDS Detect Rate

“ false positive”

“ false negative”

Accuracy = top mission of IDS ?

Alerts Alert is produced when IDS detect something as malicious. Two method of alert storage A Text Log -> terrible In Database -> mostly

Alert is produced when IDS detect something as malicious.

Two method of alert storage

A Text Log -> terrible

In Database -> mostly

What’s the problem about Alert ? Enormous Data  less Efficient Ignore the crucial information easily !!! Got Nothing if the database were crash

Enormous Data  less Efficient

Ignore the crucial information easily !!!

Got Nothing if the database were crash

Our Motivation To resolve above problems come with huge amount of anomaly information generated by IDS

To resolve above problems come with huge amount of anomaly information generated by IDS

Our IDEA - ICAS ICAS, IDS Cloud Analysis System Applying Cloud Computing technique Improve higher performance of analysis reducing redundancy Merge relation

ICAS, IDS Cloud Analysis System

Applying Cloud Computing technique

Improve higher performance of analysis

reducing redundancy

Merge relation

System Architecture ICAS Overview

System Architecture SNORT is an open source network intrusion prevention and detection system The most widely deployed intrusion detection Snort

SNORT is an open source network intrusion prevention and detection system

The most widely deployed intrusion detection

System Architecture Apache Hadoop Core is a software platform that lets one easily write and run applications that process vast amounts of data. Inspired by Google's MapReduce and Google File System (GFS) papers Implements MapReduce and Hadoop Distributed File System (HDFS) Operates <key, value> pairs Hadoop

Apache Hadoop Core is a software platform that lets one easily write and run applications that process vast amounts of data.

Inspired by Google's MapReduce and Google File System (GFS) papers

Implements MapReduce and Hadoop Distributed File System (HDFS)

Operates <key, value> pairs

System Architecture HBase is the Hadoop database An open-source, distributed, column-oriented store modeled after the Google paper, BigTable HBase

HBase is the Hadoop database

An open-source, distributed, column-oriented store modeled after the Google paper, BigTable

System Architecture Regular Parser Parsing original snort log and transfer to HDFS (hadoop file system) Analysis Procedure Dispatch job if pool is not empty and insert the result into database Data Mapper <key, value> mapping Data Reducer <“key1”, value1…valueN> <“key2”, value1…valueN> Four Components

Regular Parser

Parsing original snort log and transfer to HDFS (hadoop file system)

Analysis Procedure

Dispatch job if pool is not empty and insert the result into database

Data Mapper

<key, value> mapping

Data Reducer

<“key1”, value1…valueN>

<“key2”, value1…valueN>

Program Procedure

Alert Integration Procedure

Key - Values The victim IP addresses A unique ID used to identify attack method in Snort rules The time when the attack was launghed TCP/IP protocol Attack was lunched from this port Victim ports The IP address where malicious one launghed attack

Alert Merge Example 6007,6008 5002 4077,5002 T5 T4 T1,T2,T3 53 443 80,443 tcp, udp tcp tcp Sip3,Sip4,Sip5 ,Sip6 D.D.O.S. Host_3 Sip1 Trojan Host_2 Sip1,Sip2 Trojan Host_1 Values Key T5 tcp 6008 53 Sip6 D.D.O.S Host_3 T5 udp 6007 53 Sip5 D.D.O.S Host_3 T5 tcp 6008 53 Sip4 D.D.O.S Host_3 T5 udp 6007 53 Sip3 D.D.O.S Host_3 T4 tcp 5002 443 Sip1 Trojan Host_2 T3 tcp 5002 443 Sip1 Trojan Host_1 T2 tcp 4077 80 Sip2 Trojan Host_1 T1 tcp 4077 80 Sip1 Trojan Host_1 Timestamp Packet Protocol Source Port Destination Port Source IP Attack Signature Destination IP

Experiment Environment Machine: CPU : Intel quad-core, Memory : 2g, OS : Linux : Ubuntu 8.04 server Software Hadoop : core 0.16.4 Hbase : 0.1.3 Java : 6 Alerts Data Sets MIT Lincoln Laboratory, Lincoln Lab Data Sets Computer Security group at UCDavis, tcpdump file

Machine:

CPU : Intel quad-core, Memory : 2g,

OS : Linux : Ubuntu 8.04 server

Software

Hadoop : core 0.16.4

Hbase : 0.1.3

Java : 6

Alerts Data Sets

MIT Lincoln Laboratory, Lincoln Lab Data Sets

Computer Security group at UCDavis, tcpdump file

Experimental Result The Consuming Time of Each Number of Data Sets

Experimental Result Throughput Data Overall

Pros & Cons Legible Efficient Scalable Economical Reliable Non-realtime Latency immature

Legible

Efficient

Scalable

Economical

Reliable

Non-realtime

Latency

immature

Conclusions v.s. Future Works ICAS supplies a efficient way to analyze and merge huge number of alerts based on cloud platform. Including more IDS logs The best final result is graphical Prepare more large-scale and complete experiment

ICAS supplies a efficient way to analyze and merge huge number of alerts based on cloud platform.

Including more IDS logs

The best final result is graphical

Prepare more large-scale and complete experiment

Thank You ! & Question ? DATE:4/14/09