Multiple Classifier Systems for Adversarial  Classification Tasks
Upcoming SlideShare
Loading in...5
×
 

Multiple Classifier Systems for Adversarial Classification Tasks

on

  • 1,218 views

Pattern classification systems are currently used in security applications like intrusion detection in computer networks, spam filtering and biometric identity recognition. These are adversarial ...

Pattern classification systems are currently used in security applications like intrusion detection in computer networks, spam filtering and biometric identity recognition. These are adversarial classification problems, since the classifier faces an intelligent adversary who adaptively modifies patterns (e.g., spam e-mails) to evade it. In these tasks the goal of a classifier is to attain both a high classification accuracy
and a high hardness of evasion, but this issue has not been deeply investigated yet in the literature. We address it under the viewpoint of the choice of the architecture of a multiple classifier system. We propose a measure of the hardness of evasion of a classifier architecture, and give an analytical evaluation and comparison of an individual classifier and a classifier ensemble architecture. We finally report an experimental evaluation on a spam filtering task.

Statistics

Views

Total Views
1,218
Views on SlideShare
1,212
Embed Views
6

Actions

Likes
0
Downloads
10
Comments
0

4 Embeds 6

http://www.slideshare.net 2
http://www.linkedin.com 2
https://twitter.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Multiple Classifier Systems for Adversarial  Classification Tasks Multiple Classifier Systems for Adversarial Classification Tasks Presentation Transcript

    • Multiple Classifier Systems for Adversarial Classification Tasks Battista Biggio, Giorgio Fumera and Fabio Roli Dept. of Electrical and Electronic Eng., University of Cagliari
    • Overview  Adversarial classification  An approach to evaluate the hardness of evasion  Comparison of classifier architectures: single classifier vs MCS − analytical comparison − experimental comparison
    • Traditional pattern recognition problems Physical / logical Feature Classification process measurement
    • Adversarial classification problems Physical / logical process: legitimate samples Feature Classification measurement Adversary: malicious samples
    • Adversarial classification: previous works  Not related to concept drift  Analysis of specific vulnerabilities, proposal of specific defence strategies − Globerson and Roweis, ICML 2000 − Perdisci et al., ICDM 2006 − Jorgensen et al., JMLR 9, 2008 − Wittel and Wu, CEAS 2004 − Lowd and Meek, CEAS 2005  Theoretical frameworks − Dalvi et al., KDDM 2004 − Lowd and Meek, KDDM 2005
    • Design of pattern recognition systems Data Feature Model Classification acquisition extraction selection Goal in “traditional” applications: maximise accuracy
    • Design of pattern recognition systems Data Feature Model Classification acquisition extraction selection Goal in “traditional” applications: maximise accuracy Data Feature Model Classification acquisition extraction selection Goal in adversarial classification tasks: maximise accuracy and hardness of evasion
    • Design of pattern recognition systems Data Feature Model Classification acquisition extraction selection Goal in “traditional” applications: maximise accuracy Data Feature Model Classification acquisition extraction selection Goal in adversarial classification tasks: maximise accuracy and hardness of evasion
    • Hardness of evasion x1 ≥ 0: malicious y Î {malicious, legitimate} ... ... Decision function + < 0: legitimate xn th
    • Hardness of evasion Expected value of the minimum number of features the adversary has to modify to evade the classifier (worst case: the adversary has full knowledge on the classifier)‫‏‬ x1 ≥ 0: malicious y Î {malicious, legitimate} ... ... Decision function + < 0: legitimate xn th
    • Hardness of evasion: an example Expected value of the minimum number of features the adversary has to modify to evade the classifier x = (1 1 0 1 0) x1 = 1 0.3 x2 = 1 0.8 ≥ 0: malicious x3 = 0 3.0 + x4 = 1 1.5 < 0: legitimate x5 = 0 1.0 th = 2
    • Hardness of evasion: an example Expected value of the minimum number of features the adversary has to modify to evade the classifier x = (1 1 0 1 0) x1 = 1 0.3 x2 = 1 0.8 ≥ 0: malicious x3 = 0 3.0 + x4 = 1 1.5 < 0: legitimate x5 = 0 1.0 th = 2
    • Hardness of evasion: an example Expected value of the minimum number of features the adversary has to modify to evade the classifier x = (0 1 1 0 0) x1 = 0 0.3 x2 = 1 0.8 ≥ 0: malicious x3 = 1 3.0 + x4 = 0 1.5 < 0: legitimate x5 = 0 1.0 th = 2
    • Hardness of evasion: an example Expected value of the minimum number of features the adversary has to modify to evade the classifier x = (0 1 1 0 0) x1 = 0 0.3 x2 = 1 0.8 ≥ 0: malicious x3 = 1 3.0 + x4 = 0 1.5 < 0: legitimate x5 = 0 1.0 th = 2
    • Comparison of two classifier architectures x1 x2 w1 w2 ... wn t xn X xi Î {0,1}
    • Comparison of two classifier architectures x1 X1 ... x2 w1 t1 w2 X2 ... OR ... t2 wn ... t xn XN ... tN X xi Î {0,1} X1 È X2 È ... È XN = X Xi Ç Xj = Æ i ¹ j ,
    • Comparison of two classifier architectures x1 X1 ... x2 w1 t1 w2 X2 ... OR ... t2 wn ... t xn XN ... tN X xi Î {0,1} X1 È X2 È ... È XN = X x1, x2,..., xn i.i.d. identical weights Xi Ç Xj = Æ i ¹ j , t1 = t2 =...= tn, |Xi| = n/N
    • Comparison of two classifier architectures Details are in the paper p1A = 0.25 p1L = 0.15
    • Comparison of two classifier architectures Details are in the paper p1A = 0.25 p1L = 0.15
    • Comparison of two classifier architectures ROC working point: C=1 min (C×FP + FN)‫‏‬ C = 1, 2, 10, 100 C=2 C = 10 C = 100
    • Experimental set-up  SpamAssassin filter (open source)‫‏‬ − linear classifier: weighted sum of about N = 900 binary- valued (0/-1 or 0/1) features (tests)‫‏‬  TREC 2007 e-mail data set − 25,220 legitimate, 50,199 spam (April-July 2007)‫‏‬  Classifier architectures − linear classifier: standard SpamAssassin (linear SVM for weight computation)‫‏‬ − MCS: logical OR of N linear SVM classifiers (N = 3, 10) trained on disjoint feature subsets (identical size, random subdivision)‫‏‬ − working point: minimize FN, FP ≤ 1%
    • Experimental results
    • Conclusions  Adversarial classification tasks: accuracy and hardness of evasion  An approach for evaluating the hardness of evasion of decision functions  Multiple Classifier Systems: potentially useful to improve the hardness of evasion