Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

564 views

Published on

Presentation at SRII 2012, San Jose, CA, USA, July 24-27, 2012

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
564
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • On Imminent Threats to Learning-based Business Automation
  • Automated business processes with machine learning technologies, where machines make decisions in place of human operators, are coming into practical use [3, 11, 12, 17]. They are being used even for mission-critical operations such as credit and loan decision making [16] and fraud detection [7, 2]. While misclassifications are hard to avoid in practice, automation-based business is still viable even with limited classification accuracy by machines, thanks to business models that can cover the losses from rare mistakes with the profits from the large savings in human costs and reductions in routine losses due to relatively reliable automation.
  • Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

    1. 1. IBM Research – TokyoOpportunistic Adversaries– On Imminent Threats toLearning-based Business Automation – Michiaki Tatsubori, IBM Research – Tokyo Shohei Hido, Preferred Infrastructure, Inc.M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    2. 2. IBM Research – TokyoAbout This Talk § A business process with automated decision through machine learning is useful & promising § The “opportunistic adversaries” – potential adversaries exploiting its misclassification, which is inevitable – A case study with loan exam automation § A reference design & implementation of counter measures2 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    3. 3. IBM Research – TokyoBusiness Processes with Machine Learning– a Promising Approach An Example of Credit Card Fraud Detection BPM ML Order Validation Order parameter Transparent Transparent Fraud Decision Decision Detection Report parameter Service Service (e.g. exception) Models Models Exception? Exception Yes handling No Induce models No Yes Order accepted? Learning Training parameter Learning Service Service & decision record History History Process Repository Repository Review Order Order process histories Order rejection histories3 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    4. 4. IBM Research – TokyoPotential Application: Loan Exam Processing4 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    5. 5. IBM Research – TokyoPotential Application: Insurance Claims Processing5 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    6. 6. IBM Research – TokyoSupervised Machine Learning is the Key Technology D training = {( x1 , y1 ),..., (x n , yn )}§ Machine learning for where x i ∈ V (V : feature - vector space) and y j ∈ C (C : a set of class labels) process automation: – Learning from known Learning Data: decisions for input Approve parameters Distinction by – Allowing automated a ground-truth function (unknown) decision for unknown input parameters Models Models Ex. Insurance claim processing, credit order approval, etc. Reject6 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    7. 7. IBM Research – TokyoInevitable Misclassifications are Compensatedby Other Benefits Produce a function § Hard to avoid misclassifications h:x → y – Tradeoffs between false where x ∈ V (V : feature - vector space) positives versus false and y ∈ C (C : a set of class labels) negatives Test Data: § Overall business models can Approve compensate loss from FP Distinction by misclassifications with benefit a learned function (probabilistic) from automation: – Less human workload – Less careless misses FN Reject7 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    8. 8. IBM Research – TokyoOpportunistic Adversaries: Threats by AdversariesOutsmarting Machinery Misjudgment – Opportunistic adversaries Test Data: scenario: Approve • A user detects the misclassification by the FP FP FP FP FP system for certain input parameters • Attackers provide parameters so that they resemble the former input parameters FN misclassified Ex. A manual for “legally Reject cheating insurance claims”8 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    9. 9. IBM Research – TokyoConditions Where Opportunistic Adversaries BecomeThreats § Threat: Damages from spreading adversaries which outsmart inevitable false positives/negatives with ML, under the condition: – Attacks intentionally forge inputs (integrity attack), – Attacks start from a tiny false positive/negative case revealed to potential attackers (exploratory and indiscriminate attack), and – Unawareness of damages (stealthy attack) § Existing works didn’t address this situations or required impractical amount of learning and test samples – Transfer learning [Sugiyama 2006] – Adversarial learning [Lowd 2005] – Outlier detection [Hido 2008]9 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    10. 10. IBM Research – TokyoBPM & Abstract Decision Service + Anomaly Detection BPM Decision Service Order Validation Order parameter Transparent Transparent Fraud Decision Decision Detection Report parameter Service Service (e.g. exception) Models Models Exception? Exception Yes handling No Rule Rule Repository Repository No Yes Order accepted? History of History of Process Automated Automated Review Decisions Order Decisions Input Frequencies Input Frequencies10 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    11. 11. IBM Research – TokyoReference Countermeasure Prototype Outline § Record timestamps of training and test Record timestamps of input data: inputs A1 § Cluster training inputs to segmentalize Class A the input space into subclasses A2 A3 § Maintain frequency statistics about per-subclass probabilities of training time inputs for various times and timeframes Class B and test inputs for recent times and timeframes § Detect significant relative increases in distribution Time series analysis each subclass as anomaly to alert (telling for subclass probabilities as an exception) 1 2 3 log t – Sensing potential attacks outsmarting the trained model Score : – Giving a chance of human review Ps( test ) (l ) and model update q( x (test) )= E ( Pt ( training ) (l )) (σ ( Pt ( training) (l )) + 1) k where s = t k and l = g( xk ) (test) (test)11 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    12. 12. IBM Research – TokyoArchitecture of Reference Implementation Training Input Test Input label A B B A Time time t1 t2 t3 t4 stamp s1 s2 s3 s4 stamp Classification Output Classifier Classifier A B B Generator s1 s2 s3 classifier Sub-classifier Time Series Generator Analyzer (Test Data) sub-classifier Time Series Analyzer distribution distribution frequency (Training Data) 1 2 3 log t statistics 1 2 3 log t (test data) frequency Anomaly statistics detector notify anomaly (training data)12 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    13. 13. IBM Research – TokyoPreliminary Experimental Results§ Observed effectiveness in an Learning Data: Attack experiment with spam filtering – Experimented with Spambase (mails with some spams) in UCI data Test Data: Clusters – Used first 80% for training and last 20% for testing – Replaced 5% of testing data with misclassified Clusters Freq. Ratio / Std. Dist. samples – Observed they are Detected detected as anomaly13 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    14. 14. IBM Research – TokyoConcluding Remarks § Defined “Opportunistic Adversaries” as a threat to automated business Approve processes with machine learning FP FP FP FP FP – Integrity, exploratory, indiscriminate, and stealthy attacks FN § A reference solution architecture Reject proposed – + anomaly detection in temporal input space distribution statistics14 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation
    15. 15. IBM Research – TokyoThank you!Questions?M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation

    ×