Zero wall detecting zero-day web attacks through encoder-decoder recurrent neural networks
•
0 likes•33 views
This document describes a zero-day web attack detection system called ZeroWall. It uses an encoder-decoder neural network trained on over 1.4 billion normal requests to learn patterns in web traffic. For new requests, it acts as a "self-translation machine" by encoding the request and trying to decode it back. It uses BLEU metrics to quantify how well the request translates - anomalous requests that cannot be translated are flagged as potential attacks. In experiments on real world traffic, ZeroWall detected 28 types of zero-day attacks comprising over 100,000 requests, while having low overhead when deployed with an existing signature-based web application firewall.
Recommended
Recommended
More Related Content
Similar to Zero wall detecting zero-day web attacks through encoder-decoder recurrent neural networks
Similar to Zero wall detecting zero-day web attacks through encoder-decoder recurrent neural networks (20)
More from Danbi Cho
More from Danbi Cho (11)
Recently uploaded
Recently uploaded (20)
Zero wall detecting zero-day web attacks through encoder-decoder recurrent neural networks
- 2. Content 1. Background 2. Introduce 3. Idea 4. Model 5. Experiments 6. Summary #Kookmin_University #Natural_Language_Processing_lab. 1
- 3. Background 1) WAF(Web Application Firewall) : Detecting and Blocking Web attacks (ex: SQL Injection, Cross-Site Scripting(XSS)) - Signature-based detection : one-to-one response method - Rule-based detection : signature + AI method #Kookmin_University #Natural_Language_Processing_lab. 2 https://m.blog.naver.com/pentamkt/222084745829
- 4. Background 2) Zero-day attack : Attacks vulnerability in computer software (ex: spear phishing) 3) Whitelist : Trusted target list ( blacklist) #Kookmin_University #Natural_Language_Processing_lab. 3
- 5. Background 4) Recurrent Neural Network (RNN, LSTM) #Kookmin_University #Natural_Language_Processing_lab. 4 https://ratsgo.github.io/natural%20language%20processing/2017/03/09/rnnlstm/
- 6. Background 4) Recurrent Neural Network (RNN, LSTM) #Kookmin_University #Natural_Language_Processing_lab. 5 https://ratsgo.github.io/natural%20language%20processing/2017/03/09/rnnlstm/
- 7. Introduce > Why a zero-day attack is difficult to detect? 1) Have not been previously seen 2) Can be carried out by a single malicious HTTP request 3) Very rare within a large number of Web requests #Kookmin_University #Natural_Language_Processing_lab. 6 > Most supervised approaches are inappropriate > Contextual information is not helpful > Collective and statistical information are not effective ZeroWall - Unsupervised approach (work with an existing WAF in pipeline) - Detecting a zero-day Web attack hidden in an individual Web request
- 8. Introduce > What we want? 1) WAF detects those known attacks effectively 2) ZeroWall detects unknown attacks ignored by WAF rules #Kookmin_University #Natural_Language_Processing_lab. 7 > Filter out known attacks > Report new attack patterns to operators and security engineers to update WAF rules
- 9. Idea 1) HTTP request: string following HTTP 2) Most requests are benign, malicious requests are rare #Kookmin_University #Natural_Language_Processing_lab. 8 > Consider an HTTP request as one sentence in the HTTP request language > Train a kind of language model based on historical logs, to learn this language from benign requests Historical Web logs Language Model Train One request Can understand Cannot understand Benign Malicious
- 10. Idea : Self-Translate Machine > How to learn this ‘Hyper-Text’ language? : Use Neural Network Translation model to train a Self-Translate Machine - Encoder: encode the original request into one representation - Decoder: decode it back #Kookmin_University #Natural_Language_Processing_lab. 9
- 11. Idea : Self-Translate Machine > How to quantify the self-translation quality(anomaly score)? (Translation Quality = Anomaly Score) An attack detection problem = A machine translation quality assessment problem : use machine translation metrics #Kookmin_University #Natural_Language_Processing_lab. 10 Historical Web logs Self-Translate Machine Train One request Good Translation Bad Translation Benign Malicious
- 12. Idea : Self-Translate Machine Translation Quality = Anomaly Score : Use BLEU(Bi-Lingual Evaluation Understudy) (Malicious Score = 1-BLEU_score) #Kookmin_University #Natural_Language_Processing_lab. 11 𝐵𝐿𝐸𝑈 = 𝐵𝑃 × exp 𝑛=1 𝑁 𝑤𝑛 𝑙𝑜𝑔𝑝𝑛 𝐵𝑃 = ቐ 1, 𝑖𝑓 𝑐 > 𝑟 𝑒 1− 𝑟 𝑐 , 𝑖𝑓 𝑐 ≤ 𝑟 𝑝𝑛 = σ𝑛−𝑔𝑟𝑎𝑚∈𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 𝐶𝑜𝑢𝑛𝑡𝑐𝑙𝑖𝑝(𝑛 − 𝑔𝑟𝑎𝑚) σ𝑛−𝑔𝑟𝑎𝑚∈𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 𝐶𝑜𝑢𝑛𝑡(𝑛 − 𝑔𝑟𝑎𝑚)
- 14. Model: ZeroWall > Offline Periodic Retraining : build and update vocabulary and re-train the model #Kookmin_University #Natural_Language_Processing_lab. 13 1 2 3
- 15. Model: ZeroWall 1) Building Vocabulary #Kookmin_University #Natural_Language_Processing_lab. 14 Raw log Bag of Words Vocabulary Filtering - Stop words - variables
- 16. Model: ZeroWall 2) Parsing #Kookmin_University #Natural_Language_Processing_lab. 15 Vocabulary
- 17. Model: ZeroWall > Online Detecting : Detect anomalies in real-time requests for manual investigation #Kookmin_University #Natural_Language_Processing_lab. 16 1 2 3 4
- 18. Model: ZeroWall 1) Parsing = parsing in offline training 2) Translation #Kookmin_University #Natural_Language_Processing_lab. 17 translation
- 19. Model: ZeroWall 3) Detection 4) Investigate (1) BLEU metrics (2) Threshold [Larger? Yes ⇨ Go to step 3; No ⇨ Benign] (3) Check whitelist [Not in whitelist? Yes ⇨ Go to step 4; No ⇨ Benign] (4) Investigation [True Attacks ⇨ Update WAF/IDS; False Alarms ⇨ Update whitelist rules] #Kookmin_University #Natural_Language_Processing_lab. 18 Original sequence(token sequence) vs. Translated sequence(recovered token sequence) 1 2 3 4
- 20. Experiments > Data - 8 real world trace from an internet company - Over 1.4 billion requests in a week > Overview - Captured 28 different types of zero-day attacks - Contribute to 141,583 of zero-day attack requests in total #Kookmin_University #Natural_Language_Processing_lab. 19 # B2M: Ratio of Benign to Malicious (in WAF) # B2Z: Ratio of Benign to Zero-Day
- 21. Experiments > Baseline & labels 1) Unsupervised Approaches 2) Supervised Approaches #Kookmin_University #Natural_Language_Processing_lab. 20 > SAE(stacked auto-encoder), HMM and DFA(Deterministic Finite Automata) > Use data filtered by WAF as training set > CNN, RNN and DT > Use all data (allowed/dropped) as training set and WAF results as labels
- 22. Experiments > Evaluation Results #Kookmin_University #Natural_Language_Processing_lab. 21
- 23. Experiments > Zero-Day case - These attack is detected by ZeroWall, CNN, and RNN - WAF are usually based on keywords, e.g., eval, request, select, and execute - ZeroWall is based on the “understanding” of benign requests - The structure of this zero-day attack request is more like a programming language #Kookmin_University #Natural_Language_Processing_lab. 22 ZeroWall CNN and RNN
- 24. Experiments > Whitelist - To mitigate False Alarms - The numbers of whitelist rules refer to “how many whitelist rules are added each day”, based on the FPs labeled on that day (No rules applied on 0602 since it is the first day of testing set) - The results shows that the whitelist reduces the number of FPs with low overhead (Numbers of rules are very small) - Based on these results, we believe ZeroWall is practical in real-world deployment #Kookmin_University #Natural_Language_Processing_lab. 23
- 25. Summary 1) Present a zero-day web attack detection system ZeroWall 2) Deployed in the wild #Kookmin_University #Natural_Language_Processing_lab. 24 - Augmenting existing signature-based WAFs - Use Encoder-Decoder Network to learn patterns from normal requests - Use Self-Translation Machine & BLEU metrics - Over 1.4 billion requests - Captured 28 different types of zero-day attacks (100K of zero-day attack requests) - Low overhead