Downloaded 27 times
Uploaded byLionel Briand
Automated Vulnerability Testing Using Machine Learning and Metaheuristic Search
The document discusses automated vulnerability testing of web applications using machine learning and metaheuristic search. It discusses three key areas: 1. Testing front-end web applications for XML injection vulnerabilities by automatically generating malicious XML messages and using search-based techniques to find input strings that allow the generation of these messages. 2. Testing web application firewalls (WAFs) using machine learning to learn attack patterns from successful and blocked attacks to generate new attacks, and a multi-objective genetic algorithm to automatically repair vulnerable WAF rule sets. 3. Using machine learning to detect malicious SQL statements at the database level by training on legitimate and attack SQL logs and classifying new statements as legitimate or attacks.
More Related Content
Similar to Automated Vulnerability Testing Using Machine Learning and Metaheuristic Search
Automated Vulnerability Testing Using Machine Learning and Metaheuristic Search
- 1. .lusoftware verification &validation VVS Automated Vulnerability Testing Using Machine Learning and Metaheuristic Search PI: Lionel Briand Researchers: Annibale Panichella, Cu Nguyen, Nadia Alshahwan PhD Students: Dennis Appelt, Sadeeq Jan 1
- 2. Code Injection Manipulated datastructures Collect and analyze information Indicator Employ probabilistic techniques Manipulate system resources Subvert access control Abuse existing functionality Engage in deceptive interactions 2 % 2 % 3 % 3 % 3 % 4 % 9 % 32 % 42 % X-Force Threat Intelligence Index 2017 2 https://www.ibm.com/security/xforce/ More than 40% of all attacks were injection attacks (e.g., SQLi)
- 3. Web Applications 3 Server SQLDatabaseClient
- 4. Web Applications 4 Web form str1 str2 Username Password OK SQLquery SELECT * FROM Users WHERE (usr = ‘str1’ AND psw = ‘str2’) Name Surname … John Smith … Result Server SQL DatabaseClient
- 5. Injection Attacks 5 SQL query NameSurname … Aria Stark … John Snow … … … … Query result SELECT * FROM Users WHERE (usr = ‘’ AND psw = ‘’) OR 1=1 -- Server SQL DatabaseClient Web form ‘) OR 1=1 -- Username Password OK
- 6.
- 7.
- 8. Testing Challenges • Allprotection layers need to be tested • No single layer can possibly block all attacks • They need to be effective together • Testing is extensive: Large input space • Different test techniques for different layers • Many types of vulnerabilities 8
- 9.
- 10. Testing the Front-end(XMLi) 10 Front-end System XML I1 I2 In Generated XML Messages Back-end Systems System 1 System 2 System n Input Strings
- 11. Security Mechanisms inFront-end Web Applications • Input Sanitization: rejects inputs containing malicious characters (e.g., <) • Input Validation: converts malicious inputs to valid ones (e.g., deleting XML tags) • Other transformation: domain specific transformation (e.g., JSON to XML, calculating age) 11 Front-end System XML I1 I2 In Generated XML Messages Back-end Systems System 1 System 2 System n Input Strings
- 12. Testing of theFront-end WAs 12 Does the front-end system (SUT) allow the generation of XML injection attacks? YES The front-end is vulnerable NO The front-end is secure
- 13. Testing of theFront-end WAs 13 Front-end System XML I1 I2 In Generated XML Messages Back-end Systems System 1 System 2 System n <user> <username>Tom</username> <password>m1U9q10</password> <role>user</role> <mail>role=Adm+ tom@uni.lu</mail> </user> Step 1: Create malicious XML messages Step 2: Verify whether the SUT can generate them Malicous XML message Search for Input String
- 14. Step 1: GeneratingMalicious Messages Grammar-based Generation: automatically generating malicious messages for different type of XML injection attacks 14 Our tool SOLMI (ISSTA'16) Example of message generated by SOLMI
- 15. Step 2: Searchingfor Input Strings 15 Front-end System XML I1 I2 In Generated XML Messages Back-end Systems System 1 System 2 System n <user> <username>Tom</username> <password>m1U9q10</password> <role>user</role> <mail>role=Adm+ tom@uni.lu</mail> </user> Malicous XML message Candidate Input String The front-end web application (SUT) is a black-box The search space is very huge: all possible input strings (I1, .., In)
- 16. Step 2: Searchingfor Input Strings 16 Evaluation Selection Crossover Mutation Search Algorithm Initial Solutions Random Strings Front-end System I1 I2 In Generated Messag Email:“role=Adm” +tom@uni.lu Usr: Tom Psw: m1U9q10
- 17. Step 2: Searchingfor Input Strings 17 Evaluation Selection Crossover Mutation Search Algorithm Initial Solutions Random Strings Front-end System I1 I2 In Generated Messag Email:“role=Adm” +tom@uni.lu Usr: Tom Psw: m1U9q10 Target Edit Distance XMLXML
- 18. Step 2: Searchingfor Input Strings 18 Evaluation Selection Crossover Mutation Search Algorithm Initial Solutions Random Strings Front-end System I1 I2 In Generated Messag Email:“role=Adm” +tom@uni.lu Usr: Tom Psw: m1U9q10 XML XML XML XML New Input Strings
- 19. Some Results 19 (W/ validat.)(W/o validat.) (open source) (Industrial) %CoveredXMLiMessage 0 25 50 75 100 SBANK SSBANK XMLMAO R M RealCoded GA Standard GA Hill Climbing Random Search (Industrial)
- 20.
- 21. Web Application Firewalls(WAFs) 21 Servermalicious malicious malicious legitimate WAF
- 22. WAF Rule Set 22 Ruleset of Apache ModSecurity https://github.com/SpiderLabs/ModSecurity
- 23.
- 24. Anatomy of SQLiattacks 24 ‘ OR“a”=“a”# Bypassing Attack <START> <sq> <wsp> <sqliAttack> <cmt> <boolAttack> <opOR> <boolTrueExpr> OR <bynaryTrue> <dq> <ch> <dq> <opEq> <dq> <ch> <dq> “ a ” = “ a ” <sQuoteContext> ‘ #_ Decomposition Tree ‘ _ OR”a”=“a” # S = { Attack Slices
- 25. Learning Attack Patterns 25 S1S2 S3 S4 … Sn Outcome A1 1 1 0 0 … 0 Passed A2 0 1 0 0 … 0 Blocked … … … … … … … … Am 1 1 1 1 … 1 Blocked Training Set Sn PassedBlocked S4 YesNo YesNo YesNo S3 S1 S2 … Decision Tree
- 26. Learning Attack Patterns 26 S1S2 S3 S4 … Sn Outcome A1 1 1 0 0 … 0 Passed A2 0 1 0 0 … 0 Blocked … … … … … … … … Am 1 1 1 1 … 1 Blocked Sn PassedBlocked S4 YesNo YesNo YesNo S3 S1 S2 … Training Set Decision Tree Attack Pattern S2 ∧ ¬ Sn ∧ S1
- 27. Machine Learning Sn PassedBlocked S4 YesNo YesNo YesNo S3 S1 S2 … Generating Attacksvia ML and EAs 27 Prepare Training Data Build Classifier Mutate best attacks Execute new attacks Slice attacks Initial Attacks (μ+λ) Evolutionary Algorithm
- 28. Some Results Apache ModSecurity 28 ApacheModSecurity • ML techniques outperform random technique • ML-Driven E superior to other ML techniques DistinctAttacks Industrial Case Industrial WAFs DistinctAttacks Machine Learning-driven attack generation led to more distinct, successful attacks being discovered
- 29.
- 30. Rule Set Customization 30 Customizationis error-prone: •Complex filter rules •Limited time and resources •Lack of automated tools Rule customization is necessary: •To protect from new threats •To avoid false positives
- 31. Fixing Vulnerable WAFs 31 SQLiAttacks Attacks Decomposition Machine Learning (DT) Attack Generation Process Attack Patterns
- 32. Fixing Vulnerable WAFs 32 SQLiAttacks Attacks Decomposition Machine Learning (DT) New Regular Expressions Existing Rule Set Fixed Rule Set # Blocked Attacks # Blocked Legitimate Request
- 33. Multi-Objective Optimization 33 Problem: selectinga subset of the regular expressions produced by Decision Tree such as to (1) maximizing the recall (blocked attacks) and (2) minimizing the false positive rate.Recall False Positive Pareto Front
- 34.
- 35. Multi-Objective Genetic Algorithms 35 R1R2 R2 R4 … Rk 1 1 0 0 … 0 0 1 1 1 … 1 Initial Solutions Evaluation Selection Crossover Mutation NSGA-II Initial Solutions Solutions are evaluated and selected according to the Pareto Optimality
- 36. Some Results 36 Target WAF: ModeSecurity OWASPCore Rule Set Target Operation: doPayment() # Attacks = 1234 # Benign Req = 1567 Hypevolume(NSGAII) >Hypevolume(RS)
- 37. Hypervolume Results 37 Hypervolume 0,00 0,25 0,50 0,75 1,00 Op1 Op2Op3 Op3 NSGA-II Random Hypervolume 0,00 0,25 0,50 0,75 1,00 doPayment expireTicket simulate- Payment NSGA-II Random ModSecurity Industrial WAF
- 38. Detecting Malicious SQL Statementsat Database Level
- 39. Using ML toDetect SQLi Statements 39 SQL egitimate cution Logs Parsing Pruning Edit distance Training Phase L mate n Logs Parsing Pruning Edit distance Clustering Training Phase QL timate ion Logs Parsing Pruning Edit distance Clustering Training Phase Parsing Pruning Edit distance Clustering ng Phase Parsing Pruning Edit Distance Clustering SQL Legitimate Execution Logs Phase 1: Training SQL Security Testing Logs Parsing Pruning Testing Phase Classification SQL Security Testing Logs SQL Legitimate Execution Logs Parsing Pruning Edit distan Training Phase SQL Legitimate Execution Logs Parsing Pruning Edit distance Clu Training Phase Parsing Pruning Phase 2: Testing (Detection)
- 40. Detection Phase 40 Clustering Incoming SQL Statement1 Incoming SQL Statement 2
- 41. Detection Phase 41 Incoming SQL Statement1 Clustering Incoming SQL Statement 2 APPROVE REJECT
- 42. Some Results 42 SUT TestGen. Recall False Positive HotelRS Xavier 100% 0 % SugarCRM Xavier 100% 0% 0% TaskFreak Burpsuite 100% 0% 0%SqlMap 100% 0,1 % TheOrganizer Burpsuite 100% 0,6 % SqlMap 100% 0,3 % Wordpress-newstat Burpsuite 100% 0,2 % SqlMap 100% 0,2 % Wordpress-landingpage SqlMap 100% 0,1 %
- 43. Publications Automatic Generation ofTests to Exploit XML Injection Vulnerabilities in Web Applications. Jan, Sadeeq; Panichella, Annibale; Arcuri, Andrea; Briand, Lionel. To appear in IEEE Transaction on Software Engineering (TSE), 2017 A Machine Learning-Driven Evolutionary Approach for Testing Web Application Firewalls. Appelt, Dennis, Nguyen, Duy Cu, Panichella, Annibale, Briand, Lionel. To appear in IEEE Transaction on Reliability (TR) Automatically Repairing Web Application Firewalls Based on Successful SQL Injection Attacks. Appelt, Dennis; Annibale Panichella; Briand, Lionel. In IEEE 28th International Symposium on Software Reliability Engineering (ISSRE 2017) , Toulouse, France. Search-based Testing Approach for XML Injection Vulnerabilities in Web Applications Jan, Sadeeq; Nguyen, Duy Cu; Andrea, Arcuri; Briand, Lionel. Proc. of the 10th IEEE International Conference on Software Testing, Verification and validation (ICST 2017), Tokyo, Japan Automated and Effective Testing of Web Services for XML Injection Attacks Jan, Sadeeq; Nguyen, Duy Cu; Briand, Lionel. In Proc. the International Symposium on Software Testing and Analysis (ISSTA 2016), Saarbrücken, Germany SOFIA: An Automated Security Oracle for Black-Box Testing of SQL-Injection Vulnerabilities Ceccato, Mariano; Nguyen, Duy Cu; Appelt, Dennis; Briand, Lionel. In Proceedings of the 31th IEEE/ACM International Conference on Automated Software Engineering (ASE 2016) 43
- 44. Publications Known XML VulnerabilitiesAre Still a Threat to Popular Parsers and Open Source Systems Jan, Sadeeq; Nguyen, Duy Cu; Briand, Lionel. In The 2015 IEEE International Conference on Software Quality, Reliability & Security (QSR 2015), Vancouver, Canada Behind an Application Firewall, Are We Safe from SQL Injection Attacks? Appelt, Dennis; Nguyen, Duy Cu; Briand, Lionel. In Proc. of the 8th International Conference on Software Testing, Verification, and Validation (ICST 2015) Automated Testing for SQL Injection Vulnerabilities: An Input Mutation Approach Appelt, Dennis; Nguyen, Duy Cu; Briand, Lionel; Alshahwan, Nadia. In Proc. of the International Symposium on Software Testing and Analysis (ISSTA 2014) 44
- 45. .lusoftware verification &validation VVS Automated Vulnerability Testing Using Machine Learning and Metaheuristic Search PI: Lionel Briand Researchers: Annibale Panichella, Cu Nguyen, Nadia Alshahwan PhD Students: Dennis Appelt, Sadeeq Jan 45