Download to read offline
More Related Content
Similar to Fabrizio pastore TORACLE-2021 @ESEC/FSE 2021
Fabrizio pastore TORACLE-2021 @ESEC/FSE 2021
- 1. Metamorphic Security Testing forWeb Systems TORACLE @FSE (23/8/2021) based on paper presented at ICST’20 (+ additional results) Phu X. Mai*, Fabrizio Pastore*, Arda Goknil*, Lionel Briand*# * SnT Centre, University of Luxembourg #School of EECS, University of Ottawa
- 2. The Oracle Problem inWeb Systems Security Testing 2 Security vulnerabilities are subtle Discovered when testing with many inputs Specifying expected results is infeasible
- 3.
- 4. Metamorphic Testing Alleviates theOracle Problem • Metamorphic Relations: necessary properties of the program under test in relation to multiple inputs and their expected outputs 4 x1=(G,a,f) ∧ x2=(G,f,a) len(shortPath(x1))=len(shortPath(x2)) a b c d e f f Source input Follow-up input
- 5. Combined with an automatedstrategy to select source inputs, metamorphic relations enable automated testing 5
- 6. Metamorphic Security Testing ofWeb Systems • Source input: a sequence of valid interactions with the system • Follow-up input: generated by altering valid interactions as an attacker would do • Relations: capture properties that hold when the system is not vulnerable 6 {login(Admin), RequestURL(settings_page)} {login(User1), RequestURL(settings_page)}
- 7. Metamorphic Security Testing ofWeb Systems • Source input: a sequence of valid interactions with the system • Follow-up input: generated by altering valid interactions as an attacker would do • Relations: capture properties that hold when the system is not vulnerable 7 {login(Admin), RequestURL(settings_page)} {login(User1), RequestURL(settings_page)} if the follow-up user cannot access the URL from his GUI then the output of the source and follow-up inputs should be different
- 8. Automated Metamorphic Security TestingProcess 8 Catalog of Metamorphic Relations Selected Metamorphi c Relations Translate Metamorphic Relatio ns to Java 2 Execute a Web-crawler 3 Specify/Select Metamorphic Relations 1 Executable Metamorphic Relations in Ja va Source Inputs Data Execute the Metamorphic Testing Algorithm 4 MR1: PASS MR2: FAIL MR3: PASS MR4: PASS Test Result
- 9.
- 10. SMRL Basic Features •Extension of Xbase: • Java-like • Automated generation of Java executables from specifications • SMRL provides seven boolean operators • IMPLIES, AND, OR, TRUE, FALSE, and NOT • EQUAL • Evaluate the equality of two arguments • Define follow-up inputs • enable writing metamorphic relations in declarative form 10
- 11. Data Types 11 InputSequence InputTypes OutputSequence OutputTypes Action 1..* 0..1 SessionUser 0..1 HtmlOutput 1..* Request URL Anchor click Button click Text fill
- 12. 12 Data Representation Functions Data functionDescription Input(int n) Identifies the nth input sequence referred to in a MR. User(int n) Identifies the nth user referred in a MR. User() Identifies the 1st user referred in a MR. Action(int n) Identifies the nth input action referred in a MR. Session(int n) Identifies the nth Web session referred in a MR. Output(Input n) Identifies the sequence of outputs generated by the nth input. Output(Input n, int i) Identifies the output generated by the ith action of the nth input. …
- 13. Web-specific Functions • changeCredentials(Input i, User u) • Modifies an input sequence so that the credentials of the specifi ed user are used in the login action changeCredentials( Input(1), User() ) • Input(1): {login(Admin), RequestURL(settings_page) } • User(): User1 • Result: {login(User1), RequestURL(settings_page) } • cannotReachTroughGUI( User u, String URL) • Returns true if, based on the crawled data, a URL cannot be reached by the given user by exploring the user interface of the system 13
- 14.
- 15. Metamorphic Relation Example: Bypass AuthorizationSchema 15 Data function Boolean operat or Web specific functio n
- 16. Metamorphic Relation Example: Bypass AuthorizationSchema 16 Data function Boolean operat or Web specific functio n
- 17. Metamorphic Relation Example: Bypass AuthorizationSchema 17 Data function Boolean operat or Web specific functio n
- 18. Metamorphic Relation Example: Bypass AuthorizationSchema 18 Data function Boolean operat or Web specific functio n
- 19. Metamorphic Relation Example: Bypass AuthorizationSchema 19 Our metamorphic testing algorithm executes each MR multiple times, to ensure that every possible combination of Data function Boolean operat or Web specific functio n
- 20. Source Inputs ofType InputSequence {login(Admin), RequestURL(home_page)} {login(Admin), RequestURL(settings_page)} {login(User1), RequestURL(home_page)} {login(User1), RequestURL(project1_page)} {login(User2), RequestURL(home_page)} {login(User2), RequestURL(project2_page)} Source Inputs of Type User User1 User2 Admin Tested Combinations Source Inputs Follow-up Inputs {login(Admin), RequestURL(home_page)} + User1 / {login(Admin), RequestURL(home_page)} + User2 / {login(Admin), RequestURL(settings_page)} + User1 {login(User1), RequestURL(settings_page)} {login(Admin), RequestURL(settings_page)} + User2 {login(User2), RequestURL(settings_page)} {login(User1), RequestURL(home_page)} + User2 / {login(User1), RequestURL(project_page)} +User2 {login(User2), RequestURL(project_page)} … …
- 21. Java Integration • Metamorphicrelations are automatically translated into executable Java code • They can be selected in Junit test cases 21
- 22. Deriving Source Inputsfrom Data Crawled with Crawljax
- 23. Deriving Source Inputsfrom Test Scripts • We can derive source inputs from test scripts manually implemented for functional testing • we support Selenium-based scripts • a test script represents an InputSequence 23
- 24. 24 Executable Metamorphic Relation in Java Sourc e Inputs data Metamorphic Testing Algorithm 4 Follow-up input Test result Executor Verifyrelatio n Metamorphic Testing Execution Sourc e input Action1 Action2 Action3 Action1 Action2’ Action3 Follow-up Output Sourc e Outpu t Output1 Output2 Output3 Output1 Output2’’ Output3 MR PASS/FAIL Repeat
- 25. Metamorphic Relations Catalog • Catalog1. Includes 23 Security Metamorphic Relations derived from OWASP testing guidelines 25 Vulnerability type # Metamorphic relations Broken Authentication 6 Sensitive Data Exposure 5 Broken Access Control 8 Security Misconfiguration 3 Vulnerable Components 1 • Catalog 2. Ongoing. Shall cover 45% of vulnerability types listed in the CWE Design Principles view. Implemented 30%.
- 26. Broken Authentication • Objective:verify that a session always changes after a logout • Source input: sequence of actions performed after log-in • Follow-up input: add a logout action after an action of the source InputSequence • Relation: session IDs before and after logout action should be different 26
- 27. Sensitive Data Exposure •Objective: verify that a user cannot retrieve sensitive data by simply altering a URL parameter value • Source input: sequence of actions including at least one action containing a URL parameter (action X) • Follow-up input: change the URL parameter value for the action X (action X’) using a value observed only with other users • Relation: the output of the action X’ either contains an error message or contains data that is not sensitive • e.g., it should contain data that has been already retrieved by the same user in the recorded source inputs 27
- 28. Evaluation • RQ1. (Complementarity)What types of security testing activities not automated by other approaches can be automated by MST? • RQ2: (Completeness) What security vulnerability types can be discovered by MST? • RQ3. (Effectiveness) Is MST effective? 30
- 29. RQ1: What typesof security testing activities not automated by other approaches can be automated by our solution? • Analyzed 90 security testing activities described in OWASP Web security testing book to • identify which existing approach can be adopted to automate them • determine if MT can compensate for remaining testing activities 31
- 30. RQ1: What typesof security testing activities not automated by other approaches can be automated by our solution? • Analyzed 90 security testing activities described in OWASP Web security testing book to • identify which existing approach can be adopted to automate them • determine if MT can compensate for remaining testing activities 32 Test automation strategy # OWASP activities automated Relying on implicit oracle 2 Catalog-based 6 Not testing (e.g., architecture analysis) 19 Vulnerability-specific 22 Not addressed yet 41 Metamorphic testing 16 of 41
- 31. RQ1: What typesof security testing activities not automated by other approaches can be automated by our solution? • Analyzed 90 security testing activities described in OWASP Web security testing book to • identify which existing approach can be adopted to automate them • determine if MT can compensate for remaining testing activities 33 Test automation strategy # OWASP activities automated Relying on implicit oracle 2 Catalog-based 6 Not testing (e.g., architecture analysis) 19 Vulnerability-specific 22 Not addressed yet 41 Metamorphic testing 16 of 41
- 32. RQ2: What security vulnerabilitytypes can be discovered by MST? • Methodology • analyze each vulnerability type (weakness) reported in the Design Principles view of the CWE database (CWE-DP) • 223 weaknesses grouped by 12 design principles • manually inspected the description, few concrete vulnerabilities (CVE), and common attack patterns (CAPEC) associated to it • evaluated the possibility of writing a metamorphic relation to discover the weakness • report the number of weaknesses that can be automatically tested for each of the categories listed in CWE-DP 34
- 33. CWE-DP Weaknesses addressed by MST 35 SecurityDesign Principle # Addressed by MST Audit 6 0 (0%) Authenticate Actors 28 15 (54%) Authorize Actors 60 32 (53%) Cross Cutting 9 5 (56%) Encrypt Data 38 5 (13%) Identify Actors 12 9 (75%) Limit Access 8 3 (38%) Limit Exposure 6 0 (0%) Lock Computer 1 0 (0%) Manage User Sessions 6 4 (67%) Validate Inputs 39 26 (67%) Verify Message Integrity 10 2 (20%) Total 223 101 (45%)
- 34. RQ3 – IsMST effective? • Applied our aproach to software systems affected by known vulnerabilities • Considered all our 23 MRs 36 Vulnerability type # MRs Targeted vulnerabilities EDLAH 2 Jenkins 1.121.1 Joomla 3.8.7 Broken Authentication 6 2 2 Sensitive Data Exposure 5 1 Broken Access Control 8 5 2 Security Misconfiguration 3 1 1 Vulnerable 1
- 35. RQ3 – DataCollection • Data collection: • Executed Crawljax against the three systems for a maximum of five hours or till exploration is exhausted • Implemented test scripts that exercise features not exercised by Crawljax 37
- 36. RQ3 - Measurements •Sensitivity • ratio of vulnerabilities being discovered • Specificity • ratio of follow-up inputs that do not trigger any vulnerability and do not lead to any failure • it measures the time saved from inspecting unwarranted failures • it is the complement of the false positive rate 38
- 37. RQ3 - Sensitivity •We do not discover vulnerabilities that require specific environment settings • Jenkins configuration modified during test execution • Jenkins overloaded Case study Total Vulnerabiliti es Discovered vulnerabilities Sensitivity Crawljax Crawljax & Manual Crawljax Crawljax & Manual EDLAH2 4 3 4 75% 100% Jenkins 8 4 6 50% 75% Total 12 7 10 58.33% 83.33% 39 (ratio of vulnerabilities being discovered)
- 38. RQ3 – Sensitivity (Ratioof vulnerabilities being discovered) 40 Case study Total Vulnerabiliti es Discovered vulnerabilities Sensitivity Crawljax Crawljax & Manual Crawljax Crawljax & Manual EDLAH 2 4 3 4 75% 100% Jenkins 8 4 6 50% 75% Joomla 2 1 2 50 100 Total 14 8 12 57.14% 85.71% • We do not discover vulnerabilities that require specific environment settings • Jenkins configuration modified during test execution • Jenkins overloaded
- 39. RQ3 – Sensitivity (Ratioof vulnerabilities being discovered) • Discovered a new vulnerability in Jenkins • CVE-2020-2162: Stored XSS vulnerability in file parameters 41 Case study Total Vulnerabiliti es Discovered vulnerabilities Sensitivity Crawljax Crawljax & Manual Crawljax Crawljax & Manual EDLAH 2 4 3 4 75% 100% Jenkins 8 4 6 50% 75% Joomla 2 1 2 50 100 Total 14 8 12 57.14% 85.71%
- 40. RQ3 – Specificity (Ratioof follow-up inputs that do not trigger any vulnerability nor lead to failures) 42 Case study Follow-up inputs exercised by MST Follow-up inputs inspected Specificity Crawljax Crawljax & Manual Crawljax Crawljax & Manual Crawljax Crawljax & Manual EDLAH2 832 833 3 4 100% 100% Jenkins 31303 33495 74 176 99.889% 99.893% Joomla 22185 9395 39 43 99.525% 99.573% Total 54320 43723 136 223 99.817% 99.826% • The approach automatically tests the systems with many inputs and the number of inputs to be erroneously inspected is minimal
- 41. AutomatedMetamorphicSecurity TestingProcess 8 Catalogof Metamorphic Relations Selected Metamorphic Relations T ransform MetamorphicRelations toJava 2 Execute a W eb-crawler 3 Specify/Select Metamorphic Relations 1 Executable Metamorphic RelationsinJava Source Inputs Data Execute the MetamorphicT esting Algorithm 4 MR1:PASS MR2:FAIL MR3:PASS MR4:PASS T est Result Experimentsshow High sensitivity and specificity Discovered one unknown vulnerability in Jenkins (CVE-2020-2162) https://sntsvv.github.io/SMRL/
Editor's Notes
- #3 We address the oracle problem in the conext of security testing of web systems. Security testing is affected by the oracle problem because security vulnerabilities are often subtle and They can be discovered only by exercising a Web system with many inputs and despite there are means to automatically generate inputs we lack methods to automatically derive expected results and of course manually specifying expected results for a large number inputs tend to be infeasible For example, to test if authorization priviledges had ben configured properly
- #4 you may need to access a large number of Web pages and resources, ideally all the ones that a user can and cannot access [CLICK] And you may need to repeat it with users having different role and priveledges. ALthough identfying al the pages to be tested might be automated with a web crawler, for each of these pages, you need to manually specify the expected output, which ends up being particulally expensive. In practice, the quality of your testing campaign is limited. This is a problem that goes beyond authrization vulnerabities, as shown by the many vulnerability reports concerning well known web systems and social networks.
- #5 Metamorphic Testing is a solution to alleviate the oracle problem It relies on a set of metamorhic relations that capture properties of the program unde test n relation to inputs andoutputs. I give you an example. if you want to test an algoritm that identifies the shortest path in an acyclic graph. You would express a metamorphic relation as follows. You specify a source input. In this case a vector with a random graph, a randomply selected start node, and a randomly seletct end node. Then you specify how to derive a follow up input from the source input. In this case by swapping the last two elements. The ramianing part captures a property of interest, in this case, the leght of the shortestpath computed for the two inputs should be the same.
- #7 A generic solution to apply Metamorphic Testing to discover security vulnerabilities has never been proposed in the literature. Our objective is to find a generic strategy to apply MST to Web system. We decided to mimick attackers would do. A source input is a sequence of valid interactions with the system. For example, login as admin and then request the settings page A follow-up inputs is generated by altering the interactions as an attacker would do, fr exmple we try the sequence above with anther user And then the reation captures properties of the input and for example GUI accessibility and otputs for example seession For example, to detect autorazation problems we should check if the followng property is invalidated
- #8 basically if User1 does not see a link to the settngs page from his GUI then he should not be able to access its content #it indicates that if a user cannot reach a URL from his GUI probably he should not have access to it
- #9 We developed an approach and framework that automates metamorphic security testing for Web-systems [CLICK] In the first step the engineer can specify a metamorphic relation using a domain specific language that we developed or select the metamorphic relations to used from a catalog we provide [CLICK] The selected metamorphic relations are automatically translated into executable java code. [CLICK] A web-crawler is used to collect source inputs by interacting with the web system under test [CLICK] Then a metamorphic testing algorithm exercise the software under test with Source inputs and follow-up inputs based on the crawled data The test result consit of the outcome of the Mrs
- #10 To specify metemorphic relations, we have defined a security metamophic relation language called smerl
- #11 It is an extion of xbase, it is java-like inherits the automated generaton of executables from specifications It provides seven boolean operators Size are of ovius semantic, IMPLIES, AND, OR, TRUE, FALSE, and NOT The oeperator EQUAL is used both to evaluate the equality of two arguments And to define follow up inputs SMRL enables writing metamorphic relations in declarative form
- #12 I briefly introduce the data types we work wih. An input can be an inputsequence, so a sequence of actions performed with a web system. An action captures an interction with the system. We support all the actions support by our Web crawler, can be … An action can have an associated session And an associated user OutputTypes are The HtmlOutput generated by an action, and a sequence of such output generated with an input sequence.
- #13 We have a number of data representation functions to facilitate the writing of MRs. For example Input of n, identifies the n-th input sequence referred in a MR. Basically the parameter "n" is used as identifier to write the metamorphic relation. User of I the n-th user referred in the metamorphic relation. When we do not provide a number it’s a placeholder for using 1 We have functions for any of our data types.
- #15 I describe the metamrphic relation to detect ypass authorization schema problems
- #16 We iterate over all the actions in an source input sequence referred as Input(1)
- #17 Then we have an impliction
- #18 If a follow-up randomly selected user cannot acces the url of the action
- #19 Then if we define a follow up inpu, where we perform the input sequence ausing the follow-up user
- #20 The output observed for the action in the follow-up and source input shuld be different. Because the follow up user should not have the prviledges and thus see a different content. In this MR we have two source inputs and one follow up input. In general, they can contain any number of source and follow up inputs.
- #21 For example if we have source inputs of type inputsequence covering different interactions between users and the system and we have three source inputs of type user we will test all the combinations except the ones accessing pages already accessed in source inputs
- #22 Our relatons are transated into java code, this enables to define a metamorphic testing test suite with JUNIT.
- #23 We use a modified vesion of Crawljax to collect suce input data Basically, we configure crawljax to cawl the system with multiple users For example when it crawls Jenkins, it logs in, then clicks on some links of the main interface, then another button in the next pahe which loads a log Then it can go back an open other pages. We store the result of the crawlng as a graph. Source InputSequences is a sequnce of actions from the root to the leafs. Souce Users and Sessions are all the user and sessions appearng in the collected data
- #25 Once we have sourc einput data and metamorphic relations we execute our metamorphic testing algorithm it basically generate sequences of actions for the source and follow-up inputs, which are executed, outputs are collected, and the framework verify the relation.
- #26 We provide a catalog of 22 metamorphic relations derived from OWASP testing guidelines, they cover different vulnerability types 6 MRs concern broken authentication 5 Sensitive Data Exposure 7 Broken Access Control 3 Security misconfiguration 1 the detection of vulnerable components We are working on a second catalog based on CWE view
- #27 For example, for broken authentication we want to verify that a session always changes after a logout The source input is any sequence of actions perfrmed after. Login A follow-up input is generated by adding a logout action afer an action of the source INputSequence The relation indicates that the session IDs before and after logout should be different So basicallu, by trying all the available positions for any source input we can maximize the chances of detecting a vulnerability.
- #28 For the case of sensitive data exposre we want to verify tha a user cannot retrieve sensitive data by somply altering a parameter value, for example the ID of the owner of the sensitive data. The source input is a sequence of actions including at least one action with a parameter in the URL. Let’s call it action X. The follow up input is derived by changing the parameter vaue for actin X using a value that has been observed only with other users so there is a chance of collecting data controlled by the use of a specific identifier The relation indicates that action X’ should contain either an error message or data that is not sensitive. Our rule for determining if data is not sensitive is that the data should have been already observed by the same user in recorded source inputs So if we observe content that was never observed by the crawler for a certain user, it is likely that the user is not supposed to access the content rough the user interface
- #29 Otherwise it is likely that data has been altered or exposed
- #30 In this relation we indicate that the second parameter should not lead to any change n the behaviour, so it should be the same
- #32 To answer the first RQ, we analyzed 90 security testing activities described in OWASP Web security testing book to + identify which existing approach can be adopted to automate them + and determine if MR can compensate for testing activities depending on manual oracles
- #33 - The result shows that 41 out of 90 security testing activities are not autoated - Implicit oracle: e.g., testing for buffer overflow is automated by looking for system crashes in response to lengthy inputs - Catalog-based: based on a predefined catalog in which we specify inputs and oracles. E.g., testing for default credentials - No oracle needed: Some activities collect data to reverse engineer the system under test - Manual oracle: Some activities require humans to determine vulnerabilities based on system specifications - Vulnerability-specific approaches. Some activities can be automated by state-of-the-art tools such as Burp Suite (BS) and thus may not necessarily benefit from MT
- #34 16 can be automated by our approach
- #35 To answer the first RQ, we analyze each weakness reported in the CWE database. - We inspected the description, few concrete vulnerabilities (CVE), and common attack patterns (CAPEC) associated to it - Then, we evaluated the possibility of writing a specification capturing an attacker-system interaction following the MCP template to see if we can apply MCP to discover it. - We also evaluated the possibility of writing a metamorphic relation to discover the weakness to know if MST can cover the weakness. - Next, we report the number of weaknesses that can be automatically tested for each of the categories listed in CWE-DP. - An important point is that we study the reasons preventing the application of MCP/MST
- #36 Meanwhile, MST can detect 101 out of 223 weaknesses, or 45% of weaknesses. This number is in line with our analysis in the previous part. 3 out of 12 principles have detectable percentage over 60%. These cases are identify actors, manage user sessions and validate inputs. In comparison between MCP and MST, with MCP most categories have the detection percentage more than ones with MST.
- #37 We aim to apply MST to discover vulnerabilities of three case study systems. - For EDLAH2, we target four vulnerabilities. - For Jenkins and Joomla, we considered the vulnerabilities discovered in the 2nd half of 2018 that are targeted by our approach. Therefore. we target 8 and 2 vulnerabilities of Jenkins and Joomla respectively. - To select source inputs for these case study system + we wxecuted Crawljax against the three systems for a maximum of five hours or till exploration is exhausted + then, we implemented test scripts that exercise features, related to the vulnerabilities, not discovered by the Crawljax. There are 4 manual test scripts in total -- EDLAH2: 4 vulnerabilities Jenkins 2.121.1: 8 vulnerabilities Joomla 3.8.7: 2 vulnerabilities
- #39 Basically it’s an indication of the time saved from inspecting
- #40 Here you can see the total number of vulnerabilities for case study The number of vulnerabilities discovered when we use only the crawler t derive source inputs and when we cmbine both the crawler and the manually written test scripts. When we have manual inputs we discover more vulnerabilities. In terms of specificity, we have an extremely high specificity 83% when we combine crawljax and manual inputs. It means that we discver 83% of the vulnerabilities. Also when using only crwaljax however results are promising, indeed we discover more than 50% of the vulnerabilitis. Whch means that we discover vulnerabilities without manual effort.
- #41 - We executed each MR two times on each case study, one with the source inputs including only crawled data, another one with the source inputs including crawled data and manual test scripts. - 3 out of 4 vulnerabilities of EDLAH2, 4 out of 8 vulnerabilities of Jenkins, and 1 out of 2 vulnerabilities of Joomla were discovered in the case using only crawled data as inputs. - If we use the source inputs constructed from crawled data plus manual scripts, MST discovers 4 all vulnerabilities of EDLAH2, 6 out of 8 vulnerabilities of Jenkins, and 2 all vulnerabilities of Joomla. - In total, MST discovered 57% vulnerabilities if using only crawled data as inputs, and 85,7% vulnerabilities if using crawled data with manual scripts.
- #42 Moreover, during running test, we discovered a new vulnerability in Jenkins. We reported this vulnerability to Jenkins, and it has been listed as a CVE in the MITRE database. The high sensitivity shows that MST is feasible to apply to web security testing.
- #43 We executed all 22 generic metamorphic relations on 3 case studies, and one additional dedicated to Joomla. - In general, the specificity is extreme high, 99,82%. - For the case of EDLAH2, the specificity is 100%, about 99,89% for the case of Jenkins, and about 99.5% for the case of Joomla. - This shows that MST economize too much time engineers uses to check false alarms. Indeed, in the case using crawled data plus manual test scripts, MST executed 43723 follow-up inputs and raised 223 alarms. Only 77 out of 223 alarms are false positive. This means that only a negligible fraction of follow-up inputs inspected lead to false alarms
- #44 Concerning specificity, I report fist the number of follo-up inputs that we generate. This number is extremely high for both, mor ethan 5000. We have very few flse positives 33 and 32, manual inputs enable us to test features not discovered by crawljax for certan users and thus eliminate false positives consequently the number of follo-up inputs that either lead to true alarms or no alarms is consequently high. To summarize, specifity is above 99%. Basically, the approach automatically tests the two systems with many inputs and the number of inputs to be erronously inspected is minimal.
- #45 We address th eoracle problem in Web Systems’ Security Testing We have developed a method and a framework that automates security testing based on metamorphic relations, plus we provide a metamorphic relations catalog