Metamorphic testing is proposed to address the oracle problem in web security testing. Relations capture necessary properties between multiple inputs and outputs that must hold when a system is not vulnerable. Experiments on commercial and open source systems show the approach has high sensitivity (58.33-83.33%) and specificity (99.43-99.50%), detecting vulnerabilities without many false alarms. Extensive experiments with 22 relations achieved similar results for Jenkins and Joomla.

1. .lusoftware verification & validation
VVS
Metamorphic Security Testing
for Web Systems
Phu X. Mai*, Fabrizio Pastore*, Arda Goknil*, Lionel Briand*#
* SnT Centre, University of Luxembourg
#School of EECS, University of Ottawa

2. The Oracle Problem
in Web Systems Security Testing
2
Security vulnerabilities are subtle
Discovered when testing with many inputs
Specifying expected results is infeasible

3. 3

4. Metamorphic Testing
Alleviates the Oracle Problem
• Metamorphic Relations: necessary properties
of the program under test in relation to
multiple inputs and their expected outputs
4
x1=(G,a,f) ∧ x2=(G,f,a) len(shortPath(x1))=len(shortPath(x2))
a
b
c
d
e
f
fSource
input
Follow-up
input

5. Combined with
an automated strategy
to select source inputs,
metamorphic relations enable
automated testing
5

6. Metamorphic Security Testing
of Web Systems
• Source input: a sequence of valid interactions with the system
• Follow-up input: generated by altering valid interactions as an
attacker would do
• Relations: capture properties that hold when the system is not
vulnerable
6
{login(Admin), RequestURL(settings_page)}
{login(User1), RequestURL(settings_page)}

7. Metamorphic Security Testing
of Web Systems
• Source input: a sequence of valid interactions with the system
• Follow-up input: generated by altering valid interactions as an
attacker would do
• Relations: capture properties that hold when the system is not
vulnerable
7
{login(Admin), RequestURL(settings_page)}
{login(User1), RequestURL(settings_page)}
if the follow-up user cannot access the URL from his GUI
then the output of the source and follow-up inputs should be
different

8. Automated Metamorphic Security
Testing Process
8
Catalog of
Metamorphic
Relations
Selected
Metamorphic
Relations
Translate
Metamorphic Relations
to Java
2
Execute a
Web-crawler
3
Specify/Select
Metamorphic
Relations
1
Executable
Metamorphic
Relations in Java
Source
Inputs
Data
Execute the
Metamorphic Testing
Algorithm
4
MR1: PASS
MR2: FAIL
MR3: PASS
MR4: PASS
Test
Result

9. Security Metamorphic Relation
Language (SMRL)
9

10. SMRL Basic Features
• Extension of Xbase:
• Java-like
• Automated generation of Java executables from specifications
• SMRL provides seven boolean operators
• IMPLIES, AND, OR, TRUE, FALSE, and NOT
• EQUAL
• Evaluate the equality of two arguments
• Define follow-up inputs
• enable writing metamorphic relations in declarative form
10

11. Data Types
11
InputSequence
InputTypes
OutputSequence
OutputTypes
Action1..* 0..1
Session User
0..1
HtmlOutput
1..*
Request URL
Anchor click
Button click
Text fill

12. 12
Data Representation Functions
Data function Description
Input(int n) Identifies the nth input sequence referred to in a MR.
User(int n) Identifies the nth user referred in a MR.
User() Identifies the 1st user referred in a MR.
Action(int n) Identifies the nth input action referred in a MR.
Session(int n) Identifies the nth Web session referred in a MR.
Output(Input n) Identifies the sequence of outputs generated by the nth input.
Output(Input n, int i) Identifies the output generated by the ith action of the nth input.
…

13. Web-specific Functions
• changeCredentials( Input i, User u)
• Modifies an input sequence so that the credentials of the specified user
are used in the login action
changeCredentials( Input(1), User() )
• Input(1): {login(Admin), RequestURL(settings_page) }
• User(): User1
• Result: {login(User1), RequestURL(settings_page) }
• cannotReachTroughGUI( User u, String URL)
• Returns true if, based on the crawled data, a URL cannot be reached by
the given user by exploring the user interface of the system
13

14. Metamorphic Relation Example:
Bypass Authorization Schema
14

15. Metamorphic Relation Example:
Bypass Authorization Schema
15
Data function
Boolean operator
Web specific function

16. Metamorphic Relation Example:
Bypass Authorization Schema
16
Data function
Boolean operator
Web specific function

17. Metamorphic Relation Example:
Bypass Authorization Schema
17
Data function
Boolean operator
Web specific function

18. Metamorphic Relation Example:
Bypass Authorization Schema
18
Data function
Boolean operator
Web specific function

19. Metamorphic Relation Example:
Bypass Authorization Schema
19
Our metamorphic testing algorithm
executes each MR multiple times,
to ensure that every possible combination of
source inputs is exercised

20. 20
Source Inputs of Type InputSequence
{login(Admin), RequestURL(personal_page)}
{login(Admin), RequestURL(settings_page)}
{login(User1), RequestURL(personal_page)}
{login(User1), RequestURL(project_page)}
Source Inputs of Type User
User1
Admin
Tested Combinations
Source Inputs Follow-up Inputs
{login(Admin), RequestURL(personal_page)}
+ User1
/
{login(Admin), RequestURL(settings_page)}
+ User1
{login(User1), RequestURL(settings_page)}
{login(User1), RequestURL(personal_page)}
+ Admin
/
{login(User1), RequestURL(project_page)} {login(Admin), RequestURL(personal_page)}

21. Java Integration
• Metamorphic relations are automatically translated into
executable Java code
• They can be selected in Junit test cases
21

22. Deriving Source Inputs from Data
Crawled with Crawljax

23. Deriving Source Inputs from
Test Scripts
• For test scripts manually implemented for functional testing,
we can derive source inputs from them
• we support Selenium-based scripts
• a test script represent an InputSequence
23

24. 24
Executable
Metamorphic
Relation
in Java
Source
Inputs
data
Metamorphic
Testing
Algorithm
4
Follow-up
input
Test result
Executor
Verify relation
Metamorphic Testing Execution
Source
input
Action1
Action2
Action3
Action1
Action2’
Action3
Follow-up
Output
Source
Output
Output1
Output2
Output3
Output1
Output2’’
Output3MR PASS/FAIL
Repeat

25. Metamorphic Relations Catalog
• Includes 22 Security Metamorphic Relations derived from
OWASP testing guidelines
25
Vulnerability type # Metamorphic relations
Broken Authentication 6
Sensitive Data Exposure 5
Broken Access Control 7
Security Misconfiguration 3
Vulnerable Components 1

26. Broken Authentication
• Objective: verify that a session always changes after a logout
• Source input: sequence of actions performed after log-in
• Follow-up input: add a logout action after an action of the
source InputSequence
• Relation: session IDs before and after logout action should
be different
26

27. Sensitive Data Exposure
• Objective: verify that a user cannot retrieve sensitive data by simply
altering a URL parameter value
• Source input: sequence of actions including at least one action
containing a URL parameter (action X)
• Follow-up input: change the URL parameter value for the action X
(action X’) using a value observed only with other users
• Relation: the output of the action X’ either contains an error message or
contains data that is not sensitive
• e.g., it should contain data that has been already retrieved by the
same user in the recorded source inputs
27

28. Evaluation
• RQ1. To what extent can metamorphic testing address the oracle
problem in the context of Web security testing?
• Determine which types of security testing activities can be
automated by our solution to address the oracle problem
• RQ2. Is the proposed solution effective?
• Determine if the proposed solution enables the automated
detection of security vulnerabilities
30

29. Evaluation
• RQ1. To what extent can metamorphic testing address the oracle
problem in the context of Web security testing?
• Determine which types of security testing activities can be
automated by our solution to address the oracle problem
• RQ2. Is the proposed solution effective?
• Determine if the proposed solution enables the automated
detection of security vulnerabilities
31

30. RQ2 - Case Studies
• Case studies
• Commercial system developed in an EU project (EDLAH2)
• four vulnerabilities
• Jenkins 2.121.1
• Considered the vulnerabilities discovered in the 2nd half of
2018 that are targeted by our approach
• eight vulnerabilities
32

31. RQ2 - Setup
• Data collection:
• Executed Crawljax against the two systems for a maximum of
five hours or till exploration is exhausted
• Implemented manual test scripts that exercise features not
exercised by Crawljax
• Testing:
• Considered 11 MRs covering the type of vulnerabilities affecting
the system
33

32. RQ2 - Measurements
• Sensitivity
• ratio of vulnerabilities being discovered
• Specificity
• ratio of follow-up inputs that do not trigger any vulnerability
and do not lead to any failure
• it measures the time saved from inspecting unwarranted
failures
34

33. RQ2 - Sensitivity
• We do not discover vulnerabilities that require specific environment settings
• Jenkins configuration modified during test execution
• Jenkins overloaded
Case
study
Total
Vulnerabilities
Discovered vulnerabilities Sensitivity
Crawljax Crawljax &
Manual
Crawljax Crawljax &
Manual
EDLAH2 4 3 4 75% 100%
Jenkins 8 4 6 50% 75%
Total 12 7 10 58.33% 83.33%
35

34. RQ2 - Specificity
• Run 11 metamorphic relations
Case study Crawljax Crawljax & Manual
Follow-up Follow-up
ALL FP TP or
No alarm
Sensitivity ALL FP TP or
No alarm
Sensitivity
EDLAH2 832 0 832 100% 833 0 833 100%
Jenkins 4984 33 4951 99.34% 5568 32 5536 99.43%
Total 5816 33 5783 99.43% 6401 32 6369 99.50%
36
• The approach automatically tests the systems with many inputs and the number of
inputs to be erroneously inspected is minimal

35. The Oracle Problem
in Web Systems’ Security Testing
2
Security vulnerabilities are subtle
Discovered after testing with many inputs
Specifying expected results is infesible
Automated Metamorphic Security
Testing Process
8
Catalog of
Metamorphic
Relations
Selected
Metamorphic
Relations
Transform
Metamorphic Relations
to Java
2
Execute a
Web-crawler
3
Specify/Select
Metamorphic
Relations
1
Executable
Metamorphic
Relations in Java
Source
Inputs
Data
Execute the
Metamorphic Testing
Algorithm
4
MR1: PASS
MR2: FAIL
MR3: PASS
MR4: PASS
Test
Result
Experiments show
High sensitivity and specificity
We have recently executed
additional experiments with all the 22 MRs
to test Jenkins and Joomla
Sensitivity/Specificity rates are confirmed
Discovered one unknown vulnerability
in Jenkins (CVE-2020-2162)
https://sntsvv.github.io/SMRL/