ISSRE 2018 paper presentation

1. .lusoftware verification & validation
VVS
A Natural Language Programming Approach
for Requirements-based Security Testing
Phu X. Mai, Fabrizio Pastore, Arda Goknil, Lionel Briand
ISSRE 2018
Interdisciplinary Centre for Security, Reliability and Trust (SnT)
University of Luxembourg, Luxembourg

2. Context:
Web-based, Multi-device Systems
• Include multiple and different devices and manage personal information
• typical for different services
• case study: EDLAH2 project, platform to monitor old people
2

3. Security of
Web-based, Multi-device Systems
•Present multiple attack surfaces (e.g., parameters, URLs..)
•important to foresee the possible attacks during requirements
elicitation
•capture these in a format that is understood by all the stakeholders
(e.g., natural language)
•verify if the system is vulnerable to these attacks
•manual testing is expensive and error prone
3

4. How to automate
requirements-based security testing?
4

5. How to automate
requirements-based security testing?
5
Misuse
Case
Specifications
Executable
Security
Vulnerability
Test Cases
Automated Generation
based on Natural
Language Processing
i.e., natural language
description of attacks

6. MISUSE CASE: Bypass Authorization Schema
Description: The MALICIOUS user accesses resources that
are dedicated to a user with a different role
Precondition: The MALICIOUS user has access to one or
more accounts on the system and a list of resources (URLs)
that he should not be able to access with these accounts
Primary Actor: MALICIOUS user
Threats: Monitor Patient Progress, View Patient Information,
Configure Patient Tablet
Assets: Client DATA
6
Example of Misuse Case Specification
(Headers) RMCM template
[Mai et al., IST'18]

7. Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends username and password to the system
through the login page
3. FOREACH resource
4. The MALICIOUS user requests the resource from the system
5. The system sends a response page to the MALICIOUS user
6. The MALICIOUS user EXPLOITS the system using the response page
7. ENDFOR
8. ENDFOR
Postcondition: The MALICIOUS user has executed a function
dedicated to another user with different role
Specific Alternative Flow (SAF1)
RFS 6
1. IF the response page contains an error message THEN
2. RESUME STEP 7
3. ENDIF
Postcondition: The MALICIOUS user cannot access the resource
Example of Misuse Case Specification
(Flows)
7

8. roleIter = input["role"].__iter__()
while roleIter.__hasNext__():
role = roleIter.__next__()
arguments["password"] = role["password"]
arguments["username"] = role["username"]
system.send("login page", arguments)
resourceIter = role["resource"].__iter__()
while resourceIter.__hasNext__():
resource = resourceIter.__next__()
system.request(resource)
responsePage = system.responsePage
if not responsePage.contain(resource["error_message"]):
# the test case fails: the attacker exploits the vulnerability
maliciousUser.exploit()
else:
# the test case passes and the attacker lose
maliciousUser.abort(“malicious user cannot access resource”)
For every role
configured on the system
Login as a user
with that role
Request a resource
(not available to a user with that role)
If the response does not contain
an error message
we have accessed the resource
and can exploit the system
Otherwise the attacker
cannot access the resource8
Expected Generated Code

9. roleIter = input["role"].__iter__()
while roleIter.__hasNext__():
role = roleIter.__next__()
arguments["password"] = role["password"]
arguments["username"] = role["username"]
system.send("login page", arguments)
resourceIter = role["resource"].__iter__()
while resourceIter.__hasNext__():
resource = resourceIter.__next__()
system.request(resource)
responsePage = system.responsePage
if not responsePage.contain(resource["error_message"]):
# the test case fails: the attacker exploits the vulnerability
maliciousUser.exploit()
else:
# the test case passes and the attacker lose
maliciousUser.abort(“malicious user cannot access resource”)
For every role
configured on the system
Login as a user
with that role
Request a resource
(not available to a user with that role)
If the response does not contain
an error message
we have accessed the resource
and can exploit the system
Otherwise the attacker
cannot access the resource9
Generated code includes
input processing,
variable declarations,
cycles, conditions,
method calls, assignments,
oracles
Expected Generated Code

10. Test Generation: sub-objectives
10
• Identify input entities (e.g., ‘role’, ‘password’), input relationships (e.g., each
‘username’ is associated to a ‘role’), and input values
• Identify the instructions (e.g., API calls) that perform the operations in the
misuse case specifications
• Generate oracles from conditional sentences describing when the attack is
successful
Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends username and password to the system
through the login page
3. FOREACH resource
4. The MALICIOUS user requests the resource from the system
..
6. IF the response page contains an error message THEN
input entities
operation
conditional
sentence

11. •Misuse cases are written using a template (e.g., RMCM) that
•Enforce the use of simple sentences (no compound sentences)
•Include control flow keywords
•A test driver API is available (e.g., implemented for functional testing)
•Sentences in use case specifications are “textually similar” to the API
operations and parameters in the implemented test cases
arguments["passwd"] = role["password"]
arguments["user"] = role["username"]
system.send("login page", arguments)
/*Send inputs through a form in a specific page*/
System::send(String page, Dictionary arguments)
/*Request a resource (URL) from the system*/
System::request(String resource)
11
“The user sends username and password to the system through the login page”
Working Assumptions

12. Our Solution:
Misuse Case Programming
•Rely on Natural Language Programming concepts to perform test
generation
• Natural Language Programming: Approaches automatically
generating software programs from specifications in NL
• Mapping sentences to test API functions based on natural language
processing and an ontology that captures the test API information
•Our contributions:
• Automated generation of executable test cases
• Automated identification of test inputs
• Automated generation of test oracles
12

13. MCP Steps
13
1. Automatically generate an ontology model for the test driver API
2. Automatically generate a misuse case model (control flow)
3. Automatically identify test inputs
1. Identify input entities
2. Identify relationships among input entities
3. Generate inputs or input files to be filled by engineers
4. Automatically generate executable test cases
1. Declare variables
2. Generate method calls
3. Generate assignments
4. Generate oracles

14. Step 1: Automatically Generate an Ontology
Model for the Test Driver API
14
MCP
toolTest driver
API (Python) «Class»
System
«Method»
send
«Method»
request
methodOf
methodOf
«Parameter»
page
«Parameter»
arguments
«Parameter»
resource
parameterOf
parameterOf parameterOf
«Class»
Dictionary
«Class»
String
type type
«Class»
String
type

15. Step 2: Automatically Generate the Misuse
Case Model Capturing the Control Flow
1. FOREACH
Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends username
and password to the system through the
login page
3. FOREACH resource
4. The MALICIOUS user requests the
resource from the system.
..
6. IF the response page contains an error
message THEN
3. FOREACH
2.
4.
5.
6. IF
15
NLP relying on keywords

16. Step 3.1: Identify Input Entities
Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends
username and password to
the system through the login page
3. FOREACH resource
4. The MALICIOUS user requests
the resource from the system.
..
6. IF the response page contains an
error message THEN
• Determine which sentences
describe activities where inputs
are provided to the system
• Determine which phrases are
the inputs
16

17. Step 3.1: Identify Input Entities
Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends
username and password to
the system through the login page
3. FOREACH resource
4. The MALICIOUS user requests
the resource from the system.
..
6. IF the response page contains an
error message THEN
• Determine which sentences
describe activities where inputs
are provided to the system
• Determine which phrases are
the inputs
17
Rely on Semantic Role Labeling

18. Semantic Role Labeling (SRL)
“The MALICIOUS user sends the username to the system”
actor affected
by the activity
actor performing
an activity
verb
SRL: Automatically determines the semantic roles of
phrases in sentences
destination
18

19. Semantic Role Labeling (SRL)
“The MALICIOUS user sends the username to the system”
actor affected
by the activity
actor performing
an activity
verb
SRL: Automatically determines the semantic roles of
phrases in sentences
destination
MCP: Input sentence when the destination is the system.
The input is the actor affected by the activity.
19

20. Step 3.2: Identify and Model Input Relationships
1. FOREACH role
2. The MALICIOUS user sends username and password
to the system through the login page
3. FOREACH resource
4. The MALICIOUS user requests the resource from the system.
..
6. IF the response page contains an error message THEN
Necessary for the generation of valid test cases
• e.g., automatically determine:
• each role is associated to a username with a password,
• the resources accessed with a given role are not the same accessed with
another role
We rely on the structure of the misuse case specification:
• sentences in loop-body lead to one-to-one relationships (e.g., role-username)
• nested loops lead to one-to-many relationships (e.g., role-resources)
20

21. Step 3.2: Identify and Model Input Relationships
• Ontology: structured
representation of relationships
between inputs
• Dictionaries are used to
capture complex concepts
(e.g., a role has a username
and a password) including
one-to-many relationships
21
«Dictionary»
inputs
«Dictionary»
rolesV
«Key»
roles
valueOf
keyOf
«Key»
resources
«Key»
username
keyOf
«Dictionary»
resourcesV
valueOf
keyOf
«Class»
String
typeOf
«Key»
password
keyOf
«Class»
String
typeOf

22. Step 3.3: Automatically Generate a JSON File
Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user SENDS
username and password TO
the system through the login
page
3. FOREACH resource
4. The MALICIOUS user requests
the resource from the system.
..
6. IF the response page contains
an error message THEN
“roles”: [ {
“role” : SPECIFY;
“username” : SPECIFY;
“password” : SPECIFY;
“resources” = [
{ resource = SPECIFY },
{ resource = SPECIFY },
]}]
3.4 Input values are specified by engineers
3.3 Generate a JSON file that
matches the input structure in the ontology
22
«Dictionary»
inputs
«Dictionary»
rolesV
«Key»
roles
valueOf
keyOf
«Key»
resources
«Key»
username
keyOf
«Dictionary»
resourcesV
valueOf
keyOf
«Class»
String
typeOf
«Key»
password
keyOf
«Class»
String
typeOf

23. Step 3.4: Engineers Specify Input Values
"roles": [ {
"role": "Doctor",
"username": phu@mymail.lu,
"password": "testPassword1",
"resources": [ {
"resource": http://www.icare247.eu/?q=micare_invite&accountID=11”
}, {
"resource": "http://www.icare247.eu/?q=micare_skype/config&clientID=36"
}, ]
} , {
"role": "Patient",
…
23

24. Step 3.4: Engineers Specify Input Values
"roles": [ {
"role": "Doctor",
"username": phu@mymail.lu,
"password": "testPassword1",
"resources": [ {
"resource": http://www.icare247.eu/?q=micare_invite&accountID=11”
}, {
"resource": "http://www.icare247.eu/?q=micare_skype/config&clientID=36"
}, ]
} , {
"role": "Patient",
…
24
We automatically generate inputs when attack keywords (e.g., SQLi)
appear in the specifications
(see paper)

25. Step 4: Automatically Generate the Executable Test
• For each control flow node in the misuse case model (e.g., ‘FOREACH..’)
generate a corresponding control flow instruction in the test case
• For the other nodes
• declare required variables
• generate a method call or an assignment
• greedy strategy: generate both, and select the one more
'similar' to the text
1. FOREACH role
2. The MALICIOUS user sends
username and password to ..
3. FOREACH resource
25
roleIter = input["roles"].__iter__()
while roleIter.__hasNext__():
role = roleIter.__next__()
arguments["password"] = role["password"]
arguments["username"] = role["username"]
system.send("login page", arguments)

26. Step 4.1: Variable Declarations
26

27. Step 4.1: Variable Declarations
• Each generated test case declares three variables
• system: instance of a class that provides system operations
(e.g., request a resource)
• maliciousUser: test case instance, which implements the activities
typically performed by a malicious user (e.g., run a proxy)
• input: the input dictionary
• Additional variables are generated while the misuse case is processed
• (e.g., variables ‘role’ and ‘username’)
system = System() # the system instance
maliciousUser = self # the test instance
input = loadJSON(“input.json”) # the input dictionary
roleIter = input["role"].__iter__()
while True:
role = roleIter.__next__()
27

28. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
• actor that performs the activity (e.g., malicious user)
• actor that receives the data indicated in the sentence (e.g., system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
28

29. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
• actor that performs the activity (e.g., malicious user)
• actor that receives the data indicated in the sentence (e.g., system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
29

30. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
(e.g., malicious user, system)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
30

31. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
(e.g., malicious user, system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
31

32. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
(e.g., malicious user, system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
32

33. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
(e.g., malicious user, system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
Finally, assign variables to parameters
33

34. Step 4.2: Automatically Generate Method Calls
Using the API ontology, find a method that:
• belongs to a class instance with a name matching one of the actors in the sentence
(e.g., malicious user, system)
• has a name similar to the verb name (e.g., request)
• has a subset of parameters with names similar to terms in the spec. (e.g., resource)
«Class»
System
«Method»
request
methodOf
«Parameter»
resource
parameterOf
«Method»
send
methodOf
«Parameter»
arguments
parameterOf
2. The MALICIOUS user REQUESTS the resource FROM the system.
Given a step of the misuse case specification:
«Variable»
system
«Variable»
maliciousUser
«Class»
TestCase
typeOf typeOf
«Method»
runProxy
methodOf
system.request(resource)
Generated method call:
Finally, assign variables to parameters
34

35. Step 4.4: Generate Condition Instructions
RFS 6
1. IF the response page contains an error message THEN
2. RESUME STEP 7.
RFS 4.
1. IF the resource contains a role parameter in the URL THEN
2. The MALICIOUS user modifies the role values in the URL.
35
•Correspond to condition statements in misuse case specifications
• a condition instruction may evaluate the results of an API call
• or may evaluate a configuration given by the engineer

36. Step 4.4: Generate Condition Instructions
RFS 6
1. IF the response page contains an error message THEN
2. RESUME STEP 7.
RFS 4.
1. IF the resource contains a role parameter in the URL THEN
2. The MALICIOUS user modifies the role values in the URL.
First, try to generate a matching method call
if responsePage.contain(resource["error_message"] ) :
36
•Correspond to condition statements in misuse case specifications
• a condition instruction may evaluate the results of an API call
• or may evaluate a configuration given by the engineer

37. Step 4.4: Generate Condition Instructions
RFS 6
1. IF the response page contains an error message THEN
2. RESUME STEP 7.
RFS 4.
1. IF the resource contains a role parameter in the URL THEN
2. The MALICIOUS user modifies the role values in the URL.
First, try to generate a matching method call
if responsePage.contain(resource["error_message"] ) :
If the score of the matching call is too low (< 0.4), there is no method
implementing that activity; ask the engineer to encode the condition
if eval(input["the_resource_contains_a_role_parameter…"]) :
37
•Correspond to condition statements in misuse case specifications
• a condition instruction may evaluate the results of an API call
• or may evaluate a configuration given by the engineer

38. roleIter = input["role"].__iter__()
while roleIter.__hasNext__():
role = roleIter.__next__()
arguments["password"] = role["password"]
arguments["username"] = role["username"]
system.send("login page",arguments)
resourceIter = role["resource"].__iter__()
while resourceIter.__hasNext__():
resource = resourceIter.__next__()
system.request(resource)
if not responsePage.contain(parameter1) :
resourceIter = role["resource"].__iter__()
while True:
try:
resource = resourceIter.__next__()
system.request(resource)
maliciousUser.responsePage = system.responsePage
if not responsePage.contain( resource["error_message"] ) :
maliciousUser.exploit()
else:
maliciousUser.abort(“malicious user cannot access resource”)
Condition instruction
(oracle)
Condition Instructions Enable the Generation of
Oracles
Records failure information
Indicates that the attack fails 38
'exploit' and 'abort' are
general utility methods
provided by our test driver API

39. Empirical Evaluation
39
•Generated test cases for 12 misuse case specifications of EDLAH2
•Selected high risk ones (i.e., capable of detecting vulnerabilities,
according to previous manual testing)
•Research questions
•RQ1. Does MCP correctly identify input entities?
•RQ2. Can MCP generate executable test cases from misuse case
specifications?
•RQ3. How do the generated test cases compare to manual test
cases?

40. RQ1. Does MCP correctly identify input entities?
40
•True positives (TP): input entities correctly identified by MCP (i.e.,
necessary to perform the test)
•False positives (FP): input entities that do not correspond to software
inputs
•False negatives (FN): input entities required to perform the attack,
which have not been identified by MCP
•MCP leads to 29 TP, 1 FP, and 3 FN
•Precision = TP/(TP+FP) = 0.97
•Recall = TP/(TP+FN) = 0.91

41. RQ2. Can MCP generate executable test cases
from misuse case specifications?
•The test cases generated by MCP
•do not contain any programming error
•were all successfully executed against the EDLAH2 system
•Generated test cases are not trivial
•791 lines of code
•172 method calls
•44 assignments
•260 method arguments
41

42. RQ3. How do the generated test cases
compare to manual test cases?
42
•Effectiveness & soundness:
•Executable test cases derived manually from the same
specifications identified nine vulnerabilities
•The test cases generated by MCP
•identified all the nine vulnerabilities
•do not report any false alarm

43. RQ3. How do the generated test cases
compare to manual test cases?
43
•Costs:
•MCP
•requires a test driver API with 1053 LOC (reusable across systems)
•Manual testing:
•1523 lines of code required to implement 12 misuse cases
•more than 60 misuse cases, very high effort

44. Basic Threat Flow:
1. FOREACH role
2. The MALICIOUS user sends username and password to the system
through the login page
3. FOREACH resource
4. The MALICIOUS user requests the resource from the system
5. The system sends a response page to the MALICIOUS user
6. The MALICIOUS user EXPLOITS the system using the response page
7. ENDFOR
8. ENDFOR
Postcondition: The MALICIOUS user has executed a function
dedicated to another user with different role
Specific Alternative Flow (SAF1)
RFS 6
1. IF the response page contains an error message THEN
2. RESUME STEP 7
3. ENDIF
Postcondition: The MALICIOUS user cannot access the resource
Example of Misuse Case Specification
(Flows)
8
roleIter = input["role"].__iter__()
while roleIter.__hasNext__():
role = roleIter.__next__()
arguments["password"] = role["password"]
arguments["username"] = role["username"]
system.send("login page", arguments)
resourceIter = role["resource"].__iter__()
while resourceIter.__hasNext__():
resource = resourceIter.__next__()
system.request(resource)
responsePage = system.responsePage
if not responsePage.contain(resource["error_message"]):
# the test case fails: the attacker exploits the vulnerability
maliciousUser.exploit()
else:
# the test case passes and the attacker lose
maliciousUser.abort(“malicious user cannot access resource”)
For every role
configured on the system
Login as a user
with that role
Request a resource
(not available to a user with that role)
If the response does not contain
an error message
we have accessed the resource
and can exploit the system
Otherwise the attacker
cannot access the resource10
Generated Code
Evaluation shows that the
approach is feasible.
https://sntsvv.github.io/MCP/
44

45. .lusoftware verification & validation
VVS
A Natural Language Programming Approach
for Requirements-based Security Testing
Phu X. Mai, Fabrizio Pastore, Arda Goknil, Lionel Briand
ISSRE 2018
Interdisciplinary Centre for Security, Reliability and Trust (SnT)
University of Luxembourg, Luxembourg

46. Security Testing State-of-the-Art
•In industry, requirements-based testing is mostly performed
manually
•expensive, error prone
•Most automated security testing approaches focus on a specific
vulnerability
•e.g., buffer overflows [Haller’13; Ognawala’16], code injections [Tripp’13;
Appelt’15]
•deal with the generation of simple inputs (e.g., strings, files)
•cannot verify if the system is prone to attacks involving articulate
interactions
46

47. Security Testing State-of-the-Art
•Model-based approaches generate test cases based on interaction
protocol specifications [Veanes’05; Silva’08]
•can potentially be used to generate test cases for complex attack
scenarios
•require detailed system models, which are seldom produced by
engineers
•Approaches to generate functional test cases from natural language
[Carvalho’14; Wang’15]
•Adopted to test intended behaviours in the context of functional
testing
•Not suitable to test unintended behaviours (e.g., security attacks)
47

48. Approaches for Test Case Generation from
Use Case Specifications
are partially applicable [Wang'15,'18]
48
• Security vulnerability testing simulates the misbehaviour of a malicious user
• Malicious user behaviour depends on the system responses (no predefined flow of actions)
• No predefined flow of actions
• Necessary to generate test cases that react based on the responses received
• Functional approaches could be used to generate functional security test cases
• Specifications include attack info; domain model not usually adopted in security context
• Dedicated input generation approaches required
Use
Case
Specifications
Domain Model
for word
disambiguation and
inputs identification
UseCaseStart
Input
Condition
Condition
Output
Exit
Condition
Internal
Internal
Include
Control Flow
of activities
in specifications
One test case
per execution
flow

49. Related Approach for Test Case
Generation from Use Case Specifications
49
• Extracts control-flow test models from use case specifications [Wang‘15, Wang‘18]
• Based on Natural Language Processing, requires a mapping table and a domain model
• Domain model used to identify inputs
• Mapping table used to match inputs to API functions
• Generated test cases aim to exercise the expected system behaviour
• Since use case specifications describe how the system reacts based on user inputs, test cases are
derived from a finite set of expected behaviours
• Could also be used also to generate functional security test cases
• Security vulnerability testing simulates the misbehaviour of a malicious user
• Malicious user behaviour depends on the system responses (no predefined flow of actions)
• Necessary to generate test cases that react based on the responses received
• Multiple user-system interactions per test case (mapping table might be expensive to produce)
• Specifications include attack info; domain model not usually adopted in security context
• Dedicated input generation approaches required

50. 50
Test Driver API
provided by MCP
(possibly extended
by engineers)
Phase 1: Map the test driver API to the MCP ontology
Misuse Case Specifications
In Natural Language
Bypass Authorization
Step 1:…
Step 2:…
(A) Initial ontology provided by
MCP (models programming
language concepts).
Class
AttributeMethod
(B) Ontology including information
about the test driver API.
«Class»
HttpTest
«Class»
System
«Method»
send
Class
Method
Attribute
Phase 2: Generate Misuse Case Models
Step 1
Step 3Step 5
(C) Misuse Case Model
capturing control flow.
Phase 3: Identify Test Inputs
(D) Ontology
updated with
individuals capturing
the relations
between inputs.
«Key»
role
«Key»
username
«Key»
password
«Dictionary»
inputs
(E) Test Input files
Inputs.json
Phase 4: Generate Executable Test Cases
(G) Executable Test Cases.
reuseInvitation.py
guessUserAccount.py
bypassAuthorization.py
(F) Ontology updated with information about
instance variables in the scope of test case lines.
«Class»
HttpTest
«Class»
System
«Variable»
system
Class
Method
Attribute
«Class»
BPA
«Variable»
this
«Scope»

51. Step 4: Assigning Scores to Methods
• Compute a score for each possible method by computing the average of
• the string similarity degree between the method name and the verb
in the sentence
• the average string similarity degree of all the parameters with the
best matching input entity
• the percentage of terms (i.e., verb and noun phrases appearing in
the misuse case) that match the method parameters
• Example:
2. The MALICIOUS user requests the resource from the system.
Given a step of the misuse case specification
51
system.request(String resource): (0.875+1+0.75)/3 = 0.875
system.send(String page, Dictionary arguments): (0.1+0+0.25) = 0.17

52. Step 4: Automatically generate assignments
Based on the identification of the roles in the sentence
• ‘actor that performs the activity’ (e.g., system)
• corresponds to RHS instance
• ‘destination’ role (e.g., malicious user)
• corresponds to LHS instance
• actor that is affected by the action (e.g., response page)
• data being exchanged, i.e., field name
2. The system sends the response page to the MALICIOUS user.
Assignments are generated when data is exchanged between two actors
maliciousUser.responsePage = system.responsePage
The assignment score results from the average of:
• the average string similarity degree for the terms used to identify the left-hand side
and the right-hand side of an assignment
• the portion of terms of the misuse case step that appear in the generated assignments
52