Conference Presentation - CAiSE 2019 A security orchestration platform aims at integrating the activities performed by multi-vendor security tools to streamline the required incident response process. To make such a platform useful in practice in a Security Operation Center (SOC), we need to address three key challenges: interpretability, interoperability, and automation. In this paper, we proposed a novel semantic integration approach to automatically select and integrate security tools with essential capability for auto-execution of an incident response process in a security orchestration platform. The capability of security tools and the activities of the incident response process are formalized using ontologies, which have been used for NLP based approach to classify the activities for the emerging incident response processes. The developed ontologies and NLP approaches have been used for an interoperability model for selection and integration of security tools at runtime for the successful execution of an incident response process. Experimental results demonstrate the feasibility of the classifier and interoperability model for achieving interpretability, interoperability, and automation of security tools integrated into a security orchestration platform.

1. Automated Interpretation and Integration of
Security Tools Using Semantic Knowledge
Authors: Chadni Islam, M. Ali Babar, and Surya Nepal
CREST Research Centre, University of Adelaide, Adelaide, Australia
Data61, CSIRO, Australia
31st International Conference on
Advanced Information Systems
Engineering
Islam C., Babar M.A., Nepal S. (2019) Automated Interpretation and Integration of Security Tools Using
Semantic Knowledge. In: Giorgini P., Weber B. (eds) Advanced Information Systems Engineering. CAiSE
2019. Lecture Notes in Computer Science, vol 11483. Springer, Cham

2. Outline
Introduction
Challenges
Research Aim
Proposed Solution
Evaluation
Experiment and Results
Limitations
Future Work
CREST Centre | University of Adelaide 2

3. Security Orchestration
“Security Orchestration is the planning, integration,
cooperation, and coordination of the activities of security
systems and experts to produce and automate required
actions in response to any security incident across multiple
technology paradigms.”
CREST Centre | University of Adelaide 3
Integration
Orchestration
Automation

4. Introduction
Detect and
Prevent
Integrate
Analyze
Validate
Plan
Monitor and
Analyze
Security
Operation Centre
I1
I2
S: Security Tool
Firewall
Intrusion Detection System (IDS)
Endpoint Detection and Response (EDR)
A: Assets AC: ActivityI: Security Incident
Organization
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019 4
The Security Operation Centre (SOC) of an organizations uses a variety of
security tools, developed by different vendor’s to protect an organization’s
information and communication technology and business application.

5. Introduction
Detect and
Prevent
Integrate
Analyze
Validate
Plan
Monitor and
Analyze
Security
Operation Centre
I1
I2
S: Security Tool
Firewall
Intrusion Detection System (IDS)
Endpoint Detection and Response (EDR)
A: Assets AC: ActivityI: Security Incident
Organization
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019 5
The SOC are expected to monitory and analyse the activities (validate alerts,
correlate logs and remove malware) of these security tools to respond to a
incident. The continuous process of monitoring and analysing the security
activities are time consuming, tedious and repetitive.

6. Introduction
Detect and
Prevent
S: Security ToolA: Assets AC: Activity
I1
I2
I: Security Incident
Security
Orchestration
Platform
Firewall
Intrusion Detection System (IDS)
Endpoint Detection and Response (EDR)
Search
Organization
t: Task
CREST Centre | University of Adelaide 6
IRP: Incident
response plan
Most SOC in recent years uses security orchestration platform to
orchestrate the activities of security tools and automate the repetitive task
manually performed by the human experts.

7. Introduction
Detect and
Prevent
S: Security ToolA: Assets AC: Activity
I1
I2
I: Security Incident
Security
Orchestration
Platform
Firewall
Intrusion Detection System (IDS)
Endpoint Detection and Response (EDR)
Search
Organization
t: Task
CREST Centre | University of Adelaide 7
IRP: Incident
response plan
Emerging threat behaviours and variation in organization infrastructure cause experts to
change the deployment and execution environment of security orchestration platform,
such as integration of new tools, updates of tools capabilities or modification of IRP

8. Organizations Plan to Response to a
Security Incident
8
Example of Incident Response Plan (IRP) for Phishing Attack
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
# Response Task Activity
ac1 Is this a phishing
attack?
Determine if this is a phishing attack? In the task, select yes or
no in the outcome.
ac2 Scan endpoint –
malware found?
After running a scan, determine whether malware was found.
In the task, select yes or no in Outcome.
ac3 Remove malware –
success?
Determine whether the malware was successfully remove. In
the task, select Yes or no in outcome.
ac4 Wipe and reimage If you did not successfully remove the malware found, this task
instruct you to perform a wipe and reimage on the computers
infected with the malware.
ac5 Update email
protection software
If it was determined that this is a phishing attack, you are
prompted to update your email protection software accordingly.
ac6 Remove unread
phishing email in
queue
Perform the steps necessary to remove the phishing email still
in the queue for all of your users

9. Organizations Plan to Response to a
Security Incident
9
Example of Incident Response Plan (IRP) for Phishing Attack
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
# Response Task Activity
ac1 Is this a phishing
attack?
Determine if this is a phishing attack? In the task, select yes or
no in the outcome.
ac2 Scan endpoint –
malware found?
After running a scan, determine whether malware was found.
In the task, select yes or no in Outcome.
ac3 Remove malware –
success?
Determine whether the malware was successfully remove. In
the task, select Yes or no in outcome.
ac4 Wipe and reimage If you did not successfully remove the malware found, this task
instruct you to perform a wipe and reimage on the computers
infected with the malware.
ac5 Update email
protection software
If it was determined that this is a phishing attack, you are
prompted to update your email protection software accordingly.
ac6 Remove unread
phishing email in
queue
Perform the steps necessary to remove the phishing email still
in the queue for all of your users
A SOC does not follow any specific structure while defining the activity of an
IRP. A security orchestration platform needs to deal with different tools that are
not interoperable to automate the execution of an IRP’s activities.

10. Organizations Plan to Response to a
Security Incident
10
Example of Incident Response Plan (IRP) for Phishing Attack
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
# Response Task Activity
ac1 Is this a phishing
attack?
Determine if this is a phishing attack? In the task, select yes or
no in the outcome.
ac2 Scan endpoint –
malware found?
After running a scan, determine whether malware was found.
In the task, select yes or no in Outcome.
ac3 Remove malware –
success?
Determine whether the malware was successfully remove. In
the task, select Yes or no in outcome.
ac4 Wipe and reimage If you did not successfully remove the malware found, this task
instruct you to perform a wipe and reimage on the computers
infected with the malware.
ac5 Update email
protection software
If it was determined that this is a phishing attack, you are
prompted to update your email protection software accordingly.
ac6 Remove unread
phishing email in
queue
Perform the steps necessary to remove the phishing email still
in the queue for all of your users
For example, execution of activity ac1 may require a threat intelligence such as
MISP. A MISP is used by a security orchestration platform to validate a
incident. The execution of ac2 may require an EDR tools to scan endpoint and
SIEM to identify the malware from EDR logs.

11. Challenges
• A security orchestration platform is not adaptable
towards the change in the execution environment.
… … …
Diverse nature of data
generated and ingested
by Security Tools
Lack on Interpretability
of the generated data
Lack of
Interoperability
Integration of the Security tools is one of the most challenging task of SOC
CREST Centre | University of Adelaide 11

12. Research Aim
Automated Interpretation and
Integration of Security Tools in a
Security Orchestration Platform
12 |Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

13. Proposed Solution (1/4)
• Formalized the core concepts of security orchestration
platform in an ontology that are required to automate the
execution of an IRP.
• Follow a systematic set of guidelines to define the classes
of the ontology and the relationship among the classes.
• A prediction module to automatically classify the
activities with text description according to the ontology.
• An interoperability model to select the best suits of tools
that have the required capability to execute an IRP.
CREST Centre | University of Adelaide 13

14. Proposed Solution (2/4)
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019 14
Ontological Model to enable Semantic Integration
The proposed ontology contains three main classes, these classes are defined to formally represent
heterogenous security tools.

15. Proposed Solution (2/4)
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019 15
Ontological Model to enable Semantic Integration
The proposed ontology contains three main classes, these classes are defined to formally represent
heterogenous security tools.

16. Proposed Solution (2/4)
16
# Response Task Action
ac1 Is this a phishing
attack?
Determine if this is a phishing attack? In the
task, select yes or no in the outcome.
ac2 Scan endpoint –
malware found?
After running a scan, determine whether
malware was found. In the task, select yes or
no in Outcome.
ac3 Remove malware
– success?
Determine whether the malware was
successfully removed. In the task, select Yes
or no in outcome.
Ontological Model to enable Semantic Integration
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
We define the categories of the activities as a subclass of the activity class. The activities are
associated with detection prevention, recover and remediation actions of a threat defined.

17. Proposed Solution (2/4)
17
# Response Task Action
ac1 Is this a phishing
attack?
Determine if this is a phishing attack? In the
task, select yes or no in the outcome.
ac2 Scan endpoint –
malware found?
After running a scan, determine whether
malware was found. In the task, select yes or
no in Outcome.
ac3 Remove malware
– success?
Determine whether the malware was
successfully removed. In the task, select Yes
or no in outcome.
Ontological Model to enable Semantic Integration
ac1 : Is (Verb) this (Det) a (Det) phishing (Verb) Attack (Noun) ? (Punc)
. = Is (Validate) Phishing Attack
Subclass: Validate  Validate Phishing  Validate Phishing Email
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
We follow a system set of guidelines to define the classes of the activity class manually.

18. Proposed Solution (3/4)
CREST Centre | University of Adelaide 18 |
Classification of Activities based on Text Similarity
Prediction Module
Activity description Activity class from ontology
Textpreprocessing Data splitting N-gram generation Feature transformation Model training and evaluation
Model selection based on K fold cross validation
Dataset

19. Proposed Solution (3/4)
CREST Centre | University of Adelaide 19 |
Classification of Activities based on Text Similarity
Prediction Module
Activity description Activity class from ontology
Textpreprocessing Data splitting N-gram generation Feature transformation Model training and evaluation
Model selection based on K fold cross validation
Dataset
• Part-of-speech
tagging
• Remove Null value,
Punctuation, Stop
words and
Meaningless words
• Four classifiers
• Evaluation metrics:
Accuracy, recall,
precision, f1-score

20. Activity description Activity class from ontology
Text preprocessing Data splitting N-gram generation Feature transformation Model training and evaluation
Model selection based on K fold cross validation
N-gram generation
Optimal classifier
Model trainingFeature transformation
Feature configuration
Model building
Dataset
Trained model
Proposed Solution (3/4)
CREST Centre | University of Adelaide 20 |
Classification of Activities based on Text Similarity
Prediction Module

21. Activity description Activity class from ontology
Text preprocessing Data splitting N-gram generation Feature transformation Model training and evaluation
Model selection based on K fold cross validation
N-gram generation
Optimal classifier
Model trainingFeature transformation
Feature configuration
New activity description Text preprocessing Feature transformation Classification of activity description
Model building
Dataset
Prediction
Trained model
Feature model
Proposed Solution (3/4)
CREST Centre | University of Adelaide 21 |
Classification of Activities based on Text Similarity
Prediction Module
The prediction module reduces the manual analysis of the activity description by
classifying the activity according to the ontology.

22. Proposed Solution (4/4)
22 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

23. Proposed Solution (4/4)
23 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

24. Proposed Solution (4/4)
24 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

25. Proposed Solution (4/4)
25 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

26. Proposed Solution (4/4)
26 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

27. Proposed Solution (4/4)
27 |
An Interoperability Model to Select the Security Tools to Automate
the Sequence of Activities in an Incident Response Plan
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

28. Evaluation
How feasible is the proposed
prediction module and
interoperability model?
CREST Centre | University of Adelaide 28

29. Experiments and Results (1/3)
29 |
Preparing the Dataset for Prediction module
Activity Description Level 1 Level 2 Level 3
Scan endpoint to see whether
malware was found
Scan ScanEndpoint ScanEndpointMalware
Is this a phishing email Validate ValidatePhishing ValidatePhishingEmail
Isolate the malicious node from the
network
Isolate IsolateMalicious IsolateMaliciousNode
• 34 categories under level 1
• 67 categories under level 2
• 74 categories under level 3
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019
https://docs.servicenow.com/
Crawled 1080 activity descriptions

30. Experiments and Results (2/3)
CREST Centre | University of Adelaide 30 |
Implementing the Prediction module
0.926
0.909
0.865
0.93
0.911
0.869
0.833
0.817
0.792
0.918
0.905
0.864
LEVEL 1 LEVEL 2 LEVEL 3
SVM RF NB LR
0.96
0.94
0.9
0.97
0.93
0.87
0.96
0.94
0.9
0.96
0.93
0.88
LEVEL 1 LEVEL 2 LEVEL 3
Accuracy Precision
Recall F1-score
Validated weighted average of F1-score for optimal
configuration of different classifiers, SVM
(Support Vector Machine), RF (Random Forest),
NB (Naïve Bayes), LR (Linear Regression)
Testing results of Random Forest
for three levels of classes

31. Experiments and Results (3/3)
• Seven security solutions: Snort, Splunk, Limacharlie,
Wireshark, WinPcap, Microsoft security essential, MISP
• 21 different capabilities
• Nine incident response plans with 17 activities.
31 |
Success Rate around 90%
Developing Interoperability Model
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

32. Limitations
• The ontology is developed based on freely available and
open source security tool’s capabilities and activities.
• The selected optimal classifiers may not guarantee the
highest performance for classifying new and updated
activity description
CREST Centre | University of Adelaide 32

33. Future Work
• Use the semantic definition of the tools capabilities to
auto-create the APIs when new security tools with new
capability are integrated.
• Design a probabilistic model for selecting and integrating
security tools.
Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019 33

34. Key Takeaways
• An ontological model to formalize the security
tools, their capabilities and activities of Incident
Response Plan (IRP).
• A learning-based prediction module to
automatically define the classes for new activity
according to the ontology.
• An interoperability model to select the required
security tools that are interoperable for auto-
execution of IRP.
CREST Centre | University of Adelaide 34

35. References
• Chadni Islam, Muhammad Ali Babar, and Surya Nepal, “Automated Interpretation
and Integration of Security Tools Using Semantic Knowledge”, 31st International
Conference on Advanced Information Systems Engineering, CAiSE, 2019, Rome,
Italy, 3-7 June 2019.
• Chadni Islam, Muhammad Ali Babar, and Surya Nepal. 2019. A Multi-Vocal
Review of Security Orchestration. ACM Computing Survey. Vol 52, Issue 2,
Article 37 (April 2019), 45 pages. DOI: https://doi.org/10.1145/3305268
• Chadni Islam, Muhammad Ali Babar, and Surya Nepal, “An Ontology-Driven
Approach to Automate the Process of Integration Security Software Systems”,
International Conference of Software and Systems Process, ICSSP, 2019, Montreal,
Canada, 2019.
• https://www.securityweek.com/cost-data-breach-uk-increases-more-41-two-years
• https://www.ibm.com/downloads/cas/861MNWN2
• https://cybersecurityventures.com/cybercrime-damages-6-trillion-by-2021/
• https://bricata.com/blog/cybersecurity-alert-deluge
35 |Automated Interpretation and Integration of Security Tools Using Semantic Knowledge| CAiSE 2019

36. Questions?
Chadni Islam
CREST Centre
https://crest-centre.net/
School of Computer Science,
University of Adelaide, Australia
CSIRO’s Data61, Australia
https://research.csiro.au/distributed-
systems-security/
CREST Centre | University of Adelaide 36
M. Ali Babar
CREST Centre
https://crest-centre.net/
School of Computer Science,
University of Adelaide,
Australia
Email: ali.babar@adelaide.edu.au
Surya Nepal
CSIRO’s Data61,
Australia
Email:surya.Nepal@data61.csrio.au
https://research.csiro.au/distributed-
systems-security/
Twitter: @_chadni_
Email: chadni19@gmail.com,
chadni.islam@Adelaide.edu.au
Twitter: @alibabar