Presentation of SerIot Hypothesis Testing module developed during the SerIoT project. More: https://www.seriot-project.eu

1. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
Secure and Safe Internet ofThings (SerIoT)
1 Horizon 2020, Project No. 780139
20.01.2021
Hypothesis testing module
Information Technologies Institute, Centre of Research and Technology Hellas (CERTH/ITI)

2. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
Hypothesis testing Tool (1/2)
2 Horizon 2020, Project No. 780139
The Hypothesis testing module allows the security operator to investigate how
changing mitigation actions affects various KPI and if the KPI resulting from the
modification are statistically different when compared to those occurring from a
starting mitigation action set.
General concept: Τhe KPI values resulting from different mitigation strategies are
used to create clusters by employing a machine learning algorithm. The difference
between these clusters is evaluated by means of a p-value provided by statistical
method.
The tool uses the serIoT Mitigation Engine component as a basis, in terms of KPI
used.

3. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
Hypothesis testing Tool (2/2)
3 Horizon 2020, Project No. 780139
 The tool answers the following question "Are two clusters of mitigation actions
𝐶𝐴 or 𝐶𝐵 different in terms of their underlying distribution and is this
difference statistically significant?".
 This can be formed as a hypothesis :
 𝐻0 The clusters come from the same distribution (𝑝≤0.05) or
 𝐻1 The clusters come from a different distribution (𝑝≥0.05).
 The HDBSCAN machine learning algorithm is used to cluster the mitigation
actions while a method called Statistical Significance of Clustering using Soft
Thresholding is used to assess the difference.

4. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
High level overview of module operation
Horizon 2020, Project No. 780139
4
1. The operator can modifies existing or adds new mitigation actions via the Visual analytics module.
2. The hypothesis testing tool a) calculates KPI values for new and existing mitigations and b) a HDBSCAN
clustering model trained on historical data.
3. The clustering model is applied to see to which clusters, the existing and new mitigations belong to.
4. The SigClust model is applied and the comparison results along with a p-value become available to the
operator.

5. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
KPIs used for Clustering
5 Horizon 2020, Project No. 780139
 Common Vulnerability Scoring System (CVSS) is an Industry standard for
assessing the severity of a cybersecurity vulnerability.
 Return on response investment (RORI) is tool used to calculate (a self-named) an
index associated to the mitigation actions composing a response plan.
 The Vulnerabilities Surface Coverage (VSC) or Vulnerability Coverage of a
countermeasure cm, is found by counting the number of vulnerabilities it covers.
 The Deployment Cost KPI considers deployment time, consumed resources and
the importance of the device that deploys the countermeasure as assessed by the
network security operator .

6. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
Experimental results
6 Horizon 2020, Project No. 780139
Results of proposed method
Ground Truth
Assessment
Same Cluster Different Cluster Accuracy
Same Cluster 8 1 95.74%
Different
Cluster
1 36
Results of Jackstraw* method
Ground Truth
Assessment
Same Cluster Different Cluster Accuracy
Same Cluster 9 0 89.13%
Different
Cluster
5 32
• An experiment was performed with a dataset of 6300
instances.
• The cluster resulted to 10 dense Clusters of KPI values
(shown in figure).
• The proposed method outperforms a SoA method by
≈6% accuracy in correctly assessing Cluster
membership
*N.C. Chung (2020), “Statistical significance of cluster membership for
unsupervised evaluation of cell identities”, Bioinformatics, Volume 36,
Issue 10, 3107:3114

7. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
Hypothesis Testing Integration with
Visual Analytics Dashboard
7 Horizon 2020, Project No. 780139
Current and
modified
mitigation
KPI values
Modified
mitigation
actions can be
directly applied
to the network
Existing
mitigation
actions are
modified or new
added by clicking
a device

8. This project has received funding from
the European Union’s Horizon 2020
Research and Innovation programme
under grant agreement No. 780139
8 Horizon 2020, Project No. 780139
 SerIoT project (2020), Deliverable ‘D4.5. Unsupervised IoT-ready engine for threat
mitigation’
- Hypothesis Testing
 Huang H, Liu Y, Yuan M, Marron JS (2015). Statistical Significance of Clustering using Soft
Thresholding. J Comput Graph Stat.. Volume 24, issue 4.
 L. McInnes, J. Healy, S. Astels,(2017). “hdbscan: Hierarchical density based clustering In: Journal of
Open Source Software”, The Open Journal, volume 2, number 11.
Related Publications