Dissertation Defense: Improved Security Detection & Response via Optimized Alert Output: A Usability Study

1. Improved Security Detection & Response via
Optimized Alert Output: A Usability Study
CapitolTechnology University
Dissertation Defense
by
G. Russell McRee
Dissertation Chair: Ian McAndrew PhD FRAeS
Dissertation Committee: Dr. Atta-Ur-Rahman (Examiner), Allen H. Exner (Ex Officio)
17 AUG 2021

2. Statement of the Problem
• Organizations risk data breach, loss of valuable human resources,
reputation, and revenue due to excessive security alert volume and a lack of
fidelity in security event data
• These organizations face a large burden due to alert overload, where 99% of
security professionals surveyed acknowledge that high volumes of security
alerts are problematic

3. Rationale for the Study
• This study addresses challenges inherent in data overload and complexity,
using security data analytics derived from machine learning (ML) and data
science models that produce alert output for analysts
• Security analysts benefit in two ways:
• Efficiency of results derived at scale via ML models
• Benefit of quality alert results derived from the same models.

4. Literature Overview
• Security data visualization can be used to address related human cognitive
limitations (Rajivan, 2011)
• Giacobe (2013) discussed the effectiveness of visual analytics and data
fusion techniques on situation awareness in cyber-security, and focused on
visual analytics, data fusion, and cybersecurity
• Giacobe found that participants using the visual analytics (VA) interface performed
better than those on the text-oriented interface, where the visual analytic interface
yielded a performance that was quicker and more accurate that the text interface.
• Giacobe conducted an experiment and survey separately
• This study merged quasi-experiment in survey

5. Research Methodology/Design
• Quantitative, quasi-experimental, explanatory study
• TechnologyAcceptance Model (TAM)
• Methodology utilized to statistically measure security analysts’ acceptance
of two security alert output types: visual alert output (VAO) & text alert
output (TAO)
• A qualitative methodology & design was not considered as the business
problem is one of data.The study’s data-driven findings can contribute to
data-informed business decisions.

6. Data Analysis
• DV: level of acceptance of the security alert output and is based on the four individual
TAM components: PU, PEU, AU, and IU
• Within-subjects IV: Scenario (3x), all participants subject to all scenarios
• Between-subjects IV: Maximum Visual
• Two levels: a preference forVAO in all three scenarios, and a preference forTAO in at least one
of the scenarios
• Mixed ANOVA to test level of acceptance of alert outputs as influenced by the within-
subjects variable Scenario and the between-subjects variable Maximum Visual
• Mann-Whitney U test performed to compare level of acceptance of alert outputs of
the two levels of MaximumVisual
• Friedman test performed to compare level of acceptance across the three scenarios

7. Findings (non-parametric)
Significant difference (U = 863.5, p = 0.023) in level
of acceptance of alert output between
respondents who selected visual output across all
scenarios (n = 59) compared to the respondents
who provided mixed responses (n = 22).
No significant difference between scenarios (𝑥^2
(2)=5.496, 𝑝< .064). Scenario mean ranks did not differ
significantly from scenario to scenario when not also
factoring for responses based on output preference
(MaximumVisual).

8. Findings – Mixed ANOVA
AllTAM measures (α = .05): a significant main effect of
MaximumVisual scores (F(1, 79) = 4.111, p = .046, ηp2 = .049)
on the level of acceptance of alert output as indicated by
sum of participants' scores for allTAM components (PU,
PEU, AU, and IU) between-subjects

9. Perceived Usability (α = .0125): a significant
main effect of MaximumVisual scores (F(1, 79)
= 7.643, p = .007, ηp2 = .088) on the level of
acceptance of alert output as indicated by sum
of participants' scores for Perceived Usability
(PU) between-subjects
Perceived Ease of Use (α = .0125): an insignificant main
effect of MaximumVisual scores (F(1, 79) = .842, p = .362,
ηp2 = .011) on the level of acceptance of alert output as
indicated by sum of participants' scores for Perceived Ease
of Use (PEU) between-subjects
Findings:
Mixed
ANOVA

10. Findings:
Mixed
ANOVA
AttitudeToward Using (α = .0125): an
insignificant main effect of MaximumVisual
scores (F(1, 79) = 4.566, p = .036, ηp2 = .055) on
the level of acceptance of alert output as
indicated by sum of participants' scores for
Attitude Toward Using (AU) between-subjects
Intention To Use (α = .0125): an insignificant main
effect of MaximumVisual scores (F(1, 79) = 4.378, p =
.040, ηp2 = .053) on the level of acceptance of alert
output as indicated by sum of participants' scores for
Intention to Use (IU) between-subjects

11. Findings – RQ1
• RQ1: Is there a difference in the level of acceptance of security alert output
between those with a preference for visual alert outputs (VAO) and those
with a preference for text alert outputs (TAO), withVAO andTAO
generated via data science/machine learning methods, as predicted by the
Technology Acceptance Model (TAM)? Yes.
• Non-parametric (between-subjects): U = 863.5, p = 0.023
• Parametric:
• Within-subjects: (F (1.455, 114.915) = 5.634, p = 0.010, ηp2 = .067)
• Between-subjects: (F (1, 79) = 4.111, p = .046, ηp2 = .049)

12. Findings – SQ1
• SQ1: Does the adoption ofVAO have a significant impact on the four
individualTAM components, perceived usefulness (PU), perceived ease of
use (PEU), attitude toward using (AU), and intention to use (IU)? In part.
• TheTAM components perceived usability (PU) and perceived ease of
use (PEU) are not significantly influenced by the adoption ofVAO
within-subjects while attitude toward using (AU), and intention to use
(IU) are significantly influenced by the adoption ofVAO within-subjects.
• TheTAM component perceived usability (PU) is significantly influenced
by the adoption ofVAO between-subjects.

13. Findings – SQ2
• SQ2: Does the adoption ofTAO have a significant impact on the four
individualTAM components, perceived usefulness (PU), perceived ease of
use (PEU), attitude toward using (AU), and intention to use (IU)? No.
• No individualTAM component is significantly influenced byTAO
adoption, andTAO adoption trailedVAO in near totality.

14. Recommendations for Research
• Security analysts likely seek an initial visual alert inclusive of the options to
dive deeper into the raw data. A future study could expose the degree to
which analysts seek multifaceted options
• A future study could further explore the perceptions of, and interactions
with, dynamic visualizations versus static visualizations
• Further explore, even under online survey constraints, a framework that
more robustly assesses user experience
• Opportunity exists to develop more nuanced data where information
specific to participant gender, location, age group, company or organization
size, and business sector could lead to improved insights

15. Thank you
Questions?
Once in a while, you get shown the light
In the strangest of places if you look at it right
~Garcia/Hunter