In this talk, I cover the basic idea of hardware-assisted, two-level architectures for security monitoring and its applications to the malware detection problem. I propose detection triggers involving branch predictor, MMU, memory controller, co-processors, and FPGAs. Talk presented at the Real Time systems group seminar series at the University of York.

1. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Hardware-accelerated security monitoring
Marcus Botacin
1botacin@tamu.edu
marcusbotacin.github.io
Hardware-accelerated security monitoring 1 / 35 YORK

2. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Who Am I?
Assistant Professor (2022) - Texas A&M University (TAMU), USA
ACES Program Fellowship
PhD. in Computer Science (2021) - Federal University of Paraná (UFPR), Brazil
Thesis: “On the Malware Detection Problem: Challenges and new Approaches”
MSc. in Computer Science (2017) - University of Campinas (UNICAMP), Brazil
Dissertation: “Hardware-Assisted Malware Analysis”
Computer Engineer (2015) - University of Campinas (UNICAMP), Brazil
Final Project: “Malware detection via syscall patterns identification”
Hardware-accelerated security monitoring 2 / 35 YORK

3. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
The Problem
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 3 / 35 YORK

4. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
The Problem
Bottleneck: Real-time monitoring performance penalty
0
50
100
150
200
250
Perl Xalanc Gobmk H264 Namd Mcf
Time
(s)
Benchmark
AV’s Monitoring Performance
Filter AV SSDT AV No AV
Figure: AV Monitoring Performance.
0
50
100
150
200
250
300
perl namd Bzip milc mfc
Execution
Time
(s)
Benchmark
AV scanning overhead
Scan
Baseline
Figure: In-memory AV scans worst-case
and best-case performance penalties.
Hardware-accelerated security monitoring 4 / 35 YORK

5. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Solution
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 5 / 35 YORK

6. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Solution
Hardware AV Architecture
2-level Architecture
Do not fully replace AVs, but add effi-
cient matching capabilities to them.
Hardware-accelerated security monitoring 6 / 35 YORK

7. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Solution
Performance Characterization
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
5 10 15 20 25 30 35 40
CPU
(%)
Time (s)
AV Monitoring Overhead
HEAVEN+AV
AV
No−AV
2-Phase HEAVEN CPU Performance
The inspection phase causes occasional,
and quick bursts of CPU usage. The AV
operating alone incurs a continuous 10%
performance overhead.
Hardware-accelerated security monitoring 7 / 35 YORK

8. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 8 / 35 YORK

9. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Publication
Figure: Source:
https://www.sciencedirect.com/science/article/abs/pii/S0957417422004882
Hardware-accelerated security monitoring 9 / 35 YORK

10. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Branch patterns as signatures
Hardware-accelerated security monitoring 10 / 35 YORK

11. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Branch patterns as signatures
Figure: Two-level branch predictor. A
sequence window of taken (1) and not-taken
(0) branches is stored in the Global History
Register (GHR).
0
10
20
30
40
50
60
70
80
90
100
8 16 24 32 40
Percentage
of
signature
collision
in
the
k−bit
space
Branch pattern length (in k bits)
Percentage of signature collision per branch−pattern length (in bits)
Patterns
Figure: Branch patterns coverage.
Hardware-accelerated security monitoring 11 / 35 YORK

12. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Hardware AV Architecture
2-level Architecture
Do not fully replace AVs, but add effi-
cient matching capabilities to them.
Hardware-accelerated security monitoring 12 / 35 YORK

13. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
HEAVEN
Performance Characterization
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
5 10 15 20 25 30 35 40
CPU
(%)
Time (s)
AV Monitoring Overhead
HEAVEN+AV
AV
No−AV
2-Phase HEAVEN CPU Performance
The inspection phase causes occasional,
and quick bursts of CPU usage. The AV
operating alone incurs a continuous 10%
performance overhead.
Hardware-accelerated security monitoring 13 / 35 YORK

14. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
REHAB
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 14 / 35 YORK

15. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
REHAB
Publication
Figure: Source: https://ieeexplore.ieee.org/document/9034972/
Hardware-accelerated security monitoring 15 / 35 YORK

16. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
REHAB
Profiling-Based AV
1k
10k
100k
1M
10M
0 10 20 30 40 50 60
Classifier Separation
Branch−misses
(#)
Elapsed time (s)
Perf stat branch−miss:
Branch−miss stat comparison between goodware and malware during 60 seconds.
bluetooth−wizard
dpkg−log−summary
3f5b5...f0
3c55...50
Figure: Malware Classification using low
level features.
Figure: REHAB Architecture. CPU’s HPC
data is used as feature for a FPGA-based,
reconfigurable ML classifier updatable via
software.
Hardware-accelerated security monitoring 16 / 35 YORK

17. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
REHAB
Classifiers
b
b
b
O
u
t
p
u
t
X2
X3
X
I
n
p
u
t
X1
W
X
X
+
+
+
B
W3
W2
W1
B1
Figure: SVM.
I
n
p
u
t
W
<
<
0
O
u
t
p
u
t
X1
X2
W1
W2
Figure: Random Forest.
Hidden
Layer
Output
Input
N
e
u
r
o
n
O
u
t
p
u
t
X2
X3
X
I
n
p
u
t
X1
W
X
X
+
+
+
B
W3
W2
W1
B1
𝜑
Figure: MLP.
Hardware-accelerated security monitoring 17 / 35 YORK

18. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
REHAB
AV Checks Cost
Table: Execution Speedup per AV check. Hardware Accelerator is essential for overhead
elimination.
ML algorithm → SVM RF MLP
CPU 220µs 270µs 240µs
FPGA+Comm 124.5ns 111.2ns 158.9ns
Speedup 1.7k× 2.4k× 1.5k×
Hardware-accelerated security monitoring 18 / 35 YORK

19. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
SAP
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 19 / 35 YORK

20. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
SAP
Publication
Figure: Source: https://link.springer.com/article/10.1007/s11416-020-00348-w
Hardware-accelerated security monitoring 20 / 35 YORK

21. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
SAP
Architectural Support
Figure: Pipeline Stalls Detection. Figure: MMU Modification.
Hardware-accelerated security monitoring 21 / 35 YORK

22. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
SAP
Page Handling Overhead
Figure: Page Fault Handler. Figure: Performance Penalty.
Hardware-accelerated security monitoring 22 / 35 YORK

23. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
MINI-ME
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 23 / 35 YORK

24. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
MINI-ME
Publication
Figure: Source: https://dl.acm.org/doi/10.1145/3422575.3422775
Hardware-accelerated security monitoring 24 / 35 YORK

25. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
MINI-ME
Malware Identification based on Near- and In-Memory Evaluation
(MINIME)
Figure: MINIME Architecture.
Hardware-accelerated security monitoring 25 / 35 YORK

26. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
MINI-ME
Performance Gains
MINIME vs. On-Access AVs
Significant performance gains even
in the worst case.
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
9.0%
10.0%
11.0%
12.0%
13.0%
perl namd bzip mcf milc
Execution
Time
Overhead
(%)
Monitoring Overhead
On−Access
MINI−ME
Hardware-accelerated security monitoring 26 / 35 YORK

27. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 27 / 35 YORK

28. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Publication
Figure: Source: https://dl.acm.org/doi/10.1145/3494535
Hardware-accelerated security monitoring 28 / 35 YORK

29. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Function Arguments Inspection
Figure: Function Interposition. Figure: Matching Framework.
Hardware-accelerated security monitoring 29 / 35 YORK

30. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Inspection Triggering
Figure: Inspection Breakpoint.
Hardware-accelerated security monitoring 30 / 35 YORK

31. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Parallel Execution Constraints
Figure: Scanning Cycles Boundary.
Hardware-accelerated security monitoring 31 / 35 YORK

32. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
TERMINATOR
Performance Penalty Reduction
1×108
1×109
1×1010
1×1011
1×10
12
1×10
13
1×10
14
blender nab roms bwaves djeng perl cam4 cactusomnetpp mcf wrf x264 xzr leela parest lbm namd imagick povray xalanc gcc echg2
Cycles
(logscale)
Benchmark
AV’s Performance Overhead
AVSW
AVHW
BASE
Figure: Performance evaluation when tracking all function calls. Comparison between
execution without AV (BASE), execution with software AV, and execution with the proposed
coprocessor model.
Hardware-accelerated security monitoring 32 / 35 YORK

33. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Recap & Remarks
Topics
1 Introduction
The Problem
Solution
2 Signature Matching
HEAVEN
3 HPC Classification
REHAB
4 Packer Identification
SAP
5 Fileless Malware Detection
MINI-ME
6 Function Checking
TERMINATOR
7 Conclusions
Recap & Remarks
Hardware-accelerated security monitoring 33 / 35 YORK

34. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Recap & Remarks
Summary
Malware Detection
Huge performance penalties.
Increasing performance increases detection.
Academic Contributions
Branch patterns to replace byte-based signatures.
FPGAs to classify HPCs in runtime.
SMC-aware processor to detect packers.
Instrumented memory controller to detect fileless malware.
CPU coprocessors for real-time syscall checking.
Hardware-accelerated security monitoring 34 / 35 YORK

35. Introduction Signature Matching HPC Classification Packer Identification Fileless Malware Detection Function Checking Conclusions
Recap & Remarks
Thanks!
Questions? Comments?
@MarcusBotacin
botacin@tamu.edu
marcusbotacin.github.io
Hardware-accelerated security monitoring 35 / 35 YORK