Discusses the role of ML and AI in Security. Discusses some problems with training and decision surfaces. Explains why ML models in security overestimate their accuracy.

1. Copyright 2018 Trend Micro Inc.
1
Role of ML in Security
Jonathan Oliver

2. Copyright 2018 Trend Micro Inc.
2
“Use Machine Learning”
“Not yet…”

3. Copyright 2018 Trend Micro Inc.
3
Cybercriminal
Business
Models

4. Copyright 2018 Trend Micro Inc.
4
2000s
How to monetize a malware infection Return
Spambot $
Banking Trojan $$
Stealing Intellectual Property $$
Ransomware $$$
How to monetize a malware infection Return
Rent: Spambot / DDOS $
Banking Trojan $$
Stealing Intellectual Property $$
2014

5. Copyright 2018 Trend Micro Inc.
5
Changes brought about by Ransomware
Issue Pre Ransomware
Type of Solution Reactive Solutions were
good enough
Business Critical
Problem:
FP in the OS
Backup Solutions Sophisticated
organizations had backup
solutions

6. Copyright 2018 Trend Micro Inc.
6
Changes brought about by Ransomware
Issue Pre Ransomware Post Ransomware
Type of Solution Reactive Solutions were
good enough
Time Zero Solutions
required
=> Adoption of ML
Business Critical
Problem:
FP in the OS Ransomware infection
Backup Solutions Sophisticated
organizations had backup
solutions
Everyone needs backup
solutions

7. Copyright 2018 Trend Micro Inc.
7
2015-2018: New Business Models
Malware Type Notes
Ransomware Has to adapt to changing defensive
posture
Coinmining Up and down with value of
cryptocurrencies
InfoStealer Can be first stage of other attacks
(ransomware)

8. Copyright 2018 Trend Micro Inc.
8
ML / AI
A Silver Bullet?

9. Copyright 2018 Trend Micro Inc.
9
Copyright 2015 Trend Micro Inc. 9
No silver bullets
Industry Perspective

10. Copyright 2018 Trend Micro Inc.
10
10
Academic Perspective

11. Copyright 2018 Trend Micro Inc.
11
2011

12. Copyright 2018 Trend Micro Inc.
12
2016

13. Copyright 2018 Trend Micro Inc.
13
2016
2017

14. Copyright 2018 Trend Micro Inc.
14
Advance System Game Requirements
1997 Deep
Blue
Chess Strategy. Looking ahead.
Each piece has its own rules.
Very complex
2011 Watson Jeopardy Can be about anything.
Open ended.
2016 AlphaGo Go Amazingly complex
Every piece is identical
2017 Libratus Poker Bluffing. Lack of Knowledge.
S
E
C
U
R
I
T
Y

15. Copyright 2018 Trend Micro Inc.
15
Role of ML and AI in Security
Creating an attack surface which
1. Is difficult to attack
2. Can be maintained

16. Copyright 2018 Trend Micro Inc.
16
Attack Surface ~
ML Decision Surface
In ML process,
selected during
training / testing
https://www.kdnuggets.com/2015/06/decision-boundaries-deep-
learning-machine-learning-classifiers.html

17. Copyright 2018 Trend Micro Inc.
17
Machine Learning
Life Cycle

18. Copyright 2018 Trend Micro Inc.
18
Machine Learning Life Cycle
Identify
Problem
Data
Collection
/ Cleaning
Train /
Test
Implement /
Dev
Maintain

19. Copyright 2018 Trend Micro Inc.
19
Issue 1: Training / Algorithm Selection
Traditional Approach
1. Split the dataset into 2 (say 70%/30%)
2. Train on 70%
3. Test on 30% select the model with best results
Can result in complex models or ensemble models…

20. Copyright 2018 Trend Micro Inc.
20

21. Copyright 2018 Trend Micro Inc.
21
The Problem
Wide gap between testing and real world results.
Discussed at anti-spam conferences
• CEAS
• MIT spam conference
But not addressed
Lorenzo Cavallaro ‘inflated results’ (MRE 2018)

22. Copyright 2018 Trend Micro Inc.
22
How wide a gap?
Anti-spam Benchmarks: 97%-98%
Anti-spam Published Results: 99%+ (up to 99.87%)

23. Copyright 2018 Trend Micro Inc.
23

24. Copyright 2018 Trend Micro Inc.
24
What is Going Wrong?
Malicious events typically come in clusters:
“Outbreaks”
And this has important impact in Training / Test
ML is very effective within clusters
ML is less effective at generalizing from one cluster to
other clusters

25. Copyright 2018 Trend Micro Inc.
25
xx
x

26. Copyright 2018 Trend Micro Inc.
26
Demo
Hierarchical clustering of malware (5 million files)
Distance metric: TLSH
Daisuke Inoue
“Massiveness reveals new insights”
3,749,768 / 5,000,000 (75%) were clustered

27. Copyright 2018 Trend Micro Inc.
27
Distance metric: TLSH
https://github.com/trendmicro/tlsh
Uses byte level features (ngram-like 2skip-5gram)
4ED2054D3AA64922C2BDAB7848F5834102F282130527D7EF4CC9ACE55B772D40640DE4
File 1 File 2
TLSH
Distance

28. Copyright 2018 Trend Micro Inc.
28
Tight clusters using byte level features
Members of a cluster are very likely to have similar
feature vectors (or identical feature vectors)
Lets look at malware clusters

29. Copyright 2018 Trend Micro Inc.
29
xx
x

30. Copyright 2018 Trend Micro Inc.
30
Conclusion Issue 1
For the purposes of the traditional train/test split
Security is different than many other tasks
A key task for security is to generalise beyond clusters
to related clusters
More thorough train/test protocols need to be used.
Time based testing

31. Copyright 2018 Trend Micro Inc.
31
Observations of ML Life Cycle
Considerations from later in the life cycle may impact
steps earlier in the ML life cycle

32. Copyright 2018 Trend Micro Inc.
32
Machine Learning Life Cycle
Identify
Problem
Data
Collection
/ Cleaning
Train /
Test
Implement /
Dev
Maintain
Missed detections
FPs
Retraining
Attacks on ML
Maintenance
considerations
impact on
algorithm selection

33. Copyright 2018 Trend Micro Inc.
33
Maintenance: Issue 2 Retraining
• We collect errors
• Missed detections
• False positives
⇒ Retrain
• Standard ML retraining will typically change a
decision surface in unexpected ways

34. Copyright 2018 Trend Micro Inc.
34
Maintenance: Decision Surfaces
Before Retraining After Retraining
Customer has
samples here

35. Copyright 2018 Trend Micro Inc.
35
Issue 2: retraining
• Simple models (such linear models) can be
modified in ways that guarantee new unexpected
FPs will not be created.
• Further research that guarantees that un-related
FPs will not be created

36. Copyright 2018 Trend Micro Inc.
36
Maintenance Requirements for Security
• False Alarms / False Positives are a big deal
• Need to be explained

37. Copyright 2018 Trend Micro Inc.
37
Demo
Look at mixed clusters that need explaining

38. Copyright 2018 Trend Micro Inc.
38
Attacking ML/AI

39. Copyright 2018 Trend Micro Inc.
39
Source: “Intriguing properties of Neural Networks”, Szegedy et al., Feb 2014, https://arxiv.org/pdf/1312.6199.pdf
ML tricked to recognize right-hand photos as an ostrich

40. Copyright 2018 Trend Micro Inc.
40
Attacking ML
NIPS competition 2017
• Gradient descent attacks
• …
https://arxiv.org/pdf/1804.00097.pdf

41. Copyright 2018 Trend Micro Inc.
41

42. Copyright 2018 Trend Micro Inc.
42
Underlying Issues
White Box Attacks (access ML model) versus
Black Box Attacks (access to ML results)
Issue 3: Where should ML reside?
• In the Cloud?
• On the endpoint?

43. Copyright 2018 Trend Micro Inc.
43
ML on the client
Extract Features
Check model
Make decision
Backend Server
model
file

44. Copyright 2018 Trend Micro Inc.
44
ML in the Cloud
Extract
Features
Backend Server
Check ML model
Make decision
features
decision
file

45. Copyright 2018 Trend Micro Inc.
45

46. Title
xxx

47. Copyright 2018 Trend Micro Inc.
47
Issue 3: Where should ML reside?
Issues:
• Working on air-gapped systems
– Airgapped systems are a minority of servers
• What happens when ML is attacked
– Ease of Attack?
– Updating ML models

48. Copyright 2018 Trend Micro Inc.
48
Alternatives: Hybrid ML
ML model on the client – identifies most malware
at time zero
ML model in the cloud – for malware which is
evading ML

49. Copyright 2018 Trend Micro Inc.
49
Extra Topic:
Combining ML and
Traditional Security

50. Copyright 2018 Trend Micro Inc.
50
False Positives
• Typical anti-malware might scan 30,000 files per day (approx)
• So if we processed all files with ML – then even a 1/30,000 FP
would be very bad
• 1 in a million FP rate would result in a FP every 33 days
⇒So we sort out the good files as well

51. Copyright 2018 Trend Micro Inc.
51
FPs in OS…

52. Copyright 2018 Trend Micro Inc.
52
Web & File Reputation
Exploit Prevention
Application Control
Variant Protection
Behavioral Analysis
Pre-execution Machine Learning
LEGEND
Known
Good Data
Known
Bad Data
Unknown
Data
Noise
Cancellation
Runtime Machine Learning
The Right Technique
At The Right Time
30,000 files
per day
30 files
per day

53. Copyright 2018 Trend Micro Inc.
53
Interaction ML
Ideally our traditional security results in ML that focusses on the
most suspicious files:
– Web downloads
– Executable files in Emails
– Executable files from USB
– Executable files dropped by unexpected processes (like
SMB)
– …

54. Copyright 2018 Trend Micro Inc.
54
Interaction ML
Sandboxes are really useful…
Ideally we want to sandbox
the entire internet…
(many many files)
Use ML / Clustering to select
samples that go in the
sandbox…

55. Copyright 2018 Trend Micro Inc.
55
Conclusion
Training / Testing protocols
Retraining impact on decision surfaces
Intelligent interaction
• Traditional Security and ML / AI

56. Copyright 2018 Trend Micro Inc.
56
Backup
Slides

57. Copyright 2018 Trend Micro Inc.
57
Demo of Evolutionary Tree
Data: 5,000,000 samples
Timing
1 hour to cluster (optimized algorithms)
5 hours to create trees (needs to use optimized algorithms)
Key Feature: Automatically created index into 5 million files

58. Copyright 2018 Trend Micro Inc.
58
Summary of Clustering
Found 288131 tight clusters (T=30)
Samples Clustered
Not
clustered
Percent
Clustered
Good 3138792 2575823 562969 82.06%
Malware 1861208 1173945 687263 63.07%
Total 5000000 3749768 1250232 75.00%

59. Copyright 2018 Trend Micro Inc.
59
TLSH – Versions of Chrome
TLSH Chrome
version
09159c11f085c0b3e56311b2c869ba34427cbe68832655db639c7e1a2db47d2363b7db 55.0.2877.0
10159c11f085c0b7d57201b1d87ab775067cbc28832681db73a87e1a3eb47d2267b69b 55.0.2882.4
94158d51f0858077e57210b2d8bdba30427cbd68432a65db63987d1a3db47d23a3b6cb 56.0.2899.0
88159e51f045c1b7e5b31071d8b9bb71067cad28431691db6398be2a3db03d22e3b69b 57.0.2957.1
2b158d11f045c1b7e5b310b1d8b9bb75057cad28831691db6398be1a3db03d22e3b69b 57.0.2966.0
12158d12f485c077e5b210b1d8bd7a34427cbd28432655db63987d2a2eb43d13a3badb 57.0.2975.1
cb159e11f049c1b7d5b211b1d87dba31467cbc28832a41db63d87e1a3db47d22a3b69b 57.0.2978.4
c4159d11f445c1b7d5b211b2d47dba71467cbc28832a40db63987e1a3eb43d22a3b6db 57.0.2987.4
50159d12f081c177e9721072d8bd7a34467cad28432655db63987e1a3eb43d13a3badb 58.0.3012.1

60. Copyright 2018 Trend Micro Inc.
60
The distance between 2 versions of Chrome
chrome.exe
SHA256:c70b8cbb2ac962b343535454e4f2bcb3e48d83a04792c64bc768d59b3c1bf403
1c159d11f445c1b7e5b211b2d879ba71467cbc28832641db63987e1a3db03d23a3b6db
c4159d11f445c1b7d5b211b2d47dba71467cbc28832a40db63987e1a3eb43d22a3b6db
chrome.exe
SHA256:723aa4a407160bd99430de690f1f0d34af4a6622e2c44fe95be3bda3d7c344b3

61. Copyright 2018 Trend Micro Inc.
61
Distance calculation
1c159d11f445c1b7e5b211b2d879ba71467cbc28832641db63987e1a3db03d23a3b6db
c4159d11f445c1b7d5b211b2d47dba71467cbc28832a40db63987e1a3eb43d22a3b6db
1 1 2 1 2 1 1
8: 1000 6: 0110
4: 0100 a: 1010
Ham(8,4)=2 Ham(6,a)=2
⇒Distance = 9

62. Copyright 2018 Trend Micro Inc.
62
How Good a Ruler is TLSH?
For Security Applications, we did a study across
• Executable files
• HTML files
• Text files
• JavaScript files
• Image files
• Source code files
• …
To evaluate how well the distance matches, we use a ROC curve

63. Copyright 2018 Trend Micro Inc.
63
How Good a Ruler is TLSH: ROC Curve
https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf