The document discusses using locality sensitive hashing (LSH) to identify similar malware samples, specifically fresh ransomware. It evaluates the performance of different LSH techniques like SSDEEP, SDHASH and TLSH for clustering a dataset of malware binaries. TLSH was found to work best by grouping similar samples and completing searches faster than alternatives like YARA rules. The proposed solution uses LSH to search a database of old ransomware for similar new files, aiming to efficiently identify fresh ransomware samples. Future work could include parallelizing searches, expanding the ransomware corpus, developing improved LSH methods, and adding labels to the malware database.

1. © 2018 CrySyS Lab, BME
EXAMPLES OF LOCALITY SENSITIVE
HASHING AND THEIR USAGE FOR
MALWARE CLASSIFICATION
Csongor Tamás
Budapest University of Technology and Economics
csotam@crysys.hu
Supervisors: Dr. Boldizsár Bencsáth, Dr. Levente Buttyán
w w w . c r y s y s . h u

2. |
Problem statement
 We need a feed of fresh ransomware!
 Fresh – from last week or month
 But how?
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 2
NotPetya

3. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 3
Old ransomware Feed of new files
Method
Fresh ransomware

4. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 4
Search corpus Feed of new files
Method
Similar samples

5. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 5
YARA rules Feed of new files
YARA rule matching
Fresh samples
Bad automatic
generation
Slow (0.015 s)

6. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 6
Old ransomware Feed of new files
Method
Fresh ransomware

7. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 7
Old ransomware Feed of new files
Method
Fresh ransomware
+ database
malware

8. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 8
Old ransomware Feed of new files
Method
Fresh ransomware
+ database
malware

9. |
Solution Concept
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 9
Old ransomware Feed of new files
Method
Fresh ransomware
+ database
LSH
malware

10. |
Locality Sensitive Hashing
Examples for Locality Sensitive Hashing and their usage for malware similarity checking 10
 What is Locality Sensitive Hashing?
–similar data –> ˝similar hash˝
–„aims to maximize the probability of a
collision for similar items”
–Distance can be calculated between two
digests (hashes)
–Similar files (hashes) are ˝close˝ to each
other, others are ˝far˝

11. |
Locality Sensitive Hashing
11
 SSDEEP
–Context Triggered Piecewise Hashing
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

12. |
Locality Sensitive Hashing
12
 SSDEEP
–Context Triggered Piecewise Hashing
 SDHASH
–Statistically improbable features
 TLSH
–TrendMicro Locality Sensitive Hash
–5-grams –> statistics –> hash
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

13. |
SSDEEP
13
o r h a n d s o f g o l
da r e a l w a y s c o l
d
,
F
o r l a n d s o f g o l
da r e a l w a y s c o l
d
,
F
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

14. |
Locality Sensitive Hashing
14
 Reasons:
– Small data to store
– Fast automatic generation
– Fast comparison
Examples for Locality Sensitive Hashing and their usage for malware similarity checking
YARA SSDEEP TLSH
0.015s 0.003s 0.002s
SSDEEP TLSH
0.100s 0.037s
YARA SSDEEP TLSH
Whole binary <110 bytes 70 bytes

15. |
Locality Sensitive Hashing
15
 Reasons:
– Small data to store
– Fast automatic generation
– Fast comparison
Examples for Locality Sensitive Hashing and their usage for malware similarity checking
YARA SSDEEP TLSH
0.015s 0.003s 0.002s
SSDEEP TLSH
0.100s 0.037s
YARA SSDEEP TLSH
Whole binary <110 bytes 70 bytes
But are they applicable?

16. |
Testing LSH on a small dataset
16
 Dataset:
–34681 real binaries
–NOT classified
 Clustering algorithms:
–1. simple – if two samples are ˝close˝ they
belong to the same group
–2. k-medoids – k group centers
–3. if similar to at least a few group members
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

17. |
Testing LSH on a small dataset
17
 Results:
–(evaluation by hand)
–Samples in the same group are similar
–SDHASH is not applicable
–SSDEEP score (˝closeness˝) is badly scaled
»0 - 100 (mismatch - perfect match)
–Similar samples in different groups
– TLSH appears to be the best for this application
»With threshold = 70
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

18. |
Search SSDEEP
18
 Original sample (GandCrabV4.X):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

19. |
Search SSDEEP
19
 Original sample (GandCrabV4.X):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

20. |
Search SSDEEP
20
 Original sample (GandCrabV4.X):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

21. |
Search SSDEEP
21
 Original sample (GandCrabV4.X):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

22. |
Search TLSH
22
 Original sample (Saturn):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

23. |
Search TLSH
23
 Original sample (Saturn):
 Similars:
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

24. |
 Original sample (Saturn):
 Similars:
Search TLSH
24Examples for Locality Sensitive Hashing and their usage for malware similarity checking

25. |
Moving on to the database
25
 Generate hashes for every sample
–~ 1-2 months
 Grouping algorithms use XREF
 XREF is not scalable
 300000000
2
* 0.002s ~= 2 853 881 y
 Search will do
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

26. |
Ransomware corpus & search
26
 Currently 477 samples from 15 families
 Search currently uses 1 process, 1 thread
 Search for similars to 1 sample
–SSDEEP –> ~10-20 minutes (prefix filter)
–TLSH –> ~50 minutes
 Search for similars to 477 samples
–SSDEEP –> 14 hours
–TLSH –> 29 hours
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

27. |
Search
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 27
Search corpus Malware database
LSH
Similar samples

28. |
Final Solution
EXAMPLES OF LOCALITY SENSITIVE HASHING AND THEIR USAGE FOR MALWARE CLASSIFICATION 28
Old ransomwares Feed of new files
LSH
Fresh ransomwares

29. |
Future work
29
 Parallelization
 Widen ransomware corpus
 Develop better LSH
 Label database
Examples for Locality Sensitive Hashing and their usage for malware similarity checking

30. © 2018 CrySyS Lab, BME
Questions?