The document discusses an automatic malware comparison system that uses graph theory and entropy analysis. The system analyzes malware samples using IDA Pro to extract flow graphs and adjacency matrices. Checksums and entropy values are calculated on code blocks to select similar samples for detailed comparison. Graph matching algorithms are used to identify common functions by comparing API calls, CRC32 hashes, and control flow graphs between samples. Identifying shared functions helps classify and cluster related malware variants.

1. Graph, Entropy and Grid Computing:
Automatic Comparison of Malware
Ismael Briones
Aitor Gomez
PandaLabs Malware Researcher
Virus Bulletin 2008, Ottawa

2. ChallengesChallenges
Identify new samplesIdentify new samples
AutomaticallyAutomatically
ASAPASAP
2
10/7/2008
Improve detection rateImprove detection rate
Malware nomenclatureMalware nomenclature

3. State of the ArtState of the Art
Ero Carrera & Gergely ErdélyiEro Carrera & Gergely Erdélyi
Digital Genome MappingDigital Genome Mapping
Halvar FlakeHalvar Flake
Lot of researches in graphs analysisLot of researches in graphs analysis
3
10/7/2008
Lot of researches in graphs analysisLot of researches in graphs analysis
VxClassVxClass
Marius GheorghescuMarius Gheorghescu
An Automated Virus Classification systemAn Automated Virus Classification system

4. The SystemThe System
Grid Manager
4
10/7/2008
Grid Clients
Comparison
Engine

5. The SystemThe System
Grid Manager
Gaobot Family
5
10/7/2008
Grid Clients
Comparison
Engine

6. Grid SystemGrid System
Analyse as many samples as possibleAnalyse as many samples as possible
IDA Analysis take too much timeIDA Analysis take too much time
BOINC, GLOBUS, ARC (Advanced Resource Connector)BOINC, GLOBUS, ARC (Advanced Resource Connector)
6
10/7/2008
BOINC, GLOBUS, ARC (Advanced Resource Connector)BOINC, GLOBUS, ARC (Advanced Resource Connector)
XMLRPCXMLRPC

7. Grid Server Grid Client
1) Agent2) Join Grid
7
10/7/2008

8. Grid Server Grid Client
1) Agent2) Join Grid
3) Send soft.
Grid Server Grid Client
7) Graph2DB
8) Request
samples
10) Start
comparison
8
10/7/2008
4) Req. task
5) Sample
Analize Sample
9) Send graphs
info
comparison
11) Send Result

9. Sample AnalysisSample Analysis
Ida Pro + IdaPython
9
10/7/2008

10. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
10
10/7/2008

11. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
11
10/7/2008

12. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
Adjacency MatrixAdjacency Matrix
12
10/7/2008
Columns & rows = graph nodes
(i,j) = 1, if i calls j

13. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
13
10/7/2008
Functions Control Flow Graph (CFG) signature

14. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
14
10/7/2008
Functions Control Flow Graph (CFG) signature

15. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph Block 3Block 3
Block 2Block 2
Block 1Block 1 API 1API 1
Signature: [6:8:3]Signature: [6:8:3]
15
10/7/2008
Functions Control Flow Graph (CFG) signature
Block 5Block 5
Block 6Block 6
Block 4Block 4
API 2API 2
API 3API 3

16. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
16
10/7/2008
Functions Control Flow Graph (CFG)
Function’s crc32

17. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
CRC32(Push+mov+push+push+push+call+…)
17
10/7/2008
Functions Control Flow Graph (CFG)
Function’s crc32

18. Sample AnalysisSample Analysis
Ida Pro + IdaPython
Flow Graph
18
10/7/2008
Functions Control Flow Graph (CFG)
Function’s crc32
Operating Systems and Library Calls (API’s)
Functions names (sub_, nullsub_,….)

19. Comparison AlgorithmComparison Algorithm
Select best samples
•Compiler type (Peid Signature)
19
10/7/2008
•Compiler type (Peid Signature)
•File Size
•Number Api Functions
•Number custom functions
•Checksum & Entropy

20. Checksum
‘A checksum is a form of redundancy check, […] It‘A checksum is a form of redundancy check, […] It
works by adding up the basic components of aworks by adding up the basic components of a
message, typically the assorted bitsmessage, typically the assorted bits [in our case[in our case
each byte]each byte], and storing the resulting value.’, and storing the resulting value.’
20
10/7/2008
[from wikipedia][from wikipedia]

21. Checksum PropertiesChecksum Properties
Similar blocks has very close checksum valuesSimilar blocks has very close checksum values
Checksum: 0x260A
21
10/7/2008
Checksum: 0x276D

22. Checksum PropertiesChecksum Properties
Similar blocks has very close checksum valuesSimilar blocks has very close checksum values
Checksum: 0x260A
22
10/7/2008
Checksum: 0x276D
Checksum Diff: 0x163Checksum Diff: 0x163

23. Checksum disadvantagesChecksum disadvantages
A checksum is not changed by:A checksum is not changed by:
Inserting or deleting zeroInserting or deleting zero--valued bytesvalued bytes
0xAA+0xBB = 0x165
0xAA+0x00+0xBB+0x00=0x165
23
10/7/2008
Reordering of the bytes in the data.Reordering of the bytes in the data.
0xAA+0xBB = 0x165
0xAA+0x00+0xBB+0x00=0x165
0xAA+0xBB = 0x165
0xBB+0xAA=0x165

24. Checksum value depends on 2 factorsChecksum value depends on 2 factors::
Size of data blockSize of data block
Base (basic component: bits, bytes,…)Base (basic component: bits, bytes,…)
256 (1 byte)
255 x 1024 = 0x03FC00
(3 bytes)
24
10/7/2008
1024 bytes
256 (1 byte)
65.536 (2 bytes)
4.294.976.296 (4 bytes)
4,294,967,295 x
256 =
0xFFFFFFFF00 (5
bytes)
65,535 x 512 =
0x01FFFE00
(4 bytes)

25. Checksum value depends on 2 factorsChecksum value depends on 2 factors::
Size of data blockSize of data block
Base (basic component: bits, bytes,…)Base (basic component: bits, bytes,…)
256 (1 byte)
255 x 1024 = 0x03FC00
(3 bytes)
Bigger base implies a
25
10/7/2008
1024 bytes
256 (1 byte)
65.536 (2 bytes)
4.294.976.296 (4 bytes)
4,294,967,295 x
256 =
0xFFFFFFFF00 (5
bytes)
65,535 x 512 =
0x01FFFE00
(4 bytes)
Bigger base implies a
bigger checksum for
the same data block

26. Bigger base involve a Bigger checksumBigger base involve a Bigger checksum
Decreases the probability of collision (same
checksum)
26
10/7/2008

27. Bigger base involve bigger checksum differenceBigger base involve bigger checksum difference
Similar data blocks could get significantly different checksumSimilar data blocks could get significantly different checksum
valuesvalues
27
10/7/2008

28. Bigger base involve bigger checksum differenceBigger base involve bigger checksum difference
Similar data blocks could get significantly different checksumSimilar data blocks could get significantly different checksum
valuesvalues
As code is represented as bytes, our
approach will use a base of 256
28
10/7/2008
As code is represented as bytes, our
approach will use a base of 256
(1byte)

29. Our ChecksumOur Checksum
4KB, in blocks of 1KB,4KB, in blocks of 1KB,
from the beginning of thefrom the beginning of the
1st , 2nd and last1st , 2nd and last
sectionssections
29
10/7/2008

30. Checksum AlgorithmChecksum Algorithm
Substract checksum from both filesSubstract checksum from both files
30
10/7/2008

31. Sample A Sample B
ABS(ChksmA – ChksmB) <= EntropyDiff
31
10/7/2008

32. Sample A Sample B
ABS(ChksmA – ChksmB) <= EntropyDiff
32
10/7/2008

33. Sample A Sample B
33
10/7/2008
ABS(ChksmA – ChksmB) <= EntropyDiff

34. Sample A Sample B
Sample Selected
34
10/7/2008
Sample Selected

35. Checksum vs EntropyChecksum vs Entropy
35
10/7/2008

36. Checksum Collisions & EntropyChecksum Collisions & Entropy
High entropy
Lots of collisions
36
10/7/2008
Low entropy
Less collisions
Low entropy
Less collisions

37. Checksum Collisions & EntropyChecksum Collisions & Entropy
High entropy
Lots of collisions
Medium-low entropy
37
10/7/2008
Low entropy
Less collisions
Low entropy
Less collisions
Medium-low entropy
contain code

38. Penalized checksumPenalized checksum
Penalized checksum:Penalized checksum:
Don’t use blocks with high entropyDon’t use blocks with high entropy
38
10/7/2008
Minimal checksum changes in highMinimal checksum changes in high--entropyentropy
blocks will generate distant checksum values.blocks will generate distant checksum values.

39. Comparison AlgorithmComparison Algorithm
Select best samples
Start comparison:
Graph Comparison
39
10/7/2008
Graph Comparison

40. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4
f1f1 00 11 00 11
f2f2 00 00 00 00
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4
f1’f1’ 00 11 00 11
f2’f2’ 00 00 00 00
40
10/7/2008
f3f3 11 00 00 00
f4f4 00 00 11 00
f3’f3’ 11 00 00 00
f4’f4’ 11 00 00 00

41. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4
f1f1 00 11 00 11
f2f2 00 00 00 00
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4
f1’f1’ 00 11 00 11
f2’f2’ 00 00 00 00
Common functions:Common functions:
••API FunctionsAPI Functions
41
10/7/2008
f3f3 11 00 00 00
f4f4 00 00 11 00
f3’f3’ 11 00 00 00
f4’f4’ 11 00 00 00
••API FunctionsAPI Functions
••Same and unique CRC32Same and unique CRC32
••Same and unique CFGSame and unique CFG

42. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4
f1f1 00 11 00 11
f2f2 00 00 00 00
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4
f1’f1’ 00 11 00 11
f2’f2’ 00 00 00 00
42
10/7/2008
f3f3 11 00 00 00
f4f4 00 00 11 00
f3’f3’ 11 00 00 00
f4’f4’ 11 00 00 00

43. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4
f1f1 00 11 00 11
f2f2 00 00 00 00
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4
f1’f1’ 00 11 00 11
f2’f2’ 00 00 00 00
43
10/7/2008
f3f3 11 00 00 00
f4f4 00 00 11 00
f3’f3’ 11 00 00 00
f4’f4’ 11 00 00 00
f1=f1’ and f1 != {f2’,f3’,f4’}f1=f1’ and f1 != {f2’,f3’,f4’}

44. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4 f1f1
f2f2 00 00 00 00 11
f3f3 11 00 00 00 11
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4 f1’f1’
f2’f2’ 00 00 00 00 11
f3’f3’ 11 00 00 00 11
44
10/7/2008
f3f3 11 00 00 00 11
f4f4 00 00 11 00 00
f3’f3’ 11 00 00 00 11
f4’f4’ 11 00 00 00 00

45. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4 f1f1
f2f2 00 00 00 00 11
f3f3 11 00 00 00 11
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4 f1’f1’
f2’f2’ 00 00 00 00 11
f3’f3’ 11 00 00 00 11
45
10/7/2008
f3f3 11 00 00 00 11
f4f4 00 00 11 00 00
f3’f3’ 11 00 00 00 11
f4’f4’ 11 00 00 00 00
f2=f2’ and f2 != {f3’,f4’}f2=f2’ and f2 != {f3’,f4’}

46. Matrix AMatrix A
a1a1 a2a2 a3a3 a4a4 f1f1 f2f2
f3f3 11 00 00 00 11 11
f4f4 00 00 11 00 00 11
Matrix BMatrix B
a1a1 a2a2 a3a3 a4a4 f1’f1’ f2’f2’
f3’f3’ 11 00 00 00 11 00
f4’f4’ 11 00 00 00 00 11
46
10/7/2008
f4f4 00 00 11 00 00 11 f4’f4’ 11 00 00 00 00 11
f3 != {f3’,f4’} and f4 != {f3’,f4’}f3 != {f3’,f4’} and f4 != {f3’,f4’}
Two functions have been identified: f1 and f2Two functions have been identified: f1 and f2

47. Comparison AlgorithmComparison Algorithm
Select best samples
Start comparison:
Graph Comparison
47
10/7/2008
Graph Comparison
Match more functions with CFG

48. CFG signature = 3CFG signature = 3--tuple vector in Euclidean spacetuple vector in Euclidean space
Find minimal and unique ED among functionsFind minimal and unique ED among functions
Control Flow GraphControl Flow Graph
48
10/7/2008
ThreeThree--dimensionaldimensional
Euclidean distanceEuclidean distance

49. Comparison AlgorithmComparison Algorithm
Select best samples
Start comparison:
Graph Comparison
49
10/7/2008
Graph Comparison
Match more functions with CFG
Index of Similarity

50. Index of SimilarityIndex of Similarity
Measure how close two binaries are.Measure how close two binaries are.
50
10/7/2008

51. Index of SimilarityIndex of Similarity
Index of similarity vs number of matched functionsIndex of similarity vs number of matched functions
First Exp.
51
10/7/2008
Second Exp.

52. ImprovementsImprovements
More Initial Fixed PointsMore Initial Fixed Points
52
10/7/2008

53. More Fixed PointsMore Fixed Points
SPP (Small Prime Product)SPP (Small Prime Product)
Identical String referencesIdentical String references
53
10/7/2008
In/Out degree (similar number of calls to/calls from)In/Out degree (similar number of calls to/calls from)
Match same name (sub_XXXXXX) if same CRC32Match same name (sub_XXXXXX) if same CRC32
Stack Frame Size (similar stack frame size)Stack Frame Size (similar stack frame size)

54. ImprovementsImprovements
More Initial Fixed PointsMore Initial Fixed Points
Python version improvementsPython version improvements
54
10/7/2008

55. PsycoPsyco –– a Justa Just--InIn--Time compilerTime compiler
Python version improvementsPython version improvements
Extend Python code with CExtend Python code with C
√
55
10/7/2008
From Python to C/C++From Python to C/C++
Develop the comparison algorithm in C/C++Develop the comparison algorithm in C/C++
√

56. ImprovementsImprovements
Python version improvementsPython version improvements
Matrix rows as bitsMatrix rows as bits
More Initial Fixed PointsMore Initial Fixed Points
56
10/7/2008
Matrix rows as bitsMatrix rows as bits

57. Avoid string comparison (string comparison is a timeAvoid string comparison (string comparison is a time--
consuming task)consuming task)
Treat rows as groups of bits (100110011101…)Treat rows as groups of bits (100110011101…)
Split rows as groups of 32 bitsSplit rows as groups of 32 bits
Matrix rows as bitsMatrix rows as bits
57
10/7/2008
Compare as integers (integer comparison is faster)Compare as integers (integer comparison is faster)

58. ImprovementsImprovements
Python version improvementsPython version improvements
Matrix rows as bitsMatrix rows as bits
More Initial Fixed PointsMore Initial Fixed Points
58
10/7/2008
Matrix rows as bitsMatrix rows as bits
Stream SIMD extensionsStream SIMD extensions

59. Streaming SIMD extensionsStreaming SIMD extensions
SSE added eight 128SSE added eight 128--bits registers:bits registers: XMM0XMM0--XMM7XMM7
Each register packs together four 32Each register packs together four 32--bit integersbit integers
59
10/7/2008
Compare 4x32 (four integers) in one instruction:Compare 4x32 (four integers) in one instruction:
__m128 _mm_cmpeq_ps(__m128 a, __m128 b)__m128 _mm_cmpeq_ps(__m128 a, __m128 b)

60. ImprovementsImprovements
Python version improvementsPython version improvements
Matrix rows as bitsMatrix rows as bits
More Initial Fixed PointsMore Initial Fixed Points
60
10/7/2008
Matrix rows as bitsMatrix rows as bits
Stream SIMD extensionsStream SIMD extensions
NVIDIA CudaNVIDIA Cuda

61. NVIDIA CudaNVIDIA Cuda
Cuda: compiler and SDK for NVIDIA GPUsCuda: compiler and SDK for NVIDIA GPUs
GPUs:GPUs:
Parallel "manyParallel "many--core" architecturecore" architecture
Each core: thousands of threads simultaneouslyEach core: thousands of threads simultaneously
61
10/7/2008
Each core: thousands of threads simultaneouslyEach core: thousands of threads simultaneously
Not tried yet. Future development:Not tried yet. Future development:
Code algorithm for graphics processing unit (GPU)Code algorithm for graphics processing unit (GPU)
Launch one thread for each compared sampleLaunch one thread for each compared sample
Launch a thread for each compared rowLaunch a thread for each compared row

62. PossibilitiesPossibilities
Shared code => generic signatureShared code => generic signature
Port information from malware databasePort information from malware database
62
10/7/2008
Shared code => generic signatureShared code => generic signature
Functions with same CRC32
Clusterization of malware automaticallyClusterization of malware automatically
Classify unknown samples (index of similarity)

63. BarriersBarriers
Hard bound to IDA
Only unpacked samples
63
10/7/2008
Hard bound to IDA
Big database storage
Delphi and VB samples don’t work well

64. Demo
64
10/7/2008

65. Questions?
Thank you very much
65
10/7/2008
Questions?
ismael.briones@pandasecurity.com
aitor.gomez@pandasecurity.com