Maintaining a Malware Collection Dr. Vesselin Bontchev, anti–virus researcher FRISK Software International Thverholt 18, IS-105 Reykjavik, ICELAND National Laboratory of Computer Virology Bulgarian Academy of Sciences Acad. G. Bontchev Str., Bl. 2, 1113, BULGARIA E–mail: [email_address]

Introduction The Naïve Idea of AV Product Testing Get a large set of files from somewhere Call it a “virus collection” It’s good if most scanners report most of the files as “something” It’s even better if some files are not reported by some or all of the popular scanners as anything Run an on-demand scanner on the set Classify the results by some criteria (e.g., number of detected objects) and publish them

The Naïve Idea of AV Product Testing

Get a large set of files from somewhere

Call it a “virus collection”

It’s good if most scanners report most of the files as “something”

It’s even better if some files are not reported by some or all of the popular scanners as anything

Run an on-demand scanner on the set

Classify the results by some criteria (e.g., number of detected objects) and publish them

Let’s Concentrate on the “Collection” Part Face It: Most of the collections you can get easily consist mostly of crap (Vx) Even the good collections contain some crap (AV) Or are inadequate (WLO) So Analyze the contents and remove the crap Replicate the viruses And only then use the result for testing!

Face It:

Most of the collections you can get easily consist mostly of crap (Vx)

Even the good collections contain some crap (AV)

Or are inadequate (WLO)

So

Analyze the contents and remove the crap

Replicate the viruses

And only then use the result for testing!

Analyzing the Contents Unpack the Archives Needs LOTS of disk space! Sometimes there are nested archives And/or encrypted ones (“infected”, “virus”, etc.) The file extensions are misleading! (Or non-existent. Morons.) Some files need decoding Basically – look at the damn things! Do not assume

Unpack the Archives

Needs LOTS of disk space!

Sometimes there are nested archives

And/or encrypted ones (“infected”, “virus”, etc.)

The file extensions are misleading! (Or non-existent. Morons.)

Some files need decoding

Basically – look at the damn things! Do not assume

More Analyzing Remove the Duplicates Of which there are LOTS! Both in the same collection and across collections You need a duplicate file locator The commercial ones are crap! Big, slow, unreliable and inadequate So, write your own Beware: with the huge number of files in the contemporary collections, CRC-32 is not adequate as a hash function (due to collisions). Use MD4 (not MD5, MD2 or SHA, because MD4 is faster and is secure enough)

Remove the Duplicates

Of which there are LOTS! Both in the same collection and across collections

You need a duplicate file locator

The commercial ones are crap! Big, slow, unreliable and inadequate

So, write your own

Beware: with the huge number of files in the contemporary collections, CRC-32 is not adequate as a hash function (due to collisions). Use MD4 (not MD5, MD2 or SHA, because MD4 is faster and is secure enough)

Even More Analyzing Remove the Corrupted Files Zapped beginnings Entry points going nowhere Partially disinfected stuff Breakpoints in the code Just random garbage Basically – stuff that doesn’t work If you don’t know what the above means or how to detect its presence, you’re not qualified to test AV products – find a different job, or learn it first Unfortunately, if you do know, that still doesn’t necessarily mean that you’re qualified to test AV products!

Remove the Corrupted Files

Zapped beginnings

Entry points going nowhere

Partially disinfected stuff

Breakpoints in the code

Just random garbage

Basically – stuff that doesn’t work

If you don’t know what the above means or how to detect its presence, you’re not qualified to test AV products – find a different job, or learn it first

Unfortunately, if you do know, that still doesn’t necessarily mean that you’re qualified to test AV products!

Still More Analyzing Remove the Envelopes “Immunizations” Packers Sometimes you MUST NOT remove these! So, how do you know when to remove them? Sandwiches

Remove the Envelopes

“Immunizations”

Packers

Sometimes you MUST NOT remove these!

So, how do you know when to remove them?

Sandwiches

And Even More Analyzing Remove the Non-Malware Utilities Legitimate tools used by malware Simulators False positives Buggy programs Just text files and pictures Build a database of unwanted crap, because you’ll keep receiving it over and over! The TRASHBIN tool Mike will tell you more on this subject

Remove the Non-Malware

Utilities

Legitimate tools used by malware

Simulators

False positives

Buggy programs

Just text files and pictures

Build a database of unwanted crap, because you’ll keep receiving it over and over! The TRASHBIN tool

Mike will tell you more on this subject

And Finally… Separate the Viruses from the Non-Viral Malware Trojan Horses Dialers Password stealers Exploits Kits Germs Injectors Intended

Separate the Viruses from the Non-Viral Malware

Trojan Horses

Dialers

Password stealers

Exploits

Kits

Germs

Injectors

Intended

Basic Rules of Thumb Know what you’re doing! Look at everything personally! Don’t put a file in your collection, unless you can explain why you have done so “A scanner reports it” or “Found it in a virus collection” are NOT good explanations!

Know what you’re doing!

Look at everything personally!

Don’t put a file in your collection, unless you can explain why you have done so

“A scanner reports it” or “Found it in a virus collection” are NOT good explanations!

And Now – the Real Work Begins Replicate All the Viruses Yourselves! Yes, all of them! Yes, yourselves! If you cannot replicate something, either figure out why not and then replicate it, or don’t put it in the virus collection! Yes, this requires multiple environments (OSes and devices), a lot of knowledge and a lot of work Did I ever say that maintaining a malware collection was easy ?!

Replicate All the Viruses Yourselves!

Yes, all of them!

Yes, yourselves!

If you cannot replicate something, either figure out why not and then replicate it, or don’t put it in the virus collection!

Yes, this requires multiple environments (OSes and devices), a lot of knowledge and a lot of work

Did I ever say that maintaining a malware collection was easy ?!

More About Replication Viruses Fail to Replicate for Various Reasons CPU dependency OS dependency Memory dependency Date/time dependency File system dependency There are many others! Basically – analyze the damn thing and figure it out when/if it replicates If you cannot, you are not qualified to do AV product testing!

Viruses Fail to Replicate for Various Reasons

CPU dependency

OS dependency

Memory dependency

Date/time dependency

File system dependency

There are many others!

Basically – analyze the damn thing and figure it out when/if it replicates

If you cannot, you are not qualified to do AV product testing!

Order the Collection Separate by Malware Type Group by Malware Family Separate by Malware Variant Rules of Thumb: If two samples contain the same malware, they should be in the same directory If two samples contain two different malware programs, they should be in two different directories You need to be able to tell if two samples contain the same malware or not. Obvious, isn’t it? Scanners can help but are by far not sufficient

Separate by Malware Type

Group by Malware Family

Separate by Malware Variant

Rules of Thumb:

If two samples contain the same malware, they should be in the same directory

If two samples contain two different malware programs, they should be in two different directories

You need to be able to tell if two samples contain the same malware or not. Obvious, isn’t it?

Scanners can help but are by far not sufficient

Testing With a Collection, You Can Test: Detection Heuristic detection (Careful!) Disinfection Memory scanning (does anybody do that?) Identification (Difficult!) On-demand and on-access Test on various platforms Scanning speed? (Ooops! Never!!!) Forget the WLO/ItW crap!

With a Collection, You Can Test:

Detection

Heuristic detection (Careful!)

Disinfection

Memory scanning (does anybody do that?)

Identification (Difficult!)

On-demand and on-access

Test on various platforms

Scanning speed? (Ooops! Never!!!)

Forget the WLO/ItW crap!

That’s All, Folks Simple, Isn’t It? Just Some Common Sense Which is so uncommon nowadays A Lot of Knowledge and Experience And Work, Work, Work! Questions? Conclusion

That’s All, Folks

Simple, Isn’t It?

Just Some Common Sense

Which is so uncommon nowadays

A Lot of Knowledge and Experience

And Work, Work, Work!

Questions?