Exploiting the testing system Viorel Canja,Head of BitDefender Labs

Contents What does the title mean ? Testing detection on wildcore Testing detection on zoo collections Retrospective detection tests Examples Feedback from the industry Q&A

What does the title mean ?

Testing detection on wildcore

Testing detection on zoo collections

Retrospective detection tests

Examples

Feedback from the industry

Q&A

What does the title mean ? Purpose of tests: to define metrics and measure the performance of AV products to find am approximation for the real world performance of AV products to give feedback to AV researchers about their products to allow the users to make an informed decision

Purpose of tests:

to define metrics and measure the performance of AV products

to find am approximation for the real world performance of AV products

to give feedback to AV researchers about their products

to allow the users to make an informed decision

What does the title mean ? “ Define:exploit” use or manipulate to one's advantage draw from; make good use of overwork: work excessively hard

“ Define:exploit”

use or manipulate to one's advantage

draw from; make good use of

overwork: work excessively hard

What does the title mean ? To use the limitations of the testing procedure to one’s advantage. The focus is on those actions which have questionable benefits for the user.

To use the limitations of the testing procedure to one’s advantage.

The focus is on those actions which have questionable benefits for the user.

Types of tests detection tests on wildcore detection tests on zoo collections retrospective detection tests

detection tests on wildcore

detection tests on zoo collections

retrospective detection tests

Testing detection on wildcore What is wildcore ? “ WildCore is a set of replicated virus samples that represents the real threat to computer users.” “ When a virus is reported to us by two or more Reporters, it's a pretty good indication that the virus is out there, spreading, causing real problems to users. We consider such a virus to be 'In the Wild'.”

What is wildcore ?

“ WildCore is a set of replicated virus samples that represents the real threat to computer users.”

“ When a virus is reported to us by two or more Reporters, it's a pretty good indication that the virus is out there, spreading, causing real problems to users. We consider such a virus to be 'In the Wild'.”

Testing detection on wildcore The Wildcore samples are known to all AV companies as soon as wildcore is published. Tests are likely to be performed on exactly the same samples. This is always the case with samples of malware which does not replicate.

The Wildcore samples are known to all AV companies as soon as wildcore is published.

Tests are likely to be performed on exactly the same samples. This is always the case with samples of malware which does not replicate.

Testing detection on wildcore Quick hack: just sign all the samples with dumb ( aka automatic ) signatures. Disable heuristics to avoid false positives ( if the testbed is already known there is no need for technology that detects previously unknown threats )

Quick hack: just sign all the samples with dumb ( aka automatic ) signatures.

Disable heuristics to avoid false positives ( if the testbed is already known there is no need for technology that detects previously unknown threats )

Testing detection on zoo collections Zoo should contain a large number of files so that the statistics are as accurate as possible Threats should be replicated ( where applicable ) or large numbers of samples should be used for polymorphic malware or malware that is re-generated on the server The zoo should not contain garbage

Zoo should contain a large number of files so that the statistics are as accurate as possible

Threats should be replicated ( where applicable ) or large numbers of samples should be used for polymorphic malware or malware that is re-generated on the server

The zoo should not contain garbage

Testing detection on zoo collections Hacks: use customized settings for the test. Heuristics should be set to paranoid mode. Automatically sign all previously missed samples and white-list all previously reported false positives. automatically sign all samples detected by at least one AV product just to be on the “safe” side

Hacks:

use customized settings for the test. Heuristics should be set to paranoid mode. Automatically sign all previously missed samples and white-list all previously reported false positives.

automatically sign all samples detected by at least one AV product just to be on the “safe” side

Testing detection on zoo collections Hacks (2): - add detection routines for garbage that is usually found in collections. This includes detecting known false positives of other products, detecting damaged executables, detecting files produced by different analysis tools.

Hacks (2):

- add detection routines for garbage that is usually found in collections. This includes detecting known false positives of other products, detecting damaged executables, detecting files produced by different analysis tools.

Retrospective detection tests Signature databases are frozen at a certain moment Detection is tested against samples received after that moment Testing should be done with default settings because most of the products are marketed as “install and forget” and the majority of users will not change the settings

Signature databases are frozen at a certain moment

Detection is tested against samples received after that moment

Testing should be done with default settings because most of the products are marketed as “install and forget” and the majority of users will not change the settings

Retrospective detection tests Has the disadvantage that it will not take into account proactive detections introduced by generic routines created for malware families that appear after the signatures are frozen These routines ( or signatures ) will detect proactively subsequent variants of the same family Favors aggressive heuristics if not correlated with false positive tests

Has the disadvantage that it will not take into account proactive detections introduced by generic routines created for malware families that appear after the signatures are frozen

These routines ( or signatures ) will detect proactively subsequent variants of the same family

Favors aggressive heuristics if not correlated with false positive tests

Examples Automatic signing: Av01 (1 st pair) : TR/Zapchast.CP Av02 : Collected.Z Av03: W32/KillAV.3B84!tr Av04: Trojan.Downloader.Asks Av05: Program:Win32/SpySheriff (threat-c) Av06: Trojan.Gen Av07 : Win32:Trojan-gen. {Other} Av08: Win32/Dewnuttin.B Av09: W32/Tofger.CD Av10: Application/KillApp.A Av11: (2 nd pair) TROJ_PROCKILL.DJ Av12: Trojan.Xtssksastsm Av13: (1 st pair) Trojan.Win32.Zapchast.cp Av14: (2 nd pair) application ProcKill-DJ Av15: Win32/ProcKill.1hj!Trojan Av16: Trojan.Zapchast.CT

Automatic signing:

Av01 (1 st pair) : TR/Zapchast.CP

Av02 : Collected.Z

Av03: W32/KillAV.3B84!tr

Av04: Trojan.Downloader.Asks

Av05: Program:Win32/SpySheriff (threat-c)

Av06: Trojan.Gen

Av07 : Win32:Trojan-gen. {Other}

Av08: Win32/Dewnuttin.B

Av09: W32/Tofger.CD

Av10: Application/KillApp.A

Av11: (2 nd pair) TROJ_PROCKILL.DJ

Av12: Trojan.Xtssksastsm

Av13: (1 st pair) Trojan.Win32.Zapchast.cp

Av14: (2 nd pair) application ProcKill-DJ

Av15: Win32/ProcKill.1hj!Trojan

Av16: Trojan.Zapchast.CT

Examples Detecting other products’ false positives: Av01: Backdoor.X Av02: FalseAlarm.Av01.Backdoor.X

Detecting other products’ false positives:

Av01: Backdoor.X

Av02: FalseAlarm.Av01.Backdoor.X

Feedback from the industry Automatic sample processing … is a must given the number of samples received

Automatic sample processing …

is a must given the number of samples received

Feedback from the industry … and adding detection based on the output of other AVs illegal, immoral, plain wrong bad idea it’s common practice it probably started as an attempt to have common names there is no other way

… and adding detection based on the output of other AVs

illegal, immoral, plain wrong

bad idea

it’s common practice

it probably started as an attempt to have common names

there is no other way

Feedback from the industry Reporting packed files if they are not malicious we should not detect them some of the packers should be blacklisted while others are too widely used so must be allowed an unfortunate necessity professional companies do not need to use dodgy packers

Reporting packed files

if they are not malicious we should not detect them

some of the packers should be blacklisted while others are too widely used so must be allowed

an unfortunate necessity

professional companies do not need to use dodgy packers

Feedback from the industry White-listing clean apps instead of black-listing malware it’s not possible does not scale it’s ok in controlled environments better and better idea as time passes

White-listing clean apps instead of black-listing malware

it’s not possible

does not scale

it’s ok in controlled environments

better and better idea as time passes

The end … Q&A

Q&A