The difference between track and testing performance Roel Schouwenberg, Senior Anti-Virus Researcher Kaspersky Lab Benelux [email_address]

The difference between track and testing performance

Roel Schouwenberg, Senior Anti-Virus Researcher

Kaspersky Lab Benelux

[email_address]

About:Roel Malware analysis AV research Incident response

Malware analysis

AV research

Incident response

Overview Testing AV engine Testing AVendor’s response time Product technologies Conclusions

Testing AV engine

Testing AVendor’s response time

Product technologies

Conclusions

Current testing On-demand WildList (won’t go there) Large (zoo) test bed Retrospective using x month old product On-access (not so common or detailed)

On-demand

WildList (won’t go there)

Large (zoo) test bed

Retrospective

using x month old product

On-access (not so common or detailed)

On-demand: obvious flaws Trash files Age of samples Lack of transparency Response time is not a factor Lack of resources to perfect testing Etc.

Trash files

Age of samples

Lack of transparency

Response time is not a factor

Lack of resources to perfect testing

Etc.

Infectors / Trojanizers Trojanizers (PE, script) Real infectors Check response time for detection and disinfection Creating trojanizer test bed can take a long time

Trojanizers (PE, script)

Real infectors

Check response time for detection and disinfection

Creating trojanizer test bed can take a long time

Online scan services JottiScan, VirusTotal (and others) Much trash and ‘trash’ False positive issues Additional checks needed SFX archives and so on

JottiScan, VirusTotal (and others)

Much trash and ‘trash’

False positive issues

Additional checks needed

SFX archives and so on

Testing vs track performance Detection on/of packer/crypter Compare results with and without packer detection Differentiate between packers Regular vs custom packer/crypter Generic vs detecting specific family Age of samples 1/2/3/6/12 months old

Detection on/of packer/crypter

Compare results with and without packer detection

Differentiate between packers

Regular vs custom packer/crypter

Generic vs detecting specific family

Age of samples

1/2/3/6/12 months old

Differentiate between malware Regional malware Malware coming from a region Payload (Banker vs GameThief trojan) Automagically fabricated samples How many Zlobs do you want in the equation?

Regional malware

Malware coming from a region

Payload (Banker vs GameThief trojan)

Automagically fabricated samples

How many Zlobs do you want in the equation?

Response time Global outbreak Localized outbreak Low priority malware Infectors/trojanizers

Global outbreak

Localized outbreak

Low priority malware

Infectors/trojanizers

Retrospective testing 1 second is enough Modified ‘droppers’ Type of samples

1 second is enough

Modified ‘droppers’

Type of samples

Product technologies HIPS-like module Components working together – AV vs IS (Memory scanner) Not so relevant (in this case): Malware removal Registry cleanup Malware detection on infected system

HIPS-like module

Components working together – AV vs IS

(Memory scanner)

Not so relevant (in this case):

Malware removal

Registry cleanup

Malware detection on infected system

Conclusions Other/nicer ways to check out the competition  Product technologies make testing-life harder Testing will always be flawed

Other/nicer ways to check out the competition 

Product technologies make testing-life harder

Testing will always be flawed

The end Thank you for your attention! Questions or comments? [email_address]

Thank you for your attention!

Questions or comments?

[email_address]