Measuring the Actual Security that Vendors Provide to Customers

406 views

Published on

“There is a desperate need for new standards for today’s anti-virus products. The dominant paradigm, scanning directories of files, is focused on old and known threats, and reveals little about product efficacy in the wild.”
Williamson & Gorelik (2007)

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
406
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Measuring the Actual Security that Vendors Provide to Customers

  1. 1. Measuring the Actual Security that Vendors Provide to Customers the need for AV product testing reform An executive session with Trend Micro CTO Raimund Genes and industry guests
  2. 2. Four ways to stop malware <ul><li>Block malware from arriving at the endpoint. </li></ul><ul><ul><ul><li>e.g. , web filtering; web reputation services </li></ul></ul></ul><ul><li>Stop malware files from executing on the endpoint. </li></ul><ul><ul><ul><li>e.g. , signature-based scanning of files </li></ul></ul></ul><ul><li>Interrupt malware doing bad things on execution. </li></ul><ul><ul><ul><li>e.g. , behavior monitoring </li></ul></ul></ul><ul><li>Protect vulnerabilities from being exploited. </li></ul><ul><ul><ul><li>e.g. , disable access to known vulnerabilities until patched </li></ul></ul></ul>
  3. 3. But traditional testing only counts one <ul><li>Block malware from arriving at the endpoint. </li></ul><ul><ul><ul><li>e.g. , web filtering; web reputation services </li></ul></ul></ul><ul><li>Stop malware files from executing on the endpoint. </li></ul><ul><ul><ul><li>e.g. , signature-based scanning of files </li></ul></ul></ul><ul><li>Interrupt malware doing bad things on execution. </li></ul><ul><ul><ul><li>e.g. , behavior monitoring </li></ul></ul></ul><ul><li>Protect vulnerabilities from being exploited. </li></ul><ul><ul><ul><li>e.g. , disable access to known vulnerabilities until patched </li></ul></ul></ul>Traditional AV product testing only measures detection
  4. 4. As a result … “ There is a desperate need for new standards for today’s anti-virus products. The dominant paradigm, scanning directories of files , is focused on old and known threats, and reveals little about product efficacy in the wild .” Williamson & Gorelik (2007)
  5. 5. Test Labs are responding <ul><li>Independent testing labs have introduced new testing methods </li></ul><ul><li>Many new metrics attempt to better measure actual security </li></ul><ul><li>But the Labs have trouble keeping up with changes: </li></ul><ul><ul><li>New cybercriminal techniques </li></ul></ul><ul><ul><li>Anti-malware solution innovations </li></ul></ul><ul><li>As a result, there is now chaos in AV testing metrics & results </li></ul><ul><li>Names of Testing Metrics </li></ul><ul><li>Anti-malware detection </li></ul><ul><li>Caught initially on download </li></ul><ul><li>Caught on first exposure </li></ul><ul><li>Caught subsequently on execution </li></ul><ul><li>Caught with repeated exposure </li></ul><ul><li>Drive-by-download protection </li></ul><ul><li>Dynamic detection </li></ul><ul><li>End-to-end web threat protection </li></ul><ul><li>Exposure layer web threat protection </li></ul><ul><li>Infection layer web threat protection </li></ul><ul><li>Internet-connected detection </li></ul><ul><li>Malware blocking </li></ul><ul><li>Malware detection </li></ul><ul><li>Overall web threat protection </li></ul><ul><li>Proactive detection </li></ul><ul><li>Web security blocking </li></ul><ul><li>Web threat blocking effectiveness </li></ul><ul><li>Whole product dynamic test </li></ul><ul><li>Zero-day protection </li></ul>
  6. 6. AV Testing Metrics Chaos <ul><li>Names of Testing Metrics </li></ul><ul><li>Anti-malware detection </li></ul><ul><li>Caught initially on download </li></ul><ul><li>Caught on first exposure </li></ul><ul><li>Caught subsequently on execution </li></ul><ul><li>Caught with repeated exposure </li></ul><ul><li>Drive-by-download protection </li></ul><ul><li>Dynamic detection </li></ul><ul><li>End-to-end web threat protection </li></ul><ul><li>Exposure layer web threat protection </li></ul><ul><li>Infection layer web threat protection </li></ul><ul><li>Internet-connected detection </li></ul><ul><li>Malware blocking </li></ul><ul><li>Malware detection </li></ul><ul><li>Overall web threat protection </li></ul><ul><li>Proactive detection </li></ul><ul><li>Web security blocking </li></ul><ul><li>Web threat blocking effectiveness </li></ul><ul><li>Whole product dynamic test </li></ul><ul><li>Zero-day protection </li></ul>
  7. 7. AV Testing Metrics Chaos <ul><li>No consistency of testing method </li></ul><ul><li>No consistency of applied threat stimuli </li></ul><ul><li>No consistency of metrics definition </li></ul><ul><li>No consistency of results </li></ul><ul><li>consequently: </li></ul><ul><li>Little value to buyers of security products </li></ul>
  8. 8. Blocking in the cloud before arrival <ul><li>92% of malware arrives over the Internet. </li></ul><ul><li>The source is often easier to identify than the malware files. </li></ul><ul><li>Blocking files from a bad source does not require file detection. </li></ul><ul><li>Traditional test methods do not credit blocking by source URL. </li></ul>
  9. 9. A new threat every 1.5 sec. <ul><li>Thousands of new threats per day overwhelm test methods. </li></ul><ul><li>Stored threats become irrelevant before a test is completed. </li></ul><ul><li>Speed of response to new threats is more important than detection of old threats. </li></ul><ul><li>Many threats are “old” in hours to days – not weeks. </li></ul>
  10. 10. How long to respond to a new threat? … a metric that shows real differences among vendors.
  11. 11. Key principles for AV product testing <ul><li>Credit for “protection” instead of “detection” </li></ul><ul><li>“Real-time” or “dynamic” testing </li></ul><ul><li>Reproducibility: Statistical not deterministic </li></ul><ul><li>Broad and diverse relevant threat samples </li></ul><ul><li>Measuring the vendor response </li></ul><ul><li>e.g. , “time-to-protect” </li></ul>
  12. 12. Comments from Industry Guests <ul><li>Gerhard Eschelbeck </li></ul><ul><li>CTO & SVP Engineering at Webroot </li></ul><ul><li>Vik Phatak </li></ul><ul><li>Chairman & CTO at NSS Labs </li></ul><ul><li>Andreas Marx </li></ul><ul><li>CEO at AV-Test </li></ul><ul><li>Anil Somayaji </li></ul><ul><li>Director, Computer Security Lab, Carleton University </li></ul>

×