Testing Heuristic Detections


Published on

Presented at the International Antivirus Testing Workshop 2007 by Andrew Lee, Chief Research Officer, ESET LLC

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Testing Heuristic Detections

  1. 1. Testing Heuristics Andrew Lee CISSP Chief Research Officer ESET LLC [email_address]
  2. 2. What do you need? <ul><li>The appropriateness of the methodology (or it’s correct application) </li></ul><ul><ul><li>Repeatability </li></ul></ul><ul><ul><li>Independently verifiable </li></ul></ul><ul><ul><li>Validated sample sets </li></ul></ul><ul><li>Adherence to safe and ethical practices in handling and testing samples </li></ul><ul><li>Understanding of what heuristic detection is (and what it’s not) </li></ul>
  3. 3. A quick word on FP testing <ul><li>No ‘tricks’! </li></ul><ul><ul><li>Appropriate “ItW” false positive set </li></ul></ul><ul><ul><li>Evaluation of FP’s </li></ul></ul><ul><ul><li>‘ Grey’/unusual or very strange unlikely files will tend to penalize heuristic based products </li></ul></ul><ul><li>Defaults </li></ul><ul><li>Best settings </li></ul>
  4. 4. Junk / Corrupt files <ul><li>Poor sample sets simply reinforce the cycle - the more junk added, the more detected </li></ul><ul><li>Using AV products to determine maliciousness is silly, it simply reinforces the cycle (Kaminski - Eicar 2006?) </li></ul>
  5. 5. “ Time to Update” 6 hours 30 hours at %20 (5 upd) X4 4 hours 8 hours at %50 (10 upd) X3 4 hours 4 hours at 5% (1 upd) X2 1 hour 1 hour at 100% (20 upd) X1 Average TtU Actual Time to Update / % missed (20 Samples) Product
  6. 6. Actual TtU 30 hours 30 hours at %20 X4 8 hours 8 hours at 50% X3 4 hours 4 hours at 5% X2 1 hour 1 hour at 100% X1 Average TtU (zero removed) Actual Time to Update / % missed Product
  7. 7. Mean time Each Dot represents a different product
  8. 8. Lies, Damned Lies and Statistics <ul><li>Statistical intgrity is biased, means of more succesful product are calculated over less samples (necessarily). This is not good for comparisons. </li></ul><ul><li>Concentrating on speed of update is surely sending the wrong message to the consumers, giving them the false impression that buying a product that releases a lot of updates very quickly is going to protect them better. </li></ul>
  9. 9. Retrospective (Frozen Update) <ul><li>Selection of time period </li></ul><ul><ul><li>6 months? </li></ul></ul><ul><ul><li>3 months? </li></ul></ul><ul><ul><li>1 day? </li></ul></ul><ul><ul><li>1 hour? </li></ul></ul><ul><li>Verification (is it possible to do real time?) </li></ul>
  10. 10. Frozen Update Pt II <ul><li>What samples are important? </li></ul><ul><li>Is this a recursive process? </li></ul><ul><ul><li>Single snapshot is not necessarily the most useful information </li></ul></ul><ul><ul><li>Performance over time </li></ul></ul><ul><ul><li>Sound statistical model </li></ul></ul>
  11. 11. To quote Dr Alan Solomon. <ul><li>1. If something is superb at detecting viruses, it's no use if it gives a lot of false alarms. </li></ul><ul><li>2. Anything that relies on the user to make a correct decision, on matters that he is not likely to be able to decide about, is useless. </li></ul><ul><li>3. You can receive something that is *exactly* what the salesman promised to deliver, and it's nevertheless useless. </li></ul>
  12. 13. Shameless plug <ul><li>AVIEN Guide to Managing Malware in the Enterprise </li></ul><ul><li>http://www.smallblue-greenworld.co.uk/pages/avienguide.html </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.