Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Testing Heuristics Andrew Lee CISSP Chief Research Officer ESET LLC [email_address]
What do you need? <ul><li>The appropriateness of the methodology (or it’s correct application) </li></ul><ul><ul><li>Repea...
A quick word on FP testing <ul><li>No ‘tricks’! </li></ul><ul><ul><li>Appropriate “ItW” false positive set </li></ul></ul>...
Junk / Corrupt files <ul><li>Poor sample sets simply reinforce the cycle - the more junk added, the more detected </li></u...
“ Time to Update” 6 hours 30 hours at %20 (5 upd) X4 4 hours 8 hours at %50 (10 upd) X3 4 hours 4 hours at 5% (1 upd) X2 1...
Actual TtU 30 hours 30 hours at %20 X4 8 hours 8 hours at 50% X3 4 hours 4 hours at 5% X2 1 hour 1 hour at 100% X1 Average...
Mean time Each Dot represents a different product
Lies, Damned Lies and Statistics <ul><li>Statistical intgrity is biased, means of more succesful product are calculated ov...
Retrospective (Frozen Update) <ul><li>Selection of time period </li></ul><ul><ul><li>6 months? </li></ul></ul><ul><ul><li>...
Frozen Update Pt II <ul><li>What samples are important? </li></ul><ul><li>Is this a recursive process? </li></ul><ul><ul><...
To quote Dr Alan Solomon. <ul><li>1. If something is superb at detecting viruses, it's no use if it gives a lot of false a...
 
Shameless plug <ul><li>AVIEN Guide to Managing Malware in the Enterprise </li></ul><ul><li>http://www.smallblue-greenworld...
Upcoming SlideShare
Loading in …5
×

Testing Heuristic Detections

881 views

Published on

Presented at the International Antivirus Testing Workshop 2007 by Andrew Lee, Chief Research Officer, ESET LLC

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

Testing Heuristic Detections

  1. 1. Testing Heuristics Andrew Lee CISSP Chief Research Officer ESET LLC [email_address]
  2. 2. What do you need? <ul><li>The appropriateness of the methodology (or it’s correct application) </li></ul><ul><ul><li>Repeatability </li></ul></ul><ul><ul><li>Independently verifiable </li></ul></ul><ul><ul><li>Validated sample sets </li></ul></ul><ul><li>Adherence to safe and ethical practices in handling and testing samples </li></ul><ul><li>Understanding of what heuristic detection is (and what it’s not) </li></ul>
  3. 3. A quick word on FP testing <ul><li>No ‘tricks’! </li></ul><ul><ul><li>Appropriate “ItW” false positive set </li></ul></ul><ul><ul><li>Evaluation of FP’s </li></ul></ul><ul><ul><li>‘ Grey’/unusual or very strange unlikely files will tend to penalize heuristic based products </li></ul></ul><ul><li>Defaults </li></ul><ul><li>Best settings </li></ul>
  4. 4. Junk / Corrupt files <ul><li>Poor sample sets simply reinforce the cycle - the more junk added, the more detected </li></ul><ul><li>Using AV products to determine maliciousness is silly, it simply reinforces the cycle (Kaminski - Eicar 2006?) </li></ul>
  5. 5. “ Time to Update” 6 hours 30 hours at %20 (5 upd) X4 4 hours 8 hours at %50 (10 upd) X3 4 hours 4 hours at 5% (1 upd) X2 1 hour 1 hour at 100% (20 upd) X1 Average TtU Actual Time to Update / % missed (20 Samples) Product
  6. 6. Actual TtU 30 hours 30 hours at %20 X4 8 hours 8 hours at 50% X3 4 hours 4 hours at 5% X2 1 hour 1 hour at 100% X1 Average TtU (zero removed) Actual Time to Update / % missed Product
  7. 7. Mean time Each Dot represents a different product
  8. 8. Lies, Damned Lies and Statistics <ul><li>Statistical intgrity is biased, means of more succesful product are calculated over less samples (necessarily). This is not good for comparisons. </li></ul><ul><li>Concentrating on speed of update is surely sending the wrong message to the consumers, giving them the false impression that buying a product that releases a lot of updates very quickly is going to protect them better. </li></ul>
  9. 9. Retrospective (Frozen Update) <ul><li>Selection of time period </li></ul><ul><ul><li>6 months? </li></ul></ul><ul><ul><li>3 months? </li></ul></ul><ul><ul><li>1 day? </li></ul></ul><ul><ul><li>1 hour? </li></ul></ul><ul><li>Verification (is it possible to do real time?) </li></ul>
  10. 10. Frozen Update Pt II <ul><li>What samples are important? </li></ul><ul><li>Is this a recursive process? </li></ul><ul><ul><li>Single snapshot is not necessarily the most useful information </li></ul></ul><ul><ul><li>Performance over time </li></ul></ul><ul><ul><li>Sound statistical model </li></ul></ul>
  11. 11. To quote Dr Alan Solomon. <ul><li>1. If something is superb at detecting viruses, it's no use if it gives a lot of false alarms. </li></ul><ul><li>2. Anything that relies on the user to make a correct decision, on matters that he is not likely to be able to decide about, is useless. </li></ul><ul><li>3. You can receive something that is *exactly* what the salesman promised to deliver, and it's nevertheless useless. </li></ul>
  12. 13. Shameless plug <ul><li>AVIEN Guide to Managing Malware in the Enterprise </li></ul><ul><li>http://www.smallblue-greenworld.co.uk/pages/avienguide.html </li></ul>

×