Building and Leveraging a Whitelist Database for Anti-Virus Testing Mario Vuksan, Director, Knowledgebase Services
Agenda Growing Signature/Definition Problem Building a Global Whitelist Leveraging a Global Whitelist QA
Growing Signature Problem Cumulative unique variants have grown ten-fold over last 5 years (Yankee Group) “ Denial-Of-Service” Attacks:  Malware changing signature every 10 minutes Solutions Heuristic & Behavioral Detections New Problem: High “False Positive” Count
Whitelist: a Google-sized Project Sizing Software Universe Number of Files Released Daily by: Microsoft – 500K / IBM – 100K / Sourceforge – 500K / Mozilla.Org – 250K More Components, Daily Builds, Auto Updaters 2.7B Files Indexed, heading for 10B 30TB of Installers, heading for 100TB Daily acquiring 50M File Records, ¼ of YouTube Tracking 20,000 Software Companies E.g. DMOZ tracks 200,000+ Entities
Mechanics of a Whitelist Collect Extract Analyze Software Infrastructure Hardware Infrastructure Publish (Interfaces) Consumers Outbound Metadata Inbound User Metadata
Building a Whitelist Trusted Partners Benefits Trusted Source of Binary Material In-depth Information on the Binary Data Indexed Realities Expensive Partner Programs Complicated Applications Lack of Interest Lack of Comprehensive Repositories
Certifying Software Certificate Mechanism As a Component for Validation Costly Process, Cumbersome for QA Departments Great When Seen on Shareware Sites   Less than 10% Penetration First-Seen Date Microsoft & Shared Installer Components Long Time & No Detection    Likely Good
Challenges of Software Acquisition Buying/Getting Physical Media Retail Prices vs. Ebay How to process 35K DVDs? FTP Sites Web Sites Simple: Links and Forms Complicated: Javascript Super Complicated: Frames and AJAX Shareware Sites Warez Legal Ramifications Users vs. Collectors
Harvesting The Internet Order of Difficulty  FTPs – Wget, Curl Simple HTTPs – Open Source Spiders Try Grabbing Download.com Try Grabbing Downloads.microsoft.com Try Grabbing Canon or any Driver Site Datacenter Requirements
Assuring Software is Trustworthy Anti-Malware Scanning Name and Type Normalization Behavior Scanning Code Inspection External Meta Data Collection and Matching
Software Analysis Results Basic Embedded Data PE Header Analysis Processor, Language, Binary Type Packers and Protectors 500+ Variants ASPack and Adobe PECompact and Google Install Formats Proprietary (like Skype) Binary Diffs (Patch Factory, MS PSF) Runtime Analysis and Sandboxing
Software Classifications Classifying Source Trust-based vs. Type-based Classifying Files Functional (Font, Driver, Screensaver) vs. Descriptive  Classifying Products Basic Open Source Commercial: Driver vs. Application IM / P2P / Games Better Malware Classifications Interesting Steganography/Watermarking/Hacking/Hiding
Industry & Government Certifications Government Certifications NIAP, FIPS, DCTS Vulnerability Reports CVE, CERT, SANS, MSB, etc. For Good Software: Certification Programs Built for Vista, Windows Certified, Java Approved eTrust Download For Malware: StopBadware, CME
Leveraging the Whitelist
PE Header Subsystem
Other PE Header Data
What about False Positives? Typical Suspects: Internet Explorer Drivers (Network, File Access) OS Components Universal Installer and Uninstaller Components Optimized Applications: Using Obscure Third-Party Software ASPack, PECompact, Themida
Archive Format Distribution Most popular archive/packer formats
Or Are They False Positives? (FTP Injection Attacks) HP
Or Are They False Positives? (FTP Injection Attacks) Nero AG
Vertical Detection Malware Sample Vertical File Detection Chart Good File Vertical Analysis Anti-Malware Reports per Web Site Bit9 ISV Safe Software Program
Use Case: Anti-Malware Benefits R&D Tool Packers, Metadata, Sources QA Tool False Positives Performance Accelerator Robin Bloor’s AVID Next Generation Anti-Malware
About Bit9 What We Do: Application and Device Control Solutions and Software Metadata Reporting What We Offer: Bit9 Parity Protects against Malicious Software and Data Leakage The Bit9 Knowledgebase is the Largest Collection of Actionable Intelligence about the World’s Software Background Founded in 2002 by founders of Okena (Cisco) $2 Million NIST ATP Grant in 2003 Headquartered in Cambridge, Mass. Venture Funded

Building & Leveraging White Database for Antivirus Testing

  • 1.
    Building and Leveraginga Whitelist Database for Anti-Virus Testing Mario Vuksan, Director, Knowledgebase Services
  • 2.
    Agenda Growing Signature/DefinitionProblem Building a Global Whitelist Leveraging a Global Whitelist QA
  • 3.
    Growing Signature ProblemCumulative unique variants have grown ten-fold over last 5 years (Yankee Group) “ Denial-Of-Service” Attacks: Malware changing signature every 10 minutes Solutions Heuristic & Behavioral Detections New Problem: High “False Positive” Count
  • 4.
    Whitelist: a Google-sizedProject Sizing Software Universe Number of Files Released Daily by: Microsoft – 500K / IBM – 100K / Sourceforge – 500K / Mozilla.Org – 250K More Components, Daily Builds, Auto Updaters 2.7B Files Indexed, heading for 10B 30TB of Installers, heading for 100TB Daily acquiring 50M File Records, ¼ of YouTube Tracking 20,000 Software Companies E.g. DMOZ tracks 200,000+ Entities
  • 5.
    Mechanics of aWhitelist Collect Extract Analyze Software Infrastructure Hardware Infrastructure Publish (Interfaces) Consumers Outbound Metadata Inbound User Metadata
  • 6.
    Building a WhitelistTrusted Partners Benefits Trusted Source of Binary Material In-depth Information on the Binary Data Indexed Realities Expensive Partner Programs Complicated Applications Lack of Interest Lack of Comprehensive Repositories
  • 7.
    Certifying Software CertificateMechanism As a Component for Validation Costly Process, Cumbersome for QA Departments Great When Seen on Shareware Sites  Less than 10% Penetration First-Seen Date Microsoft & Shared Installer Components Long Time & No Detection  Likely Good
  • 8.
    Challenges of SoftwareAcquisition Buying/Getting Physical Media Retail Prices vs. Ebay How to process 35K DVDs? FTP Sites Web Sites Simple: Links and Forms Complicated: Javascript Super Complicated: Frames and AJAX Shareware Sites Warez Legal Ramifications Users vs. Collectors
  • 9.
    Harvesting The InternetOrder of Difficulty FTPs – Wget, Curl Simple HTTPs – Open Source Spiders Try Grabbing Download.com Try Grabbing Downloads.microsoft.com Try Grabbing Canon or any Driver Site Datacenter Requirements
  • 10.
    Assuring Software isTrustworthy Anti-Malware Scanning Name and Type Normalization Behavior Scanning Code Inspection External Meta Data Collection and Matching
  • 11.
    Software Analysis ResultsBasic Embedded Data PE Header Analysis Processor, Language, Binary Type Packers and Protectors 500+ Variants ASPack and Adobe PECompact and Google Install Formats Proprietary (like Skype) Binary Diffs (Patch Factory, MS PSF) Runtime Analysis and Sandboxing
  • 12.
    Software Classifications ClassifyingSource Trust-based vs. Type-based Classifying Files Functional (Font, Driver, Screensaver) vs. Descriptive Classifying Products Basic Open Source Commercial: Driver vs. Application IM / P2P / Games Better Malware Classifications Interesting Steganography/Watermarking/Hacking/Hiding
  • 13.
    Industry & GovernmentCertifications Government Certifications NIAP, FIPS, DCTS Vulnerability Reports CVE, CERT, SANS, MSB, etc. For Good Software: Certification Programs Built for Vista, Windows Certified, Java Approved eTrust Download For Malware: StopBadware, CME
  • 14.
  • 15.
  • 16.
  • 17.
    What about FalsePositives? Typical Suspects: Internet Explorer Drivers (Network, File Access) OS Components Universal Installer and Uninstaller Components Optimized Applications: Using Obscure Third-Party Software ASPack, PECompact, Themida
  • 18.
    Archive Format DistributionMost popular archive/packer formats
  • 19.
    Or Are TheyFalse Positives? (FTP Injection Attacks) HP
  • 20.
    Or Are TheyFalse Positives? (FTP Injection Attacks) Nero AG
  • 21.
    Vertical Detection MalwareSample Vertical File Detection Chart Good File Vertical Analysis Anti-Malware Reports per Web Site Bit9 ISV Safe Software Program
  • 22.
    Use Case: Anti-MalwareBenefits R&D Tool Packers, Metadata, Sources QA Tool False Positives Performance Accelerator Robin Bloor’s AVID Next Generation Anti-Malware
  • 23.
    About Bit9 WhatWe Do: Application and Device Control Solutions and Software Metadata Reporting What We Offer: Bit9 Parity Protects against Malicious Software and Data Leakage The Bit9 Knowledgebase is the Largest Collection of Actionable Intelligence about the World’s Software Background Founded in 2002 by founders of Okena (Cisco) $2 Million NIST ATP Grant in 2003 Headquartered in Cambridge, Mass. Venture Funded

Editor's Notes

  • #2 I'm Mario Vuksan, Director of Knowledgebase Services for Bit9. In case you haven't heard of Bit9, we're a leading application control and device control solution provider. Part of our offering is the Bit9 Knowledgebase, the  largest collection of actionable intelligence about the world's software and today I am here to talk with you about trust-based computing.