This talk was given at DEF CON 2010 by Jeremy Chiu, Benson Wu, and Wayne Huang
For antivirus vendors and malware researchers today, the challenge lies not in "obtaining" the malware samples - they have too many already. What's needed is automated tools to speed up the analysis process. Many sandboxes exist for behavior profiling, but it still remains a challenge to handle anti-analysis techniques and to generate useful reports.
The problem with current tools is the monitoring mechanism - there's always a "sandbox" or some type of monitoring mechanism that must be loaded BEFORE malware execution. This allows malware to detect whether such monitoring mechanisms exist, and to bail out thus avoiding detection and analysis.
Here we release 0box--an afterDark analyser that loads AFTER malware execution. No matter how well a piece of malware hides itself, there will be runtime forensics data that can be analyzed to identify "traces" of a process trying to hide itself. For example, evidences within the process module lists or discrepancies between kernel- and user-space datastructures. Since analysis is done post mortem, it is very hard for malware to detect the analysis.
By using runtime forensics to extract evidences, we turn a piece of malware from its original binary space into a feature space, with each feature representing the existence or non-existence of a certain behavior (ex, process table tampering, unpacking oneself, adding hooks, etc). By running clustering algorithms in this space, we show that this technique not only is very effective and very fast at detecting malware, but is also very accurate at clustering the malware into existing malware families. Such clustering is helpful for deciding whether a piece of malware is just a variation or repacking of an existing malware family, or is a brand new find.
Using three case studies, we will demo 0box, compare 0box with 0box with recent talks at BlackHat and other security conferences, and explain how 0box is different and why it is very effective. 0box will be released at the conference as a free tool.