FFRI,Inc.
Fourteenforty Research Institute, Inc.
FFRI,Inc.
http://www.ffri.jp
Automated on-execute test using VirtualBox
J...
FFRI,Inc.
1. Background and motivation
2. Overview of a test
– automated on-execute test
– virtualization software and aut...
FFRI,Inc.
• Automated test against a large amount of malware is required to evaluate a
malware detection engine
• Testing ...
FFRI,Inc.
• Basic steps are following
1. Copy malware into a guest
2. Execute copied malware in the guest
3. Analyze or de...
FFRI,Inc.
software Licence copy-to copy-from exec revert method
VMware Workstation Proprietary ○ ○ ○ ○ VIX API
VMware ESX(...
FFRI,Inc.
• A kind of x86 virtualization software, currently developed by Oracle
• Version 4.0 and later, fully open sourc...
FFRI,Inc.
2.4.Example of VBoxManage
7
% vboxmanage startvm vm
% vboxmanage controlvm vm poweroff
% vboxmanage snapshot vm ...
FFRI,Inc.
• Automation script using VBoxManage, just a shell script
– auto-monkey.sh:automation for copy, exec, copy, reve...
FFRI,Inc.
• conform to KISS principle
• Estimation of remaining time is important for this kind of test
– we cannot determ...
FFRI,Inc.
• Testing under 1host and 7guest environment
• Processed 20,000 malware, each execution time was 60 seconds
– to...
FFRI,Inc.
• About 70% of total processor is idle state(each core also indicates same trend)
3.4.performance - processor
11...
FFRI,Inc.
• Consuming about 80% - 90% memory steadily
– real memory usage is between 2.5 and 4.0GB
3.4.performance - memor...
FFRI,Inc.
• Disk busy ratio(%util) stays around 30% steadily
• The number of queued requests is between 4 and 8
3.4.perfor...
FFRI,Inc.
• None of CPU, memory and IO wasn’t bottleneck under 1host and 7guest
environment
• It seems we can add some mor...
FFRI,Inc.
• http://www.ffri.jp/assets/files/research/freeware/FFRIAutoMonkey-1.0.tgz
• https://www.virtualbox.org/manual/U...
FFRI,Inc.
• E-Mail
– research-feedback@ffri.jp
• Twitter
– @FFRI_Research
Contact Information
16
Upcoming SlideShare
Loading in …5
×

Mr201309 automated on-execute_test_using_virtual_box_eng

460 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
460
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Mr201309 automated on-execute_test_using_virtual_box_eng

  1. 1. FFRI,Inc. Fourteenforty Research Institute, Inc. FFRI,Inc. http://www.ffri.jp Automated on-execute test using VirtualBox Junichi Murakami Executive Officer, Director of Advanced Development Division Ver2.00.01
  2. 2. FFRI,Inc. 1. Background and motivation 2. Overview of a test – automated on-execute test – virtualization software and automation methods – Oracle VM VirtualBox and its automation – example of VBoxManage 3. Automation script – FFRI AutoMonkey – design concept – throughput – performance 4. References 5. Contact information Agenda 2
  3. 3. FFRI,Inc. • Automated test against a large amount of malware is required to evaluate a malware detection engine • Testing methods are classified into on-demand and on-access testing • on-execute test which is a kind of on-access test has to execute malware one by one • Therefore automation based on virtualization is required • This slides describes automated on-execute test method using VirtualBox 1.Background and motivation 3 malware scanning on-demand on-access on-read/write on-execute * type of malware scanning
  4. 4. FFRI,Inc. • Basic steps are following 1. Copy malware into a guest 2. Execute copied malware in the guest 3. Analyze or detect malware in the guest 4. Preserve the result after execution is terminated 5. Revert the guest back to original condition 6. Go to 1. • Required functions to execute above are following a. Copying a file to a guest from a host (copy-to) b. Executing arbitrary a program in a guest from a host(exec) c. Copying a file from guest to a host (copy-from) d. Reverting a guest condition based on a snapshot(revert) 2.1.Automated on-execute testing 4 → All functions can be achieved by making a communication interface between a host and a guest using TCP/IP. We considered the way we do not need to involve developing software as possible as we could
  5. 5. FFRI,Inc. software Licence copy-to copy-from exec revert method VMware Workstation Proprietary ○ ○ ○ ○ VIX API VMware ESX(#1) Proprietary ○ ○ ○ ○ VIX API Oracle VM VirtualBox GPL2 ○ ○ ○ ○ VBoxManage QEMU + KVM GPL2(#2) × × × ○ Libvirt 2.2.virtualization software and automation methods 5 #1 ESXi can also use VIX API for 60days by registering a evaluation license. #2 KVM's parts are licensed under various GNU licenses(GPL, GPL2, LGPL2, etc.) • Use functions which virtualization software has natively • VMware(licensed) and VritualBox have all the features we need → We considered using VirtualBox because of the cost advantage • QEMU+KVM can be used by 3rd party software(ex: libguestfs + winexe) – “Malware Analysis: Collaboration, Automation & Tuning”, Shmoocon 2013 http://www.slideshare.net/xabean/malware-analysis-16674048
  6. 6. FFRI,Inc. • A kind of x86 virtualization software, currently developed by Oracle • Version 4.0 and later, fully open source software (GPL2) • Supporting various host and guest environments – HostOS:Windows, Linux, Mac OS X, Solaris – GuestOS:Windows, Linux, FreeBSD, OpenBSD, Mac OS X Server, Solaris,etc. • CLI is available (VBoxManage), friendly to automation – startvm , pause, resume, poweroff, clonevm, showvinfo – copyto, copyfrom, exec – taking snapshot and reverting – control virtual machine devices status, etc. 2.3.Oracle VM VirtualBox and its automation 6
  7. 7. FFRI,Inc. 2.4.Example of VBoxManage 7 % vboxmanage startvm vm % vboxmanage controlvm vm poweroff % vboxmanage snapshot vm restore snapshot-1 % vboxmanage guestcontrol exec vm --image “c:/windows/system32/calc.exe” ¥ --username admin --timeout 60000 --wait-exit % vboxmanage guestcontrol vm copyto “/some/file” “c:/file.txt” --username admin * starting a guest * power off a guest * reverting a guest based on a snapshot * execute a program in a guest from a host * copying a file to a guest from a host
  8. 8. FFRI,Inc. • Automation script using VBoxManage, just a shell script – auto-monkey.sh:automation for copy, exec, copy, revert steps – watch-monkey.sh:watch dog script for the monkey • It can execute multiple test simultaneously, works individually • Published at our website below, see README for the detail (License: BSD) – http://www.ffri.jp/research/freeware.htm 3.1.FFRI AutoMonkey 8 VirtualBox(guest) malware log auto-monkey.sh watch-monkey.sh VirtualBox(guest) malware log auto-monkey.sh watch-monkey.sh image snapshot image snapshot
  9. 9. FFRI,Inc. • conform to KISS principle • Estimation of remaining time is important for this kind of test – we cannot determine when it would finish if the script hangs up • Stability of VBoxManage (and VIX API) is the lifeline for the automation • In fact, error occurs when it runs long time – Failure by error • exits immediately • resumed a test automatically by watch-monkey.sh – Hanging up(stuck) by error • watch-monkey.sh monitors lifetime of a VirtualBox process • if it is stuck, kill and resume 3.2.Design concept 9
  10. 10. FFRI,Inc. • Testing under 1host and 7guest environment • Processed 20,000 malware, each execution time was 60 seconds – total elapsed time: 37h15m – throughput:8.95 malware/minute # if malware execution terminated less than 60 seconds, the script processes next item. • Host and guest environment is following 3.3.Throughput 10 Hardware CPU: Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz Memory: 8GB HDD: 1.8TB x 1 Host OS Ubuntu 13.04 + VirtualBox 4.2 Guest OS Windows XP SP3(x86) + FFR yarai 2.3 CPU:1 CPU Memory:750MB
  11. 11. FFRI,Inc. • About 70% of total processor is idle state(each core also indicates same trend) 3.4.performance - processor 11 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0:05 0:55 1:45 2:35 3:25 4:15 5:05 5:55 6:45 7:35 8:25 9:15 10:05 10:55 11:45 12:35 13:25 14:15 15:05 15:55 16:45 17:35 18:25 19:15 20:05 20:55 21:45 22:35 23:25 load all processsor statistics %idle %steal %iowait %system %nice %user
  12. 12. FFRI,Inc. • Consuming about 80% - 90% memory steadily – real memory usage is between 2.5 and 4.0GB 3.4.performance - memory 12 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000 5000000 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0:05 1:05 2:05 3:05 4:05 5:05 6:05 7:05 8:05 9:05 10:05 11:05 12:05 13:05 14:05 15:05 16:05 17:05 18:05 19:05 20:05 21:05 22:05 23:05 memory utilization statistics kbmemused kbmemfree kbmemused -(kbbuffers+kbcached)
  13. 13. FFRI,Inc. • Disk busy ratio(%util) stays around 30% steadily • The number of queued requests is between 4 and 8 3.4.performance – Disk IO 13 0 10 20 30 40 50 60 70 80 90 0 2 4 6 8 10 12 0:05 0:55 1:45 2:35 3:25 4:15 5:05 5:55 6:45 7:35 8:25 9:15 10:05 10:55 11:45 12:35 13:25 14:15 15:05 15:55 16:45 17:35 18:25 19:15 20:05 20:55 21:45 22:35 23:25 0:00 Usage of block device(/dev/sda) avgqu-sz %util
  14. 14. FFRI,Inc. • None of CPU, memory and IO wasn’t bottleneck under 1host and 7guest environment • It seems we can add some more guests up to around 10 VMs according to memory usage • However, we have to consider requirement of a process which is executed in a guest (cpu, memory) 3.4.performance - consideration 14
  15. 15. FFRI,Inc. • http://www.ffri.jp/assets/files/research/freeware/FFRIAutoMonkey-1.0.tgz • https://www.virtualbox.org/manual/UserManual.html • http://www.slideshare.net/xabean/malware-analysis-16674048 • http://www.youtube.com/watch?v=peHdyUlchSM • http://libguestfs.org/ • http://sourceforge.net/projects/winexe/files/ References 15
  16. 16. FFRI,Inc. • E-Mail – research-feedback@ffri.jp • Twitter – @FFRI_Research Contact Information 16

×