What is Cloud Computing?• The delivery of computing as a service as opposed to a product• Three types: – SaaS (Software as a Service) – PaaS (Platform as a Service) – IaaS (Infrastructure as a Service)
SaaS (Software as a Service)• Provider controls infrastructure• Client uses a hosted application
PaaS (Platform as a Service)• Provider controls operating system and hardware• Client controls middleware and interfaces to allow for software development
IaaS (Infrastructure as a Service)• Provider rents hardware and storage space as service• Client can install virtualized operating systems on which their applications can run
(IaaS Cloud Platform)• What if you want to create your own cloud?• Eucalyptus is a leading private cloud platform• Allows organizations to use existing infrastructure to create IaaS clouds• Can become a hybrid cloud when interfaced with Amazon Web Services for migration of workloads
Project Description1. Implementation of Eucalyptus cloud2. Testing potential for live forensics via virtual introspection3. Testing potential for recovering previous cloud tenant ephemeral data
1. Virtualization Definitions:• Physical host – Computer or server that will host virtual instances• Virtual Instance – Guest operating system that runs on top of physical host• Hypervisor – Allows multiple virtual instances to run concurrently on the physical host• KVM – One hypervisor option for Linux• QEMU – Processor emulator and virtualizer
Hypervisor• KVM turns Linux Kernel into hypervisor and virtual instance becomes Linux process• Host processor must support virtualization extensions: egrep ‘(vmx|svm)’ /proc/cpuinfo• Originally used Shadow page tables for virtual to physical memory translation• Now uses Intel’s Extended Page Tables or AMD’s Nested Page Tables for faster memory translation
Processor Emulator• Runs instance code on host CPU• Provides ability for virtual instance to access physical host I/O resources• Uses malloc() function for memory allocation• Virtual instance sees malloc() defined memory as its “physical” memory
- Front End Server• Manages underlying resources• Bucket storage (images, data)• Provides block level storage• Controls execution of instances
- Node• Uses KVM hypervisor to control instance• Kernel interfaces with host hardware• Runs instance code on host CPU)• Virtual instance that holds operating system• Linux Kernel based full virtualization solution
Virtual Introspection• Process of monitoring virtual instance state from a virtual machine monitor (VMM)• Two Examples: – QEMU-Monitor • QEMU provides a monitoring interface to control and inspect virtual instance – Libvirt • Toolkit to interact with KVM/QEMU in order to control virtual instance
Example 1: QEMU-Monitor• Can inspect running virtual instance (screenshots, memory dump, information about instance)• Can be accessed through: – Holding down CTRL-ALT plus Shift-2 which brings up a new window with the QEMU-Monitor – AQEMU (QEMU GUI) – Libvirt
Example 2: libvirt• A toolkit to interact with QEMU and hypervisor• 3 main pieces: – API library – Libvirtd daemon – Virsh command line utility• libvirt allows for scripting of the QEMU-Monitor:
QEMU Monitor – ‘pmemsave’• Command that dumps virtual instance’s “RAM” to file• The instance see the “RAM” as its physical memory but it is really virtual• pmemsave 0 536870912 memory.dump – 0 = start of memory offset in bytes – 536870912 = end of memory offset in bytes (512 MB) – memory.dump = output file name
Virtual Introspection - Scenario• A forensics examiner would like to crack the password of username shawn on virtual instance Shawn2• Here is a video of the manual process:
What is Ephemeral Storage?• Left over space after file system is installed and swap space is allocated• Virtual instances without persistent storage will utilize ephemeral storage for user data.• Example: – Virtual Disk Total – Filesystem – Swap = Ephemeral
Scenario 1:A. A cloud tenant cancels their subscriber agreementB. Cloud provider shuts down and terminates previous tenant’s instanceC. New tenant signs up and instance is launchedD. Is it possible for new tenant to recover previous tenant’s ephemeral data?
Scenario 1: Item A. (Old Tenant)A cloud tenant cancels their subscriber agreement• Node B has an 80GB physical drive• We created and launched a virtual instance sized to 107GB (Instance ID - i-47AC0940) – Allows majority of physical drive to be allocated for ephemeral storage to ensure some overlap with next tenant instance
Scenario 1: Item A. (Ephemeral)• Ephemeral Space of Instance ID – i47AC0940• c1.xlarge - /dev/sda = 107.4GB /dev/sda1 = root filesystem (1.5GB) /dev/sda2 = ephemeral (103GB) /dev/sda3 = swap (3.1GB)
Scenario 1: Item A. (Seed Data)• A unique seed was needed to simulate the prior tenant ephemeral data• We picked: – SecurityByObscurityIsNoSecurityAtAll! – Hex: 536563757269747942794f627363757269747949734 e6f53656375726974794174416c6c21• Two Python scripts used to create and plant seed throughout instance ephemeral space
Scenario 1: Item B. (Termination) Cloud provider shuts down and terminates previous tenant’s instance• Search performed with od and grep to verify seed data plant successful• In HybridFox we terminated instance i-47AC0940
Scenario 1: Item C. (New Tenant)New tenant signs up and instance is launched• A new instance with same 107GB size created and launched which ensures some overlap with prior terminated instance• (New Instance ID- i-476B083A)
Scenario 1: Item D. (Analysis) Is it possible for new tenant to recover previous tenant’s ephemeral data?• Search run with linux tool od and mmcat, img_cat, and sigfind from The Sleuth Kit (TSK)• No traces of the original seed were found in the new instance!
Scenario 1: Conclusion A new Eucalyptus cloud tenant is NOT able to recover a previous tenant’s ephemeral data! Scenario 2: What about a forensics examiner looking at the entire physical disk after termination??
Scenario 2: Physical Disk - Analysis• After new instance creation, we used Helix 2009 on Node B and took a bit for bit level copy of the entire physical drive with the enhanced dd program dcfldd• We then loaded the dd image into forensics analysis software EnCase and ran a search for the planted seed string.
Scenario 2: Physical Disk – Analysis• Results: – SecurityByObscurityIsNoSecurityAtAll!
Scenario 2: Physical Disk – Conclusion• Seed data is found all over the physical drive! – Why is the seed data not found from within the new instance but found on the physical drive?? – Sparse Files!
Sparse Files• Uses file system space more efficiently on empty blocks allocated to a file• Writes metadata representing empty blocks until block contains actual (non-empty) data – Is the reason a 107GB disk file can be created on an 80GB node controller disk – Reason why virtual disk can be created so quickly
Eucalyptus – Ephemeral Partition• An ephemeral partition can be created where all space is pre-allocated or it can use sparse files to simply reserve the empty space.• If Eucalyptus were to allocate the entire space upfront without sparse files, it would use the following dd command to sanitize prior session data:
Ephemeral Fully Allocated“dd bs=1M count=%11d if=dev/zero of=$s/ephemeral 2>/dev/null” – (if=dev/zero) destroys preexisting data by filling the ephemeral partition with zeroes
Ephemeral Sparsely Allocated• If Eucalyptus thin provisions the disk via the use of sparse files:• Outside virtual instance: – Physical host sees sparse space as empty holes• Inside virtual instance: – Instance sees sparse space as zeroes even though zeroes are not physically written
Final Conclusions• Virtual instance can’t see seed because KVM translates sparse space into zeroes.• Seed can be seen on physical drive because the sparse file concept doesn’t really write zeroes to the space, it only uses metadata to “reserve” the space.
Non-Eucalyptus Environments• libvirt also has a secure wiping utility: – Forensics examiner could check virsh.log to see if either of these commands were used on a non- eucalyptus system: • Overwrites existing data with all zeroes or a specific pattern: – #virsh vol-wipe <volume> • Deletes volume file but data still present on storage device: – #virsh vol-delete <volume>• Libvirt supports: – KVM/QEMU, Xen, Vmware, Microsoft Hyper-V, etc.
Documents• Please email us if you would like a copy of our documentation: – Technical Document – User Manual (Cloud Creation, Introspection Tools, Script Code)