This document summarizes a project on cloud forensics. It discusses cloud computing models like SaaS, PaaS, and IaaS. It describes implementing a private Eucalyptus cloud and testing live forensics via virtual introspection and recovering ephemeral data from previous cloud tenants. It demonstrates recovering data from a physical disk but not from a new virtual instance due to sparse files. The document concludes ephemeral data is not accessible to new tenants in Eucalyptus clouds due to sparse files and zero-filling.

1. Cloud Forensics
ForenSecure 2012
Shawn Davis
Terence Fernandes
Kenny Warren

2. What is Cloud Computing?
• The delivery of computing as a service as
opposed to a product
• Three types:
– SaaS (Software as a Service)
– PaaS (Platform as a Service)
– IaaS (Infrastructure as a Service)

3. SaaS (Software as a Service)
• Provider controls infrastructure
• Client uses a hosted application

4. PaaS (Platform as a Service)
• Provider controls operating system and hardware
• Client controls middleware and interfaces to
allow for software development

5. IaaS (Infrastructure as a Service)
• Provider rents hardware and storage space as
service
• Client can install virtualized operating systems on
which their applications can run

6. (IaaS Cloud Platform)
• What if you want to create your own cloud?
• Eucalyptus is a leading private cloud platform
• Allows organizations to use existing infrastructure
to create IaaS clouds
• Can become a hybrid cloud when interfaced with
Amazon Web Services for migration of workloads

7. Project Description
1. Implementation of Eucalyptus cloud
2. Testing potential for live forensics via virtual
introspection
3. Testing potential for recovering previous cloud
tenant ephemeral data

8. 1. Implementation – Eucalyptus Cloud

9. 1. Virtualization Definitions:
• Physical host – Computer or server that will host
virtual instances
• Virtual Instance – Guest operating system that
runs on top of physical host
• Hypervisor – Allows multiple virtual instances to
run concurrently on the physical host
• KVM – One hypervisor option for Linux
• QEMU – Processor emulator and virtualizer

10. Hypervisor
• KVM turns Linux Kernel into hypervisor and
virtual instance becomes Linux process
• Host processor must support virtualization
extensions: egrep ‘(vmx|svm)’ /proc/cpuinfo
• Originally used Shadow page tables for virtual to
physical memory translation
• Now uses Intel’s Extended Page Tables or AMD’s
Nested Page Tables for faster memory translation

11. Processor Emulator
• Runs instance code on host CPU
• Provides ability for virtual instance to access
physical host I/O resources
• Uses malloc() function for memory allocation
• Virtual instance sees malloc() defined memory as
its “physical” memory

13. HybridFox

14. - Front End Server
• Manages underlying resources
• Bucket storage (images, data)
• Provides block level storage
• Controls execution of instances

15. - Node
• Uses KVM hypervisor to control instance
• Kernel interfaces with host hardware
• Runs instance code on host CPU)
• Virtual instance that holds operating system
• Linux Kernel based full virtualization solution

16. 2. Live Forensics – Virtual Introspection

17. Virtual Introspection
• Process of monitoring virtual instance state from
a virtual machine monitor (VMM)
• Two Examples:
– QEMU-Monitor
• QEMU provides a monitoring interface to control and
inspect virtual instance
– Libvirt
• Toolkit to interact with KVM/QEMU in order to control
virtual instance

18. Example 1: QEMU-Monitor
• Can inspect running virtual instance
(screenshots, memory dump, information about
instance)
• Can be accessed through:
– Holding down CTRL-ALT plus Shift-2 which brings up a
new window with the QEMU-Monitor
– AQEMU (QEMU GUI)
– Libvirt

19. Example 2: libvirt
• A toolkit to interact with QEMU and hypervisor
• 3 main pieces:
– API library
– Libvirtd daemon
– Virsh command line utility
• libvirt allows for scripting of the QEMU-Monitor:

20. libvirt

21. QEMU Monitor – ‘pmemsave’
• Command that dumps virtual instance’s “RAM” to
file
• The instance see the “RAM” as its physical
memory but it is really virtual
• pmemsave 0 536870912 memory.dump
– 0 = start of memory offset in bytes
– 536870912 = end of memory offset in bytes (512
MB)
– memory.dump = output file name

22. Virtual Introspection - Scenario
• A forensics examiner would like to crack the
password of username shawn on virtual instance
Shawn2
• Here is a video of the manual process:

24. Could there be
an easier way???

25. Cloudypass.py
• Don’t want to type all of that???
• We have created a script!
• Here is a video of the automated process:

27. 3. Cloud Ephemeral Data

28. What is Ephemeral Storage?
• Left over space after file system is installed and
swap space is allocated
• Virtual instances without persistent storage will
utilize ephemeral storage for user data.
• Example:
– Virtual Disk Total – Filesystem – Swap = Ephemeral

29. Scenario 1:
A. A cloud tenant cancels their subscriber
agreement
B. Cloud provider shuts down and terminates
previous tenant’s instance
C. New tenant signs up and instance is launched
D. Is it possible for new tenant to recover previous
tenant’s ephemeral data?

30. Scenario 1: Item A. (Old Tenant)
A cloud tenant cancels their subscriber agreement
• Node B has an 80GB physical drive
• We created and launched a virtual instance sized
to 107GB (Instance ID - i-47AC0940)
– Allows majority of physical drive to be allocated for
ephemeral storage to ensure some overlap with next
tenant instance

31. Scenario 1: Item A. (Ephemeral)
• Ephemeral Space of Instance ID – i47AC0940
• c1.xlarge - /dev/sda = 107.4GB
/dev/sda1 = root filesystem (1.5GB)
/dev/sda2 = ephemeral (103GB)
/dev/sda3 = swap (3.1GB)

32. Scenario 1: Item A. (Seed Data)
• A unique seed was needed to simulate the prior
tenant ephemeral data
• We picked:
– SecurityByObscurityIsNoSecurityAtAll!
– Hex:
536563757269747942794f627363757269747949734
e6f53656375726974794174416c6c21
• Two Python scripts used to create and plant seed
throughout instance ephemeral space

33. Scenario 1: Item B. (Termination)
Cloud provider shuts down and terminates
previous tenant’s instance
• Search performed with od and grep to verify seed
data plant successful
• In HybridFox we terminated instance i-47AC0940

34. Scenario 1: Item C. (New Tenant)
New tenant signs up and instance is launched
• A new instance with same 107GB size created
and launched which ensures some overlap with
prior terminated instance
• (New Instance ID- i-476B083A)

35. Scenario 1: Item D. (Analysis)
Is it possible for new tenant to recover previous
tenant’s ephemeral data?
• Search run with linux tool od and
mmcat, img_cat, and sigfind from The Sleuth Kit
(TSK)
• No traces of the original seed were found in the
new instance!

36. Scenario 1: Conclusion
A new Eucalyptus cloud tenant is NOT
able to recover a previous tenant’s
ephemeral data!
Scenario 2:
What about a forensics examiner looking at the
entire physical disk after termination??

37. Scenario 2: Physical Disk - Analysis
• After new instance creation, we used Helix 2009
on Node B and took a bit for bit level copy of the
entire physical drive with the enhanced dd
program dcfldd
• We then loaded the dd image into forensics
analysis software EnCase and ran a search for the
planted seed string.

38. Scenario 2: Physical Disk – Analysis
• Results:
– SecurityByObscurityIsNoSecurityAtAll!

39. Scenario 2: Physical Disk – Conclusion
• Seed data is found all over the physical drive!
– Why is the seed data not found from within the new
instance but found on the physical drive??
– Sparse Files!

40. Sparse Files
• Uses file system space more efficiently on empty
blocks allocated to a file
• Writes metadata representing empty blocks until
block contains actual (non-empty) data
– Is the reason a 107GB disk file can be created on an
80GB node controller disk
– Reason why virtual disk can be created so quickly

41. Eucalyptus – Ephemeral Partition
• An ephemeral partition can be created where all
space is pre-allocated or it can use sparse files to
simply reserve the empty space.
• If Eucalyptus were to allocate the entire space
upfront without sparse files, it would use the
following dd command to sanitize prior session
data:

42. Ephemeral Fully Allocated
“dd bs=1M count=%11d if=dev/zero of=$s/
ephemeral 2>/dev/null”
– (if=dev/zero) destroys preexisting data by filling the
ephemeral partition with zeroes

43. Ephemeral Sparsely Allocated
• If Eucalyptus thin provisions the disk via the use
of sparse files:
• Outside virtual instance:
– Physical host sees sparse space as empty holes
• Inside virtual instance:
– Instance sees sparse space as zeroes even though
zeroes are not physically written

44. Final Conclusions
• Virtual instance can’t see seed because KVM
translates sparse space into zeroes.
• Seed can be seen on physical drive because the
sparse file concept doesn’t really write zeroes to
the space, it only uses metadata to “reserve” the
space.

45. Non-Eucalyptus Environments
• libvirt also has a secure wiping utility:
– Forensics examiner could check virsh.log to see if
either of these commands were used on a non-
eucalyptus system:
• Overwrites existing data with all zeroes or a specific pattern:
– #virsh vol-wipe <volume>
• Deletes volume file but data still present on storage device:
– #virsh vol-delete <volume>
• Libvirt supports:
– KVM/QEMU, Xen, Vmware, Microsoft Hyper-V, etc.

46. Documents
• Please email us if you would like a copy of our
documentation:
– Technical Document
– User Manual (Cloud Creation, Introspection
Tools, Script Code)

47. Questions?
• Shawn Davis – sdavis17@hawk.iit.edu
• Terence Fernandes – tfernand@hawk.iit.edu
• Kenny Warren – warrken@hawk.iit.edu

48. Thanks for Attending!