As the Xen hypervisor evolves so do the chances for a potentially exploitable bug to be introduced. This is the case for XSA-105/106, where a number of oversights in the x86 instruction emulator created the opportunity for a number of exploits.

1. www.bitdefender.com 5/12/2015• 1
Zero-Footprint Guest Memory
Introspection from Xen
Xen Project DEVELOPER SUMMIT
August 18th, 2014
Mihai DONȚU – Technical Project Manager, Bitdefender
Ravi SAHITA – Principal Engineer, Intel
Improving VM Introspection
Using Hardware Virtualization Extensions

2. www.bitdefender.com 5/12/2015• 2
Outline
• Security issues
• Memory introspection
o Operation
o Evolution
• XenAccess and mem-events enhancements
• Sample usages
• Hardware Acceleration for memory introspection
• Conclusions

3. www.bitdefender.com 5/12/2015• 3
Security issues we are facing today
• Advanced malware types
o Rootkits
o Kernel exploits
o Zero-days
• APTs, botnets, cyber-espionage
and so on heavily rely on those…

4. www.bitdefender.com 5/12/2015• 4
Security issues we are facing today 2

5. www.bitdefender.com 5/12/2015• 5
Security issues we are facing today 3
Malware today execute in the same context and with the same
privileges as anti-malware software  lack of isolation problem
Common Malware Advanced Malware
Drivers
App1
(Office)
OS Kernel
ISOLATION
Kernel Controlled
App2
(Browser)
Security
Solution
Security
Filter Drivers
App1
(Office)
OS Kernel
App2
(Browser)
Security
Solution
Security
Filter
ISOLATION
Kernel Controlled
Isolation Bypassed & Malware Controlled

6. www.bitdefender.com 5/12/2015• 6
Conclusion: advanced attacks evade
traditional security solutions

7. www.bitdefender.com 5/12/2015• 7
Envision the big picture
so… what’s the big difference?...
Xen Hypervisor
dom0
SVA
(domU0)
domU1Memory
Introspection
Engine
domU2 domUN
Protected area

8. www.bitdefender.com 5/12/2015• 8
Memory introspection
Envision the big picture
Xen Hypervisor
dom0
SVA
(domU0)
domU1Memory
Introspection
Engine
domU2
domUN
Protected area
Hypervisor Controlled, Hardware Enforced
STRONG ISOLATION
Introspection Engine Alternative 2
Introspection
Engine
Alternative 3

9. www.bitdefender.com 5/12/2015• 9
• Address a number of security issues from outside the
guest OS without relying on functionality that can be
rendered unreliable by advanced malware
• Analyze raw memory image of guest OS, services and
user mode applications, then identify
o kernel memory areas
o driver objects, driver code, IDT, etc.
o user memory areas
o process code, process stack, process heap, etc.
What is memory introspection?
Envision the big picture

10. www.bitdefender.com 5/12/2015• 10
• Use existent hardware virtualization extensions (Intel EPT / AMD RVI)
• Set hooks on guest OS memory
o mark 4K pages as non-execute or non-writable
o hooking & notification must be supported efficiently by HV & CPU
• Audit access of those areas by the code running in VM (OS or apps)
o write attempts (driver objects, fast I/O tables, page tables)
o execution attempts
• Allow or deny attempts – decision provided by security logic
How does it work?
Memory introspection

11. www.bitdefender.com 5/12/2015• 11
How does it work? 2
Memory introspection
EPT protected areas
provide detection for various OS level
changes (ex. new module load,
new process start, …)
EPT protected areas
provide detection for attempts &
protection against integrity violation
Guest VM Physical
Memory Space
OS
kernel
code
kernel
driver code
and data
kernel data
SSDT, IDT,
…
user
mode
code
user
mode
stacks &
heaps

12. www.bitdefender.com 5/12/2015• 12
• Building a reliable image of the guest OS state
o what objects are inside a guest VM?
o what operations are being performed inside a guest VM?
o object and event identification and correlation is done by the
introspection engine – to decide event and object maliciousness
• Using hooks we can detect numerous events, including
o a driver / kernel module is loaded or unloaded
o a new user process or thread is created
o user stack / heap is allocated
o memory is being paged in / out
How does it work? 3
Memory introspection

13. www.bitdefender.com 5/12/2015• 13
How does it work? 4
Memory introspection
Traditional in-guest
security solution
Out-of-guest memory
introspection
OSKernel
TypicalAnti-MalwareKernelModule
Read mem by Virtual Addr,
read files, registry, …
Enum processes, files, …
01010
10011 MZ…
Setup well-known
callbacks & notifications
New process
PID 0x1234
HKLM...
XENHypervisor
MemoryIntrospectionEngine
Read mem by Physical Addr
Read vCPU registers
01010
10011
Setup EPT hooks on mem
pages (by Physical Addr)
Write / Execute attempt on
PA 0x000A12345678
RAX = 0x1234
RIP = 0x7890

14. www.bitdefender.com 5/12/2015• 14
• bridging the semantic gap – obtain rich semantics from only
raw physical memory pages and virtual CPU registers
o how do we correlate 4K memory pages with semantically
rich and meaningful OS specific data structures?
o to be solved by security solution vendors
• forward lots of mem-event notifications with low overhead to
introspection engine
o to be solved by hypervisor and CPU vendors
Two big challenges
Memory introspection

15. www.bitdefender.com 5/12/2015• 15
Memory introspection evolution
• 2003 – Garfinkel & Rosenblum: “A Virtual Machine Introspection Based Architecture
for Intrusion Detection”
o the starting point for a considerable amount of academic research
• 2006 – Jiang & Wang: “’Out-of-the-box’Monitoring of VM-based High-Interaction Honeypots”
• 2008 – Dinaburg et al.: “Ether: MalwareAnalysis via Hardware Virtualization Extensions”
o Built on top of Xen 3.1
• 2008 – VMsafe API announced by VMware, which provides access to a guest’s:
o CPU, memory, disk, I/O devices etc.
o supported memory introspection for vSphere / ESXi
• 2010 – VMware vShield Endpoint (as a replacement for VMsafe API)
o in-guest agent based
o file introspection only
• 2012 – VMware deprecates VMsafe

16. www.bitdefender.com 5/12/2015• 16
Memory introspection in Xen
• 2007 – XenAccess, XenProbes
• 2008 – Lares
• 2009 – first patches for the mem-events API
• 2010 – LibVMI – uses XenAccess and XenStore
o targets Xen, but support for other HV-s can be added
o insufficient flexibility in changing page permissions
o no support for mapping guest memory RW
o insufficient information about the guest CPUs
o high overhead when reading the vCPU register state
o a ‘complete’ initialization requires intimate knowledge about the guest OS
o code for handling specific guest OS-s (Windows, with Linux in the works)

17. www.bitdefender.com 5/12/2015• 17
XenAccess and mem-events
enhancements
• 2014 – Bitdefender published a set of patches (as RFC) to enhance
the mem-events API
o implements its own version of LibVMI
o simpler API
o nothing [guest] OS specific
o support for other HV-s can be added
o allows to map guest memory (via EPT)
o uses a very simple page cache to optimize (un)maps
o optimized access to specific resources
• Some patches went into mainline, others will follow shortly

18. www.bitdefender.com 5/12/2015• 18
XenAccess and mem-events
enhancements 2
mem-event notification
Xen Hypervisor
SVA
(domU0)
Memory
Introspection
Engine
domU1
uint32_t flags;
uint32_t vcpu_id;
uint64_t gfn;
…
mem_event_regs_t regs;

19. www.bitdefender.com 5/12/2015• 19
Example use of the extended API
Xen Hypervisor
dom0
SVA
(domU0)
Memory
Introspection
Engine
Protected areas
Critical Kernel Module
Critical Kernel Module
App1 App2
…Code, stacks,
heaps, IAT, …
Code, stacks,
heaps, IAT, …
Code, stacks,
heaps, IAT, …
Mem-events and
VMCALLs facilitated by XEN

20. www.bitdefender.com 5/12/2015• 20
Example use of the extended API 2
• Bitdefender’s own introspection engine can
o protect the kernel from known rootkit hooking techniques
o protect user processes (e.g. browsers, MS Office, Adobe Reader, …) from
o code injection
o function detouring
o code execution from stack / heap
o unpacked malicious code
o inject remediation tools into the guest on-the-fly (no help from ‘within’ needed)
• Runs in userspace in a user domain (SVA – Security Virtual Appliance)
• Introspection logic has very small overhead
o bulk of the overhead is given by sending / receiving events and calls between
protected guest VMs and SVA

21. 21
Hardware Acceleration for Memory
Introspection
Ravi Sahita
Intel

22. 22
Hardware Acceleration for Memory
Introspection
 Factors Limiting VM Memory Monitoring
Performance
 Addressing Lack of Memory Isolation
 Addressing Performance gaps for execution
and data access-control policies
 Xen Extensions

23. 23
Factors Limiting VM Memory
Monitoring Performance
 Round-trip time
–Monitoring execution and data accesses
–Dynamic data structures imply high frequency events
 Filtering events
–Monitoring data accesses requires filtering non-
interesting events due to 4K page sharing
 Further, round-trip time is amplified with VMMs nesting

24. 24
Multiple EPTs as Protection Domains
CPU0
Hypervisor
Intel® VT-x
with EPT
VM0
Extended Page Table
(EPT) Domains
EPT
Walker
Execution crossing
EPT domains or data
Accesses causing eventshost physical
address
Intel® Virtualization Technology for IA-32, Intel® 64 and Intel® Architecture (Intel® VT-x)
OS kernel
Code/data
(RX/RW)
Driver
Code (RX)
Driver
Code (RO)
OS kernel
Code/data
(RO/NP)
Data
(RW)
Data
(NP)

25. 25
Addressing Lack of Isolation…
Xen
EPT DomainsEPT Domains
policy events
CPU
Hypervisor-
derived
isolation

26. 26
Kernel
module Kernel module
Security
module
…Without Sacrificing Performance
 Must allow for legal execution
of components isolated in
permission domains
 Each execution transfer across
domains leads to VM exits that
the hypervisor must mediate
 As components isolated via
domains, numerous execution
transfers are induced
 High Frequency of such VM
exits to the hypervisor causes
overhead
Process
Hypervisor
Eliminate these induced VM exits
on legal control transfers
Legal control
transfers

27. 27
VM Functions:
Hypercalls Without VM Exits
 VM Functions: Intel® VT-x extensions for services
configured by the hypervisor
– Different VM Functions correspond to different services
 Hypervisor configures VM Functions via new fields in
VMCS
 Guest software invokes VM Functions via new
instruction (VMFUNC<leaf>)
– Value in EAX specifies which VM Function leaf is invoked
 CPU provides services as configured with no VM exit
Intel® Virtualization Technology for IA-32, Intel® 64 and Intel® Architecture (Intel® VT-x)

28. 28
VMFUNC-based Domain Switching
 Paravirtualized
software can
request protection
domain switch via
VMFUNC (specifying
domain index)
 Hypervisor pre-
configures domain
index to EPTPs
 Hypervisor pre-
configures domain
boundaries
 View switching to
speed up access
control policies
CPU0
Hypervisor
Intel® VT-x
with EPT
VM0
Extended Page Table
(EPT) domains
EPT
Walker
VMFUNC (EPTP switch)
Host physical
address
Aternate Extended
Page Table
(EPT) list
Intel® Virtualization Technology for IA-32, Intel® 64 and Intel® Architecture (Intel® VT-x)

29. 29
Asynchronous Induced VMExits
 In VM-introspection scenarios critical
data is monitored in place
 Legacy approaches are to VMExit and
emulate access
 Alternatively, VMExit and switch
views to single step the guest (MTF)
 High frequency of writes to
monitored data cause high overhead
 Requires custom logic in the VMM
increasing complexity/state in the
hypervisor
Kernel (Guest)
Hypervisor
EPT Domains
Ring-0
Monitoring
Agent
Monitored
Memory
WRITE
2. EPT-V
VMexit
3. VMResume to
single step
access
Single
Step Memory
View
4. VMexit
after access
completed
Access
Policy
1. View
Policies
5. VMResume
EPT Domains Emulation

30. 30
Accelerating Induced Events
 Via Virtualization Exception (#VE)
– Mutates EPT violations into a new IA
exception – delivered through
guest IDT
 VMM enables EPT violation
conversion to #VE
 Data access monitoring view
policies setup in EPT domains
 Data access causes #VE instead
of VMexit
 Guest monitoring agent can
emulate in guest OR use VMFUNC
to single step access
Hypervisor (VMX-root)
Kernel/VMX-non-root
EPT Domains
Monitoring
Service
Monitored
Memory
1. Handle #VE
3. Complete single step
or emulate in guest
Single Step
Memory
View
2. Setup single
step or emulate
WRITE
Access
Policy
1. Policies
VMFUNC
VMFUNC
#VE
#VE info
VMFUNC
EPT Domains

31. 31
VMFUNC Configuration
 Hypervisor checks IA32_VMX_VMFUNC
MSR for allowed VM-Function controls
 Opts-in by setting “Enable VM functions”
in the secondary processor-based VM-
execution controls field
 Activates “EPTP switching” by setting
bit-0 in the VM Function Control
 Configure alternate EPTP values in
memory referenced via VMCS field
 Guest software uses VMFUNC opcode
with leaf selector EAX=0 and ECX
containing the index of EPTP (view)
selected
 Errors reported to the hypervisor via VM
Exits
“Enable VM functions” VM-
execution control (bit 13)
EPTP-list address
0:Alternate-EPTP
1:Alternate-EPTP
511:Alternate-EPTP
“EPTP switching” (bit 0)
Secondary processor-based
VM-execution controls
VMFunction Control (new)
In
memory
VMCS
In VMCS
(new)
IA32_VMX_VMFUNC MSR
(index 491H)

32. 32
Virtualization Exception (VE)
Configuration
 Enumerated by the VMM via
capability MSR
 Set VMCS “Enable VE” bit
 Negotiate “VE Info” page with
Hypervisor
 #VE delivered through guest
IDT
 Suppress VE EPTE Bit 63
– Set on pages the VMM does not
want to cause a #VE for
“Enable VE” VM-execution
control (bit 18)
VE Information Address
Exit Reason
Suppress
Exit Qualification
Secondary processor-based
VM-execution controls
In
memory
Virtual Machine Control
Structure (VMCS)
In VMCS
(new)
IA32_VMX_PROCBASED_
CTLS2 (48Bh) Bit 50
Guest Linear Address
Guest Physical Address
EPTP Index

33. 33
Xen Extensions
1. Efficiently creating and maintaining alternate EPT
views/domains via extension of p2m
2. Hypercalls to edit EPT permissions without conflicting with Xen
EPT management
3. Report guest-specific memory events via #VE in a Xen
compatible manner (Suppress #VE EPTE bit)
4. Enabling CPU acceleration if VMFUNC and #VE CPU enumerated
and opted-in
Xen Hypervisor
domUN
domU0
Security
Engine
EPT DomainsEPT Domains
Security
Engine
CPU
#VE
EPT DomainsEPT Domains
#VE

34. 34
VM Introspection Performance Improvements
 Round-trip time
– VMFUNC to allow safe, fast Memory View (EPT) switches
– VMexits mutated to #VE for guest memory monitoring
 Filtering events
– Reduced latency of #VE event handling reduces overhead of
filtering events
 Round-trip time amplification due to VMM nesting
– No VMexits to root VMM implies no amplification of VMexits due
to EPT violations
Scalable Runtime Memory Introspection w/o VT overheads

35. www.bitdefender.com 5/12/2015• 35
• Today Xen can be the base for providing a much improved layer of
security – serves as a model for other HV vendors
o Truly agentless security (zero in guest footprint)
o IT Admins can deploy introspection based solutions seamlessly,
without changing a single line of config inside the guest VMs
• Hardware enforced isolation (against kernel exploits, zero days, …)
• Hardware extensions enable intra-VM isolation to enable protected
agent based introspection for high frequency access monitoring and
agent isolation
• Both models require straight-forward Xen infrastructure changes
(multi-EPT views, hardware acceleration capabilities)
Conclusions

36. www.bitdefender.com 5/12/2015• 36
Thank you!
enterprise.bitdefender.com