The OS kernel is critical to the security of a computer system. Many systems have been proposed to improve its security. A fundamental weakness of those systems is that page tables, the data structures that control the memory protection, are not isolated from the vulnerable kernel, and thus subject to tampering. To address that, researchers have relied on virtualization for reliable kernel memory protection. Unfortunately, such memory protection requires to monitor every update to the guest’s page tables. This fundamentally conflicts with the recent advances in the hardware virtualization support. In this paper, we propose SecPod, an extensible framework for virtualization-based security systems that can provide both strong isolation and the compatibility with modern hardware. SecPod has two key techniques: paging delegation delegates and audits the kernel’s paging operations to a secure space; execution trapping intercepts the (compromised) kernel’s attempts to subvert SecPod by misusing privileged instructions. We have implemented a prototype of SecPod based on KVM. Our experiments show that SecPod is both effective and efficient.

1. SecPod: A Framework for Virtualization-based
Security Systems
Xiaoguang Wang:
6, Yue Chen:, Zhi Wang:, Yong Qi6, Yajin Zhou;
Florida State University:
Xi’an Jiaotong University6
Qihoo 360;
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 1/22

2. Outline
1. Motivation
2. SecPod Design
3. Implementation
4. Evaluation
5. Related Work
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 2/22

3. Motivation
Page Table Integrity
Kernel protection requires page table integrity
§ Page tables decide address translation (from VA to PA)
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 3/22

4. Motivation
Page Table Integrity
Kernel protection requires page table integrity
§ Page tables decide address translation (from VA to PA)
§ Page tables control memory protection
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 3/22

5. Motivation
Page Table Integrity
Kernel protection requires page table integrity
§ Page tables decide address translation (from VA to PA)
§ Page tables control memory protection
§ e.g. Data Execution Prevention (Write ‘ eXecute)
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 3/22

6. Motivation
Page Table Integrity
Kernel protection requires page table integrity
§ Page tables decide address translation (from VA to PA)
§ Page tables control memory protection
§ e.g. Data Execution Prevention (Write ‘ eXecute)
However, page tables are always writable in the kernel
§ Kernel needs to frequently change memory mapping
§ Kernel protection can be subverted by manipulating page tables
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 3/22

7. Motivation
Virtualization-based Kernel Protection
§ Security tools are isolated “out-of-the-box”, but need to
intercept key guest events
Ż e.g., guest page table updates, control-register updates
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 4/22

8. Motivation
Virtualization-based Kernel Protection
§ Security tools are isolated “out-of-the-box”, but need to
intercept key guest events
Ż e.g., guest page table updates, control-register updates
§ Shadow paging enables reliable kernel memory protection
Ż Hypervisor uses shadow paging to virtualize memory
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 4/22

9. Motivation
Virtualization-based Kernel Protection
§ Security tools are isolated “out-of-the-box”, but need to
intercept key guest events
Ż e.g., guest page table updates, control-register updates
§ Shadow paging enables reliable kernel memory protection
Ż Hypervisor uses shadow paging to virtualize memory
Ż SPTs are synchronized with GPTs by the hypervisor
Ñsecurity tools can intercept guest page table updates
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 4/22

10. Motivation
Virtualization-based Kernel Protection
§ Security tools are isolated “out-of-the-box”, but need to
intercept key guest events
Ż e.g., guest page table updates, control-register updates
§ Shadow paging enables reliable kernel memory protection
Ż Hypervisor uses shadow paging to virtualize memory
Ż SPTs are synchronized with GPTs by the hypervisor
Ñsecurity tools can intercept guest page table updates
Ż SPTs supersede GPTs for address translation
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 4/22

11. Motivation
Virtualization Hardware Obsoletes Shadow Paging
§ Nested paging introduces two-level address translation for VMs
Ż Both GPT and NPT are used by CPU for address translation
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
Guest Physical
Address
Guest
Page Table
Guest Virtual
Address
Physical
Address
Nested
Page Table
Hypervisor
Guest
1
VMware: performance evaluation of Intel EPT hardware assist.
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 5/22

12. Motivation
Virtualization Hardware Obsoletes Shadow Paging
§ Nested paging introduces two-level address translation for VMs
Ż Both GPT and NPT are used by CPU for address translation
§ Nested paging has big performance advantage over SPT
Ż An acceleration of up to 48% for MMU-intensive tasks 1
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
Guest Physical
Address
Guest
Page Table
Guest Virtual
Address
Physical
Address
Nested
Page Table
Hypervisor
Guest
1
VMware: performance evaluation of Intel EPT hardware assist.
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 5/22

13. Motivation
Virtualization Hardware Obsoletes Shadow Paging
§ Nested paging introduces two-level address translation for VMs
Ż Both GPT and NPT are used by CPU for address translation
§ Nested paging has big performance advantage over SPT
Ż An acceleration of up to 48% for MMU-intensive tasks 1
§ Security tools cannot intercept guest memory updates with NPT
Ż Guest is free to change its GPTs, without notifying hypervisor
Shadow
Page Table
Guest Virtual
Address
Physical
Address
Hypervisor
Guest
Guest Physical
Address
Guest
Page Table
Guest Virtual
Address
Physical
Address
Nested
Page Table
Hypervisor
Guest
1
VMware: performance evaluation of Intel EPT hardware assist.
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 5/22

14. Motivation
Why SecPod
Our Goal:
A framework for virtualization-based security tools on the modern
virtualization hardware with nested paging
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 6/22

15. Motivation
Why SecPod
Our Goal:
A framework for virtualization-based security tools on the modern
virtualization hardware with nested paging
Our Solution:
SecPod: A Framework for Virtualization-based Security Systems
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 6/22

16. Motivation
Threat Model and Assumption
§ Trustworthy hardware and trusted booting
Ż Load-time integrity is protected by trusted booting
Ż IOMMU is properly configured to prevent DMA attacks
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 7/22

17. Motivation
Threat Model and Assumption
§ Trustworthy hardware and trusted booting
Ż Load-time integrity is protected by trusted booting
Ż IOMMU is properly configured to prevent DMA attacks
§ Hypervisor is trusted
Ż Formal verification [seL4, SOSP’09], integrity protection and
monitoring [HyperSafe, S&P’10]
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 7/22

18. Motivation
Threat Model and Assumption
§ Trustworthy hardware and trusted booting
Ż Load-time integrity is protected by trusted booting
Ż IOMMU is properly configured to prevent DMA attacks
§ Hypervisor is trusted
Ż Formal verification [seL4, SOSP’09], integrity protection and
monitoring [HyperSafe, S&P’10]
§ Kernel is benign but contains vulnerabilities
Ż Powerful attackers can change arbitrary memory of the kernel
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 7/22

19. SecPod Design
SecPod Architecture
SecPod &
Security Tool
Code/Data
SPT Pool
Entry Gate
pv_mmu_ops.alloc_pud
pv_mmu_ops.set_pud
pv_mmu_ops.write_cr3
...
Normal Space Secure Space
Hypervisor
Trusted
Boot
DMA
Protection
Guest
GPT
Exit Gate
User Mode
Kernel Mode
Key Technique I:
Paging Delegation
VMExit
Handler
Sensitive
Instructions
Key Technique II:
Execution Trapping
Up Call
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 8/22

20. SecPod Design
Key Technique I: Paging Delegation
§ SecPod creates an isolated address space from the kernel
GPT
SPT
NormalSpace
GPT SPT
Securespace
Traditional Shadow Paging Shadow Paging in SecPod
NPT
Guest
HypervisorHypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 9/22

21. SecPod Design
Key Technique I: Paging Delegation
§ SecPod creates an isolated address space from the kernel
§ Secure space maintains SPTs for the guest
Ż SPTs are the only effective page tables for the guest
Ż SPTs mirror GPTs (if no memory protection violation)
GPT
SPT
NormalSpace
GPT SPT
Securespace
Traditional Shadow Paging Shadow Paging in SecPod
NPT
Guest
HypervisorHypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 9/22

22. SecPod Design
Key Technique I: Paging Delegation
§ SecPod creates an isolated address space from the kernel
§ Secure space maintains SPTs for the guest
Ż SPTs are the only effective page tables for the guest
Ż SPTs mirror GPTs (if no memory protection violation)
§ SecPod forwards guest page table updates to secure space
GPT
SPT
NormalSpace
GPT SPT
Securespace
Traditional Shadow Paging Shadow Paging in SecPod
NPT
Guest
HypervisorHypervisor
Guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 9/22

23. SecPod Design
SecPod Address Space Layout
§ Normal/secure spaces use page-table based isolation
Ż Entry/exit gates are the only passage
Normal Space Secure Space
KernelUserProcess
Kernel Data
RW-
Kernel RO Data
R--
Kernel Code
R-X
Kernel Data
RW-
Kernel RO Data
R--
Kernel Code
R--
SecPod Data
RW-
SecPod Code
R-X
Security Tool Data
RW-
Security Tool Code
R-X
Gate Data
RW-
Entry/Exit Gates
R-XUser Data
RW-
User Code
R-X
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 10/22

24. SecPod Design
SecPod Address Space Layout
§ Normal/secure spaces use page-table based isolation
Ż Entry/exit gates are the only passage
§ Guest kernel is mapped in secure space
Ż Security tools can access guest memory, but not execute it
Normal Space Secure Space
KernelUserProcess
Kernel Data
RW-
Kernel RO Data
R--
Kernel Code
R-X
Kernel Data
RW-
Kernel RO Data
R--
Kernel Code
R--
SecPod Data
RW-
SecPod Code
R-X
Security Tool Data
RW-
Security Tool Code
R-X
Gate Data
RW-
Entry/Exit Gates
R-XUser Data
RW-
User Code
R-X
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 10/22

25. SecPod Design
Protecting Secure Space
Attacker might try to:
§ Enter secure space without security checks
§ Request malicious page table updates
Ż e.g., to map secure memory in guest
§ Misuse privileged instructions
Ż e.g., to load a malicious page table, to disable paging...
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 11/22

26. SecPod Design
Protecting Secure Space
Attacker might try to:
§ Enter secure space without security checks
§ Request malicious page table updates
Ż e.g., to map secure memory in guest
§ Misuse privileged instructions
Ż e.g., to load a malicious page table, to disable paging...
Our countermeasures:
§ Secure and efficient context switch
§ Page table update validation
§ Execution trapping of privileged instructions
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 11/22

27. SecPod Design
Secure and Efficient Context Switch
§ Entry/exit gates are only passage between secure/normal spaces
Ż Each gate switches the page table, the stack...
Ż Entry gate runs atomically by disabling interrupts (SIM [CCS ’09])
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 12/22

28. SecPod Design
Secure and Efficient Context Switch
§ Entry/exit gates are only passage between secure/normal spaces
Ż Each gate switches the page table, the stack...
Ż Entry gate runs atomically by disabling interrupts (SIM [CCS ’09])
§ Loading CR3 is a privileged instruction trapped by SecPod
Ż Performance overhead is high if each context switch is trapped
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 12/22

29. SecPod Design
Secure and Efficient Context Switch
§ Entry/exit gates are only passage between secure/normal spaces
Ż Each gate switches the page table, the stack...
Ż Entry gate runs atomically by disabling interrupts (SIM [CCS ’09])
§ Loading CR3 is a privileged instruction trapped by SecPod
Ż Performance overhead is high if each context switch is trapped
§ Intel CR3 target list to the rescue:
Ż Four page tables can be loaded without being trapped by CPU
Ż There are many SPTs for the guest
Ñ use a fixed top-Level page table for all SPTs
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 12/22

30. SecPod Design
Page Table Update Validation
§ SecPod enforces basic kernel memory integrity for guest
Ż No mapping is allowed to the secure space code/data
Ż Enforce kernel W‘X
Guest Page Table Shadow Page Table
SPT Update
Verification
①
③
④
②
Fast Index
Tables
PT
PD SPD
SPT
NormalSpace
SecureSpace
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 13/22

31. SecPod Design
Key Technique II: Execute Trapping
§ SecPod traps malicious privileged instructions executed by guest
Ż It can trap intended and unintended2
privileged instructions
§ Hypervisor notifies secure space trapped instructions via upcalls
Ż Similar to signal delivery in the traditional OS
2
X86 has variable-length instructions, unintended instructions can be “created” by jumping to the middle of an instruction.
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 14/22

32. SecPod Design
Trapped Sensitive Instructions
Instruction Semantics
LGDT Load global descriptor table
LLDT load local descriptor table
LIDT load interrupt descriptor table
LMSW load machine status word
MOV to CR0 write to CR0
MOV to CR4 write to CR4
MOV to CR8 write to CR8
MOV to CR3 load a new page table
WRMSR write machine-specific registers
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 15/22

33. Implementation
Implementation
§ Paging delegation
Ż Leverage Linux paravirtualization interface: pv mmu ops
§ Execution trapping implemented in the Hypervisor (KVM)
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 16/22

34. Implementation
Implementation
§ Paging delegation
Ż Leverage Linux paravirtualization interface: pv mmu ops
§ Execution trapping implemented in the Hypervisor (KVM)
§ Security tools:
Ż Compiled as ELF libraries and loaded into secure space
Ż Implemented an example tool to prevent unauthorized kernel
code from execution (Patagonix [USENIX Sec ’08], NICKLE [RAID ’08])
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 16/22

35. Evaluation
Security Analysis
§ Maliciously modify secure space memory
Ż Secure space memory is not mapped in the normal space
Ñ try to map the secure space memory in the guest
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 17/22

36. Evaluation
Security Analysis
§ Maliciously modify secure space memory
Ż Secure space memory is not mapped in the normal space
Ñ try to map the secure space memory in the guest
Ż Directly change the page mapping
Ż Ask SecPod to map secure memory
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 17/22

37. Evaluation
Security Analysis
§ Maliciously modify secure space memory
Ż Secure space memory is not mapped in the normal space
Ñ try to map the secure space memory in the guest
Ż Directly change the page mapping Ð SPT is isolated
Ż Ask SecPod to map secure memory Ð SPT update validation
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 17/22

38. Evaluation
Security Analysis
§ Maliciously modify secure space memory
Ż Secure space memory is not mapped in the normal space
Ñ try to map the secure space memory in the guest
Ż Directly change the page mapping Ð SPT is isolated
Ż Ask SecPod to map secure memory Ð SPT update validation
§ Misuse privileged instructions
Ż Privileged instructions by guest are trapped and verified
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 17/22

39. Evaluation
Performance Evaluation: LMBench
0%
20%
40%
60%
80%
100% null
open/close
fork
signal_installm
m
ap
TCP_bandw
idth
file(create)select(250fd)stat
ctxsw
(4p/16k)
Bcopy(libc)m
ain_m
em
PerformanceOverhead
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 18/22

40. Evaluation
Performance Evaluation: SysBench OLTP
0
100
200
300
400
500
600
700
2 4 8 16 32 64 128
Throughput(trans/sec)
Number of Threads
Linux
SecPod
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 19/22

41. Related Work
Related Work
§ Virtualization-based security
Ż Malware analysis: Ether[CCS’08]
Ż Rootkit detection and prevention: PoKeR[EuroSys’09]
Ż Virtual machine introspection: Virtuoso[S&P’11], SIM[CCS’09]
§ Kernel/user application security
Ż Exploit mitigation techniques: ASLR, DEP, CFI[CCS’07]
Ż Kernel/hypervisor memory integrity: TZ-RKP[CCS’14],
HyperSafe[S&P’10], Nested Kernel[ASPLOS’15]
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 20/22

42. Summary
Summary
SecPod &
Security Tool
Code/Data
SPT Pool
Entry Gate
pv_mmu_ops.alloc_pud
pv_mmu_ops.set_pud
pv_mmu_ops.write_cr3
...
Normal Space Secure SpaceHypervisor
Trusted
Boot
DMA
Protection
Guest
GPT
Exit Gate
User Mode
Kernel Mode
Key Technique I:
Paging Delegation
VMExit
Handler
Sensitive
Instructions
Key Technique II:
Execution Trapping
Up Call
§ SecPod: A Framework for Virtualization-based Security Systems
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 21/22

43. Summary
Thank you & Questions?
SecPod: A Framework for Virtualization-based Security Systems 2015 USENIX ATC July 8-10 2015 Santa Clara, CA 22/22