SlideShare a Scribd company logo

CrySys guest-lecture: Virtual machine introspection on modern hardware

Slides of guest lecture at CrySys Lab, Budapest

1 of 36
Download to read offline
Virtual machine introspection on
modern hardware
Tamas K Lengyel
@tklengyel
tamas@tklengyel.com
2/19/2015 – CrySys Lab, Budapest
Agenda
1. VMI intro
2. Intel’s split-TLB
3. Intel’s EPT and its limitations
4. Intel’s #VE / VMFUNC / EPTP-switching
5. Intel’s SMM/DMM
6. ARM
7. Conclusion
Virtual Machine Introspection (VMI)
Interpret virtual hardware
● Network, Disk, vCPU & Memory
● The semantic gap problem:
Reconstruct high-level state information from low-
level data-sources.
Virtual Machine Introspection (VMI)
Bridging the semantic gap
● The guest OS is in charge of managing the virtual
hardware
● How to get the info from it?
o Install in-guest agent to query using standard
interfaces
o If OS is compromised in-guest agent can be
disabled / tampered with
o Just as vulnerable as your AntiVirus
Virtual Machine Introspection (VMI)
Bridging the semantic gap
● Replicate guest OS functions externally
o Requires expert knowledge on OS internals and
hardware behavior
o Requires debug data to understand in-memory
structures
● Need access to VM memory, vCPU registers, etc.
o Hypervisor support required (or custom VMM)
o Need to emulate hardware v2p translation
Translation lookaside buffer (TLB) poisoning
Translation lookaside buffer (TLB)
● Virtual to physical address translation is
expensive
● Hardware managed transparent cache
of the results
● Separate cache for data read/write and instruction fetch
(Harvard-type architecture)!
● The OS can flush it, but cannot query it
o Opportunity to whack it out of sync!
o Shadow Walker / FU rootkit

Recommended

Scalability, Fidelity and Stealth in the DRAKVUF Dynamic Malware Analysis System
Scalability, Fidelity and Stealth in the DRAKVUF Dynamic Malware Analysis SystemScalability, Fidelity and Stealth in the DRAKVUF Dynamic Malware Analysis System
Scalability, Fidelity and Stealth in the DRAKVUF Dynamic Malware Analysis SystemTamas K Lengyel
 
Virtual Machine Introspection with Xen
Virtual Machine Introspection with XenVirtual Machine Introspection with Xen
Virtual Machine Introspection with XenTamas K Lengyel
 
Pitfalls and limits of dynamic malware analysis
Pitfalls and limits of dynamic malware analysisPitfalls and limits of dynamic malware analysis
Pitfalls and limits of dynamic malware analysisTamas K Lengyel
 
BSides Denver: Stealthy, hypervisor-based malware analysis
BSides Denver: Stealthy, hypervisor-based malware analysisBSides Denver: Stealthy, hypervisor-based malware analysis
BSides Denver: Stealthy, hypervisor-based malware analysisTamas K Lengyel
 
Pitfalls of virtual machine introspection on modern hardware
Pitfalls of virtual machine introspection on modern hardwarePitfalls of virtual machine introspection on modern hardware
Pitfalls of virtual machine introspection on modern hardwareTamas K Lengyel
 
Hacktivity2014: Virtual Machine Introspection to Detect and Protect
Hacktivity2014: Virtual Machine Introspection to Detect and ProtectHacktivity2014: Virtual Machine Introspection to Detect and Protect
Hacktivity2014: Virtual Machine Introspection to Detect and ProtectTamas K Lengyel
 
Hacktivity 2016: Stealthy, hypervisor based malware analysis
Hacktivity 2016: Stealthy, hypervisor based malware analysisHacktivity 2016: Stealthy, hypervisor based malware analysis
Hacktivity 2016: Stealthy, hypervisor based malware analysisTamas K Lengyel
 
Cloud Security with LibVMI
Cloud Security with LibVMICloud Security with LibVMI
Cloud Security with LibVMITamas K Lengyel
 

More Related Content

What's hot

31c3 Presentation - Virtual Machine Introspection
31c3 Presentation - Virtual Machine Introspection31c3 Presentation - Virtual Machine Introspection
31c3 Presentation - Virtual Machine IntrospectionTamas K Lengyel
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenTamas K Lengyel
 
Virtual Machine Introspection with Xen on ARM
Virtual Machine Introspection with Xen on ARMVirtual Machine Introspection with Xen on ARM
Virtual Machine Introspection with Xen on ARMTamas K Lengyel
 
VM Forking and Hypervisor-based Fuzzing with Xen
VM Forking and Hypervisor-based Fuzzing with XenVM Forking and Hypervisor-based Fuzzing with Xen
VM Forking and Hypervisor-based Fuzzing with XenTamas K Lengyel
 
VM Forking and Hypervisor-based fuzzing
VM Forking and Hypervisor-based fuzzingVM Forking and Hypervisor-based fuzzing
VM Forking and Hypervisor-based fuzzingTamas K Lengyel
 
Troopers15 Lightning talk: VMI & DRAKVUF
Troopers15 Lightning talk: VMI & DRAKVUFTroopers15 Lightning talk: VMI & DRAKVUF
Troopers15 Lightning talk: VMI & DRAKVUFTamas K Lengyel
 
Malware Collection and Analysis via Hardware Virtualization
Malware Collection and Analysis via Hardware VirtualizationMalware Collection and Analysis via Hardware Virtualization
Malware Collection and Analysis via Hardware VirtualizationTamas K Lengyel
 
Virtual Machine Introspection in a Hyberid Honeypot Architecture
Virtual Machine Introspection in a Hyberid Honeypot ArchitectureVirtual Machine Introspection in a Hyberid Honeypot Architecture
Virtual Machine Introspection in a Hyberid Honeypot ArchitectureTamas K Lengyel
 
Practical Windows Kernel Exploitation
Practical Windows Kernel ExploitationPractical Windows Kernel Exploitation
Practical Windows Kernel ExploitationzeroSteiner
 
Масштабируемый и эффективный фаззинг Google Chrome
Масштабируемый и эффективный фаззинг Google ChromeМасштабируемый и эффективный фаззинг Google Chrome
Масштабируемый и эффективный фаззинг Google ChromePositive Hack Days
 
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytes
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytesWindows Kernel Exploitation : This Time Font hunt you down in 4 bytes
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytesPeter Hlavaty
 
Применение виртуализации для динамического анализа
Применение виртуализации для динамического анализаПрименение виртуализации для динамического анализа
Применение виртуализации для динамического анализаPositive Hack Days
 
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)Shota Shinogi
 
Is That A Penguin In My Windows?
Is That A Penguin In My Windows?Is That A Penguin In My Windows?
Is That A Penguin In My Windows?zeroSteiner
 
Guardians of your CODE
Guardians of your CODEGuardians of your CODE
Guardians of your CODEPeter Hlavaty
 
Fun With Dr Brown
Fun With Dr BrownFun With Dr Brown
Fun With Dr BrownzeroSteiner
 
Csw2017 bazhaniuk exploring_yoursystemdeeper_updated
Csw2017 bazhaniuk exploring_yoursystemdeeper_updatedCsw2017 bazhaniuk exploring_yoursystemdeeper_updated
Csw2017 bazhaniuk exploring_yoursystemdeeper_updatedCanSecWest
 
Security research over Windows #defcon china
Security research over Windows #defcon chinaSecurity research over Windows #defcon china
Security research over Windows #defcon chinaPeter Hlavaty
 
Breaking paravirtualized devices
Breaking paravirtualized devicesBreaking paravirtualized devices
Breaking paravirtualized devicesPriyanka Aash
 

What's hot (20)

31c3 Presentation - Virtual Machine Introspection
31c3 Presentation - Virtual Machine Introspection31c3 Presentation - Virtual Machine Introspection
31c3 Presentation - Virtual Machine Introspection
 
OffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with XenOffensiveCon2022: Case Studies of Fuzzing with Xen
OffensiveCon2022: Case Studies of Fuzzing with Xen
 
Virtual Machine Introspection with Xen on ARM
Virtual Machine Introspection with Xen on ARMVirtual Machine Introspection with Xen on ARM
Virtual Machine Introspection with Xen on ARM
 
VM Forking and Hypervisor-based Fuzzing with Xen
VM Forking and Hypervisor-based Fuzzing with XenVM Forking and Hypervisor-based Fuzzing with Xen
VM Forking and Hypervisor-based Fuzzing with Xen
 
VM Forking and Hypervisor-based fuzzing
VM Forking and Hypervisor-based fuzzingVM Forking and Hypervisor-based fuzzing
VM Forking and Hypervisor-based fuzzing
 
Troopers15 Lightning talk: VMI & DRAKVUF
Troopers15 Lightning talk: VMI & DRAKVUFTroopers15 Lightning talk: VMI & DRAKVUF
Troopers15 Lightning talk: VMI & DRAKVUF
 
Malware Collection and Analysis via Hardware Virtualization
Malware Collection and Analysis via Hardware VirtualizationMalware Collection and Analysis via Hardware Virtualization
Malware Collection and Analysis via Hardware Virtualization
 
Virtual Machine Introspection in a Hyberid Honeypot Architecture
Virtual Machine Introspection in a Hyberid Honeypot ArchitectureVirtual Machine Introspection in a Hyberid Honeypot Architecture
Virtual Machine Introspection in a Hyberid Honeypot Architecture
 
Practical Windows Kernel Exploitation
Practical Windows Kernel ExploitationPractical Windows Kernel Exploitation
Practical Windows Kernel Exploitation
 
Масштабируемый и эффективный фаззинг Google Chrome
Масштабируемый и эффективный фаззинг Google ChromeМасштабируемый и эффективный фаззинг Google Chrome
Масштабируемый и эффективный фаззинг Google Chrome
 
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytes
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytesWindows Kernel Exploitation : This Time Font hunt you down in 4 bytes
Windows Kernel Exploitation : This Time Font hunt you down in 4 bytes
 
Применение виртуализации для динамического анализа
Применение виртуализации для динамического анализаПрименение виртуализации для динамического анализа
Применение виртуализации для динамического анализа
 
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)
Introduction of ShinoBOT (Black Hat USA 2013 Arsenal)
 
ShinoBOT Suite
ShinoBOT SuiteShinoBOT Suite
ShinoBOT Suite
 
Is That A Penguin In My Windows?
Is That A Penguin In My Windows?Is That A Penguin In My Windows?
Is That A Penguin In My Windows?
 
Guardians of your CODE
Guardians of your CODEGuardians of your CODE
Guardians of your CODE
 
Fun With Dr Brown
Fun With Dr BrownFun With Dr Brown
Fun With Dr Brown
 
Csw2017 bazhaniuk exploring_yoursystemdeeper_updated
Csw2017 bazhaniuk exploring_yoursystemdeeper_updatedCsw2017 bazhaniuk exploring_yoursystemdeeper_updated
Csw2017 bazhaniuk exploring_yoursystemdeeper_updated
 
Security research over Windows #defcon china
Security research over Windows #defcon chinaSecurity research over Windows #defcon china
Security research over Windows #defcon china
 
Breaking paravirtualized devices
Breaking paravirtualized devicesBreaking paravirtualized devices
Breaking paravirtualized devices
 

Similar to CrySys guest-lecture: Virtual machine introspection on modern hardware

SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMUSFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMULinaro
 
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...The Linux Foundation
 
Core Scheduling for Virtualization: Where are We? (If we Want it!)
Core Scheduling for Virtualization: Where are We? (If we Want it!)Core Scheduling for Virtualization: Where are We? (If we Want it!)
Core Scheduling for Virtualization: Where are We? (If we Want it!)Dario Faggioli
 
Enhance Virtualization Stack with Intel CET and MPX
Enhance Virtualization Stack with Intel CET and MPXEnhance Virtualization Stack with Intel CET and MPX
Enhance Virtualization Stack with Intel CET and MPXPriyanka Aash
 
Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overviewLinaro
 
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelXPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelThe Linux Foundation
 
TSC Summit #3 - Reverse engineering and anti debugging techniques
TSC Summit #3 - Reverse engineering and anti debugging techniquesTSC Summit #3 - Reverse engineering and anti debugging techniques
TSC Summit #3 - Reverse engineering and anti debugging techniquesMikal Villa
 
Detecting hardware virtualization rootkits
Detecting hardware virtualization rootkitsDetecting hardware virtualization rootkits
Detecting hardware virtualization rootkitsEdgar Barbosa
 
Current and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on LinuxCurrent and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on Linuxmountpoint.io
 
Breaking hardware enforced security with hypervisors
Breaking hardware enforced security with hypervisorsBreaking hardware enforced security with hypervisors
Breaking hardware enforced security with hypervisorsPriyanka Aash
 
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORS
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORSDEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORS
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORSFelipe Prado
 
Owasp AppSecEU 2015 - BeEF Session
Owasp AppSecEU 2015 - BeEF SessionOwasp AppSecEU 2015 - BeEF Session
Owasp AppSecEU 2015 - BeEF SessionBart Leppens
 

Similar to CrySys guest-lecture: Virtual machine introspection on modern hardware (20)

SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMUSFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
 
Fuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdfFuzzing_with_Xen.pdf
Fuzzing_with_Xen.pdf
 
Intel update
Intel updateIntel update
Intel update
 
VXCON 2017
VXCON 2017VXCON 2017
VXCON 2017
 
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...
XPDDS17: Approach to Native Applications in XEN on ARM - Volodymyr Babchuk, E...
 
ECI OpenFlow 2.0 the Future of SDN
ECI OpenFlow 2.0 the Future of SDN ECI OpenFlow 2.0 the Future of SDN
ECI OpenFlow 2.0 the Future of SDN
 
SR-IOV Introduce
SR-IOV IntroduceSR-IOV Introduce
SR-IOV Introduce
 
Esp 100107093030-phpapp02
Esp 100107093030-phpapp02Esp 100107093030-phpapp02
Esp 100107093030-phpapp02
 
Core Scheduling for Virtualization: Where are We? (If we Want it!)
Core Scheduling for Virtualization: Where are We? (If we Want it!)Core Scheduling for Virtualization: Where are We? (If we Want it!)
Core Scheduling for Virtualization: Where are We? (If we Want it!)
 
Enhance Virtualization Stack with Intel CET and MPX
Enhance Virtualization Stack with Intel CET and MPXEnhance Virtualization Stack with Intel CET and MPX
Enhance Virtualization Stack with Intel CET and MPX
 
Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overview
 
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, IntelXPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
XPDS14 - Xen as High-Performance NFV Platform - Jun Nakajima, Intel
 
TSC Summit #3 - Reverse engineering and anti debugging techniques
TSC Summit #3 - Reverse engineering and anti debugging techniquesTSC Summit #3 - Reverse engineering and anti debugging techniques
TSC Summit #3 - Reverse engineering and anti debugging techniques
 
Detecting hardware virtualization rootkits
Detecting hardware virtualization rootkitsDetecting hardware virtualization rootkits
Detecting hardware virtualization rootkits
 
Current and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on LinuxCurrent and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on Linux
 
Breaking hardware enforced security with hypervisors
Breaking hardware enforced security with hypervisorsBreaking hardware enforced security with hypervisors
Breaking hardware enforced security with hypervisors
 
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORS
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORSDEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORS
DEF CON 27 - ALI ISLAM and DAN REGALADO WEAPONIZING HYPERVISORS
 
003-vmm.pptx
003-vmm.pptx003-vmm.pptx
003-vmm.pptx
 
Hacker bootcamp
Hacker bootcampHacker bootcamp
Hacker bootcamp
 
Owasp AppSecEU 2015 - BeEF Session
Owasp AppSecEU 2015 - BeEF SessionOwasp AppSecEU 2015 - BeEF Session
Owasp AppSecEU 2015 - BeEF Session
 

More from Tamas K Lengyel

Estimating Security Risk Through Repository Mining
Estimating Security Risk Through Repository MiningEstimating Security Risk Through Repository Mining
Estimating Security Risk Through Repository MiningTamas K Lengyel
 
Anti-evil maid with UEFI and Xen
Anti-evil maid with UEFI and XenAnti-evil maid with UEFI and Xen
Anti-evil maid with UEFI and XenTamas K Lengyel
 
Stealthy, Hypervisor-based Malware Analysis
Stealthy, Hypervisor-based Malware AnalysisStealthy, Hypervisor-based Malware Analysis
Stealthy, Hypervisor-based Malware AnalysisTamas K Lengyel
 
CyberSEED: Virtual Machine Introspection to Detect and Protect
CyberSEED: Virtual Machine Introspection to Detect and ProtectCyberSEED: Virtual Machine Introspection to Detect and Protect
CyberSEED: Virtual Machine Introspection to Detect and ProtectTamas K Lengyel
 
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and Cloning
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and CloningNSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and Cloning
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and CloningTamas K Lengyel
 
Dfrws eu 2014 rekall workshop
Dfrws eu 2014 rekall workshopDfrws eu 2014 rekall workshop
Dfrws eu 2014 rekall workshopTamas K Lengyel
 

More from Tamas K Lengyel (6)

Estimating Security Risk Through Repository Mining
Estimating Security Risk Through Repository MiningEstimating Security Risk Through Repository Mining
Estimating Security Risk Through Repository Mining
 
Anti-evil maid with UEFI and Xen
Anti-evil maid with UEFI and XenAnti-evil maid with UEFI and Xen
Anti-evil maid with UEFI and Xen
 
Stealthy, Hypervisor-based Malware Analysis
Stealthy, Hypervisor-based Malware AnalysisStealthy, Hypervisor-based Malware Analysis
Stealthy, Hypervisor-based Malware Analysis
 
CyberSEED: Virtual Machine Introspection to Detect and Protect
CyberSEED: Virtual Machine Introspection to Detect and ProtectCyberSEED: Virtual Machine Introspection to Detect and Protect
CyberSEED: Virtual Machine Introspection to Detect and Protect
 
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and Cloning
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and CloningNSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and Cloning
NSS 2013: Towards Hybrid Honeynets via Virtual Machine Introspection and Cloning
 
Dfrws eu 2014 rekall workshop
Dfrws eu 2014 rekall workshopDfrws eu 2014 rekall workshop
Dfrws eu 2014 rekall workshop
 

CrySys guest-lecture: Virtual machine introspection on modern hardware

  • 1. Virtual machine introspection on modern hardware Tamas K Lengyel @tklengyel tamas@tklengyel.com 2/19/2015 – CrySys Lab, Budapest
  • 2. Agenda 1. VMI intro 2. Intel’s split-TLB 3. Intel’s EPT and its limitations 4. Intel’s #VE / VMFUNC / EPTP-switching 5. Intel’s SMM/DMM 6. ARM 7. Conclusion
  • 3. Virtual Machine Introspection (VMI) Interpret virtual hardware ● Network, Disk, vCPU & Memory ● The semantic gap problem: Reconstruct high-level state information from low- level data-sources.
  • 4. Virtual Machine Introspection (VMI) Bridging the semantic gap ● The guest OS is in charge of managing the virtual hardware ● How to get the info from it? o Install in-guest agent to query using standard interfaces o If OS is compromised in-guest agent can be disabled / tampered with o Just as vulnerable as your AntiVirus
  • 5. Virtual Machine Introspection (VMI) Bridging the semantic gap ● Replicate guest OS functions externally o Requires expert knowledge on OS internals and hardware behavior o Requires debug data to understand in-memory structures ● Need access to VM memory, vCPU registers, etc. o Hypervisor support required (or custom VMM) o Need to emulate hardware v2p translation
  • 6. Translation lookaside buffer (TLB) poisoning Translation lookaside buffer (TLB) ● Virtual to physical address translation is expensive ● Hardware managed transparent cache of the results ● Separate cache for data read/write and instruction fetch (Harvard-type architecture)! ● The OS can flush it, but cannot query it o Opportunity to whack it out of sync! o Shadow Walker / FU rootkit
  • 7. TLB poisoning Original algorithm: Input: Splitting Page Address (addr) Pagetable Entry for addr (pte) invalidate_instr_tlb (pte); // flush TLB pte = the_shadow_code_page (addr); // replace PTE in memory mark_global (pte); // disable auto-flush reload_instr_tlb (pte); // load it into TLB pte = the_orig_code_page (addr); // put original entry back
  • 8. TLB poisoning and virtualization VMENTRY/VMEXIT automatically flushes the entire TLB ● TLB poisoning is impossible ● Performance hit Introduction of TLB tagging (VPID) in Intel Nehalem (2008) ● 16-bit field specified in the VMCS for each vCPU ● Performance boost! ● VM TLB entries invisible to the VMM ● The problem is not the split TLB, it’s the TLB itself
  • 9. TLB poisoning with Windows / Linux TLB poisoning uses global pages ● CR4.PGE (bit 7) ● Makes PTE’s marked as global survive context-switches ● Great performance boost for kernel pages! Windows 7 ● Regularly flushes global pages by disabled & re-enabling CR4.PGE Linux ● Doesn’t touch CR4.PGE after boot
  • 10. The tagged TLB in Xen The TLB tag is assigned to the vCPU from a global counter ● asid->asid = data->next_asid++; No flushes, just assign a new tag when needed! When counter is overflown, flush everything and restart from 1 A new TLB tag is assigned to the vCPU every time a MOV- TO-CR3 is caught! ● The use of global pages is negated in the guest! ● The TLB needs to be primed on each context-switch!
  • 11. The tagged TLB in KVM Tag is assigned when vCPU structure is created • Doesn’t matter if the vCPU is activated or not. • Disable tagging and revert to old VMENTRY/VMEXIT TLB flushing if out of tags Priming the TLB in Linux guests on KVM is a problem! • However, the split TLB still has issues
  • 12. The sTLB! Intel Nehalem introduced second-level (victim) cache The problem: ● Split-TLB relies on a custom page-fault handler being called to re-split the TLB when it’s evicted ● With sTLB, the entry is brought back.. o Both into the iTLB & the dTLB o Split-TLB becomes unsplit! ● Split-TLB poisoning is unreliable in VMs!
  • 13. Disabling the sTLB 2014: MoRE Shadow Walker - The evolution of TLB splitting on x86 ● sTLB doesn’t merge entries with conflicting PTE permissions ● Definitely applies to EPT PTEs but hasn’t been tested with regular PTEs ● Make shadow code-page X only before loading and the custom #PF handler will be triggered when evicted
  • 14. Reconstructing the guest OS’s view Use standard OS structures at known locations ● KPCR, KDBG ● Linux Sysmap Scan for objects of interest with signatures ● Memory is in fast-flux ● False positives ● Weak signatures ● Cross-view validation is costly
  • 15. Interposition on x86 Intel Need to trap VM execution for coherent view ● Avoids memory fast-flux problems ● Can avoid DKOM/DKSM Two main options ● Tracing via EPT o Granularity is that of a page but can trap R/W/X ● Tracing via #BP o Only traps X on direct hit but is guest-visible o Can be protected with EPT to hide it
  • 16. Extended Page Tables (EPT) Speed up guest virtual to machine physical address translation. Two sets of tables: 1st layer managed by guest OS 2nd layer managed by the VMM Permissions can be different in the two layers!
  • 17. Extended Page Tables (EPT) Up to 512 EPTs per VM! ● Most VMMs just assign 1 EPT / VM ● Value defined in VMCS for each vCPU ● Xen 4.6 will allow different EPT / vCPU Permission stored in bit 0-2 of EPT PTE ● r : 1, /* bit 0 - Read permission */ ● w : 1, /* bit 1 - Write permission */ ● x : 1, /* bit 2 - Execute permission */ Lots of software programmable bits in EPT PTE available ● access : 4, /* bits 61:58 - p2m_access_t */
  • 18. Extended Page Tables (EPT) Unlike normal PTEs, permission can be: • R • X • R/W • R/X • R/W/X • W and W/X triggers EPT misconfiguration • Still traps to the VMM but viewer info transfered
  • 19. EPT limitations ● When violation is trapped, the information only gives the start address and type of the violation. ● A single R/W violation may touch up to 8-bytes! ● Not sufficient to match violation offset against known locations ● Violations in the vicinity of a watched area need to be treated as potential hits as well!
  • 20. EPT limitations Read/Write violation ambiguities "An EPT violation that occurs during as a result of execution of a read- modify-write operation sets bit 1 (data write). Whether it also sets bit 0 (data read) is implementation-specific and, for a given implementation, may differ for different kinds of read-modify-write operations.“ - Intel SDM It is possible to siphon data using r-m-w operations from a page that doesn’t allow reading! Fixed in Xen 4.5
  • 21. EPT limitations How to let the VM progress without missing a potential event?
  • 22. EPT limitations How to let the VM progress without missing a potential event? Emulate the instruction! • Xen comes with a baked in x86 emulator! • Option to “emulate with no write” • Perform the instruction but don’t let it write to memory
  • 23. Intel #VE (Virtualization Exceptions) EPT based tracing has a 4k granularity ● Too much overhead when we only care about certain points ● Handle EPT violations in the guest! #VE Interrupt Service Routine (ISR) ● Interrupt handler within the guest ● Defined in guest IDT #20 ● EPT PTE bit 63 determines if violation triggers #VE or VMEXIT
  • 24. VMFUNC and EPTP switching “This instruction allows software in VMX non-root operation to invoke a VM function, which is processor functionality enabled and configured by software in VMX root operation. No VM exit occurs.” VMFUNC with EAX=0 => EPTP switching ● Remember how we can have up to 512 EPT’s per VM? ● Pass EPT IDX as parameter in ECX (0-512) ● Faulting translation of GPFN is performed via the new table! ● Guest never really “knows” the value of EPTP
  • 25. EPTP switching How do we go “back” to the original EPTP afterwards? Single-step and restore? “The key observation is that at any single point in time, a given hardware thread can be fetching an instruction or reading data, but not both. There are ways of avoiding the single-step too. [...] It's an optimization certainly, but it's not required, and it's not a technique we have placed in the public domain. You could try talking to us under NDA or figure it out for yourself.” Ed White (Intel) xen-devel, 1/16/2015
  • 26. 1-setting of the “EPT-violation #VE” VM- execution control Optional CPU feature baked into the chip During VMFUNC[0] saves the updated EPT index (ECX[15:0]) and gives it to the next #VE Allows you to juggle the “views” without missing anything! Index 0 := selective X traps; 1 := only X; 2 := only R/W • If #VE with IDX=0, switch to IDX=1 (VMEXIT if of interest) • If #VE with IDX=1, switch to IDX=2 • If #VE with IDX=2, switch to IDX=0
  • 27. System Management Mode (SMM) SMM intended for low-level services, such as: ● thermal (fan) control ● USB emulation ● hw errata workarounds Can be used for: ● VMI ● Anti-VMI! Hard to take control of it on (most) Intel devices as it is loaded by the signed BIOS.
  • 28. SMM ● Normal mode SMM is triggered by interrupts (SMI) ● Can be configured to happen periodically ● Always returns to the same execution mode afterwards
  • 29. SMM The problem for SMM based VMI systems: “A limitation of any SMM-based solution [...] is that a malicious hypervisor could block SMI interrupts on every CPU in the APIC, effectively starving the introspection tool. For VMI, trusting the hypervisor is not a problem, but the hardware isolation from the hypervisor is incomplete.” Jain et al. “SoK: Introspections on Trust and the Semantic Gap” 2014, IEEE S&P
  • 30. Intel Dual-monitor mode SMM – the DMM Available on all CPUs with VT-x SMM can become an independent hypervisor! SMIs are still available but.. A VMCALL executed by the VMM automatically and unconditionally traps into the DMM
  • 31. Intel DMM The VMCALL instruction can be used to instrument the VMM. ● Same way the VMM can use #BP to instrument a VM. The DMM can enter any execution mode on the system! ● Full control over the execution flow ● Hidden VMs The DMM can disable SMIs for a VM! ● Forced execution ● Nothing in the system can preempt it (SMIs, NMIs, etc.)
  • 32. ARM
  • 33. ARM 2-stage paging Very similar to EPT • Fewer software available bits available • We need to store our custom permissions somewhere so we know if a violation was induced by us or not • With EPT its just stored directly in the PTE Xen mem_access for ARM • VMM maintains a Radix tree to store custom permissions • During violation the tree is queried • If violation induced by monitor, forward to monitor
  • 34. ARM 2-stage paging How do we let the VM progress without potentially missing the next event? No MTF-like singlestep available • Emulation? • On x86 Xen already comes with a built-in emulator • Not so much on ARM.. The same trick we did in the VMFUNC! • Only this time it traps into the VMM
  • 35. ARM SMC is trappable to the VMM! ARM has no #BP that traps to the VMM • Secure Monitor Call (SMC) can be configured to do so! • SMC allowed only from VMM or guest kernel • Better than nothing What to use the TrustZone for? • Integrity check the VMM! • Make all VMM pages non-writable • VMM #PF handler executes SMC • Verify if change is allowed
  • 36. Conclusion Modern hardware is complex Behavior sometimes vaguely defined Intel really made a lot of progress since VT-x was introduced but the DMM is very odd ARM generally needs to catch-up but its progressing rapidly Thanks! Questions?