The document discusses the development of a secret-free hypervisor prompted by issues like Meltdown and Spectre, focusing on rethinking Xen's strategy to eliminate sensitive data leaks. Key targets include removing direct memory mapping, implementing per-CPU stacks, and redefining the memory allocation structure within Xen. The project involves extensive code revisions and patch development to enhance security while maintaining performance.

1. Secret-free Hypervisor: Now and Future
Wei Liu <wl@xen.org>

2. Agenda
• Why this project
• What and how
• Primary targets
• Project scope
• Project showcase
• Project status

3. Why this project
• Prompted by Meltdown and Spectre
• Side-channels are here to stay
• Hardware bugs are here to stay
• Well-established isolation mechanisms and security boundaries
are broken
• Rethink Xen’s strategy against a whole class of issues
• What if there is no secret to leak in the first place?

4. Primary targets
• Direct map
• Xen has a 1:1 mapping of all physical memory in the machine for
performance reason
• If you can read from that mapping, you can read everything in the machine
• Stack
• Stack in Xen is allocated from an always mapped heap (xenheap) visible
to all PCPUs
• Stack content is considered sensitive

5. Fixes?
• Direct map
• Remove it completely
• Map / unmap pages as required when accessing memory
• Stack
• Switch to per-PCPU stack which is not visible to other PCPUs in the same
machine

6. Xen code is … J
• Xen’s academic origin and organic growth for years brings some
challenging aspects:
• Messy or not so modular code structure
• Missing design documents
• Lacking comments
• Interesting-looking code and design decisions

7. A taste of the scope
Secret-free hypervisor
Remove direct map Use per-PCPU stack
Implement Xenheap with vmap
Remove mapping infrastructure’s
dependency on direct map
Implement per-PCPU mapping
infrastructure
Implement Persistent MAP
(PMAP) infrastructure for
bootstrapping
Use Domheap for page tables
Carve out address space
Adjust early boot / AP bringup
code
Maintain correct mapping while
context switching

8. Project showcase
Secret-free hypervisor
Remove direct map Use per-PCPU stack
Implement Xenheap with vmap
Remove mapping infrastructure’s
dependency on direct map
Implement per-PCPU mapping
infrastructure
Implement Persistent MAP
(PMAP) infrastructure for
bootstrapping
Use Domheap for page tables
Carve out address space
Adjust early boot / AP bringup
code
Maintain correct mapping while
context switching

9. Domheap for page tables
• Currently page tables are xenheap pages (always mapped pages)
• All code is written with the “always mapped” property in mind
• Want to use domheap for page tables
• Need to (painstakingly) rewrite internal APIs and dependent code
without breaking bisection
• More importantly, the changes need to be broken down to
reviewable patches

10. Project showcase
Secret-free hypervisor
Remove direct map Use per-PCPU stack
Implement Xenheap with vmap
Remove mapping infrastructure’s
dependency on direct map
Implement per-PCPU mapping
infrastructure
Implement Persistent MAP
(PMAP) infrastructure for
bootstrapping
Use Domheap for page tables
Carve out address space
Adjust early boot / AP bringup
code
Maintain correct mapping while
context switching

11. Implement xenheap with vmap
• Less secrets in xenheap but keep its property
• Currently vmap is implemented on top of xenheap -- need to flip
their relationship
• Problems so far:
• Make sure page <-> virt conversion functions continue working (or remove
them entirely with lots of code churn…)
• Make vmap able to unmap a portion of a mapping – there is an
optimization in xmalloc which depends on such behavior
• Need to break a xenheap-vmap call loop in the page table allocation APIs -
- this is reason for the previous project
• Patches need not break bisection and be reviewable

12. Project showcase
Secret-free hypervisor
Remove direct map Use per-PCPU stack
Implement Xenheap with vmap
Remove mapping infrastructure’s
dependency on direct map
Implement per-PCPU mapping
infrastructure
Implement Persistent MAP
(PMAP) infrastructure for
bootstrapping
Use Domheap for page tables
Carve out address space
Adjust early boot / AP bringup
code
Maintain correct mapping while
context switching

13. Implement PMAP infrastructure
• Named (badly) after an infrastructure in Linux
• The only similarity is they both have a small-ish address space with
statically set up page tables
• To solve a bootstrapping problem:
• Per-PCPU mapping infrastructure needs some mapping mechanism to set
itself up without a direct map
• Only used during CPU onlining / offlining

14. Status
Secret-free hypervisor
Remove direct map Use per-PCPU stack
Implement Xenheap with vmap
Remove mapping infrastructure’s
dependency on direct map
Implement per-PCPU mapping
infrastructure
Implement Persistent MAP
(PMAP) infrastructure for
bootstrapping
Use Domheap for page tables
Carve out address space
Adjust early boot / AP bringup
code
Maintain correct mapping while
context switching
50+ patches
~5 patches
10+ patches (incomplete)
~5 patches
~10 patches
~15 patches
Legend:
Patches written from scratch
Patches from earlier work that
are maybe salvageable

15. Missing aspect: performance
• Can’t really test performance at this stage
• May not be too bad (but I could be wrong):
• Removing direct map
• Xen only needs to access a few pages while a guest is running
• PV guest page manipulation code is already very long
• Per-PCPU stack
• A PCPU only cares about its own stack during normal operation

16. More information
• https://xenproject.atlassian.net/browse/XEN-119
• https://lists.xenproject.org/archives/html/xen-devel/2019-
01/msg01923.html
• https://lists.xenproject.org/archives/html/xen-devel/2018-
10/msg01477.html