Xen Memory Management

Vancouver, February 2009

Memory management in (x86) Xen

Tim Deegan

Xen’s memory services
• Memory management
• Allocating memory to guests, scrubbing free memory
• Tracking memory usage with reference counts and types
 Heap allocators and the frametable.
• Virtual memory
• Protecting guests from each other
• Enforcing typing rules, e.g. read-only areas
• Providing translation services between address spaces
 MMU hypercalls, shadow pagetables, hardware-assisted paging

© 2008 Citrix Systems, Inc. — All rights reserved 2

Terminology
• Virtual address/Physical address/Machine address
• Frame vs. Page
• PFN: physical frame number
• Guest’s abstraction for tracking/allocating RAM
• Usually fairly contiguous
• GFN: guest frame number
• Guest’s idea of what hardware addresses are
• Used in guest pagetables
• MFN: machine frame number
• Actual hardware addresses

Basic memory management
• Buddy allocator hands out frames
• Each guest has a max number of frames
• Frame-table records for each frame:
• Owner, if any
• Linked list of other frames owned by this guest
• Reference count (must be zero to free the frame)
• Type, and a refcount for the type (must be zero to change type)
• TLB-flush-avoidance timestamp


PV pagetables, a.k.a. direct paging
• PFN  MFN table managed by the guest
• Shared MFN  PFN table provided by Xen
• GFN == MFN, so pagetables can be used directly
by the hardware
• Xen checks the contents of the guest pagetables
before allowing the hardware to see them.


Enforcing isolation
• Guest pagetables must have a pagetable type
• Xen checks that page contents obey the typing
rules before allowing them to take on PT type
• Typing rules:
• No mapping other guests’ frames
• No read-write mappings of frames with PT type
• Modifying an already-typed PT needs a call to Xen
to check the modification obeys the rules.
(Or trap-and-emulate assistance from Xen.)


Grant Tables
• Guest-supplied ACLs allowing other guests to map
their frames
• Mapper makes a hypercall with a domid, an
opaque index, and the address of a PTE
• Xen checks that entry in the mappee’s grant table
and if it’s OK, modifies the PTE
• Needs explicit unmap hypercall when finished
• Also available: grant-copy, where Xen memcpy()s
from/to a granted frame instead of mapping it.


HVM pagetables
• PFN  MFN table managed by Xen
• GFN == PFN so need another layer of translation
• Guest won’t cooperate in enforcing access control
• Two options:
• Xen builds shadow copies of guest pagetables
with the extra translations and controls added; or
• Hardware support for using a second set of
pagetables containing extra translations and
controls

Shadow pagetables
• Keep Xen-maintained copies of guest frames that
we think are being used as pagetables
• Guest never sees the shadows so we can add any
translations and restrictions we like
• 13 different kinds of shadows depending on what
kind of pagetable we think it is: a single frame can
have up to 10 shadows at once
• Also have three kinds of shadows for faking out
superpages (2MB of contiguous PFNs does not
mean 2MB of contiguous MFNs)

Shadow pagetables: building
• Start with an empty top-level shadow of the PFN in
CR3
• On pagefault, shadow the entries in the PT walk,
making new shadows at each level if necessary.
• Each shadow entry is the guest entry with the GFN
replaces by an MFN (of the next-level shadow or of
guest memory) and extra access restrictions:
• Pages that have shadows are mapped read-only.
• Extra restrictions can be specified in the PFN  MFN table.
• We can restrict write access to guest’s frames for tracking page-
dirtying during live migration.


Shadow pagetables: maintenance
• Shadowed pages are always kept read-only.
• When the guest writes to a shadowed frame, Xen’s
pagefault handler must:
• Emulate the current instruction to figure out what’s being written;
• Write the new value into the guest pagetable; and
• Update the equivalent parts of all shadows of the frame.


Shadow pagetables: tearing back down
• Shadowing a frame is expensive
• Thousands of cycles for trap and emulation of every write.
• Easy to tell when a page becomes a PT; harder to
tell when it stops:
• Reference count based on higher-level shadows and CR3 contents,
but hard to know when a PFN’s been used in CR3 for the last time
• Guess based on odd-looking page contents
• Guess based on memory access patterns
• Get PV drivers to give us hints
• Recycle under memory pressure by approximating LRU


Optimizations
• Tagged TLBs (AMD’s ASID; Intel’s VPID) allow us
to avoid a TLB flush on every VMEXIT/VMENTER
• In theory can do even better now that Win2k8 supports context
switching without TLB flushing.

• Shadowing not-present entries with invalid entries
lets us fast-track “real” pagefaults back to the guest
• Out-of-sync shadows: let the guest write directly to
the lowest level of pagetables and sync up the
shadows whenever a hardware TLB would re-read
(TLB flush, page faults, higher-level writes)


Hardware-assisted paging
• Xen supplies a second set of pagetables describing
the PFN  MFN translation and extra restrictions
• CPU takes a pointer to this as well as a (PFN-
space) CR3 value from the guest
• MMU hardware applies the composition of the two
translations and the intersection of the access
rights


Hardware-assisted paging: performance
Avoid expensive trap + emulate on writes to PTs,
and extra logic on pagefault path
TLB fill can now take 20 memory accesses!
CPU’s TLB is much smaller than the set of
shadows we can maintain
• AMD’s RVI gives +10% performance over shadows
on some workloads, -10% on others; Intel’s EPT
seems more consistently better than shadowing
• Performance depends heavily on using superpage
mappings in the second pagetable

Fin


Xen Memory Management

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Xen Memory Management

Similar to Xen Memory Management (20)

More from The Linux Foundation

More from The Linux Foundation (20)

Recently uploaded

Recently uploaded (20)

Xen Memory Management