XS Oracle 2009 PVOps

  • 454 views
Uploaded on

Jeremy Fitzhardinge: PVOPS Status

Jeremy Fitzhardinge: PVOPS Status

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
454
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
13
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Xen in Mainline Linux Status Update Jeremy Fitzhardinge Citrix Xen Summit at Oracle Feb 24-25, 2009
  • 2. Upstreaming Xen Xen domU support ● Upstream and stable for well over a year – Shipped as standard in several distros – Fedora ● Debian ● ...? ● (SuSE is the odd one out.) – Dom0 support is the next frontier ● Xen Summit at Oracle Feb 24-25, 2009
  • 3. git migration Migrated upstream Xen work to git ● Mercurial + patchqueue was getting awkward ● Git makes is more conventional: ● Easier for upstream developers to work with – Easier for Xen developers to get a working tree – Hosted on git.kernel.org: “jeremy/xen.git” ● Xen Summit at Oracle Feb 24-25, 2009
  • 4. Git Branches Two main merge branches ● xen/master – core Xen, domU – xen/dom0/hackery – dom0 work branch – Lots of topic branches ● Merged into merge branches for use – Will add more as needed – Room for lots more ● Will take anything – Even merged if you don't break things! – Xen Summit at Oracle Feb 24-25, 2009
  • 5. General improvements Preemptable lazy mmu updates ● Allow more use of lazy updates – Less scheduling effect – /proc/xen + usermode xenbus for tools support ● New calling convention for reduced register ● pressure Still need a concerted performance-oriented ● push Xen Summit at Oracle Feb 24-25, 2009
  • 6. Paravirt performance Early design goal of paravirt_ops was zero- ● overhead native execution Some benchmarking showed we were falling ● short ~5% impact on mmap heavy benchmark – The problem: inlines -> function calls cause lots ● more register pressure Resulting in higher cache traffic due to spills ● Xen Summit at Oracle Feb 24-25, 2009
  • 7. New calling convention Solution: define new calling convention ● Normal convention trashes many registers ● Define new one which preserves everything ● Except return value – Result: ● compiler can generate better code – Inline patching more effective – But how to call normal code? ● Xen Summit at Oracle Feb 24-25, 2009
  • 8. Generate reg-saver thunks Add macro to generate register saving wrapper ● around conventional code Pushes cost of call to complex code ● Simple callees – written in asm – need no ● wrapper Overall: reduce overhead to 1-2% ● Xen Summit at Oracle Feb 24-25, 2009
  • 9. Dom0 Boot-time control domain ● Also basis for driver domains ● At heart, a normal PV Xen domain ● But with extra bits ● APIC – ACPI – Device mappings – DMA / SWIOTLB – Xen Summit at Oracle Feb 24-25, 2009
  • 10. Dom 0: APICs APIC = interrupt controller ● Two parts: ● IO APICs connected to PCI slots – Local ACPI in each CPU – Xen owns Local APIC, since dom0 has ● virtualized CPUs Dom0 owns IO APICs because it has the ● device drivers Xen Summit at Oracle Feb 24-25, 2009
  • 11. Dom 0: APICs (2) Changes surprisingly small ● Some changes to APIC discovery to avoid local – apics Hook acpi_register_gsi to direct all setup to Xen – code Only support ACPI interrupt routing – IRQ space reserved for 1:1 GSI mapping – Caveat: APIC and ACPI use different senses ● for interrupt triggering Getting it wrong works surprisingly well – Xen Summit at Oracle Feb 24-25, 2009
  • 12. Dom 0: ACPI ACPI = Amazingly Complex Piece of Interface ● Used for everything from device discovery, ● interrupt routing to power management To start with, mostly interested in devices and ● interrupts Main changes to APIC code were to make it ● map properly Seems well-behaved after that ● Xen Summit at Oracle Feb 24-25, 2009
  • 13. Dom 0: device mappings Dom0 kernel has two address spaces: ● Pseudo-physical domain memory – Machine-physical memory – Must know which is which for any given mapping ● Add _PAGE_IOMAP flag to mark hardware ptes ● No pfn->mfn conversion – VM_IO segments mapped with _PAGE_IOMAP ● E820 map avoids RAM holes ● Xen Summit at Oracle Feb 24-25, 2009
  • 14. Dom 0: DMA / SWIOTLB Make sure devices are talking to the memory ● they think they are Hook DMA operations to ● Convert pfn<->mfn – Make memory machine contiguous – SWIOTLB deals with a lot of the tricky cases ● Involves copying, so should be avoided for high – performance devices Xen Summit at Oracle Feb 24-25, 2009
  • 15. *** SUBJECT HERE *** *** BLURB HERE *** Xen Summit at Oracle Feb 24-25, 2009
  • 16. Dom 0: upstream progress Most core patches posted, out for review ● Looks good for next merge window (2.6.30) ● Core support at least – Not necessarily every feature – ● Xen Summit at Oracle Feb 24-25, 2009
  • 17. DEMO Xen Summit at Oracle Feb 24-25, 2009
  • 18. State: Good Generally very stable ● AHCI, PIIX, mpt drive controllers work ● E1000, iwlagn networking OK ● Intel graphics fully accelerated ● Radeon X server starts OK too – Sound fine ● USB good too ● Oh, and you can start domains ● Xen Summit at Oracle Feb 24-25, 2009
  • 19. State: Meh S3 suspend not done ● Cpufreq doesn't work ● ACPI hotkeys seem dead ● Need wider hardware testing for more ● confidence HVM was working, but now not (general xen- ● unstable issue?) Xen Summit at Oracle Feb 24-25, 2009
  • 20. TODO Host S3 suspend resume ● Should just be a matter of bringing over patches – Upstreaming could be awkward – MSI ● Hoping its no more complex than APIC – Pciback ● Blktap2 ● Pvhvm support – started, need more work ● Wider hardware testing ● Xen Summit at Oracle Feb 24-25, 2009
  • 21. No Excuses Now is the time to base development on the ● mainline kernel Core Xen support is stable ● Dom0 support is fairly stable ● Remaining work is in self-contained chunks ● Xen Summit at Oracle Feb 24-25, 2009
  • 22. The Kittens are Thinking of You... Xen Summit at Oracle Feb 24-25, 2009
  • 23. Thanks! GIT repository: git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git or... 1. Go to git.kernel.org 2. Look for “xen.git” on the page 3. Cut long url 4. git clone <paste> linux-xen Xen Summit at Oracle Feb 24-25, 2009
  • 24. Xen in Mainline Linux Status Update Jeremy Fitzhardinge Citrix Xen Summit at Oracle Feb 24-25, 2009
  • 25. Upstreaming Xen Xen domU support ● Upstream and stable for well over a year – Shipped as standard in several distros – Fedora ● Debian ● ...? ● (SuSE is the odd one out.) – Dom0 support is the next frontier ● Xen Summit at Oracle Feb 24-25, 2009
  • 26. git migration Migrated upstream Xen work to git ● Mercurial + patchqueue was getting awkward ● Git makes is more conventional: ● Easier for upstream developers to work with – Easier for Xen developers to get a working tree – Hosted on git.kernel.org: “jeremy/xen.git” ● Xen Summit at Oracle Feb 24-25, 2009
  • 27. Git Branches Two main merge branches ● xen/master – core Xen, domU – xen/dom0/hackery – dom0 work branch – Lots of topic branches ● Merged into merge branches for use – Will add more as needed – Room for lots more ● Will take anything – Even merged if you don't break things! – Xen Summit at Oracle Feb 24-25, 2009
  • 28. General improvements Preemptable lazy mmu updates ● Allow more use of lazy updates – Less scheduling effect – /proc/xen + usermode xenbus for tools support ● New calling convention for reduced register ● pressure Still need a concerted performance-oriented ● push Xen Summit at Oracle Feb 24-25, 2009
  • 29. Paravirt performance Early design goal of paravirt_ops was zero- ● overhead native execution Some benchmarking showed we were falling ● short ~5% impact on mmap heavy benchmark – The problem: inlines -> function calls cause lots ● more register pressure Resulting in higher cache traffic due to spills ● Xen Summit at Oracle Feb 24-25, 2009
  • 30. New calling convention Solution: define new calling convention ● Normal convention trashes many registers ● Define new one which preserves everything ● Except return value – Result: ● compiler can generate better code – Inline patching more effective – But how to call normal code? ● Xen Summit at Oracle Feb 24-25, 2009
  • 31. Generate reg-saver thunks Add macro to generate register saving wrapper ● around conventional code Pushes cost of call to complex code ● Simple callees – written in asm – need no ● wrapper Overall: reduce overhead to 1-2% ● Xen Summit at Oracle Feb 24-25, 2009
  • 32. Dom0 Boot-time control domain ● Also basis for driver domains ● At heart, a normal PV Xen domain ● But with extra bits ● APIC – ACPI – Device mappings – DMA / SWIOTLB – Xen Summit at Oracle Feb 24-25, 2009
  • 33. Dom 0: APICs APIC = interrupt controller ● Two parts: ● IO APICs connected to PCI slots – Local ACPI in each CPU – Xen owns Local APIC, since dom0 has ● virtualized CPUs Dom0 owns IO APICs because it has the ● device drivers Xen Summit at Oracle Feb 24-25, 2009
  • 34. Dom 0: APICs (2) Changes surprisingly small ● Some changes to APIC discovery to avoid local – apics Hook acpi_register_gsi to direct all setup to Xen – code Only support ACPI interrupt routing – IRQ space reserved for 1:1 GSI mapping – Caveat: APIC and ACPI use different senses ● for interrupt triggering Getting it wrong works surprisingly well – Xen Summit at Oracle Feb 24-25, 2009
  • 35. Dom 0: ACPI ACPI = Amazingly Complex Piece of Interface ● Used for everything from device discovery, ● interrupt routing to power management To start with, mostly interested in devices and ● interrupts Main changes to APIC code were to make it ● map properly Seems well-behaved after that ● Xen Summit at Oracle Feb 24-25, 2009
  • 36. Dom 0: device mappings Dom0 kernel has two address spaces: ● Pseudo-physical domain memory – Machine-physical memory – Must know which is which for any given mapping ● Add _PAGE_IOMAP flag to mark hardware ptes ● No pfn->mfn conversion – VM_IO segments mapped with _PAGE_IOMAP ● E820 map avoids RAM holes ● Xen Summit at Oracle Feb 24-25, 2009
  • 37. Dom 0: DMA / SWIOTLB Make sure devices are talking to the memory ● they think they are Hook DMA operations to ● Convert pfn<->mfn – Make memory machine contiguous – SWIOTLB deals with a lot of the tricky cases ● Involves copying, so should be avoided for high – performance devices Xen Summit at Oracle Feb 24-25, 2009
  • 38. *** SUBJECT HERE *** *** BLURB HERE *** Xen Summit at Oracle Feb 24-25, 2009
  • 39. Dom 0: upstream progress Most core patches posted, out for review ● Looks good for next merge window (2.6.30) ● Core support at least – Not necessarily every feature – ● Xen Summit at Oracle Feb 24-25, 2009
  • 40. DEMO Xen Summit at Oracle Feb 24-25, 2009
  • 41. State: Good Generally very stable ● AHCI, PIIX, mpt drive controllers work ● E1000, iwlagn networking OK ● Intel graphics fully accelerated ● Radeon X server starts OK too – Sound fine ● USB good too ● Oh, and you can start domains ● Xen Summit at Oracle Feb 24-25, 2009
  • 42. State: Meh S3 suspend not done ● Cpufreq doesn't work ● ACPI hotkeys seem dead ● Need wider hardware testing for more ● confidence HVM was working, but now not (general xen- ● unstable issue?) Xen Summit at Oracle Feb 24-25, 2009
  • 43. TODO Host S3 suspend resume ● Should just be a matter of bringing over patches – Upstreaming could be awkward – MSI ● Hoping its no more complex than APIC – Pciback ● Blktap2 ● Pvhvm support – started, need more work ● Wider hardware testing ● Xen Summit at Oracle Feb 24-25, 2009
  • 44. No Excuses Now is the time to base development on the ● mainline kernel Core Xen support is stable ● Dom0 support is fairly stable ● Remaining work is in self-contained chunks ● Xen Summit at Oracle Feb 24-25, 2009
  • 45. The Kittens are Thinking of You... Xen Summit at Oracle Feb 24-25, 2009
  • 46. Thanks! GIT repository: git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git or... 1. Go to git.kernel.org 2. Look for “xen.git” on the page 3. Cut long url 4. git clone <paste> linux-xen Xen Summit at Oracle Feb 24-25, 2009