XS Oracle 2009 PVOps

816 views

Published on

Jeremy Fitzhardinge: PVOPS Status

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
816
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

XS Oracle 2009 PVOps

  1. 1. Xen in Mainline Linux Status Update Jeremy Fitzhardinge Citrix Xen Summit at Oracle Feb 24-25, 2009
  2. 2. Upstreaming Xen Xen domU support ● Upstream and stable for well over a year – Shipped as standard in several distros – Fedora ● Debian ● ...? ● (SuSE is the odd one out.) – Dom0 support is the next frontier ● Xen Summit at Oracle Feb 24-25, 2009
  3. 3. git migration Migrated upstream Xen work to git ● Mercurial + patchqueue was getting awkward ● Git makes is more conventional: ● Easier for upstream developers to work with – Easier for Xen developers to get a working tree – Hosted on git.kernel.org: “jeremy/xen.git” ● Xen Summit at Oracle Feb 24-25, 2009
  4. 4. Git Branches Two main merge branches ● xen/master – core Xen, domU – xen/dom0/hackery – dom0 work branch – Lots of topic branches ● Merged into merge branches for use – Will add more as needed – Room for lots more ● Will take anything – Even merged if you don't break things! – Xen Summit at Oracle Feb 24-25, 2009
  5. 5. General improvements Preemptable lazy mmu updates ● Allow more use of lazy updates – Less scheduling effect – /proc/xen + usermode xenbus for tools support ● New calling convention for reduced register ● pressure Still need a concerted performance-oriented ● push Xen Summit at Oracle Feb 24-25, 2009
  6. 6. Paravirt performance Early design goal of paravirt_ops was zero- ● overhead native execution Some benchmarking showed we were falling ● short ~5% impact on mmap heavy benchmark – The problem: inlines -> function calls cause lots ● more register pressure Resulting in higher cache traffic due to spills ● Xen Summit at Oracle Feb 24-25, 2009
  7. 7. New calling convention Solution: define new calling convention ● Normal convention trashes many registers ● Define new one which preserves everything ● Except return value – Result: ● compiler can generate better code – Inline patching more effective – But how to call normal code? ● Xen Summit at Oracle Feb 24-25, 2009
  8. 8. Generate reg-saver thunks Add macro to generate register saving wrapper ● around conventional code Pushes cost of call to complex code ● Simple callees – written in asm – need no ● wrapper Overall: reduce overhead to 1-2% ● Xen Summit at Oracle Feb 24-25, 2009
  9. 9. Dom0 Boot-time control domain ● Also basis for driver domains ● At heart, a normal PV Xen domain ● But with extra bits ● APIC – ACPI – Device mappings – DMA / SWIOTLB – Xen Summit at Oracle Feb 24-25, 2009
  10. 10. Dom 0: APICs APIC = interrupt controller ● Two parts: ● IO APICs connected to PCI slots – Local ACPI in each CPU – Xen owns Local APIC, since dom0 has ● virtualized CPUs Dom0 owns IO APICs because it has the ● device drivers Xen Summit at Oracle Feb 24-25, 2009
  11. 11. Dom 0: APICs (2) Changes surprisingly small ● Some changes to APIC discovery to avoid local – apics Hook acpi_register_gsi to direct all setup to Xen – code Only support ACPI interrupt routing – IRQ space reserved for 1:1 GSI mapping – Caveat: APIC and ACPI use different senses ● for interrupt triggering Getting it wrong works surprisingly well – Xen Summit at Oracle Feb 24-25, 2009
  12. 12. Dom 0: ACPI ACPI = Amazingly Complex Piece of Interface ● Used for everything from device discovery, ● interrupt routing to power management To start with, mostly interested in devices and ● interrupts Main changes to APIC code were to make it ● map properly Seems well-behaved after that ● Xen Summit at Oracle Feb 24-25, 2009
  13. 13. Dom 0: device mappings Dom0 kernel has two address spaces: ● Pseudo-physical domain memory – Machine-physical memory – Must know which is which for any given mapping ● Add _PAGE_IOMAP flag to mark hardware ptes ● No pfn->mfn conversion – VM_IO segments mapped with _PAGE_IOMAP ● E820 map avoids RAM holes ● Xen Summit at Oracle Feb 24-25, 2009
  14. 14. Dom 0: DMA / SWIOTLB Make sure devices are talking to the memory ● they think they are Hook DMA operations to ● Convert pfn<->mfn – Make memory machine contiguous – SWIOTLB deals with a lot of the tricky cases ● Involves copying, so should be avoided for high – performance devices Xen Summit at Oracle Feb 24-25, 2009
  15. 15. *** SUBJECT HERE *** *** BLURB HERE *** Xen Summit at Oracle Feb 24-25, 2009
  16. 16. Dom 0: upstream progress Most core patches posted, out for review ● Looks good for next merge window (2.6.30) ● Core support at least – Not necessarily every feature – ● Xen Summit at Oracle Feb 24-25, 2009
  17. 17. DEMO Xen Summit at Oracle Feb 24-25, 2009
  18. 18. State: Good Generally very stable ● AHCI, PIIX, mpt drive controllers work ● E1000, iwlagn networking OK ● Intel graphics fully accelerated ● Radeon X server starts OK too – Sound fine ● USB good too ● Oh, and you can start domains ● Xen Summit at Oracle Feb 24-25, 2009
  19. 19. State: Meh S3 suspend not done ● Cpufreq doesn't work ● ACPI hotkeys seem dead ● Need wider hardware testing for more ● confidence HVM was working, but now not (general xen- ● unstable issue?) Xen Summit at Oracle Feb 24-25, 2009
  20. 20. TODO Host S3 suspend resume ● Should just be a matter of bringing over patches – Upstreaming could be awkward – MSI ● Hoping its no more complex than APIC – Pciback ● Blktap2 ● Pvhvm support – started, need more work ● Wider hardware testing ● Xen Summit at Oracle Feb 24-25, 2009
  21. 21. No Excuses Now is the time to base development on the ● mainline kernel Core Xen support is stable ● Dom0 support is fairly stable ● Remaining work is in self-contained chunks ● Xen Summit at Oracle Feb 24-25, 2009
  22. 22. The Kittens are Thinking of You... Xen Summit at Oracle Feb 24-25, 2009
  23. 23. Thanks! GIT repository: git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git or... 1. Go to git.kernel.org 2. Look for “xen.git” on the page 3. Cut long url 4. git clone <paste> linux-xen Xen Summit at Oracle Feb 24-25, 2009
  24. 24. Xen in Mainline Linux Status Update Jeremy Fitzhardinge Citrix Xen Summit at Oracle Feb 24-25, 2009
  25. 25. Upstreaming Xen Xen domU support ● Upstream and stable for well over a year – Shipped as standard in several distros – Fedora ● Debian ● ...? ● (SuSE is the odd one out.) – Dom0 support is the next frontier ● Xen Summit at Oracle Feb 24-25, 2009
  26. 26. git migration Migrated upstream Xen work to git ● Mercurial + patchqueue was getting awkward ● Git makes is more conventional: ● Easier for upstream developers to work with – Easier for Xen developers to get a working tree – Hosted on git.kernel.org: “jeremy/xen.git” ● Xen Summit at Oracle Feb 24-25, 2009
  27. 27. Git Branches Two main merge branches ● xen/master – core Xen, domU – xen/dom0/hackery – dom0 work branch – Lots of topic branches ● Merged into merge branches for use – Will add more as needed – Room for lots more ● Will take anything – Even merged if you don't break things! – Xen Summit at Oracle Feb 24-25, 2009
  28. 28. General improvements Preemptable lazy mmu updates ● Allow more use of lazy updates – Less scheduling effect – /proc/xen + usermode xenbus for tools support ● New calling convention for reduced register ● pressure Still need a concerted performance-oriented ● push Xen Summit at Oracle Feb 24-25, 2009
  29. 29. Paravirt performance Early design goal of paravirt_ops was zero- ● overhead native execution Some benchmarking showed we were falling ● short ~5% impact on mmap heavy benchmark – The problem: inlines -> function calls cause lots ● more register pressure Resulting in higher cache traffic due to spills ● Xen Summit at Oracle Feb 24-25, 2009
  30. 30. New calling convention Solution: define new calling convention ● Normal convention trashes many registers ● Define new one which preserves everything ● Except return value – Result: ● compiler can generate better code – Inline patching more effective – But how to call normal code? ● Xen Summit at Oracle Feb 24-25, 2009
  31. 31. Generate reg-saver thunks Add macro to generate register saving wrapper ● around conventional code Pushes cost of call to complex code ● Simple callees – written in asm – need no ● wrapper Overall: reduce overhead to 1-2% ● Xen Summit at Oracle Feb 24-25, 2009
  32. 32. Dom0 Boot-time control domain ● Also basis for driver domains ● At heart, a normal PV Xen domain ● But with extra bits ● APIC – ACPI – Device mappings – DMA / SWIOTLB – Xen Summit at Oracle Feb 24-25, 2009
  33. 33. Dom 0: APICs APIC = interrupt controller ● Two parts: ● IO APICs connected to PCI slots – Local ACPI in each CPU – Xen owns Local APIC, since dom0 has ● virtualized CPUs Dom0 owns IO APICs because it has the ● device drivers Xen Summit at Oracle Feb 24-25, 2009
  34. 34. Dom 0: APICs (2) Changes surprisingly small ● Some changes to APIC discovery to avoid local – apics Hook acpi_register_gsi to direct all setup to Xen – code Only support ACPI interrupt routing – IRQ space reserved for 1:1 GSI mapping – Caveat: APIC and ACPI use different senses ● for interrupt triggering Getting it wrong works surprisingly well – Xen Summit at Oracle Feb 24-25, 2009
  35. 35. Dom 0: ACPI ACPI = Amazingly Complex Piece of Interface ● Used for everything from device discovery, ● interrupt routing to power management To start with, mostly interested in devices and ● interrupts Main changes to APIC code were to make it ● map properly Seems well-behaved after that ● Xen Summit at Oracle Feb 24-25, 2009
  36. 36. Dom 0: device mappings Dom0 kernel has two address spaces: ● Pseudo-physical domain memory – Machine-physical memory – Must know which is which for any given mapping ● Add _PAGE_IOMAP flag to mark hardware ptes ● No pfn->mfn conversion – VM_IO segments mapped with _PAGE_IOMAP ● E820 map avoids RAM holes ● Xen Summit at Oracle Feb 24-25, 2009
  37. 37. Dom 0: DMA / SWIOTLB Make sure devices are talking to the memory ● they think they are Hook DMA operations to ● Convert pfn<->mfn – Make memory machine contiguous – SWIOTLB deals with a lot of the tricky cases ● Involves copying, so should be avoided for high – performance devices Xen Summit at Oracle Feb 24-25, 2009
  38. 38. *** SUBJECT HERE *** *** BLURB HERE *** Xen Summit at Oracle Feb 24-25, 2009
  39. 39. Dom 0: upstream progress Most core patches posted, out for review ● Looks good for next merge window (2.6.30) ● Core support at least – Not necessarily every feature – ● Xen Summit at Oracle Feb 24-25, 2009
  40. 40. DEMO Xen Summit at Oracle Feb 24-25, 2009
  41. 41. State: Good Generally very stable ● AHCI, PIIX, mpt drive controllers work ● E1000, iwlagn networking OK ● Intel graphics fully accelerated ● Radeon X server starts OK too – Sound fine ● USB good too ● Oh, and you can start domains ● Xen Summit at Oracle Feb 24-25, 2009
  42. 42. State: Meh S3 suspend not done ● Cpufreq doesn't work ● ACPI hotkeys seem dead ● Need wider hardware testing for more ● confidence HVM was working, but now not (general xen- ● unstable issue?) Xen Summit at Oracle Feb 24-25, 2009
  43. 43. TODO Host S3 suspend resume ● Should just be a matter of bringing over patches – Upstreaming could be awkward – MSI ● Hoping its no more complex than APIC – Pciback ● Blktap2 ● Pvhvm support – started, need more work ● Wider hardware testing ● Xen Summit at Oracle Feb 24-25, 2009
  44. 44. No Excuses Now is the time to base development on the ● mainline kernel Core Xen support is stable ● Dom0 support is fairly stable ● Remaining work is in self-contained chunks ● Xen Summit at Oracle Feb 24-25, 2009
  45. 45. The Kittens are Thinking of You... Xen Summit at Oracle Feb 24-25, 2009
  46. 46. Thanks! GIT repository: git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git or... 1. Go to git.kernel.org 2. Look for “xen.git” on the page 3. Cut long url 4. git clone <paste> linux-xen Xen Summit at Oracle Feb 24-25, 2009

×