XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citrix Systems
•Download as PPTX, PDF•
2 likes•44,074 views
This talk will give a brief background to the Xen Project Windows PV driver architecture for those who are not already familiar. It will then go on to update the community on recent changes to the drivers, and planned future changed. It will also cover the new HID and console drivers that have been introduced to the supported set, including demonstrations of those drivers.
More Related Content
What's hot
What's hot (20)
Similar to XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citrix Systems
Similar to XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citrix Systems (20)
More from The Linux Foundation
More from The Linux Foundation (20)
Recently uploaded
Recently uploaded (20)
XPDDS18: Windows PV Drivers Project: Status and Updates - Paul Durrant, Citrix Systems
- 1. Windows PV Drivers: Status and Updates Paul Durrant, Senior Principal Software Engineer, Citrix Systems R&D 1
- 3. ● Closed source drivers specifically for XenServer ○ Assumptions and custom hacks 3 XenServer Legacy PV Drivers
- 4. ● Closed source drivers specifically for XenServer ○ Assumptions and custom hacks ● Lots of interdependencies ○ Direct function calls between modules 4 XenServer Legacy PV Drivers
- 5. ● Closed source drivers specifically for XenServer ○ Assumptions and custom hacks ● Lots of interdependencies ○ Direct function calls between modules ● Installed together via an NSIS[1] installer 5 XenServer Legacy PV Drivers [1] See https://sourceforge.net/projects/nsis
- 6. ● Closed source drivers specifically for XenServer ○ Assumptions and custom hacks ● Lots of interdependencies ○ Direct function calls between modules ● Installed together via an NSIS[1] installer ● Used prior to XenServer 7.0 6 XenServer Legacy PV Drivers [1] See https://sourceforge.net/projects/nsis
- 7. 7 New XenServer PV Drivers
- 8. ● Most drivers completely re-written ○ No assumptions or custom hacks 8 New XenServer PV Drivers
- 9. ● Most drivers completely re-written ○ No assumptions or custom hacks ● Interdependencies removed or controlled ○ No direct function calls ○ Use interfaces 9 New XenServer PV Drivers
- 10. ● Most drivers completely re-written ○ No assumptions or custom hacks ● Interdependencies removed or controlled ○ No direct function calls ○ Use interfaces ● Can be separately installed ○ Available on Windows Update 10 New XenServer PV Drivers
- 11. ● Most drivers completely re-written ○ No assumptions or custom hacks ● Interdependencies removed or controlled ○ No direct function calls ○ Use interfaces ● Can be separately installed ○ Available on Windows Update ● Open source ○ Adopted by Xen Project 11 New XenServer PV Drivers
- 12. URLs ● Project Front Page: https://www.xenproject.org/developers/teams/windows-pv-drivers.html ● Driver Repositories: http://xenbits.xen.org/gitweb/?a=project_list;pf=pvdrivers ● Driver Binaries: http://xenbits.xen.org/pvdrivers/win ● Mailing List: win-pv-devel@lists.xenproject.org
- 14. Background: Device Objects PDO Physical Device Object
- 15. Background: Device Objects PDO Physical Device Object FDO Function Device Object
- 16. Background: Device Objects PDO Physical Device Object FDO Function Device Object FiltDO Filter Device Object
- 17. Background: Device Driver Hierarchy Bus Driver
- 18. Background: Device Driver Hierarchy Function Driver
- 19. Background: Device Driver Hierarchy Function Driver Function driver FDO binds to bus driver PDO
- 20. Background: Device Driver Hierarchy BAR FOO DRV.SYS DRV.INF [Inst.NT$ARCH$] Name = Tag, FOOBAR [Tag] ; Installation details Function driver binds to PDO by name Binding is controlled by driver INF file
- 21. Background: Device Driver Hierarchy Filter Driver
- 22. Background: Device Driver Hierarchy Filter Driver Interposes on IRP communication
- 23. Background: Device Driver Hierarchy Filter Driver Insertion controlled by registry
- 24. Background: Device Driver Hierarchy Function drivers can also be bus drivers
- 26. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS
- 27. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS Main PV bus infrastructure
- 28. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS PV network subsystem
- 29. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS PV storage subsystem
- 30. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS Xen Platform or Xen PV PCI device
- 31. Core PV Driver Hierarchy XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS PCI Class filter
- 32. Core PV Driver Hierarchy Core Xen interface XEN.SYS XENFILT.SYS XENBUS.SYS XENVIF.SYS XENVBD.SYS XENNET.SYS
- 33. Core PV Driver Packaging XENBUS.INF XENVBD.INFXENVIF.INF XENNET.INF
- 34. Core PV Driver Packaging XENBUS.INF XENVBD.INFXENVIF.INF XENNET.INF No direct function calls
- 36. Interface Negotiation IRP_MN_QUERY_INTERFACE Subscriber specifies interface GUID and version Provider Subscriber
- 37. Interface Negotiation IRP_MN_QUERY_INTERFACE Provider hands back a table containing methods and contextContext Method 1 Method 2
- 38. Interface Negotiation Acquire and Release are present in all interfaces for reference counting purposes Context Acquire Release
- 39. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 39 Driver Interface Examples ● XENVIF ○ VIF
- 40. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 40 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed ● XENVIF ○ VIF
- 41. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 41 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed Drivers can register functions to be called on resume from suspend ● XENVIF ○ VIF
- 42. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 42 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed Drivers can register functions to be called on resume from suspend These are self explanatory ● XENVIF ○ VIF
- 43. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 43 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed Drivers can register functions to be called on resume from suspend These are self explanatory ● XENVIF ○ VIF Useful for managing resource allocation
- 44. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 44 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed Drivers can register functions to be called on resume from suspend These are self explanatory ● XENVIF ○ VIF Useful for managing resource allocation Pseudo slab allocator
- 45. ● XENBUS ○ DEBUG ○ SUSPEND ○ EVTCHN ○ GNTTAB ○ STORE ○ RANGE_SET ○ CACHE 45 Driver Interface Examples Drivers can register a function to be called when Xen debug-key ‘q’ is pressed Drivers can register functions to be called on resume from suspend These are self explanatory ● XENVIF ○ VIF Useful for managing resource allocation Pseudo slab allocator Implementation of Xen netif protocol
- 46. Interface Compatibility IRP_MN_QUERY_INTERFACE GUID = DEBUG Version = X
- 47. Interface Compatibility IRP_MN_QUERY_INTERFACE GUID = DEBUG Version = X DEBUG code only implements version Y
- 48. Interface Compatibility IRP_MN_QUERY_INTERFACE GUID = DEBUG Version = X STATUS_NOT_SUPPORTED DEBUG code only implements version Y
- 49. Interface Compatibility IRP_MN_QUERY_INTERFACE GUID = DEBUG Version = X STATUS_NOT_SUPPORTED Need to stop this from happening DEBUG code only implements version Y
- 50. Interface Compatibility BAR ● Provider implementations should be compatible with older versions of interfaces FOO
- 51. Interface Compatibility BAR ● Provider implementations should be compatible with older versions of interfaces ● For each new version, modify the PDO binding name FOO Version X FOOBAR_1 Version Y FOOBAR_2
- 52. Interface Compatibility BAR ● Provider implementations should be compatible with older versions of interfaces ● For each new version, modify the PDO binding name FOO Version X FOOBAR_1 Version Y FOOBAR_2 [Inst.NT$ARCH$] Name = Tag, FOOBAR_1 ● Subscriber chooses the corresponding binding name
- 53. Interface Compatibility Upgrade Provider Upgrade Subscriber
- 54. Interface Compatibility Upgrade Provider Providers backwards compatible with older subscribers Upgrade Subscriber
- 55. Interface Compatibility Upgrade Provider Providers backwards compatible with older subscribers Upgrade Subscriber Won’t displace older subscriber until new provider binding is available
- 57. 57 We graduated!
- 58. 58 New stable release tagged, built, and signed
- 59. New HID drivers
- 60. Previously, when you commented these lines out of your xl.cfg...
- 61. This is what you get ...
- 62. Problem solved!
- 64. Means we can now do this...
- 65. Q&A 65
Editor's Notes
- Hi. My name is Paul Durrant. I’m a Senior Principal Software Engineer in the XenServer group at Citrix Systems R&D, based in Cambridge UK. I’ve been a contributor to the Xen Project for many years now. I’m community lead for the Windows PV Drivers but I also contribute to Linux, QEMU and the hypervisor itself. Today I’m going to talk about the PV drivers, their history, their architecture and the current status of the project.
- So, first some history… The Xen Project PV drivers ancestry lies in the PV drivers that were written to support the Citrix XenServer product (lately rebranded Citrix Hypervisor)
- These drivers were closed source Citrix proprietary drivers to support only XenServer. They contained assumptions about their environment that were only true for VMs running on XenServer, and they also relied on custom hacks in the hypervisor and QEMU.
- There were lots of non-obvious interdependencies between drivers. Some drivers event made direct function calls to code supplied by other drivers, which was the cause of many blue screens during driver updates (where it was easy for binary incompatibilities to creep in).
- So, to mitigate against incompatibilities all the drivers were remove or installed together as a single unit via an NSIS installer. So this meant, for instance, that a relatively trivial fix to a nework driver would require re-installation of the storage driver… And because the storage driver is responsible for the system disk then a reboot would be needed… This was not popular.
- These drivers were used for all versons of XenServer prior to 7.0, at which point...
- We switched over to a new set of drivers…
- We got rid of all the assumptions and dependencies on custom hacks, which improved compatibility across different versions of XenServer and also meant that drivers worked on upstream Xen installations… Which helped development a lot, since we could easily develop and test against e.g. new PV backends in upstream versions of Linux, or new hypervisor features (such as per-vcpu event channel interrupts).
- All the direct inter-module function calls were removed and, where necessary, replaced with something we called interfaces which I’ll talk about later
- And because the direct function calls are gone, the drivers no longer need to be removed or installed as a unit... They can be installed independently, in any order, using standard mechanisms (via device manager if you like) and that means… We could post them on Windows Update, so no need to manually install if you don’t want to.
- And they are all open source. So, because they are no longer tied to XenServer by custom hacks or assumptions, they have now been adopted by the Xen Project as the official Windows PV drivers.
- These are the relevant URLs where you can get information about the project. You can download the code and build it yourself (using VS2015 or VS2017 for the master branches)... Or you can get the binaries that are built by our public build infrastructure (which are currently built with VS2015)... You can also find Linux Foundation signed releases built from the stable branch on the project front page too.
- Now I’ll go on to describe the architecture of the drivers… But before I can do that I need to give those of you who are not experienced Windows driver developers a quick tutorial...
- Windows drivers deal with device objects, but there are several types of these... The first is the Physical Device Object (or PDO) and this essentially represents a piece of hardware i.e. a node in the machine’s bus topology… I’ll colour these pink like so.
- The next type is the Function Device Object (or FDO)… I’ll colour these blue...
- ...And the last type is the Filter Device Object, which are not called FDOs… They are generally shortened to FiltDO and I’ll colour these green. Now let’s look are how FDOs and FiltDOs relate to PDOs...
- Windows objects form a hierarchy and the crucial entity in that hierarchy is a Bus Driver… This is a driver that creates FDOs and PDOs. Let’s ignore the FDO for the moment and consider just the PDOs... As I said, PDOs generally relate to nodes and machines bus topology so a Bus Driver is typically enumerating some layer in that topology and creating a PDO for everything it finds.
- Now there are some PDOs we can look at what an FDO is… FDOs are created by Function Drivers...
- And a function driver creates an FDO in response to being told about a PDO by the Windows PnP subsystem… This is called binding But how does Windows know which function driver to tell about which PDO?
- Well PDOs have names which are determined by the bus driver that created them… And function drivers have a companion file as part of their package, called an INF file… And that INF file contains information about the names of the PDOs to which the function driver can bind… So when the function driver is installed into Windows, the PnP subsystem uses the information in the INF file to figure out which PDOs to tell it about. But what about FiltDOs?
- Well, FiltDOs are created by Filter Drivers and they logically sit between FDOs and PDOs...
- And they get to interpose on the communications between and FDO and the PDO to which it is bound… This communication is in the form of messages, called I/O Request Packets or IRPs and a FiltDO can create new IRPs to send to the PDO, completely block IRPs from the FDO, or make modifications to them as they pass. But how does Windows decide which filter drivers to put into the hierarchy?
- Well, insertion of filters is managed by certain registry keys and they can be specified to interpose on PDOs with certain names, or a certain class.., For example, you can set up registry keys to load a filter on top of every PDO created the main PCI bus driver… and indeed, we’ll see in a bit, this is exactly what one of the PV driver packages does.
- Now, the last thing to note is that the reason that top level bus driver had an FDO is that bus drivers are also function drivers… So the driver hierarchy in Windows grows tree-fashion… The top level bus driver (which doesn’t actually bind to a real PDO itself) creates its own PDOs, then a bus driver is loaded (because of its INF file) and binds an FDO… The it creates its own PDOs, and then another bus driver may loaded… or maybe just a plain function driver this time… And so it goes on.
- Now let’s look at a concrete exmaple… the core PV drivers… By core, I mean the drivers involved in providing the PV network and storage data paths...
- This is their hierarchy…
- We have the main bus infrastructure here, which enumerates its PDOs from the various PV device classes it finds in xenstore...
- Here we have the PV network drivers… XENVIF creates a PDO for every vif node it finds in xenstore… And then XENNET is an NDIS (network) miniport driver that binds to those PDOs.
- And over here we have the PV storage driver… XENVBD creates a PDO for every vbd node it finds in xenstore… There’s no function driver in this case, because Windows supplies its own generic function driver called DISK.SYS. All XENVBD has to do is create PDOs with the right names for that driver to bind.
- The hierarchy is all rooted on a PCI device that is synthesized by QEMU. The XENBUS driver there in the middle can bind to a couple of different PCI PDOs.... They both have a Vendor ID registered to the Xen Project, but one has device id 0x0001 (or 0x0002 for XenServer… for historical reasons) and the other has device id 0xC000… The 0xC000 device is the magic one you need if you want to get drivers from Windows Update…
- You’ll also notice that between XENBUS’s FDO and the PCI PDO there is a FiltDO created by XENFILT. XENFILT is actually installed as a class filter, and so it actually creates a FiltDO for Core Xen interface every PCI PDO in the system… And one of the reasons this we do this is so that we can find out about what emulated network and storage devices are present in the system
- Actually there’s one more driver… and it has no objects at all! Windows also has a special type of driver that is neither a bus driver, nor a function driver, and it’s called an export driver. It’s basically just a DLL for the kernel, and indeed it’s init hook is called DllInitialize() rather than DriverEntry(). And the purpose of this driver, as you might guess from the name, is to be the interface to Xen… It owns the hypercall page and contains all the hypercall stubs. It’s also the only driver into which other drivers make direct function calls, XENBUS and XENFILT in this case… but that is ok because...
- As we can see from this diagram, the XEN driver is controlled by the same INF file as XENBUS and XENFILT, and so all three are always installed together as a unit...
- What would not be ok, as I mentioned in the history section, is a direct call like this… Because it’s to easy for binary compatibility issues to creep when doing driver updates… In this example, any change in the XEN driver’s implementation could cause XENVIF to call the wrong thing, or worse, pass the wrong number of arguments and mess up the stack. But we want XENVIF to be able to interact with XEN, for things like event channels and grant tables, so how are we going to allow this?
- Well that’s where interfaces come in… I mentioned them before but now I’ll go into some detail...
- Happily Windows specifies a type of IRP to use in this kind of thing, so we use that… When a subscriber wants to access an interface then it sends the QUERY_INTERFACE IRP to the provider specifying the ID of the interface (Windows uses GUIDs for this kind of thing) and the version of the interface it knows about.
- The provider then hands back a table to the subscriber with a set of methods it can call, and a private context pointer to pass to each of the methods (which it may need to disambiguate between multiple subscribers).
- All of the PV driver interfaces implement these two methods… Acquire() and Release() These are to track actual use of interfaces by subscribers and in most cases the first call to Acquire() will cause the interface implementation’s internal state to be initialized… And similarly the last call to Release() will tear down that state. Note that the jump table is valid regardless of calls Acquire() or Release()... The implementation can’t be unloaded because of the parent/child relationship of the drivers in Windows’ device object hierarchy. The reason for the methods is that the internal state of all interfaces must be torn down before the system state can transition out of S0, so all the drivers have power management code that makes sure they clean up during a system sleep or hibernate.... And the reason for that is that the OS instance may resume in a new domain (certainly this is true of hibernate) and so all the interfaces that talk to the hypervisor must start from scratch.
- So here’s a list of some of the interfaces that the PV drivers provide… This list is not exhaustive but let’s go through it...
- The XENBUS DEBUG interface at the top is pretty simple. It just allows other drivers to register a callback that will be invoked (at interrupt level) when the Xen debug key ‘q’ is pressed… either on the serial console or by issuing an xl debug-keys command.
- Next, there’s the SUSPEND interface. Drivers use this to register callbacks the are invoked early or late in a resume from suspend (which may be a restore or a migrate-in). Early callbacks are invoked with interrupts disabled. Late callbacks are invoked with interrupts (i.e. events) enabled. All callbacks are invoked whilst the system has all CPUs, other than the one handling the resume, spinning in a tight loop to make sure everything is set up before arbitrary system code can run.
- Then there are EVTCHN, GNTTAB and STORE… and you can probably guess what they do.
- Then there’s RANGE_SET which is a set of methods for allocating and freeing ranges of numbers. This can be useful in resource management and indeed the GNTTAB code uses this interface internally for managing grant references.
- Then there’s the CACHE interface which provides a pseudo slab allocator. I say pseudo because it doesn’t actually allocate slabs, it just uses non-paged pool, but it implements the object constructor/destructor primitives and these are heavily used by XENVIF.
- And then lastly we have the one interface that XENVIF provides… The VIF interface is basically a complete implementation of the Xen netif protocol and this underpins the operation of XENNET… That means XENNET is actually a pretty small driver. All it has to do is marshal network packets between the VIF interface and the NDIS stack. The majority of the code in the PV network subsystem is actually in XENVIF.
- Now, the main reason we want to use interfaces is to avoid those direct function calls which make driver updates unsafe… But how do we make driver updates safe when interface versions might need to change… say when we add a feature... Well let’s look at what might go wrong… Say we update the provider, but the subscriber is still an older version… It now looks for DEBUG interface version X
- But we’ve updated the implementation of the DEBUG interface to version Y...
- So, now the subscriber can’t access the functionality it needs...
- We really want to avoid this happening… In certain circumstances, say where the subscriber is XENVBD and it’s providing the OS system disk, an interface negotiation failure like this might lead to a blue screen and a non-bootable VM.
- So there are some rules that providers and subscribers need to follow… The first for providers is that interface implementations should be compatible with older versions… Just because we add a feature to an implementation doesn’t mean we can’t support older subscribers that don’t know about that feature...
- The next is a bit more subtle… Remember we talked about PDO names… Well PDOs can actually have more than one name… So for each combination of interface versions that a provider supports, it should define a new name, and all of these names should be applied to the PDOs it creates… So in this case we have version X of an interface referencing a PDO name of BAR_1 and version Y of the interface referencing a PDO name of BAR_2. Remember that the PDO simultaneously has both of those names…
- Those are the main provider rules… Now the subscriber needs to follow a rule too, which is that it must bind to the PDO name that corresponds to the versions of the interfaces it’s going to use. So, in this example, because the subscriber wants to use DEBUG version X, it needs to bind to PDO name BAR_1.
- So bearing those rules in mind let’s think about the safety of upgrading providers and subscribers… Can we safely upgrade a provider?
- Well, yes, because we maintain backwards compatibility and… Because we maintain backwards compatibility the old PDO name is still valid, so the old subscribers can still bind to it. Now, can we upgrade a subscriber?
- Yes, because we will try to bind to the provider’s new PDO name so… If the old provider is still in place, Windows will not load the new version of the subscriber (because it sees it as a driver with nothing to bind to)... But as soon as the new version of the provider is installed then Windows will favour the new version of the subscriber because it will have a newer build date and a more recent version number. Remember than driver removal is not something that Windows really supports… old versions are not removed, they are just displaced from use. They remain in driverstore forever.
- So, that completes my explanation of the driver architecture.... Now a quick summary of the project status...
- Well firstly… We graduated… That means we are out of incubation and we are a fully fledged sub-project of the Xen Project.
- Next, a few weeks ago I did a new tag of all the driver stable branches and built and signed version 8.2.1 of the drivers. You can get the binaries from this link on the project front page. These drivers have various bug fixes in and I would recommend you upgrade if you’re still on the 8.2.0 release.
- Also, we have some new drivers! There are a couple of drivers now that use the PV vkbd protocol for paravirtual keyboard and tablet device. XENVKBD is the class bus driver (analogous to XENVIF or XENVBD) and XENHID is the function driver that handles both keyboard and tablet HID protocols. Let’s have a look at why these are a nice addition to the PV driver set...
- Well, before they existed, and you commented these lines out of your xl.cfg to turn off USB emulation (which is actually quite a dom0 cpu hog)...
- This happened… As you can see, the mouse pointer in the VM’s console is not in-sync with the VNC client’s mouse pointer… and that’s because Windows has fallen back to using QEMU’s implementation of a PS/2 mouse… which is, of course, a relative (rather than absolute) pointing device.
- Well, install XENVKBD and XENHID and that problem goes away… So no need for emulated USB any more.
- But that’s not all… There’s also a new console driver called XENCONS… Let’s have a look what that lets us do...
- You can see that, from a dom0 shell we can use xl console to get access to a command shell in the VM… And then we can use a sysinternals tool called psexec to actually kick off a process in the interactive console. And this is not a security hole… I logged in as the same used in both places!
- I’ll end there. Thanks for listening. If anyone has any questions then please fire away...