QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix

Uploaded on

Windows Hardware Certification Kit (HCK) is a set of tools, processes, and tests for certifying HW devices, device drivers and systems. Being a great test environment for QEMU devices, Windows Guest …

Windows Hardware Certification Kit (HCK) is a set of tools, processes, and tests for certifying HW devices, device drivers and systems. Being a great test environment for QEMU devices, Windows Guest device drivers and related Host subsystems, it's still frightening due to deployment complexity. We'll share a way to deploy HCK setup(s) on top of QEMU VMs in just a few minutes.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1.  QEMU development and testing automation using MS HCK Anton Nayshtut, anton@daynix.com Yan Vugenfirer, yan@daynix.com Daynix Computing LTD
  • 2. Agenda What is WHQL/HCK? Introducing VirtHCK Using VirtHCK Call to action Daynix Computing LTD 2
  • 3. What is HCK/WHQL? The Windows certification program (previously known as the Windows Logo Program) for devices and system A comprehensive devicesystem test toolkit by Microsoft Devices which pass HCK are certified and their binaries are digitally signed Daynix Computing LTD 3
  • 4. HCK test kit Introduced with Windows 8 (previous kit know as WLK) Predefined test suites for each device type Ability to add your own tests for additional testing outside of the certification process Reproducible scenarios Daynix Computing LTD 4
  • 5. Benefits for QEMU and Linux community QEMU device testing (not only virtio) Host subsystems testing, for example networking Extensive coverage for defined device categories Regression testing Stress testing Daynix Computing LTD 5
  • 6. HCK PCI compliance test Device under test compliance to PCI spec Applicable for: New QEMU chipsets PCIe devices Next generation of virtio devices Existing PCI devices Daynix Computing LTD 6
  • 7. HCK NDIS (network) tester (1) Comprehensive networking test suite Send/receive, Stress tests (MPE) Offload, VLAN, priority, packet filtering etc Not once used to test the robustness of the host networking stack Daynix Computing LTD 7
  • 8. HCK NDIS (network) tester (2) Applicable for QEMU network devices New devices New features Host network components Bridge Open vSwitch TAP Kernel Daynix Computing LTD 8
  • 9. HCK Windows OS tests Device path exerciser Identifies drivers that do not correctly handle the different IO calls Driver verifier Captures deadlocks, memory leaks, memory corruption Verified correctness of kernel API usage Daynix Computing LTD 9
  • 10. HCK power management tests The test cycling through different sleep states There are variations of IO and PNP (disable/enable) during, before and after power transitions Applicable for: QEMU Devices power management support SeaBIOS Daynix Computing LTD 10
  • 11. Our past experience Transfer hangs with vhost. Offload/VLAN related corner cases with network devices (virtio-net, vmxnet3, vmxnet). vmxnet3 and pvscsi in QEMU Open vSwitch out of order issues RSC (Windows GRO) PCI compliance Daynix Computing LTD 11
  • 12. HCK - great, but frightening Cumbersome setup requirements Complex network configuration Hard to replicate setups Setup conflicts No power management tests automation support Daynix Computing LTD 12
  • 13. VirtHCK Daynix Computing LTD 13
  • 14. Introducing VirtHCK Framework for automating HCK test execution Based on easily configurable host and guest scripts Creates isolated virtual network environments Allows running different HCK scenarios in automated manner OSS, distributed under BSD license Daynix Computing LTD 14
  • 15. Repository and WiKi Get VirtHCK from https://github.com/daynix/ bricklets.git VirtHCK wiki with steps for deployment and usage: https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation Daynix Computing LTD 15
  • 16. HCK issues solved by VirtHCK Setup replication Multiple setups sharing the same host Setup isolation solves the following issues: HCK controller controls all HCK clients in the network Inability to use VM templates without manual VM reconfiguration Automatic network topology setup during network device tests Daynix Computing LTD 16
  • 17. Setup types supported Network (virtio-net, vmxnet3, vmxnet) Storage (virtio-block, virtio-scsi, pvscsi) Simple PCI (virtio-serial, virtio-balloon) Easily extendable by small changes in configuration files Daynix Computing LTD 17
  • 18. Typical HCK deployment Test Client Control NIC Test Client Control NIC Test Client Control NIC WHQL control network Control NIC - e1000 HCK Controller Daynix Computing LTD 18
  • 19. VirtHCK setup Disconnected from outside world network during test run Isolated NICs under test network Isolated WHQL control network A different number of test clients can run according to test type Daynix Computing LTD 19
  • 20. VirtHCK setup for storage and other devices Test VM 1 Control NIC - e1000 WHQL control bridge Control NIC - e1000 Studio VM “World” bridge Physical NIC Outside world NIC Daynix Computing LTD 20
  • 21. VirtHCK setup for storage and other devices - running Test VM 1 Control NIC - e1000 WHQL control bridge Control NIC - e1000 “World” bridge Studio VM Outside world NIC Physical NIC X Daynix Computing LTD 21
  • 22. VirtHCK setup for NICs NIC under test Test bridge NIC under test Test VM 2 Test VM 1 Control NIC - e1000 Control NIC - e1000 WHQL control bridge Control NIC - e1000 Studio VM “World” bridge Physical NIC Outside world NIC Daynix Computing LTD 22
  • 23. VirtHCK setup for NICs - running NIC under test NIC under test Test bridge Test VM 2 Test VM 1 Control NIC - e1000 Control NIC - e1000 WHQL control bridge Control NIC - e1000 “World” bridge Studio VM Outside world NIC Physical NIC X Daynix Computing LTD 23
  • 24. VirtHCK - quick start Clone VirtHCK from github Create VM images in /images directory (studio and test client) Update STUDIO_IMAGE/CLIENT1_IMAGE/ CLIENT2_IMAGE variables in hck_setup.cfg Run VirtHCK - “sudo ./VirtHCK/hck.sh” Install Microsoft HCK Studio and Clients software according to Microsoft documentation Windows HCK Getting Started and following Checklist for a new client VM guidelines. Daynix Computing LTD 24
  • 25. Directory tree VirtHCK/ - Directory of all VirtHCK scripts, supposed to run in Linux environment VirtHCK/guest_tools/ - Directory for scripts required for Windows guests Images directory where VM's images should be located Daynix Computing LTD 25
  • 26. VirtHCK walkthrough - host Configuration: network, VMs etc Host scripts Images Daynix Computing LTD 26
  • 27. hck_setup.cfg VirtHCK setup configuration file Environment configuration VM start up parameters Configured accordingly to required HCK setup type Allows multiple setups on one host Daynix Computing LTD 27
  • 28. Running multiple setups Use UNIQ_ID to configure multiple setups UNIQ_ID will be used as a base for creation of: VNC or Spice ports Bridges names Network interface names, MAC addresses and etc. Daynix Computing LTD 28
  • 29. Test device types Network Storage (also boot) Balloon Serial More can be added Daynix Computing LTD 29
  • 30. Remote access VNC or Spice Spice is needed for power management tests Daynix Computing LTD 30
  • 31. Network infrastructure Linux bridge or Open vSwitch can be used Daynix Computing LTD 31
  • 32. Images Configure Studio VM image Test VMs images Daynix Computing LTD 32
  • 33. VM resources Number of vCPUs Memory NICs for control channels Daynix Computing LTD 33
  • 34. Extra parameters QEMU binary VHOST, snapshots, cache mode for fast install ISOs for guest installation BIOS Daynix Computing LTD 34
  • 35. Configuration summary Daynix Computing LTD 35
  • 36. Host scripts - hck.sh Main script to run HCK enviroment Dumps configuration Creates network configuration for HCK setup Runs VMs in loop (crucial during PM tests) Cleans up all the configurations on exit Daynix Computing LTD 36
  • 37. VM execution scripts Executed by hck.sh run_hck_studio.sh - starts up Studio VM run_hck_client.sh -starts up Client VM Configuration according hck_setup.cfg Daynix Computing LTD 37
  • 38. Network scripts Passed as parameter to QEMU to add network interface to appropriate bridge hck_ctrl_bridge_ifup.sh hck_test_bridge_ifup.sh hck_world_bridge_ifup.sh Daynix Computing LTD 38
  • 39. Guests configuration Daynix Computing LTD 39
  • 40. Studio VM Setup HCK according to Microsoft’s documentation Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation#wiki-Checklist_for_a_new_studio_VM: Do not run HCK studio directly Use run_hck_studio.bat or run_hck_studio_8.1.bat located in guest_tools The above scripts will disconnect the HCK Studio VM from outside network Daynix Computing LTD 40
  • 41. Test client VMs Follow the checklist for test client - https://github.com/ daynix/bricklets/wiki/VirtHCK-brickletdocumentation#wiki-Checklist_for_a_new_client_VM: Install HCK client according to Microsoft’s documentation from hck-studioHCKInstallClient setup.exe Daynix Computing LTD 41
  • 42. Daynix Computing LTD 42
  • 43. What’s next HCK automation on the guest SVVP configuration Xen support (other hypervisors?) Adding more device types Using external configuration Using libvirt More guest scripts to automate configuration of test clients Daynix Computing LTD 43
  • 44. HCK automation on the guest Goal: Run full test cycle for specific device type without user interaction. Daynix Computing LTD 44
  • 45. Adding more device types Goal Test all the devices in QEMU and not only virtio (PCI bridges, USB controllers and etc) Daynix Computing LTD 45
  • 46. More plans Using external configuration Multiple setups on one host without replicating VirtHCK and images Use libvirt Arbitrary number of test clients on one setup Xen support Daynix Computing LTD 46
  • 47. Call to action Test QEMU devices with VirtHCK (not only virtio devices) Contribute Daynix Computing LTD 47
  • 48. Q&A 48
  • 49. Links WiKi - https://github.com/daynix/bricklets/wiki/ VirtHCK-bricklet-documentation Repository - https://github.com/daynix/bricklets.git HCK kit download - http://msdn.microsoft.com/en-us/ windows/hardware/hh833788.aspx Windows HCK Getting Started - http:// msdn.microsoft.com/en-us/library/windows/hardware/ jj123537.aspx Daynix - www.daynix.com Daynix Computing LTD 49