The document discusses QEMU development and testing automation with the introduction of a framework called VirthCK, designed to streamline the execution of the HCK (Hardware Certification Kit) tests for device certification on Microsoft Windows. It highlights the benefits for the QEMU and Linux community, including extensive device testing and automation features that address previous challenges in the HCK setup. The document also provides guidance on how to use VirthCK, its setup configurations, and future plans for enhancing device type testing.

1. QEMU development and testing automation
using MS HCK
Anton Nayshtut, anton@daynix.com
Yan Vugenfirer, yan@daynix.com
Daynix Computing LTD

2. Agenda
What is WHQL/HCK?
Introducing VirtHCK
Using VirtHCK
Call to action
Daynix Computing LTD
2

3. What is HCK/WHQL?
The Windows certification program (previously known
as the Windows Logo Program) for devices and system
A comprehensive devicesystem test toolkit by
Microsoft
Devices which pass HCK are certified and their binaries
are digitally signed
Daynix Computing LTD
3

4. HCK test kit
Introduced with Windows 8 (previous kit know as WLK)
Predefined test suites for each device type
Ability to add your own tests for additional testing
outside of the certification process
Reproducible scenarios
Daynix Computing LTD
4

5. Benefits for QEMU and
Linux community
QEMU device testing (not only virtio)
Host subsystems testing, for example networking
Extensive coverage for defined device categories
Regression testing
Stress testing
Daynix Computing LTD
5

6. HCK PCI compliance test
Device under test compliance to PCI spec
Applicable for:
New QEMU chipsets
PCIe devices
Next generation of virtio devices
Existing PCI devices
Daynix Computing LTD
6

7. HCK NDIS (network) tester (1)
Comprehensive networking test suite
Send/receive,
Stress tests (MPE)
Offload, VLAN, priority, packet filtering
etc
Not once used to test the robustness of the host networking
stack
Daynix Computing LTD
7

8. HCK NDIS (network) tester (2)
Applicable for
QEMU network devices
New devices
New features
Host network components
Bridge
Open vSwitch
TAP
Kernel
Daynix Computing LTD
8

9. HCK Windows OS tests
Device path exerciser
Identifies drivers that do not correctly handle the
different IO calls
Driver verifier
Captures deadlocks, memory leaks, memory
corruption
Verified correctness of kernel API usage
Daynix Computing LTD
9

10. HCK power management tests
The test cycling through different sleep states
There are variations of IO and PNP (disable/enable)
during, before and after power transitions
Applicable for:
QEMU Devices power management support
SeaBIOS
Daynix Computing LTD
10

11. Our past experience
Transfer hangs with vhost.
Offload/VLAN related corner cases with network
devices (virtio-net, vmxnet3, vmxnet).
vmxnet3 and pvscsi in QEMU
Open vSwitch out of order issues
RSC (Windows GRO)
PCI compliance
Daynix Computing LTD
11

12. HCK - great, but frightening
Cumbersome setup requirements
Complex network configuration
Hard to replicate setups
Setup conflicts
No power management tests automation support
Daynix Computing LTD
12

13. VirtHCK
Daynix Computing LTD
13

14. Introducing VirtHCK
Framework for automating HCK test execution
Based on easily configurable host and guest scripts
Creates isolated virtual network environments
Allows running different HCK scenarios in automated
manner
OSS, distributed under BSD license
Daynix Computing LTD
14

15. Repository and WiKi
Get VirtHCK from https://github.com/daynix/
bricklets.git
VirtHCK wiki with steps for deployment and usage:
https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation
Daynix Computing LTD
15

16. HCK issues solved by VirtHCK
Setup replication
Multiple setups sharing the same host
Setup isolation solves the following issues:
HCK controller controls all HCK clients in the network
Inability to use VM templates without manual VM
reconfiguration
Automatic network topology setup during network
device tests
Daynix Computing LTD
16

17. Setup types supported
Network (virtio-net, vmxnet3, vmxnet)
Storage (virtio-block, virtio-scsi, pvscsi)
Simple PCI (virtio-serial, virtio-balloon)
Easily extendable by small changes in configuration
files
Daynix Computing LTD
17

18. Typical HCK deployment
Test Client
Control NIC
Test Client
Control NIC
Test Client
Control NIC
WHQL control network
Control NIC - e1000
HCK
Controller
Daynix Computing LTD
18

19. VirtHCK setup
Disconnected from outside world network during test
run
Isolated NICs under test network
Isolated WHQL control network
A different number of test clients can run according to
test type
Daynix Computing LTD
19

20. VirtHCK setup for storage and
other devices
Test VM 1
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
Studio VM
“World” bridge
Physical NIC
Outside world NIC
Daynix Computing LTD
20

21. VirtHCK setup for storage and
other devices - running
Test VM 1
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
“World” bridge
Studio VM
Outside world NIC
Physical NIC
X
Daynix Computing LTD
21

22. VirtHCK setup for NICs
NIC under test
Test bridge
NIC under test
Test VM 2
Test VM 1
Control NIC - e1000
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
Studio VM
“World” bridge
Physical NIC
Outside world NIC
Daynix Computing LTD
22

23. VirtHCK setup for NICs - running
NIC under test
NIC under test
Test bridge
Test VM 2
Test VM 1
Control NIC - e1000
Control NIC - e1000
WHQL control bridge
Control NIC - e1000
“World” bridge
Studio VM
Outside world NIC
Physical NIC
X
Daynix Computing LTD
23

24. VirtHCK - quick start
Clone VirtHCK from github
Create VM images in /images directory (studio and test
client)
Update STUDIO_IMAGE/CLIENT1_IMAGE/
CLIENT2_IMAGE variables in hck_setup.cfg
Run VirtHCK - “sudo ./VirtHCK/hck.sh”
Install Microsoft HCK Studio and Clients software
according to Microsoft documentation Windows HCK
Getting Started and following Checklist for a new client
VM guidelines.
Daynix Computing LTD
24

25. Directory tree
VirtHCK/ - Directory of all VirtHCK scripts, supposed
to run in Linux environment
VirtHCK/guest_tools/ - Directory for scripts required
for Windows guests
Images directory where VM's images should be located
Daynix Computing LTD
25

26. VirtHCK walkthrough - host
Configuration: network, VMs etc
Host scripts
Images
Daynix Computing LTD
26

27. hck_setup.cfg
VirtHCK setup configuration file
Environment configuration
VM start up parameters
Configured accordingly to required HCK setup type
Allows multiple setups on one host
Daynix Computing LTD
27

28. Running multiple setups
Use UNIQ_ID to configure multiple setups
UNIQ_ID will be used as a base for creation of:
VNC or Spice ports
Bridges names
Network interface names, MAC addresses and etc.
Daynix Computing LTD
28

29. Test device types
Network
Storage (also boot)
Balloon
Serial
More can be added
Daynix Computing LTD
29

30. Remote access
VNC or Spice
Spice is needed for power management tests
Daynix Computing LTD
30

31. Network infrastructure
Linux bridge or Open vSwitch can be used
Daynix Computing LTD
31

32. Images
Configure
Studio VM image
Test VMs images
Daynix Computing LTD
32

33. VM resources
Number of vCPUs
Memory
NICs for control channels
Daynix Computing LTD
33

34. Extra parameters
QEMU binary
VHOST, snapshots, cache mode for fast install
ISOs for guest installation
BIOS
Daynix Computing LTD
34

35. Configuration summary
Daynix Computing LTD
35

36. Host scripts - hck.sh
Main script to run HCK enviroment
Dumps configuration
Creates network configuration for HCK setup
Runs VMs in loop (crucial during PM tests)
Cleans up all the configurations on exit
Daynix Computing LTD
36

37. VM execution scripts
Executed by hck.sh
run_hck_studio.sh - starts up Studio VM
run_hck_client.sh -starts up Client VM
Configuration according hck_setup.cfg
Daynix Computing LTD
37

38. Network scripts
Passed as parameter to QEMU to add network
interface to appropriate bridge
hck_ctrl_bridge_ifup.sh
hck_test_bridge_ifup.sh
hck_world_bridge_ifup.sh
Daynix Computing LTD
38

39. Guests configuration
Daynix Computing LTD
39

40. Studio VM
Setup HCK according to Microsoft’s documentation
Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation#wiki-Checklist_for_a_new_studio_VM:
Do not run HCK studio directly
Use run_hck_studio.bat or run_hck_studio_8.1.bat located in
guest_tools
The above scripts will disconnect the HCK Studio VM from outside
network
Daynix Computing LTD
40

41. Test client VMs
Follow the checklist for test client - https://github.com/
daynix/bricklets/wiki/VirtHCK-brickletdocumentation#wiki-Checklist_for_a_new_client_VM:
Install HCK client according to Microsoft’s
documentation from hck-studioHCKInstallClient
setup.exe
Daynix Computing LTD
41

42. Daynix Computing LTD
42

43. What’s next
HCK automation on the guest
SVVP configuration
Xen support (other hypervisors?)
Adding more device types
Using external configuration
Using libvirt
More guest scripts to automate configuration of test
clients
Daynix Computing LTD
43

44. HCK automation on the
guest
Goal:
Run full test cycle for specific device type without
user interaction.
Daynix Computing LTD
44

45. Adding more device types
Goal
Test all the devices in QEMU and not only virtio (PCI
bridges, USB controllers and etc)
Daynix Computing LTD
45

46. More plans
Using external configuration
Multiple setups on one host without replicating
VirtHCK and images
Use libvirt
Arbitrary number of test clients on one setup
Xen support
Daynix Computing LTD
46

47. Call to action
Test QEMU devices with VirtHCK (not only virtio
devices)
Contribute
Daynix Computing LTD
47

48. Q&A
48

49. Links
WiKi - https://github.com/daynix/bricklets/wiki/
VirtHCK-bricklet-documentation
Repository - https://github.com/daynix/bricklets.git
HCK kit download - http://msdn.microsoft.com/en-us/
windows/hardware/hh833788.aspx
Windows HCK Getting Started - http://
msdn.microsoft.com/en-us/library/windows/hardware/
jj123537.aspx
Daynix - www.daynix.com
Daynix Computing LTD
49