QEMU development and testing automation
using MS HCK
Anton Nayshtut, anton@daynix.com
Yan Vugenfirer, yan@daynix.com
Dayni...
Agenda
What is WHQL/HCK?
Introducing VirtHCK
Using VirtHCK
Call to action

Daynix Computing LTD

2
What is HCK/WHQL?
The Windows certification program (previously known
as the Windows Logo Program) for devices and system
A...
HCK test kit
Introduced with Windows 8 (previous kit know as WLK)
Predefined test suites for each device type
Ability to ad...
Benefits for QEMU and
Linux community
QEMU device testing (not only virtio)
Host subsystems testing, for example networking...
HCK PCI compliance test
Device under test compliance to PCI spec
Applicable for:
New QEMU chipsets
PCIe devices
Next gener...
HCK NDIS (network) tester (1)
Comprehensive networking test suite
Send/receive,
Stress tests (MPE)
Offload, VLAN, priority,...
HCK NDIS (network) tester (2)
Applicable for
QEMU network devices
New devices
New features
Host network components
Bridge
...
HCK Windows OS tests
Device path exerciser
Identifies drivers that do not correctly handle the
different IO calls
Driver ve...
HCK power management tests
The test cycling through different sleep states
There are variations of IO and PNP (disable/ena...
Our past experience
Transfer hangs with vhost.
Offload/VLAN related corner cases with network
devices (virtio-net, vmxnet3,...
HCK - great, but frightening
Cumbersome setup requirements
Complex network configuration
Hard to replicate setups
Setup con...
VirtHCK
Daynix Computing LTD

13
Introducing VirtHCK
Framework for automating HCK test execution
Based on easily configurable host and guest scripts
Creates...
Repository and WiKi
Get VirtHCK from https://github.com/daynix/
bricklets.git
VirtHCK wiki with steps for deployment and u...
HCK issues solved by VirtHCK
Setup replication
Multiple setups sharing the same host
Setup isolation solves the following ...
Setup types supported
Network (virtio-net, vmxnet3, vmxnet)
Storage (virtio-block, virtio-scsi, pvscsi)
Simple PCI (virtio...
Typical HCK deployment
Test Client
Control NIC

Test Client
Control NIC

Test Client
Control NIC

WHQL control network

Co...
VirtHCK setup
Disconnected from outside world network during test
run
Isolated NICs under test network
Isolated WHQL contr...
VirtHCK setup for storage and
other devices
Test VM 1
Control NIC - e1000

WHQL control bridge

Control NIC - e1000

Studi...
VirtHCK setup for storage and
other devices - running
Test VM 1
Control NIC - e1000

WHQL control bridge

Control NIC - e1...
VirtHCK setup for NICs
NIC under test

Test bridge

NIC under test

Test VM 2

Test VM 1

Control NIC - e1000

Control NIC...
VirtHCK setup for NICs - running
NIC under test

NIC under test

Test bridge

Test VM 2

Test VM 1

Control NIC - e1000

C...
VirtHCK - quick start
Clone VirtHCK from github
Create VM images in /images directory (studio and test
client)
Update STUD...
Directory tree
VirtHCK/ - Directory of all VirtHCK scripts, supposed
to run in Linux environment
VirtHCK/guest_tools/ - Di...
VirtHCK walkthrough - host

Configuration: network, VMs etc
Host scripts
Images

Daynix Computing LTD

26
hck_setup.cfg
VirtHCK setup configuration file
Environment configuration
VM start up parameters
Configured accordingly to requ...
Running multiple setups
Use UNIQ_ID to configure multiple setups
UNIQ_ID will be used as a base for creation of:
VNC or Spi...
Test device types
Network
Storage (also boot)
Balloon
Serial
More can be added

Daynix Computing LTD

29
Remote access
VNC or Spice
Spice is needed for power management tests

Daynix Computing LTD

30
Network infrastructure
Linux bridge or Open vSwitch can be used

Daynix Computing LTD

31
Images
Configure
Studio VM image
Test VMs images

Daynix Computing LTD

32
VM resources

Number of vCPUs
Memory
NICs for control channels

Daynix Computing LTD

33
Extra parameters
QEMU binary
VHOST, snapshots, cache mode for fast install
ISOs for guest installation
BIOS

Daynix Comput...
Configuration summary

Daynix Computing LTD

35
Host scripts - hck.sh
Main script to run HCK enviroment
Dumps configuration
Creates network configuration for HCK setup
Runs...
VM execution scripts
Executed by hck.sh
run_hck_studio.sh - starts up Studio VM
run_hck_client.sh -starts up Client VM
Con...
Network scripts
Passed as parameter to QEMU to add network
interface to appropriate bridge
hck_ctrl_bridge_ifup.sh
hck_tes...
Guests configuration

Daynix Computing LTD

39
Studio VM
Setup HCK according to Microsoft’s documentation
Follow the checklist - https://github.com/daynix/bricklets/wiki...
Test client VMs
Follow the checklist for test client - https://github.com/
daynix/bricklets/wiki/VirtHCK-brickletdocumenta...
Daynix Computing LTD

42
What’s next
HCK automation on the guest
SVVP configuration
Xen support (other hypervisors?)
Adding more device types
Using ...
HCK automation on the
guest
Goal:
Run full test cycle for specific device type without
user interaction.

Daynix Computing ...
Adding more device types

Goal
Test all the devices in QEMU and not only virtio (PCI
bridges, USB controllers and etc)

Da...
More plans
Using external configuration
Multiple setups on one host without replicating
VirtHCK and images
Use libvirt
Arbi...
Call to action

Test QEMU devices with VirtHCK (not only virtio
devices)
Contribute

Daynix Computing LTD

47
Q&A

48
Links
WiKi - https://github.com/daynix/bricklets/wiki/
VirtHCK-bricklet-documentation
Repository - https://github.com/dayn...
Upcoming SlideShare
Loading in...5
×

QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix

1,756

Published on

Windows Hardware Certification Kit (HCK) is a set of tools, processes, and tests for certifying HW devices, device drivers and systems. Being a great test environment for QEMU devices, Windows Guest device drivers and related Host subsystems, it's still frightening due to deployment complexity. We'll share a way to deploy HCK setup(s) on top of QEMU VMs in just a few minutes.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,756
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix

  1. 1.  QEMU development and testing automation using MS HCK Anton Nayshtut, anton@daynix.com Yan Vugenfirer, yan@daynix.com Daynix Computing LTD
  2. 2. Agenda What is WHQL/HCK? Introducing VirtHCK Using VirtHCK Call to action Daynix Computing LTD 2
  3. 3. What is HCK/WHQL? The Windows certification program (previously known as the Windows Logo Program) for devices and system A comprehensive devicesystem test toolkit by Microsoft Devices which pass HCK are certified and their binaries are digitally signed Daynix Computing LTD 3
  4. 4. HCK test kit Introduced with Windows 8 (previous kit know as WLK) Predefined test suites for each device type Ability to add your own tests for additional testing outside of the certification process Reproducible scenarios Daynix Computing LTD 4
  5. 5. Benefits for QEMU and Linux community QEMU device testing (not only virtio) Host subsystems testing, for example networking Extensive coverage for defined device categories Regression testing Stress testing Daynix Computing LTD 5
  6. 6. HCK PCI compliance test Device under test compliance to PCI spec Applicable for: New QEMU chipsets PCIe devices Next generation of virtio devices Existing PCI devices Daynix Computing LTD 6
  7. 7. HCK NDIS (network) tester (1) Comprehensive networking test suite Send/receive, Stress tests (MPE) Offload, VLAN, priority, packet filtering etc Not once used to test the robustness of the host networking stack Daynix Computing LTD 7
  8. 8. HCK NDIS (network) tester (2) Applicable for QEMU network devices New devices New features Host network components Bridge Open vSwitch TAP Kernel Daynix Computing LTD 8
  9. 9. HCK Windows OS tests Device path exerciser Identifies drivers that do not correctly handle the different IO calls Driver verifier Captures deadlocks, memory leaks, memory corruption Verified correctness of kernel API usage Daynix Computing LTD 9
  10. 10. HCK power management tests The test cycling through different sleep states There are variations of IO and PNP (disable/enable) during, before and after power transitions Applicable for: QEMU Devices power management support SeaBIOS Daynix Computing LTD 10
  11. 11. Our past experience Transfer hangs with vhost. Offload/VLAN related corner cases with network devices (virtio-net, vmxnet3, vmxnet). vmxnet3 and pvscsi in QEMU Open vSwitch out of order issues RSC (Windows GRO) PCI compliance Daynix Computing LTD 11
  12. 12. HCK - great, but frightening Cumbersome setup requirements Complex network configuration Hard to replicate setups Setup conflicts No power management tests automation support Daynix Computing LTD 12
  13. 13. VirtHCK Daynix Computing LTD 13
  14. 14. Introducing VirtHCK Framework for automating HCK test execution Based on easily configurable host and guest scripts Creates isolated virtual network environments Allows running different HCK scenarios in automated manner OSS, distributed under BSD license Daynix Computing LTD 14
  15. 15. Repository and WiKi Get VirtHCK from https://github.com/daynix/ bricklets.git VirtHCK wiki with steps for deployment and usage: https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation Daynix Computing LTD 15
  16. 16. HCK issues solved by VirtHCK Setup replication Multiple setups sharing the same host Setup isolation solves the following issues: HCK controller controls all HCK clients in the network Inability to use VM templates without manual VM reconfiguration Automatic network topology setup during network device tests Daynix Computing LTD 16
  17. 17. Setup types supported Network (virtio-net, vmxnet3, vmxnet) Storage (virtio-block, virtio-scsi, pvscsi) Simple PCI (virtio-serial, virtio-balloon) Easily extendable by small changes in configuration files Daynix Computing LTD 17
  18. 18. Typical HCK deployment Test Client Control NIC Test Client Control NIC Test Client Control NIC WHQL control network Control NIC - e1000 HCK Controller Daynix Computing LTD 18
  19. 19. VirtHCK setup Disconnected from outside world network during test run Isolated NICs under test network Isolated WHQL control network A different number of test clients can run according to test type Daynix Computing LTD 19
  20. 20. VirtHCK setup for storage and other devices Test VM 1 Control NIC - e1000 WHQL control bridge Control NIC - e1000 Studio VM “World” bridge Physical NIC Outside world NIC Daynix Computing LTD 20
  21. 21. VirtHCK setup for storage and other devices - running Test VM 1 Control NIC - e1000 WHQL control bridge Control NIC - e1000 “World” bridge Studio VM Outside world NIC Physical NIC X Daynix Computing LTD 21
  22. 22. VirtHCK setup for NICs NIC under test Test bridge NIC under test Test VM 2 Test VM 1 Control NIC - e1000 Control NIC - e1000 WHQL control bridge Control NIC - e1000 Studio VM “World” bridge Physical NIC Outside world NIC Daynix Computing LTD 22
  23. 23. VirtHCK setup for NICs - running NIC under test NIC under test Test bridge Test VM 2 Test VM 1 Control NIC - e1000 Control NIC - e1000 WHQL control bridge Control NIC - e1000 “World” bridge Studio VM Outside world NIC Physical NIC X Daynix Computing LTD 23
  24. 24. VirtHCK - quick start Clone VirtHCK from github Create VM images in /images directory (studio and test client) Update STUDIO_IMAGE/CLIENT1_IMAGE/ CLIENT2_IMAGE variables in hck_setup.cfg Run VirtHCK - “sudo ./VirtHCK/hck.sh” Install Microsoft HCK Studio and Clients software according to Microsoft documentation Windows HCK Getting Started and following Checklist for a new client VM guidelines. Daynix Computing LTD 24
  25. 25. Directory tree VirtHCK/ - Directory of all VirtHCK scripts, supposed to run in Linux environment VirtHCK/guest_tools/ - Directory for scripts required for Windows guests Images directory where VM's images should be located Daynix Computing LTD 25
  26. 26. VirtHCK walkthrough - host Configuration: network, VMs etc Host scripts Images Daynix Computing LTD 26
  27. 27. hck_setup.cfg VirtHCK setup configuration file Environment configuration VM start up parameters Configured accordingly to required HCK setup type Allows multiple setups on one host Daynix Computing LTD 27
  28. 28. Running multiple setups Use UNIQ_ID to configure multiple setups UNIQ_ID will be used as a base for creation of: VNC or Spice ports Bridges names Network interface names, MAC addresses and etc. Daynix Computing LTD 28
  29. 29. Test device types Network Storage (also boot) Balloon Serial More can be added Daynix Computing LTD 29
  30. 30. Remote access VNC or Spice Spice is needed for power management tests Daynix Computing LTD 30
  31. 31. Network infrastructure Linux bridge or Open vSwitch can be used Daynix Computing LTD 31
  32. 32. Images Configure Studio VM image Test VMs images Daynix Computing LTD 32
  33. 33. VM resources Number of vCPUs Memory NICs for control channels Daynix Computing LTD 33
  34. 34. Extra parameters QEMU binary VHOST, snapshots, cache mode for fast install ISOs for guest installation BIOS Daynix Computing LTD 34
  35. 35. Configuration summary Daynix Computing LTD 35
  36. 36. Host scripts - hck.sh Main script to run HCK enviroment Dumps configuration Creates network configuration for HCK setup Runs VMs in loop (crucial during PM tests) Cleans up all the configurations on exit Daynix Computing LTD 36
  37. 37. VM execution scripts Executed by hck.sh run_hck_studio.sh - starts up Studio VM run_hck_client.sh -starts up Client VM Configuration according hck_setup.cfg Daynix Computing LTD 37
  38. 38. Network scripts Passed as parameter to QEMU to add network interface to appropriate bridge hck_ctrl_bridge_ifup.sh hck_test_bridge_ifup.sh hck_world_bridge_ifup.sh Daynix Computing LTD 38
  39. 39. Guests configuration Daynix Computing LTD 39
  40. 40. Studio VM Setup HCK according to Microsoft’s documentation Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCKbricklet-documentation#wiki-Checklist_for_a_new_studio_VM: Do not run HCK studio directly Use run_hck_studio.bat or run_hck_studio_8.1.bat located in guest_tools The above scripts will disconnect the HCK Studio VM from outside network Daynix Computing LTD 40
  41. 41. Test client VMs Follow the checklist for test client - https://github.com/ daynix/bricklets/wiki/VirtHCK-brickletdocumentation#wiki-Checklist_for_a_new_client_VM: Install HCK client according to Microsoft’s documentation from hck-studioHCKInstallClient setup.exe Daynix Computing LTD 41
  42. 42. Daynix Computing LTD 42
  43. 43. What’s next HCK automation on the guest SVVP configuration Xen support (other hypervisors?) Adding more device types Using external configuration Using libvirt More guest scripts to automate configuration of test clients Daynix Computing LTD 43
  44. 44. HCK automation on the guest Goal: Run full test cycle for specific device type without user interaction. Daynix Computing LTD 44
  45. 45. Adding more device types Goal Test all the devices in QEMU and not only virtio (PCI bridges, USB controllers and etc) Daynix Computing LTD 45
  46. 46. More plans Using external configuration Multiple setups on one host without replicating VirtHCK and images Use libvirt Arbitrary number of test clients on one setup Xen support Daynix Computing LTD 46
  47. 47. Call to action Test QEMU devices with VirtHCK (not only virtio devices) Contribute Daynix Computing LTD 47
  48. 48. Q&A 48
  49. 49. Links WiKi - https://github.com/daynix/bricklets/wiki/ VirtHCK-bricklet-documentation Repository - https://github.com/daynix/bricklets.git HCK kit download - http://msdn.microsoft.com/en-us/ windows/hardware/hh833788.aspx Windows HCK Getting Started - http:// msdn.microsoft.com/en-us/library/windows/hardware/ jj123537.aspx Daynix - www.daynix.com Daynix Computing LTD 49
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×