The document discusses practical Xen testing approaches at Intel. It outlines Intel's progressive testing of Xen features over time to increase test coverage and ensure release quality. The test infrastructure is described as being automated, extendable, and centered around a test result center to timely expose regressions. Best practices proposed include extending the test framework, using an automated build system, and a patch testing system.

1. Practical Xen Testing at Intel
Nov. 2009
Haicheng Li <haicheng.li@intel.com>
Jiajun Xu <jiajun.xu@intel.com>
Xen Summit Asia 2009

2. Legal Information
• INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL
PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO
ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT
AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS,
INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS
OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS
INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR
PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR
OTHER INTELLECTUAL PROPERTY RIGHT.
• Intel may make changes to specifications, product descriptions, and plans at any time, without
notice.
• All products, computer systems, dates, and figures specified are preliminary based on current
expectations, and are subject to change without notice.
• Performance tests and ratings are measured using specific computer systems and/or
components and reflect the approximate performance of Intel products as measured by those
tests. Any difference in system hardware or software design or configuration may affect
actual performance.
• Intel is a trademark of Intel Corporation in the U.S. and other countries.
• *Other names and brands may be claimed as the property of others.
• Copyright © 2009, Intel Corporation. All rights are protected.
Software and Services Group
2
2

3. Agenda
• The Overview – how test helps Xen?
• Test Infrastructure – the world under automation
• Best Practices – practical test approaches
• Some Proposals – to improve Xen quality
Software and Services Group
3
3

4. The Overview
Software and Services Group
4
4

5. Progressive Xen Testing
VT-x SR-IOV
Intel VT
VT- i VT-d
Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009
Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0
XEN
• Hundreds of Xen patches Monthly !!!
XEN Development
• How can TEST help?
•
Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS
Coverage
SMP VM Expose regression timely
Live Migration
TXT Real Mode Device Pass-Through SR-IOV
•
3.0.x Release
Ensure release quality
3.1.x Release 3.2.x Release 3.3.x Release 3.4.x Release
Test Test Reports Test Reports Test Reports Test Reports Test Reports
Deliverables
Nightly Test Reports Weekly Test Reports Biweekly Test Reports
Actively Bug Tracking
Software and Services Group
5
5

6. Progressive Xen Testing
VT-x SR-IOV
Intel VT
VT- i VT-d
Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009
Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0
XEN
XEN Development
Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS
Coverage
SMP VM Live Migration TXT Real Mode Device Pass-Through SR-IOV
3.0.x Release 3.1.x Release
~300 major Release reported,
3.2.x Release 3.3.x
bugs 3.4.x Release
Test
Deliverables
Test Reports Test Reports 20% of Xen Bugzilla
Test Reports Test Reports Test Reports
Nightly Test Reports Weekly Test Reports Biweekly Test Reports
Actively Bug Tracking
Software and Services Group
6
6

7. Test Infrastructure
Software and Services Group
7
7

8. Solid Test Infrastructure
• Test Automation in Everywhere
– auto daily testing
– auto release resting
– auto patch testing
• Test Infrastructure is The Basis
• Extendable Test Framework
• Auto Build System
• Test Result Center
• Patch Test System
Software and Services Group
8
8

9. Extendable Test Framework
Manual Testing
Regular Testing
Auto Build *XVS Result submitter to Result Center
Common lib *XVS: eXtendable Validation System
Management Module
VT-d/
Controller Report Generator VT-d | -- VT-d lib
| -- test1
| -- test2
Call with Scenario file
SR-IOV
Test Suite Pool
…
Harness SR-IOV/
Power | -- IOV lib
Test Harness
| -- test1
| -- test2
RAS
Software and Services Group
9
9

10. Test Result Center
Find Defects Out
Timely!
Software and Services Group
10
10

11. Patch Test System
Recipe Submit UI Machine Status UI Result UI
DB Tool Web UI
Receive
Core Management
Runtime DB Recipe
Test Schedule Test Box 1
Local Control
Scheduler
Machine Test Box 2
Recipe
…
Test Info Test Machine
R-Queue Pool
Test Info XVS
Users Test Box N
Test Resources
OSD
Results Results Processing
Machines
Software and Services Group
11
11

12. Patch Test System
Ensure Patch Quality Before
Submitting Out!
Software and Services Group
12
12

13. Best Practices
Software and Services Group
13
13

14. Power Management Testing
• Challenge
– How to achieve integrated Power Management testing?
• Test Approach
– Automated functionality testing.
• Interface : xenpm, hypercall, sysfs interface
• Coverage : idle, runtime, Dom0/HVM S3
– Performance measurement.
• Scenario : idle, runtime, benchmarks
• Platform : laptop & server
• OS : Linux, Windows
• Publish : web site
Software and Services Group
14
14

15. Power Management Testing
• Challenge
– How to achieve integrated power testing?
• Test Approach
– Automated functionality testing.
• Interface : xenpm, hypercall, sys i/f
• Coverage : idle, runtime, Dom0/HVM S3
– Performance measurement.
• Scenario : idle, runtime, benchmarks
• Platform : laptop & server
• OS : Linux, Windows
• Publish : web site
Software and Services Group
15
15

16. RAS Testing
• Challenge
– How to achieve good test coverage?
• Test Approach
– Add test stub to Xen to dump page types.
– Add test stub to Xen to translate address from guest to host.
– Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages.
• To Do
– How to stress?
• Option: add more software test interface to *MCE handlers.
– CPU Migration
– MEM hot-plug
*MCE: Machine Check Exception
Software and Services Group
16
16

17. RAS Testing
• Challenge
– How to achieve good test coverage?
• Test Approach
– Add test stub to Xen to dump page types.
– Add test stub to Xen to What MCE test i/f guest to host.
translate address from Linux provides?
– Use ACPI4.0/APEI to trigger CPU/Mem to trigger fake MCE
• MCE injector *MCE to specific pages.
• madvise syscall to poison page
• To Do • HWPosion injector to poison page
– How to stress? •……
• Option: add more software test interface to *MCE handlers.
– CPU Migration
– MEM hot-plug
*MCE: Machine Check Error
Software and Services Group
17
17

18. VT-d/SR-IOV Testing
• Challenge
– Hardware variety & complexity : platforms + devices
• Test Approach
– Select typical test configurations.
• Platform : laptop -> desktop -> server
• Device : NIC, USB, SATA
• PCI type : PCIE MSI, PCIE INTx, PCI INTx
• OS : Linux, Windows
– Use NIC to achieve test automation.
• Methods : ping, scp (file copy), ssh
• Coverage : static assignment + VT-d hotplug
• Multiple-devices assignment.
Software and Services Group
18
18

19. Some Proposals
Software and Services Group
19
19

20. Improve Xen Quality
• Source Control
– Synchronize source control for Ioemu, Dom0 and Xen.
• E.g. VT-d is usually broken by ioemu check-in
• Bug Tracking
– Well track bug state in Xen Bugzilla
• Current Xen BZ full of obsolete bug data
• ~500 bugs , 30% Xen Bugzilla, are NEW!
• Xen Testability
– More built-in test features for Xen?
• It’s a trend for Linux: memleak checker, fault injector, ...
Software and Services Group
20
20

21. Questions?
Software and Services Group
21
21

22. Backup
Software and Services Group
22
22

23. Regular Testing
To keep Xen from regressions
Build Check Booting Tests Basic Functional Tests
C/S Pull Auto Build Auto Install Regression Testing Bug Verification
Daily Build *BAT Testing
Daily *BAT Test Cycle
*BAT: Build Acceptance Test
Full Scope Functional Tests
Functionality Testing Performance Testing Stress Testing Others
System Testing
Monthly System Test Cycle
Software and Services Group
23
23

24. Release Testing
To make qualified Xen release
Build Check Booting Tests Basic Functional Tests
C/S Pull Auto Build Auto Install Regression Testing Bug Verification
Release Build *BAT Testing
Release *BAT Test Cycle
*BAT: Build Acceptance Test
Full Scope Functional Tests
Functionality Testing Performance Testing Stress Testing Others
System Testing
Release System Test Cycle
Software and Services Group
24
24

25. Patch Testing
To ensure patch quality for Xen
Build Check Booting Tests Basic Functional Tests
Auto Build Auto Install Regression Testing
Unit Testing Patch Testing
Patch Test Cycle
Software and Services Group
25
25