Practical Xen Testing at Intel
                              Nov. 2009


        Haicheng Li <haicheng.li@intel.com>
     ...
Legal Information

•   INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL
    PRODUCTS. NO LICENSE, EXPRESS...
Agenda
• The Overview – how test helps Xen?
• Test Infrastructure – the world under automation
• Best Practices – practica...
The Overview




               Software and Services Group
                                             4
               ...
Progressive Xen Testing
                                              VT-x                                                ...
Progressive Xen Testing
                                              VT-x                                               S...
Test Infrastructure




                  Software and Services Group
                                                7
  ...
Solid Test Infrastructure
• Test Automation in Everywhere
   – auto daily testing
   – auto release resting
   – auto patc...
Extendable Test Framework
                                                         Manual Testing
                        ...
Test Result Center




    Find Defects Out
         Timely!




                  Software and Services Group
           ...
Patch Test System

           Recipe Submit UI         Machine Status UI           Result UI

        DB Tool             ...
Patch Test System




Ensure Patch Quality Before
     Submitting Out!




                     Software and Services Grou...
Best Practices




                 Software and Services Group
                                               13
        ...
Power Management Testing
• Challenge
   – How to achieve integrated Power Management testing?
• Test Approach
   – Automat...
Power Management Testing
• Challenge
   – How to achieve integrated power testing?
• Test Approach
   – Automated function...
RAS Testing
• Challenge
   – How to achieve good test coverage?
• Test Approach
   – Add test stub to Xen to dump page typ...
RAS Testing
• Challenge
   – How to achieve good test coverage?
• Test Approach
   – Add test stub to Xen to dump page typ...
VT-d/SR-IOV Testing
• Challenge
   – Hardware variety & complexity : platforms + devices
• Test Approach
   – Select typic...
Some Proposals




                 Software and Services Group
                                               19
        ...
Improve Xen Quality
• Source Control
   – Synchronize source control for Ioemu, Dom0 and Xen.
        •   E.g. VT-d is usu...
Questions?




             Software and Services Group
                                           21
                    ...
Backup




         Software and Services Group
                                       22
                                ...
Regular Testing

To keep Xen from regressions

                 Build Check        Booting Tests      Basic Functional Tes...
Release Testing

To make qualified Xen release

                 Build Check        Booting Tests      Basic Functional Te...
Patch Testing

To ensure patch quality for Xen



                        Build Check   Booting Tests      Basic Functiona...
Upcoming SlideShare
Loading in …5
×

Practical Xen Testing At Intel

1,477 views

Published on

Xen.org community presentation from Xen Summit Asia 2

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,477
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
22
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Practical Xen Testing At Intel

  1. 1. Practical Xen Testing at Intel Nov. 2009 Haicheng Li <haicheng.li@intel.com> Jiajun Xu <jiajun.xu@intel.com> Xen Summit Asia 2009
  2. 2. Legal Information • INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. • Intel may make changes to specifications, product descriptions, and plans at any time, without notice. • All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. • Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. • Intel is a trademark of Intel Corporation in the U.S. and other countries. • *Other names and brands may be claimed as the property of others. • Copyright © 2009, Intel Corporation. All rights are protected. Software and Services Group 2 2
  3. 3. Agenda • The Overview – how test helps Xen? • Test Infrastructure – the world under automation • Best Practices – practical test approaches • Some Proposals – to improve Xen quality Software and Services Group 3 3
  4. 4. The Overview Software and Services Group 4 4
  5. 5. Progressive Xen Testing VT-x SR-IOV Intel VT VT- i VT-d Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009 Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0 XEN • Hundreds of Xen patches Monthly !!! XEN Development • How can TEST help? • Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS Coverage SMP VM Expose regression timely Live Migration TXT Real Mode Device Pass-Through SR-IOV • 3.0.x Release Ensure release quality 3.1.x Release 3.2.x Release 3.3.x Release 3.4.x Release Test Test Reports Test Reports Test Reports Test Reports Test Reports Deliverables Nightly Test Reports Weekly Test Reports Biweekly Test Reports Actively Bug Tracking Software and Services Group 5 5
  6. 6. Progressive Xen Testing VT-x SR-IOV Intel VT VT- i VT-d Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009 Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0 XEN XEN Development Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS Coverage SMP VM Live Migration TXT Real Mode Device Pass-Through SR-IOV 3.0.x Release 3.1.x Release ~300 major Release reported, 3.2.x Release 3.3.x bugs 3.4.x Release Test Deliverables Test Reports Test Reports 20% of Xen Bugzilla Test Reports Test Reports Test Reports Nightly Test Reports Weekly Test Reports Biweekly Test Reports Actively Bug Tracking Software and Services Group 6 6
  7. 7. Test Infrastructure Software and Services Group 7 7
  8. 8. Solid Test Infrastructure • Test Automation in Everywhere – auto daily testing – auto release resting – auto patch testing • Test Infrastructure is The Basis • Extendable Test Framework • Auto Build System • Test Result Center • Patch Test System Software and Services Group 8 8
  9. 9. Extendable Test Framework Manual Testing Regular Testing Auto Build *XVS Result submitter to Result Center Common lib *XVS: eXtendable Validation System Management Module VT-d/ Controller Report Generator VT-d | -- VT-d lib | -- test1 | -- test2 Call with Scenario file SR-IOV Test Suite Pool … Harness SR-IOV/ Power | -- IOV lib Test Harness | -- test1 | -- test2 RAS Software and Services Group 9 9
  10. 10. Test Result Center Find Defects Out Timely! Software and Services Group 10 10
  11. 11. Patch Test System Recipe Submit UI Machine Status UI Result UI DB Tool Web UI Receive Core Management Runtime DB Recipe Test Schedule Test Box 1 Local Control Scheduler Machine Test Box 2 Recipe … Test Info Test Machine R-Queue Pool Test Info XVS Users Test Box N Test Resources OSD Results Results Processing Machines Software and Services Group 11 11
  12. 12. Patch Test System Ensure Patch Quality Before Submitting Out! Software and Services Group 12 12
  13. 13. Best Practices Software and Services Group 13 13
  14. 14. Power Management Testing • Challenge – How to achieve integrated Power Management testing? • Test Approach – Automated functionality testing. • Interface : xenpm, hypercall, sysfs interface • Coverage : idle, runtime, Dom0/HVM S3 – Performance measurement. • Scenario : idle, runtime, benchmarks • Platform : laptop & server • OS : Linux, Windows • Publish : web site Software and Services Group 14 14
  15. 15. Power Management Testing • Challenge – How to achieve integrated power testing? • Test Approach – Automated functionality testing. • Interface : xenpm, hypercall, sys i/f • Coverage : idle, runtime, Dom0/HVM S3 – Performance measurement. • Scenario : idle, runtime, benchmarks • Platform : laptop & server • OS : Linux, Windows • Publish : web site Software and Services Group 15 15
  16. 16. RAS Testing • Challenge – How to achieve good test coverage? • Test Approach – Add test stub to Xen to dump page types. – Add test stub to Xen to translate address from guest to host. – Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages. • To Do – How to stress? • Option: add more software test interface to *MCE handlers. – CPU Migration – MEM hot-plug *MCE: Machine Check Exception Software and Services Group 16 16
  17. 17. RAS Testing • Challenge – How to achieve good test coverage? • Test Approach – Add test stub to Xen to dump page types. – Add test stub to Xen to What MCE test i/f guest to host. translate address from Linux provides? – Use ACPI4.0/APEI to trigger CPU/Mem to trigger fake MCE • MCE injector *MCE to specific pages. • madvise syscall to poison page • To Do • HWPosion injector to poison page – How to stress? •…… • Option: add more software test interface to *MCE handlers. – CPU Migration – MEM hot-plug *MCE: Machine Check Error Software and Services Group 17 17
  18. 18. VT-d/SR-IOV Testing • Challenge – Hardware variety & complexity : platforms + devices • Test Approach – Select typical test configurations. • Platform : laptop -> desktop -> server • Device : NIC, USB, SATA • PCI type : PCIE MSI, PCIE INTx, PCI INTx • OS : Linux, Windows – Use NIC to achieve test automation. • Methods : ping, scp (file copy), ssh • Coverage : static assignment + VT-d hotplug • Multiple-devices assignment. Software and Services Group 18 18
  19. 19. Some Proposals Software and Services Group 19 19
  20. 20. Improve Xen Quality • Source Control – Synchronize source control for Ioemu, Dom0 and Xen. • E.g. VT-d is usually broken by ioemu check-in • Bug Tracking – Well track bug state in Xen Bugzilla • Current Xen BZ full of obsolete bug data • ~500 bugs , 30% Xen Bugzilla, are NEW! • Xen Testability – More built-in test features for Xen? • It’s a trend for Linux: memleak checker, fault injector, ... Software and Services Group 20 20
  21. 21. Questions? Software and Services Group 21 21
  22. 22. Backup Software and Services Group 22 22
  23. 23. Regular Testing To keep Xen from regressions Build Check Booting Tests Basic Functional Tests C/S Pull Auto Build Auto Install Regression Testing Bug Verification Daily Build *BAT Testing Daily *BAT Test Cycle *BAT: Build Acceptance Test Full Scope Functional Tests Functionality Testing Performance Testing Stress Testing Others System Testing Monthly System Test Cycle Software and Services Group 23 23
  24. 24. Release Testing To make qualified Xen release Build Check Booting Tests Basic Functional Tests C/S Pull Auto Build Auto Install Regression Testing Bug Verification Release Build *BAT Testing Release *BAT Test Cycle *BAT: Build Acceptance Test Full Scope Functional Tests Functionality Testing Performance Testing Stress Testing Others System Testing Release System Test Cycle Software and Services Group 24 24
  25. 25. Patch Testing To ensure patch quality for Xen Build Check Booting Tests Basic Functional Tests Auto Build Auto Install Regression Testing Unit Testing Patch Testing Patch Test Cycle Software and Services Group 25 25

×