osstest, Xen's automatic testing facility - In the last year, osstest has gained contributors and many new test cases. A wider range of guests is being tested in a wider range of configurations; osstest's scope has increased to a wider set of the software components used in a Xen system. In this presentation, Ian Jackson will highlight some of the most interesting of these contributions (and their contributors). The presentation will include some figures showing osstests's community development. We'll then look forward to changes in the pipeline, including planned new features and infrastructure changes.

1. osstest
Xen Project automatic test system
Community participation
Recent developments
Future plans
Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com>
Xen Developer Summit Chicago 2014

2. To: <xen−devel@xxxxxxxxxxxxxxxxxxx>
Subject: [libvirt test] 29207: regressions − FAIL
flight 29207 libvirt real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/29207/
Regressions :−(
Tests which did not succeed and are blocking,
including tests which could not be run:
build−amd64−libvirt 5 libvirt−build fail REGR. vs. 29190
build−i386−libvirt 5 libvirt−build fail REGR. vs. 29190
build−armhf−libvirt 5 libvirt−build fail REGR. vs. 29190
Tests which did not succeed, but are not blocking:
test−amd64−i386−libvirt 1 build−check(1) blocked n/a
test−amd64−amd64−libvirt 1 build−check(1) blocked n/a
test−armhf−armhf−libvirt 1 build−check(1) blocked n/a
version targeted for testing:
libvirt ddf6b139cfebee4193bf6f0b13c995a182d7bc35
baseline version:
libvirt 72a8453948691b75a1802524837b6543353bf829
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
People who touched revisions under test:
....

3. To: <xen−devel@lists.xensource.com>
Subject: [xen−unstable bisection] complete build−armhf−libvirt
branch xen−unstable
xen branch xen−unstable
job build−armhf−libvirt
test libvirt−build
Tree: gnulib_libvirt git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
Tree: libvirt git://xenbits.xen.org/libvirt.git
Tree: qemuu git://xenbits.xen.org/staging/qemu−upstream−unstable.git
Tree: xen git://xenbits.xen.org/xen.git
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 871b43a309d80ac99458c13c2c3da8d15c482d30
Bug not present: 6cc89d3101d8874e01a69a89a65736a2adfbd199
commit 871b43a309d80ac99458c13c2c3da8d15c482d30
Author: Senatus Populus <senatus.populus@example.com>
Date: Fri Jun 20 18:19:12 2014 +0200
libxl: get and set soft affinity
Make space a new cpumap in vcpu_info, called cpumap_soft,
for retrieving soft affinity, and amend the relevant API
accordingly.
libxl_set_vcpuaffinity() now takes two cpumaps, one for hard
and one for soft affinity (LIBXL_API_VERSION is exploited to
retain source level backword compatibility). Either of the
two cpumap can be NULL, in which case, only the affinity
corresponding to the non−NULL cpumap will be affected.
...
For bisection revision−tuple graph see:
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen−unstable.build−armhf−libvirt.libvirt−build.html
Revision IDs in each graph node refer, respectively, to the Trees above.
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
Searching for failure / basis pass:
27871 fail [host=army] / 27481 ok.
Failure / basis pass flights: 27871 / 27481
Tree: gnulib_libvirt git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
Tree: libvirt git://xenbits.xen.org/libvirt.git
Tree: qemuu git://xenbits.xen.org/staging/qemu−upstream−unstable.git
Tree: xen git://xenbits.xen.org/xen.git

4. branches jobs
libvirt
linux−3.10
linux−3.4
linux−arm−xen
linux−linus
linux−mingo−tip−master
linux−next
osstest
qemu−mainline
qemu−upstream−4.2−testing
qemu−upstream−4.3−testing
qemu−upstream−4.4−testing
qemu−upstream−unstable
rumpuserxen
seabios
xen−4.0−testing
xen−4.1−testing
xen−4.2−testing
xen−4.3−testing
xen−4.4−testing
xen−unstable
build−amd64
build−amd64−libvirt
build−amd64−oldkern
build−amd64−pvops
build−amd64−rumpuserxen
build−amd64−xend
build−armhf
build−armhf−libvirt
build−armhf−pvops
build−i386
build−i386−libvirt
build−i386−oldkern
build−i386−pvops
build−i386−rumpuserxen
build−i386−xend
test−amd64−amd64−libvirt
test−amd64−amd64−pair
test−amd64−amd64−pv
test−amd64−amd64−rumpuserxen−amd64
test−amd64−amd64−rumpuserxen−i386
test−amd64−amd64−xl
test−amd64−amd64−xl−pcipt−intel
test−amd64−amd64−xl−qemut−debianhvm−amd64
test−amd64−amd64−xl−qemut−win7−amd64
test−amd64−amd64−xl−qemut−winxpsp3
test−amd64−amd64−xl−qemuu−debianhvm−amd64
test−amd64−amd64−xl−qemuu−ovmf−amd64
test−amd64−amd64−xl−qemuu−win7−amd64
test−amd64−amd64−xl−qemuu−winxpsp3
test−amd64−amd64−xl−sedf
test−amd64−amd64−xl−sedf−pin
test−amd64−amd64−xl−win7−amd64
test−amd64−amd64−xl−winxpsp3
test−amd64−i386−freebsd10−amd64
test−amd64−i386−freebsd10−i386
test−amd64−i386−libvirt
test−amd64−i386−pair
test−amd64−i386−pv
test−amd64−i386−qemut−rhel6hvm−amd
test−amd64−i386−qemut−rhel6hvm−intel
test−amd64−i386−qemuu−freebsd10−amd64
test−amd64−i386−qemuu−freebsd10−i386
test−amd64−i386−qemuu−rhel6hvm−amd
test−amd64−i386−qemuu−rhel6hvm−intel
test−amd64−i386−rhel6hvm−amd
test−amd64−i386−rhel6hvm−intel
test−amd64−i386−rumpuserxen−i386
test−amd64−i386−xend−qemut−winxpsp3
test−amd64−i386−xend−qemuu−winxpsp3
test−amd64−i386−xend−winxpsp3
test−amd64−i386−xl
test−amd64−i386−xl−credit2
test−amd64−i386−xl−multivcpu
test−amd64−i386−xl−qemut−debianhvm−amd64
test−amd64−i386−xl−qemut−win7−amd64
test−amd64−i386−xl−qemut−winxpsp3
test−amd64−i386−xl−qemut−winxpsp3−vcpus1
test−amd64−i386−xl−qemuu−debianhvm−amd64
test−amd64−i386−xl−qemuu−ovmf−amd64
test−amd64−i386−xl−qemuu−win7−amd64
test−amd64−i386−xl−qemuu−winxpsp3
test−amd64−i386−xl−qemuu−winxpsp3−vcpus1
test−amd64−i386−xl−win7−amd64
test−amd64−i386−xl−winxpsp3
test−amd64−i386−xl−winxpsp3−vcpus1
test−armhf−armhf−libvirt
test−armhf−armhf−xl
test−i386−i386−libvirt
test−i386−i386−pair
test−i386−i386−pv
test−i386−i386−rumpuserxen−i386
test−i386−i386−xl
test−i386−i386−xl−qemut−winxpsp3
test−i386−i386−xl−qemuu−winxpsp3
test−i386−i386−xl−winxpsp3

5. Introduction
============
OSStest is the Xen Project automated test infrastructure.
...
Terminology
===========
"flight":
Each run of osstest is referred to as a "flight". Each flight is
given a unique ID (a number or name).
"job":
Each flight consists of one or more "jobs". These are a sequence
of test steps run in order and correspond to a column in the test
report grid. They have names like "build−amd64" or
"test−amd64−amd64−pv". A job can depend on the output of another
job in the flight −− e.g. most test−* jobs depend on one or more
build−* jobs.
...
Standalone Mode
===============
To run osstest in standalone mode:
− You need to install
sqlite3
tcl8.4 tclx8.4 libsqlite3−tcl
libdbi−perl libdbd−sqlite3−perl
pax rsync
curl
netcat
− Optional: ipmitool
− Write a config file
~/.xen−osstest/config
See below.
− Select the "branch" and job to reproduce. By default the system
gives you the "branch" consisting of tests run for the xen−unstable
push gate. You need to select a job. The list of available jobs
is that shown in the publicly emailed test reports on xen−devel, eg
http://lists.xen.org/archives/html/xen−devel/2014−01/msg01614.html

6. | army earwig spider potato−beetle fire−frog itch−mite rice−weevil
| bedbug insider leaf−beetle field−cricket marilith−n4
| hydrazine lace−bug lake−frog marilith−n5
| woodlouse moss−bug chicken−mite grain−weevil
| bush−cricket gall−mite
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
arch−amd64 | x x x x x x x x x x x x x x x x x x
arch−armhf | x x x
arch−i386 | x x x x x x x x x x x x x x x x x
arch−xen−amd64 | x x x x x x x x x x x x x x x x x x
arch−xen−armhf | x x
arch−xen−i386 | x x x x x x x x x x x x x x x x x
blessed−adhoc | x x x x x x x x x x x x x x x x
blessed−play | x x x x x x x x x x x x x x x x
blessed−real | x x x x x x x x x x x x x x x x
equiv−dell−r310 | x x
equiv−marilith | x x
equiv−rackservers−q21011 | x x
equiv−rackservers−s40663 | x x
equiv−rackservers−s40670 | x x
equiv−rackservers−s40679 | x x
equiv−rackservers−s40680 | x x
hvm | x x x x x x x x x x x x x x x x x x
hvm−amd | x x x x x x x
hvm−intel | x x x x x x x x x x x
need−firmware−deb−firmware−bnx2 | x x
need−kernel−deb−armmp | x x
need−uboot−bootscr | x x
no−di−kernel | x x
no−reinstall | x
pcipassthrough−nic | x x
purpose−build | x x x x x x x x x x x x x x x x
purpose−test | x x x x x x x x x x x x x x x x x x x
spong | x
suite−squeeze | x x x x x x x x x x x x x x x x x x x x x
suite−wheezy | x x x x x x x x x x x x x x x x x x x x x x

7. 800
700
600
500
400
300
200
100
0
osstest loc by author date
2014−06−09 Anthony Perard
2013−11−21 Roger Pau Monné
2013−11−06 Dario Faggioli
2013−09−16 Wei Liu
2013−09−03 Ian Campbell
2009−11−23 Ian Jackson
01/01/12 01/07/12 01/01/13 01/07/13 01/01/14 01/07/14

8. 60
50
40
30
20
10
0
16
14
12
10
8
6
4
2
0
jobs in xen-unstable
branches ("[BRANCH test]")
01/06/13 01/08/13 01/10/13 01/12/13 01/02/14 01/04/14 01/06/14 01/08/14

9. References
Source repository
http://xenbits.xen.org/gitweb/?p=osstest.git;a=summary
README
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=HEAD
Output example
http://xenbits.xen.org/docs/osstest−output−example/24438/
Contact
xen−devel@lists.xenproject.org
Ian.Jackson@eu.citrix.com
Ian.Campbell@citrix.com