An overview of the libvirt+xen OpenStack CI, explaining the various components, how they fit together and the specific customisations needed to test libvirt+xen under OpenStack.

1. XenProject OpenStack CI overview

2. © 2014 Citrix. Confidential.2
Libvirt+Xen CI: Major components
Nodepool
Gerrit Zuul
Jenkins Devstack-GateGearman
JJB
os-ext-testing

3. © 2014 Citrix. Confidential.3
Nodepool
Used to speed up deploying of new nodes
• Devstack-gate from scratch can take >1h before it’s ready to run tests
Creates base image once a day
• Caches git repositories (reduces issues from unreliable github)
• Includes custom Xen + libvirt packages (Needs libxl fixes on top of Xen 4.5)
Creates nodes from this base image to satisfy demand
• Specify minimum-ready for the pool and maximum in each cloud region
• Advertises these nodes to Gearman
Deletes nodes (so, single-use) once Jenkins has finished running tests

4. © 2014 Citrix. Confidential.4
Zuul
Monitors event stream from review.openstack.org
Identifies dependent changes
Creates merged branches for testing using current master and dependent
changes
Adds to a pipeline (e.g. check, silent, pass-only)
Each pipeline has a set of jobs to run for a given project
Waits for node to be available and submits job to Jenkins
Comments / votes back on review.openstack.org depending on result and pipeline

5. © 2014 Citrix. Confidential.5
Libvirt+Xen CI: The Good, The Bad and The Ugly
Single-use VMs
Based on upstream
Swift upload
Highly reliable
Steep learning curve
Swift upload
Backport upstream Xen fixes
Changes to puppet scripts
No monitoring
Single Cloud
Swift upload
No pre-prod env

6. © 2014 Citrix. Confidential.6
Libvirt+Xen CI: Upgrades and backport
No “hacks” required
Libvirt 1.2.14 with:
•f86ae40 libxl: Move job acquisition in libxlDomainStart to callers
•894d2ff libxl: acquire a job when destroying a domain
•6dfec1e libxl: drop virDomainObj lock when destroying a domain
Xen 4.4 (ubuntu package) with:
•9369988 libxl: event handling: Break out ao_work_outstanding
•f1335f0 libxl: event handling: ao_inprogress does waits while reports outstanding
•4783c99 libxl: In domain death search, start search at first domid we want
•188e9c5 libxl: Domain destroy: fork

7. © 2014 Citrix. Confidential.7
Libvirt+Xen CI: customisations
Nodepool: os-ext-testing.git/nodepool-scripts
• Custom Xen+libvirt packages: Includes libxl changes not present in latest releases
• static-network-config / read-vendor-json.py: Use configdrive to get IP address, rather than xenstore.
• prepare_node_devstack_xen.sh: Install Xen, update grub, setup configdrive reading and create a small swap
Jenkins Job Builder: os-ext-testing.git/etc/jenkins_jobs/
• pre_test_hook to force-install specific version of libvirt (late to ensure it is not replaced by devstack)
• cleanup_host_hook: exit trap to copy /var/log/xen logs to output directory
• Exclusion of some jobs (test_volume_boot_pattern) pending kernel fix
• Also some customisations to add swift-upload to macros at
os-ext-testing.git/puppet/modules/os_ext_testing/templates/jenkins_job_builder/

8. © 2014 Citrix. Confidential.8
Deployment details
Public-visible entry points:
• zuul.openstack.xenproject.org: Overview of which jobs are waiting to be tested
• jenkins.openstack.xenproject.org: Historical + running jobs
Infrastructure on single host running in Rackspace Cloud
• Individual nodes to run tests also created in Rackspace Cloud
Deployed using puppet scripts
• github.com/citrix-openstack/os-ext-testing
• Currently a fork: New effort to refactor upstream puppet scripts and these to reduce rebasing
Private repository containing credentials

9. © 2014 Citrix. Confidential.9
Useful Links
Recent build trend
jenkins.openstack.xenproject.org/job/dsvm-tempest-xen/buildTimeTrend
All changes voted upon
review.openstack.org/#/q/reviewer:openstack-ci%2540xenproject.org,n,z
Overview + issue investigation
wiki.xenproject.org/wiki/OpenStack_CI_Loop_for_Xen-Libvirt

10. © 2014 Citrix. Confidential.10
Next steps
All interested parties please join wg-openstack@lists.xenproject.org
Improving stability
• Need fix for stalled device model startup https://marc.info/?l=xen-devel&m=143153926817067
• Need fix for network-setup race https://bugs.launchpad.net/nova/+bug/1461642
Testing with Neutron
• Need fix for VIF name too long https://bugs.launchpad.net/nova/+bug/1450465
Monitoring – possibly using github.com/stackforge/third-party-ci-tools
• Instance running at zuul.openstack.xenproject.org/scoreboard/
Group A

11. © 2014 Citrix. Confidential.11
WORK BETTER. LIVE BETTER.