CentOS, the Community Enterprise OS, uses Opennebula as virtualization plattform for its automated QA-process. The opennebula setup consists of 3 nodes, all running CentOS-6, who handle the following tasks: – sunstone as cloud controller – local mirror/DNS-Server/http-Server for the VMs to pull in packages – one VM to run a jenkins instance to launch the various tests (ci.de.centos.org) – nginx on the cloud controller to forward http traffic to the jenkins VM A public git repository (http://www.gitorious.org/testautomation) is used to allow whoever wants to contribute to pull the current test suite – t_functional, a series of bash scripts used to do funtional tests of various applications, binaries, configuration files and Trademark issues. As new tests are added to the repo via personal clones and merge requests, those tests first need to complete a test run via jenkins. Each test run currently consists of 4 VMs (one for each arch for C5 and C6 – C7 to come), which run the complete test suite. All VMs used for theses tests are instantiated and torn down on demand, whenever the call to testrun a personal clone is issued (via IRC). Once completed successfully, the request is merged into the main repo. The jenkins node monitors this repository and which automatically triggers another complete test run. Besides these triggered test runs, the test suite is automatically triggered daily to run. This is used to verify functionality of published updates – a handfull of failty updates have allready been discovered this way. Besides t_functional, the Linux Test Project Suite of tests is also run on a daily basis, also to verify functionality of the OS and all updates. The third setup is used to test the available and functional integrity of published docker images for CentOS. All these tests are later – during the QA-phase of a point release – used to verify functionality of new packages inside the CentOS QA-Setup.

1. CentOS, QA and OpenNebula
Opennebula Conference 2014
Berlin
December 2nd to 4th
Christoph Galuschka
tigalch@tigalch.org

2. Who am I
● Christoph Galuschka, age 37
● from Innsbruck, Austria
● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT
operations department
● Using Linux since 1998, CentOS since 2005
● Part of the CentOS-QA-Team since April 2012
● Mainly working on release-QA, t_functional git repo for automated testing
and the testing infrastructure, wiki and forums

3. CentOS CI
● created in june 2011
● an approach to automate QA-release testing
● create a set of tests to run for each point release QA-cycle
● use git as a common repository for other users (yes you too) to contribute
new tests
● currently contains more then 430 scripts/functional/trademark tests for more
than 110 packages
● additionally uses the Linux Test Project
● upcoming: to test every update before public release

4. History
● First setup of CI: lost from memory :) and in a different DC
● Second setup - move to opennebula
● started with 4.0, 4.2 later (RPMs from CentOS-Developers)
● 4.8 since November 2014
● only “issue” during update: ruby gem
● ‘/usr/share/one/install_gems’ vs. rpm

5. Setup I
● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each
● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors
are solely used for test VMs

6. Setup II

7. Setup III
● OpenNebula is used as the cloud controller
● The sunstone interface is available on Hypervisor1
● All VMs instantiated only available on the private network
● Datastore transport → SSH
● Hosts → KVM
● The ci.dev.centos.org site runs as VM in that setup
● nginx is used to forward http(s) from the public internet to the VM
● local mirror for all CentOS-releases
● mimics the setup of mirror.centos.org for all VMs
● Jenkins is used to handle all running tests/VMs

8. Jenkins
● 4 different
testgroups
● docker image
(C6/C7 only),
t_functional, LTP
and reimzul
● run on a daily
basis
● Infrastructure is
hosted by
GoDaddy

9. Jenkins II
● All tests run the current versions of CentOS (C5.11, C6.6, C7.0)
Docker image
● pulls and runs
current C5/6/7
docker image
● checks
installation and
command
execution
inside image
LTP
● Linux Test
Project1
● comprehensive
checks of basic
linux
functionality
t_functional
● functional tests
of packages
● also checks ™
issues if valid
● 14 contributors
● C7 running
since Nov. 7th
reimzul
● called on de-mand
via IRC
● used for
verifiying
merge
requests
1) visit http://ltp.sourceforge.net/

10. Images used for CI
● Created by Karanbir Singh and Jaime Melis
● Creates images for C5 and C6 with generic kickstarts and
configurations
● Images are
custombuilt on
the fly and are
downloadable
afterwards
● C7 image comes
from cloud.
centos.org

11. KS-Sample for prjrd.net
install
url --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/
lang en_US.UTF-8
keyboard uk
network --device eth0 --bootproto dhcp
rootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI.
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
selinux --enforcing
timezone --utc Europe/London
bootloader --location=mbr --driveorder=sda
repo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100
repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100
zerombr yes
clearpart --all --initlabel
part /boot --fstype ext3 --size=250
part pv.2 --size=5000 --grow
volgroup VolGroup00 --pesize=32768 pv.2
logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512
reboot
%packages
@Base
@Core
opennebula-context
%end
%post --log=/root/post.log --nochroot
...

12. KS-Sample for prjrd.net
sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init
sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth*
rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules
for f in /boot/grub/grub.conf; do
/bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f}
/bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f}
done
rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd
cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF
/sbin/restorecon -Rv /root/.ssh
if [ ! -e /var/update_done ]
then
yum update -y
touch /var/update_done
chkconfig sshd on
chkconfig auditd on
reboot
fi
EOF
%end
● This kickstart creates an image with disabled ssh-server. After boot it pulls
all updates, enables ssh and reboots once (ensuring current kernel for all
things to come for this VM).

13. introducing t_functional
● Git repo hosting all the functional tests (packages, roles)
● Using #!/bin/bash (but other scripting languages are possible!)
● Hosted on gitorious and git.centos.org (for QA)
● Community driven (yes, you too)
● Daily results at: http://ci.dev.centos.org
● https://git.centos.org/summary/sig-core!t_functional.git
● https://gitorious.org/testautomation/t_functional

14. Distro and functional tests
● Is release X.y installable?
● Through ISO?
● Through network (nfs/http/ftp) + pxe?
● Mixed (netinstall.iso + network)?
● Usb image?
● UEFI vs Bios?
● Are all packages in the distro doing what they're supposed to do ?
● Still working with different sets of packages (php vs php53 for example in
CentOS 5.x)

15. t_functional git workflow
gitorious
Clone &
change
Request
to
merge
Jenkins CI merge
request
Fix it No? No?
gitorious push git.centos.org pull QA-Env

16. further opennebula usage
● Two further setups in DCs in London
● controls four hosts in both cases
● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy,
some internal stuff
● www and forum site to come
● uses 4.8
● GlusterFS over Infiniband
● more on that: http://arrfab.net/blog/?p=485

17. Any Questions?
Thank you for your time