CentOS, QA and OpenNebula 
Opennebula Conference 2014 
Berlin 
December 2nd to 4th 
Christoph Galuschka 
tigalch@tigalch.org
Who am I 
● Christoph Galuschka, age 37 
● from Innsbruck, Austria 
● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT 
operations department 
● Using Linux since 1998, CentOS since 2005 
● Part of the CentOS-QA-Team since April 2012 
● Mainly working on release-QA, t_functional git repo for automated testing 
and the testing infrastructure, wiki and forums
CentOS CI 
● created in june 2011 
● an approach to automate QA-release testing 
● create a set of tests to run for each point release QA-cycle 
● use git as a common repository for other users (yes you too) to contribute 
new tests 
● currently contains more then 430 scripts/functional/trademark tests for more 
than 110 packages 
● additionally uses the Linux Test Project 
● upcoming: to test every update before public release
History 
● First setup of CI: lost from memory :) and in a different DC 
● Second setup - move to opennebula 
● started with 4.0, 4.2 later (RPMs from CentOS-Developers) 
● 4.8 since November 2014 
● only “issue” during update: ruby gem 
● ‘/usr/share/one/install_gems’ vs. rpm
Setup I 
● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each 
● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors 
are solely used for test VMs
Setup II
Setup III 
● OpenNebula is used as the cloud controller 
● The sunstone interface is available on Hypervisor1 
● All VMs instantiated only available on the private network 
● Datastore transport → SSH 
● Hosts → KVM 
● The ci.dev.centos.org site runs as VM in that setup 
● nginx is used to forward http(s) from the public internet to the VM 
● local mirror for all CentOS-releases 
● mimics the setup of mirror.centos.org for all VMs 
● Jenkins is used to handle all running tests/VMs
Jenkins 
● 4 different 
testgroups 
● docker image 
(C6/C7 only), 
t_functional, LTP 
and reimzul 
● run on a daily 
basis 
● Infrastructure is 
hosted by 
GoDaddy
Jenkins II 
● All tests run the current versions of CentOS (C5.11, C6.6, C7.0) 
Docker image 
● pulls and runs 
current C5/6/7 
docker image 
● checks 
installation and 
command 
execution 
inside image 
LTP 
● Linux Test 
Project1 
● comprehensive 
checks of basic 
linux 
functionality 
t_functional 
● functional tests 
of packages 
● also checks ™ 
issues if valid 
● 14 contributors 
● C7 running 
since Nov. 7th 
reimzul 
● called on de-mand 
via IRC 
● used for 
verifiying 
merge 
requests 
1) visit http://ltp.sourceforge.net/
Images used for CI 
● Created by Karanbir Singh and Jaime Melis 
● Creates images for C5 and C6 with generic kickstarts and 
configurations 
● Images are 
custombuilt on 
the fly and are 
downloadable 
afterwards 
● C7 image comes 
from cloud. 
centos.org
KS-Sample for prjrd.net 
install 
url --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ 
lang en_US.UTF-8 
keyboard uk 
network --device eth0 --bootproto dhcp 
rootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI. 
firewall --service=ssh 
authconfig --enableshadow --passalgo=sha512 --enablefingerprint 
selinux --enforcing 
timezone --utc Europe/London 
bootloader --location=mbr --driveorder=sda 
repo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100 
repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100 
zerombr yes 
clearpart --all --initlabel 
part /boot --fstype ext3 --size=250 
part pv.2 --size=5000 --grow 
volgroup VolGroup00 --pesize=32768 pv.2 
logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow 
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512 
reboot 
%packages 
@Base 
@Core 
opennebula-context 
%end 
%post --log=/root/post.log --nochroot 
...
KS-Sample for prjrd.net 
sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init 
sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth* 
rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules 
for f in /boot/grub/grub.conf; do 
/bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f} 
/bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f} 
/bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f} 
done 
rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd 
cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF 
/sbin/restorecon -Rv /root/.ssh 
if [ ! -e /var/update_done ] 
then 
yum update -y 
touch /var/update_done 
chkconfig sshd on 
chkconfig auditd on 
reboot 
fi 
EOF 
%end 
● This kickstart creates an image with disabled ssh-server. After boot it pulls 
all updates, enables ssh and reboots once (ensuring current kernel for all 
things to come for this VM).
introducing t_functional 
● Git repo hosting all the functional tests (packages, roles) 
● Using #!/bin/bash (but other scripting languages are possible!) 
● Hosted on gitorious and git.centos.org (for QA) 
● Community driven (yes, you too) 
● Daily results at: http://ci.dev.centos.org 
● https://git.centos.org/summary/sig-core!t_functional.git 
● https://gitorious.org/testautomation/t_functional
Distro and functional tests 
● Is release X.y installable? 
● Through ISO? 
● Through network (nfs/http/ftp) + pxe? 
● Mixed (netinstall.iso + network)? 
● Usb image? 
● UEFI vs Bios? 
● Are all packages in the distro doing what they're supposed to do ? 
● Still working with different sets of packages (php vs php53 for example in 
CentOS 5.x)
t_functional git workflow 
gitorious 
Clone & 
change 
Request 
to 
merge 
Jenkins CI merge 
request 
Fix it No? No? 
gitorious push git.centos.org pull QA-Env
further opennebula usage 
● Two further setups in DCs in London 
● controls four hosts in both cases 
● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy, 
some internal stuff 
● www and forum site to come 
● uses 4.8 
● GlusterFS over Infiniband 
● more on that: http://arrfab.net/blog/?p=485
Any Questions? 
Thank you for your time

OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka

  • 1.
    CentOS, QA andOpenNebula Opennebula Conference 2014 Berlin December 2nd to 4th Christoph Galuschka tigalch@tigalch.org
  • 2.
    Who am I ● Christoph Galuschka, age 37 ● from Innsbruck, Austria ● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT operations department ● Using Linux since 1998, CentOS since 2005 ● Part of the CentOS-QA-Team since April 2012 ● Mainly working on release-QA, t_functional git repo for automated testing and the testing infrastructure, wiki and forums
  • 3.
    CentOS CI ●created in june 2011 ● an approach to automate QA-release testing ● create a set of tests to run for each point release QA-cycle ● use git as a common repository for other users (yes you too) to contribute new tests ● currently contains more then 430 scripts/functional/trademark tests for more than 110 packages ● additionally uses the Linux Test Project ● upcoming: to test every update before public release
  • 4.
    History ● Firstsetup of CI: lost from memory :) and in a different DC ● Second setup - move to opennebula ● started with 4.0, 4.2 later (RPMs from CentOS-Developers) ● 4.8 since November 2014 ● only “issue” during update: ruby gem ● ‘/usr/share/one/install_gems’ vs. rpm
  • 5.
    Setup I ●3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each ● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors are solely used for test VMs
  • 6.
  • 7.
    Setup III ●OpenNebula is used as the cloud controller ● The sunstone interface is available on Hypervisor1 ● All VMs instantiated only available on the private network ● Datastore transport → SSH ● Hosts → KVM ● The ci.dev.centos.org site runs as VM in that setup ● nginx is used to forward http(s) from the public internet to the VM ● local mirror for all CentOS-releases ● mimics the setup of mirror.centos.org for all VMs ● Jenkins is used to handle all running tests/VMs
  • 8.
    Jenkins ● 4different testgroups ● docker image (C6/C7 only), t_functional, LTP and reimzul ● run on a daily basis ● Infrastructure is hosted by GoDaddy
  • 9.
    Jenkins II ●All tests run the current versions of CentOS (C5.11, C6.6, C7.0) Docker image ● pulls and runs current C5/6/7 docker image ● checks installation and command execution inside image LTP ● Linux Test Project1 ● comprehensive checks of basic linux functionality t_functional ● functional tests of packages ● also checks ™ issues if valid ● 14 contributors ● C7 running since Nov. 7th reimzul ● called on de-mand via IRC ● used for verifiying merge requests 1) visit http://ltp.sourceforge.net/
  • 10.
    Images used forCI ● Created by Karanbir Singh and Jaime Melis ● Creates images for C5 and C6 with generic kickstarts and configurations ● Images are custombuilt on the fly and are downloadable afterwards ● C7 image comes from cloud. centos.org
  • 11.
    KS-Sample for prjrd.net install url --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ lang en_US.UTF-8 keyboard uk network --device eth0 --bootproto dhcp rootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI. firewall --service=ssh authconfig --enableshadow --passalgo=sha512 --enablefingerprint selinux --enforcing timezone --utc Europe/London bootloader --location=mbr --driveorder=sda repo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100 repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100 zerombr yes clearpart --all --initlabel part /boot --fstype ext3 --size=250 part pv.2 --size=5000 --grow volgroup VolGroup00 --pesize=32768 pv.2 logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512 reboot %packages @Base @Core opennebula-context %end %post --log=/root/post.log --nochroot ...
  • 12.
    KS-Sample for prjrd.net sed -i "s/^ACTIVE_CONSOLES=/dev/tty[1-6]/ACTIVE_CONSOLES=/dev/tty1/" /mnt/sysimage/etc/sysconfig/init sed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth* rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rules for f in /boot/grub/grub.conf; do /bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f} /bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f} /bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f} done rm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshd cat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF /sbin/restorecon -Rv /root/.ssh if [ ! -e /var/update_done ] then yum update -y touch /var/update_done chkconfig sshd on chkconfig auditd on reboot fi EOF %end ● This kickstart creates an image with disabled ssh-server. After boot it pulls all updates, enables ssh and reboots once (ensuring current kernel for all things to come for this VM).
  • 13.
    introducing t_functional ●Git repo hosting all the functional tests (packages, roles) ● Using #!/bin/bash (but other scripting languages are possible!) ● Hosted on gitorious and git.centos.org (for QA) ● Community driven (yes, you too) ● Daily results at: http://ci.dev.centos.org ● https://git.centos.org/summary/sig-core!t_functional.git ● https://gitorious.org/testautomation/t_functional
  • 14.
    Distro and functionaltests ● Is release X.y installable? ● Through ISO? ● Through network (nfs/http/ftp) + pxe? ● Mixed (netinstall.iso + network)? ● Usb image? ● UEFI vs Bios? ● Are all packages in the distro doing what they're supposed to do ? ● Still working with different sets of packages (php vs php53 for example in CentOS 5.x)
  • 15.
    t_functional git workflow gitorious Clone & change Request to merge Jenkins CI merge request Fix it No? No? gitorious push git.centos.org pull QA-Env
  • 16.
    further opennebula usage ● Two further setups in DCs in London ● controls four hosts in both cases ● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy, some internal stuff ● www and forum site to come ● uses 4.8 ● GlusterFS over Infiniband ● more on that: http://arrfab.net/blog/?p=485
  • 17.
    Any Questions? Thankyou for your time