Presentation for QA:Conference held in Moscow, Russia on April 23rd. Author: Evgeny Antyshev, Virtuozzo These slide discover some Openstack Infrastructure tools to ease the task of creating generic CI systems. As an illustration, I setup "model" CI stand to test Libvirt project. Important ideas originated from Openstack testing also mentioned: pre-review integration testing, testing infrastructure in a cloud, project gating, etc.

1. Openstack Third-Party CI и
обзор компонент тестовой
инфраструктуры
Евгений Антышев, Virtuozzo
eantyshev@virtuozzo.com
skype: Evgeny.Antyshev

2. Более 600 активных разработчиков(!)
54 команды, у каждой в среднем 10 проектов, от 1 до 138 (Infrastructure)
http://git.openstack.org/cgit/openstack/governance/plain/reference/projects.yaml
1. Основной язык -- Python. Затем: Shell, Puppet. Для данных: Yaml, ini-config, json.
2. Общая структура репозиториев: README.md, testr unittests, tox virtualenv wrapper
3. Общий процесс разработки: git, gerrit review

3. Gerrit workflow

4. Задачи third-party CI:
1)Верификация входящих патчей на
своем окружении
2)Верификация third-party компонент как
необходимое условие для их принятия
сообществом
3)Стабильность third-party CI создает
хорошую репутацию в сообществе

5. Openstack CI scaling
“Jenkins”:
upstream CI robot
780 проектов
~900 workers
~20k events/day
Virtuozzo CI,
Virtuozzo Storage CI
3 проекта
3-6 workers
150 events/day

6. “Легенда”: удаленное
тестирование кода libvirt
• Разворачивание и обновление
тестовых сред с Ubuntu 14.04 LTS
(nodepool, diskimage-builder)
• Конфигурация тестовых задач для
Jenkins (Jenkins Jobs Builder)
• Управление запуском тестов (gearman)

7. Nodepool
Управление пулом тестовых серверов и
загрузочными образами в Openstack
облаке

8. nodepool
Jenkins +
plugins:
ZMQ, Node
and Label,
Gearman
WorkerN
“Trusty-label”
qemu-kvm
ssh
REST API
“Trusty-label”
image
Ubuntu 14.04
(Trusty Tahr)
base image
my-setup.sh
1) Create/update images
2) Deploy nodes for Jenkins
and utilize when it
finishes with them
Gearman
server
Provider cloud

9. Nodepool: установка
1. PyPi
# pip install nodepool
2. Git sources
# git clone git://git.openstack.org/openstack-infra/nodepool.git
../nodepool # pip install -r requirements.txt
../nodepool # pip install .
3. puppet-nodepool project
# git clone git://git.openstack.org/openstack-infra/puppet-nodepool.git
Use README

10. Nodepool: конфигурация
/etc/nodepool/secure.conf
1) Создать пользователя jenkins, настроить ssh авторизацию и
sudo
2) apt-get update и установить dev пакеты
3) git clone git://libvirt.org/libvirt.git в /opt/git
/etc/nodepool-scripts/my-setup.sh
/etc/nodepool/nodepool.yaml
[database]
dburi=mysql+pymysql://nodepool:passwd@localhost/nodepooldb
[jenkins "jenkins1"]
user=jenkins
apikey=f828c94dbd53b7e2a055112ebbc90ea8
credentials=28755fdb-244a-4fcf-baee-496031323c12
url=http://10.94.136.33:8080/

11. Nodepool.yaml: header
script-dir: /etc/nodepool-scripts
elements-dir: /etc/nodepool-elements
images-dir: /opt/nodepool_dib
cron:
check: '*/15 * * * *'
cleanup: '*/1 * * * *'
image-update: '14 2 * * *'
zmq-publishers:
- tcp://localhost:8888

12. Nodepool.yaml
Jenkins label → image name
labels:
- name: trusty-label
image: trusty1
min-ready: 1
providers:
- name: openstack001-test

13. nodepool.yaml
Cloud providers
providers:
- name: openstack001-test
username: 'admin'
password: '5fcf6292e9a240e0'
auth-url: 'http://10.94.2.99:5000/v2.0/'
project-name: 'admin'
max-servers: 2
images:
- name: trusty1
base-image: trusty
setup: my-setup.sh
min-ram: 2048
username: jenkins
private-key: /opt/nodepool/jenkins_key

14. Nodepool command line
> nodepool hold 123
> nodepool delete 123
> nodepool image-update all trusty1

15. Nodepool: создание
загрузочных образов
Snapshot images
Base image
Temporary VM
my-setup.sh
Take a snapshot
diskimage-builder
(DIB) images
Base image
qemu-nbd mount
Prep. in chroot
Convert
Upload to provider

16. Преимущества diskimage-
builder:
1) Можно запускать в VM
1) Более структурированный способ задания
конфигурации
1) Легкое добавление готовых элементов в
конфигурацию

17. diskimage-builder: типичные
задачи для автоматизации
■адреса репозиториев
■пользователи, ключи ssh
■установка и конфигурация приложений
■разделы фс, настройки загрузки
(growroot и пр.)
■кеширование git репозиториев и пр.

18. DIB stages & elements
Stages: root.d, extra-data.d, install.d (in
chrooted env.), finalise.d, etc.
Elements:
• ubuntu-minimal: base OS
• vm: preparation to boot in QEMU
• simple-init: boot scripts to enable
DHCP for NICs
• growroot: expand root FS to cover
available disk space

19. DIB element “my-slave”
(derived from my-setup.sh)
/etc/nodepool-elements/my-slave/
package-installs.yaml -- sshd, dev packages
extra-data.d/
50-jenkins-pubkey -- copy key from host to mounted
image
install.d/
50-jenkins-user 60-cache-libvirt
finalise.d/
99-nodepool-dir -- nodepool needs /etc/nodepool

20. /etc/nodepool/nodepool.yaml:
diskimages:
- name: ubuntu-trusty
elements:
- ubuntu-minimal
- vm
- simple-init
- growroot
- my-slave
release: trusty
env-vars:
TMPDIR: /opt/dib_tmp
DIB_IMAGE_CACHE: /opt/dib_cache
/var/log/nodepool.log: disk-image-create -x -t qcow2 --no-tmpfs --qemu-img-
options 'compat=0.10' -o /opt/nodepool_dib/42 ubuntu-minimal vm simple-
init growroot my-slave
…
Image file /opt/nodepool_dib/42.qcow2 created…
nodepool.yaml: diskimages

21. Jenkins Job Builder
Конфигурация как код
Yaml вместо Xml
Шаблоны, группы, макросы

22. JJB builder: copy-update-source
- builder:
name: copy-update-source
builders:
- shell: |
mv /opt/git/libvirt $WORKSPACE
cd libvirt
git clean -f -x
git fetch -f --recurse-submodules=yes
origin {branch}
git checkout FETCH_HEAD

23. JJB builder: test-libvirt
- builder:
name: test-libvirt
builders:
- shell: |
cd libvirt
./autogen.sh --system
make
sudo make install
libvirtd --version
sudo libvirtd -d
virsh list --all

24. JJB template
- job-template:
name: smoke-libvirt-{branch}
node: trusty-label || dib-trusty-label
builders:
- copy-update-source:
branch: '{branch}'
- test-libvirt
publishers:
- email:
recepients: 'eantyshev@virtuozzo.com'

25. JJB project
- project:
name: libvirt
branch:
- master
- v1.3.1-maint
jobs:
- smoke-libvirt-{branch}

26. Jenkins view

27. Gearman
Сервис распределения задач и
client/worker API
http://gearman.org
“We on Openstack infrastructure team use Jenkins extensively. Our jenkins servers, at peak load,
runs 20,000+ jobs per day. At that load we require many jenkins slaves (900+) to process all of
those build jobs. We have found that our requirement was pushing Jenkins beyond it's limits
therefore we've decided to create the Gearman Plugin to support multiple Jenkins masters. The
gearman plugin was designed to support extra slaves, allow load balancing of build jobs, and
provide redundancy.”
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin

28. Gearman workflow example
1. Worker registers “reverse” function on Gearman server.
2. Client sends “reverse” request and parameters.
3. Gearman server chooses the worker and assigns the job.
4. Worker sends result to Client via Server.

29. Ubuntu 14.04:
apt-get install gearman-job-server
/etc/nodepool/nodepool.yaml:
gearman-servers:
- host: localhost
Jenkins -> Configure System -> Enable
Gearman
Установка gearman-server,
Gearman plugin (client)

30. Gearman interactions
in “libvirt CI”
gearmand
Gearman Client
(Python API)
Jenkins Gearman
plugin (Java API)
Nodepool (status
TCP requests +
images tasks)
Gear worker

31. Диагностика gearman: status
request
Status line fmt: Function <enqeued> <executing> <Nworkers>
root@test-nodepool:/etc/jenkins_jobs/config# telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
build:smoke-libvirt-v1.3.1-maint 0 0 1
set_description:10.94.136.33 0 0 1
build:smoke-libvirt-master 0 0 1
stop:10.94.136.33 0 0 1
build:smoke-libvirt-v1.3.1-maint:dib-trusty-label 0 0 1
build:smoke-libvirt-master:dib-trusty-label 0 0 1
.
^]
telnet> Connection closed.

32. Job requests via Gear client
Python implementation:
http://pypi.python.org/pypi/gear
https://github.com/zaro0508/gearman-plugin-client
>python gear_client.py -s localhost -p 4730 --function build:smoke-libvirt-master:trusty-label --params
'{"OFFLINE_NODE_WHEN_COMPLETE":"true"}'

33. Zuul
Планирование тестовых
запусков для per-patch CI,
project gating для gerrit
http://docs.openstack.org/infra/zuul/

34. Zuul pipelines
Independent
check, post-
merge, periodic
Dependent
gate
HEAD 1
HEAD 2
HEAD 3
HEAD 1
2
3

35. Independent pipelines Dependent pipeline
http://status.openstack.org/zuul/

36. Материалы “libvirt CI”:
https://github.com/eantyshev/libvirt_ci
Полезные ссылки:
http://docs.openstack.org/infra/manual/
http://docs.openstack.org/infra/nodepool/
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
http://docs.openstack.org/infra/jenkins-job-builder/
Спасибо за внимание!
Вопросы?

38. Система ревью для git
Робот, управляющий запуском тестов
(gear python client)
Сервис очередей для распределения задач
Jenkins master (Gearman
worker)
Jenkins slave
(рабочие среды)
Создание и удаление рабочих сред,
обновление образов (Gearman queue
monitoring)

39. /etc/jenkins-jobs/config/libvirt.yaml:
- job-template:
name: smoke-{name}-{branch}
node: trusty-label || dib-trusty-label
JJB: node labels

40. smoke-libvirt job, take 2: split
builders
- job:
name: smoke-libvirt
project-type: freestyle
node: trusty-label
builders:
- shell: |
mv /opt/git/libvirt $WORKSPACE
cd libvirt
git clean -f -x
git fetch -f --recurse-submodules=yes
origin master
git checkout FETCH_HEAD
- shell: |
./autogen.sh --system
make
sudo make install
libvirtd --version
sudo libvirtd -d
virsh list --all
publishers:
- email:
recepients: 'eantyshev@virtuozzo.com'

41. ● root.d: Download cloud image (qcow2)
mount by qemu-nbd, submounts: /dev, /dev/pts, /sys, /proc + hooks dir
● environment.d: Set default environment vars
● extra-files.d: copy files to mounted image, f.e. ssh keys
Go chroot under mounted path
● pre-install.d, install.d, post-install.d: Executed in chroot
environment: set repo mirrors, install packages, configure apps
Copy root content to temp. Unmount amended image.
Make clean raw image file
● block-device.d: (auxiliary) Partition loopback block device
Create filesystem, mount the image (+/proc,dev,sys submounts), copy rootfs
contents from temp
● finalise.d: install/configure bootloader, other tools
● cleanup.d: cleanup temporary repos, unblock daemons
Unmount, convert from raw to qcow2
DIB stages

42. smoke-libvirt job, take 1
Недостатки: copy-paste when >1 job
Неструктурированная логика в
builders
1) Макрос copy-update-source с
параметром branch и макрос
test-libvirt
2) Job -> Job Template + Project
3) “project-type: freestyle” ->
defaults.yaml
- job:
name: smoke-libvirt
project-type: freestyle
node: trusty-label
builders:
- shell: |
mv /opt/git/libvirt $WORKSPACE
cd libvirt
git clean -f -x
git fetch -f --recurse-submodules=yes
origin master
git checkout FETCH_HEAD
./autogen.sh --system
make
sudo make install
libvirtd --version
sudo libvirtd -d
virsh list --all
publishers:
- email:
recepients: 'eantyshev@virtuozzo.com'

43. JJB defaults
- defaults:
name: global
project-type: freestyle
concurrent: true
wrappers:
- timeout:
timeout: 30
fail: true
- timestamps
logrotate:
daysToKeep: 7
numToKeep: -1
artifactDaysToKeep: -1
artifactNumToKeep: -1

44. type: patchset-created, patchSet.ref: 'refs/changes/38/307638/1'
type: comment-added, comment: “Looks good overall, couple of ....”
type: patchset-created, patchSet.ref: ‘refs/changes/78/216378/38’
type: comment-added, comment: “blablabla...”
zuul-server (scheduler) zuul-merger (git server)
> gerrit event-stream
Gearman server
merger:merge 0 0 1
merger:update 0 0 1
Job requests:
merger:merge args{gerrit_url, project, ref}*
merger:update args{gerrit_url, project}
Merge job complete:
Zuul ref: commit id
git clone <url> /var/lib/zuul/git/<project>
git merge <ref>
In Jenkins job:
git fetch $ZUUL_URL $ref
Zuul workflow

45. Что дальше..?
• Как запускать тесты автоматически?
• Как регулировать число тестовых нод в
зависимости от нагрузки?
• Как масштабировать CI на несколько
Jenkins серверов?

46. - job:
name: test-example
builders:
- shell: ‘echo Success!’
<?xml version="1.0" encoding="utf-8"?>
<project>
<actions/>
<description>&lt;!-- Managed by Jenkins Job Builder --&gt;</description>
<keepDependencies>false</keepDependencies>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<concurrentBuild>false</concurrentBuild>
<canRoam>true</canRoam>
<properties/>
<scm class="hudson.scm.NullSCM"/>
<builders>
<hudson.tasks.Shell>
<command>echo Success!</command>
</hudson.tasks.Shell>
</builders>
<buildWrappers/>
</project>
Yaml to Xml
> jenkins-jobs test sample-job.yaml