A short presentation about our experience of using Jenkins and Jenkins Job Builder with Vagrant as a backend tool to manage complex environment (tens of virtual machines for every test run) for database proxy server testing.

1. Managing big test
environment and
running tests with
Jenkins, Jenkins Job
Builder + Vagrant
Our experience
Open Source
and Linux Lab
OSLL

2. Our setup: why is it so complex to run test?
• Product to build and test: Maxscale – database proxy, lots of plugins (routers, filters,
protocols, etc, etc, etc)
• Integration tests, Performance tests, “scientific experiments”
• Test environment:
– at least 9 VM (1 Maxscale server, 4 Master/Slave servers, 4-servers Galera cluster)
– libvirt/qemu, AWS virtual machines + Docker containers
– need to configure backend for every test run
– need to carefully clean up
• 12 Linux distributions, 3 major MariaDB version, 2 major MySQL version, a lot of
Maxscale plugins – any combination should be testable
• dependency on VM provider: it can easily FAIL

3. What does it mean for test management system?
• Lots of parameters for every test job
• Cascades of jobs: e.g. prepare backend, install Maxscale, run test, cleanup
• Error processing (e.g. backend VMs preparation failure) – how to do cleanup in this
case?
• Matrix jobs: test a lot of combinations of distros, backend MariaDB version, backend
parameters
• very dynamic setup
We wanted to put all test system into GIT repository

4. MariaDB Server uses BuildBot!
• not very popular tool with limited support
• lots of complains from MariaDB Server team
• but
– it has been in use from very beginning (since middle of 90th
)
– do we want different tools for every our product?
– there are many useful scripts for BuildBot

5. Jenkins
• “hey guys, you can of cause try” and “do not disturb our BuildBot guy too much”
• Parameterized Trigger Plugin
• matrix jobs
• well, UI

6. Jenkins Job Builder
• A tool from OpenStack developers https://docs.openstack.org/infra/jenkins-job-builder/
• YAML
• Pipelines did not exist when we started
• our jobs: https://github.com/mariadb-corporation/maxscale-jenkins-jobs

7. What can we get from JJB?
• All our jobs (>60) are represented as a code (branches, forks, review, pull requests, …)
• Parameters values are in one place for all jobs (easy maintenance of parameters defaults
and lists)
• Automatic validation of jobs takes place at every push to jobs repo
• very easy jobs deployment
• at the beginning: app. 20 jobs, we spent 1 week to convert to JJB code (1.5 persons)

8. Our “run_test” job
Build
Create repo
Setup backend
Install Maxscale
Run tests
Extract logs, coredumps
cleanup VMs (or not)
Parse logs
Send report email

9. - job:
name: run_test
description: 'This job perform integration testing of maxscale'
parameters:
- !include: './maxscale_jobs/include/boxes_all_incl.yaml'
- !include: './maxscale_jobs/include/products_incl.yaml'
- !include: './maxscale_jobs/include/versions_incl.yaml'
....
- !include: './maxscale_jobs/include/test_branch.yaml'
- !include: './maxscale_jobs/include/slave.yaml'
scm:
- git:
# TODO parametrize this url
url: https://github.com/mariadb-corporation/maxscale-system-test.git
branches:
- $test_branch
builders:
- !include: './maxscale_jobs/include/build_parser/create_env_vars.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_initial_env.yaml'
- !include: './maxscale_jobs/include/build_parser/run_test_and_collect.yaml'
- !include: './maxscale_jobs/include/build_parser/parse_build_log.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_build_results.yaml'
- !include: './maxscale_jobs/include/build_parser/create_env_coredumps.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_coredumps_var.yaml'
- !include: './maxscale_jobs/include/build_parser/write_build_results.yaml'
publishers:
- !include: './maxscale_jobs/include/build_parser_mail_subject_with_name.yaml'
- !include: './maxscale_jobs/include/call_cleanup.yaml'
wrappers:
- !include: './maxscale_jobs/include/workspace-cleanup.yaml'
- !include: './maxscale_jobs/include/timeout.yaml'
concurrent: true
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_all.yaml'
description: 'Virtual machine OS (name of Vagrant box)'
- centos_7_libvirt
- centos_6_libvirt
- centos_5_libvirt
- ubuntu_wily_libvirt
- ubuntu_wily_aws
…
- suse_13_libvirt
boxes_all_incl.yaml
boxes_all_incl.yaml
shell:
'$HOME/build-
scripts/test/run_test.sh | tee
$WORKSPACE/build_log_$BUI
LD_ID; echo ${PIPESTATUS[0]}
> result_$BUILD_ID'
boxes_all_incl.yaml
email-ext:
recipients: !include-raw: "./maxscale_jobs/include/mail_recipients.yaml"
reply-to: $DEFAULT_REPLYTO
content-type: default
subject: $DEFAULT_SUBJECT ($name)
body: !include-raw: "./maxscale_jobs/include/build_parser_email_body"
attach-build-log: false
...
boxes_all_incl.yaml

10. Problems: !include
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_all.yaml'
description: 'Virtual machine OS (name of Vagrant box)'
boxes_all.yaml
!include: './maxscale_jobs/include/boxes_libvirt.yaml'
!include: './maxscale_jobs/include/boxes_aws.yaml'
boxes_libvirt.yaml
- centos_7_libvirt
- ubuntu_trusty_libvirt
- debian_jessie_libvirt
- suse_13_libvirt
boxes_aws.yaml
- ubuntu_wily_aws
- centos_7_aws
- sles_12_aws
- rhel_7_aws
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_libvirt.yaml'
description: 'Virtual machine OS (name of Vagrant box)'
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_aws.yaml'
description: 'Virtual machine OS (name of Vagrant box)'

11. Problems: !include
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_all.yaml'
description: 'Virtual machine OS (name of Vagrant box)'
boxes_all.yaml
- centos_7_libvirt
- ubuntu_trusty_libvirt
- debian_jessie_libvirt
- suse_13_libvirt
- ubuntu_wily_aws
- centos_7_aws
- sles_12_aws
- rhel_7_aws

12. Problems: !include
- job:
name: build
description: 'This job builds Maxscale'
parameters:
- !include: './maxscale_jobs/include/value.yaml'
- !include: './maxscale_jobs/include/source.yaml'
- !include: './maxscale_jobs/include/boxes_all_incl.yaml'
- !include: './maxscale_jobs/include/target.yaml'
- !include: './maxscale_jobs/include/cmake_flags.yaml'
- !include: './maxscale_jobs/include/do_not_destroy_vm.yaml'
- !include: './maxscale_jobs/include/build_experimental.yaml'
- !include: './maxscale_jobs/include/run_upgrade_test.yaml'
- !include: './maxscale_jobs/include/old_target.yaml'
- !include: './maxscale_jobs/include/cnf_file.yaml'
- !include: './maxscale_jobs/include/maxadmin_command.yaml'
- !include: './maxscale_jobs/include/slave.yaml'
scm:
- git:
url: !include: './maxscale_jobs/include/maxscale_repo.yaml'
branches:
- 'develop'
builders:
- shell:
'$HOME/build-scripts/prepare_and_build.sh'
- shell:
'export new_target=$target; $HOME/build-
build.yaml
- job:
name: build_all
description: 'This job builds Maxscale'
project-type: matrix
execution-strategy:
sequential: false
axes:
- axis:
type: user-defined
name: box
values:
!include: './maxscale_jobs/include/boxes.yaml'
parameters:
- !include: './maxscale_jobs/include/value.yaml'
- !include: './maxscale_jobs/include/source.yaml'
- !include: './maxscale_jobs/include/target.yaml'
- !include: './maxscale_jobs/include/cmake_flags.yaml'
- !include: './maxscale_jobs/include/do_not_destroy_vm.yaml'
- !include: './maxscale_jobs/include/build_experimental.yaml'
- !include: './maxscale_jobs/include/run_upgrade_test.yaml'
- !include: './maxscale_jobs/include/old_target.yaml'
- !include: './maxscale_jobs/include/cnf_file.yaml'
- !include: './maxscale_jobs/include/maxadmin_command.yaml'
- !include: './maxscale_jobs/include/slave.yaml'
scm:
- git:
url: !include './maxscale_jobs/include/maxscale_repo.yaml'
branches:
- 'develop'
builders:
- !include: './maxscale_jobs/include/build_all/call_build.yaml'
….
build_all.yaml

13. Problems
• No parametrized !include
• a bit tricky to put right number of spaces in the beginning of every line (YAML problem)
• can not delete job automatically (rename case)
• Jenkins plugins have to be installed manually
• Views have to be configured manually
Jenkins CLI can solve it!

14. Problem number one: VM management
• obvious idea: “Create VM” job – run it 9 times (Matrix job)
– e.g. Amazon cloud: 8 of 10 attempts to create 9 VM fail! (out of our control)
– VM creation/provision restart management
– performance vs robustness problem
– too complex Vagrantfile
• so, Jenkins can not help! Own tool created: wrapper over Vagrant – MDBCI
https://github.com/mariadb-corporation/mdbci
• Jenkins only triggers 'mdbci up'

15. Problem number one: VM management
• “Setup Environment” job leaves running VMs: job can got stuck
– VM provider tool got stuck (timeouts?)
– ssh got stuck (started daemon, output from daemon blocks ssh, etc)
• Matrix job cancellation: leaves of lot of artifacts running
– our sub-jobs cleaning script
https://github.com/mariadb-corporation/maxscale-jenkins-jobs/tree/master/maxscale_jo
bs/include/parametrized_triggerred_subjobs_killer
– no easy solution for VMs: need to use VM provider tools (sometimes manually)

16. Problem number two: run test, report result
• Jenkins: all “run_test” jobs are the
same in the list
• any search engine in Jenkins?
• our solution:
– report email
– test script puts results into …
MariaDB, developers query DB
by writing SQL manually

17. Problem number two: run test, report result
• “run_test” throttle: test run is long, several runs on one host can cause trouble
• “Throttle Concurrent Builds Plugin” (build_all and run_test_matrix examples)
– we want: e.g. 5 'build' running on the node, one onle 'build_all' on the node, but several
'build_all' can be running on several nodes – i.e. different settings form 'build_all' and
sub-jobs ('build') – plugin can't do it (or can it?)
– matrix-configs: true – need to use categories from global Jenkins config (does not work if
throttle parameters defined in the job definition – plugin bug? JJB bug?)
– more flexibility wanted: throttle parameters should depend on 'run_test' parameters

18. Problem number three: server state monitoring
• monitoring jobs triggered by timer
• email warnings

19. Full automatic deployment
(our tool deployment out of scope now)
• load config
• install plugins
• load jobs
• configure views
Jenkins CLI
Ooops!
Jenkins Remote
CLI Vulnerability
Our Jenkins master was hacked,
bitcoin mining was running on it
only CLI via SSH is enabled now,
Jenkins bind to local host (use SSH tunnel)

20. Our additions
• simple wrapper to load jobs https://github.com/OSLL/jjg.git
– adds timestamps and git commit ID to job description
– easier way to manage Jenkins creds
– manage Views ('description' contains reference to 'View' to put the job)
• Job Visualizer (not ready yet, but working)
https://github.com/OSLL/jenkins_job_builder_visualization
Authors:
Evgeny Bovykin rebovykin@gmail.com
Mark Zaslavskiy mark.zaslavskiy@gmail.com

21. Job Visualizer: Call Graph
●
Do you call all the jobs
correctly?
●
Do you have cyclic
dependencies?

22. Job Visualizer: Include Graph
If you change one
file, which jobs you
can break?

23. Plans
• full automatic deployment
• move Jenkins to VM or container
• try JobDSL (via JJB?)
• BuildBot is still in the view
• results visualization: plugin? 3rd
party tool? something own?

24. MariaDB Maxscale
Continuous Integration
Open Source
and Linux Lab
OSLL
in cooperation with
Open Source
and Linux Lab
Open Innovations Association FRUCT
and
http://fruct.org/
https://osllblog.wordpress.com/
https://mariadb.com/products/mariadb-maxscale
timofey.turenko@mariadb.com
Mark Zaslavskiy <mark.zaslavskiy@gmail.com>
Sergey Balandin <Sergey.Balandin@fruct.org>