The presentation describes CI environment for our product - Maxscale - database proxy server. To test such product we need a setup that consists of tens of machines: locally hosted virtual machines as well as machines from different clouds. All our Jenkins jobs are implemented in the form of Jenkins Job Builder code. Presentations also tells about our tool to manage virtual machines (wrapper over Vagrant)- MDBCI.

1. DB proxy server test:
run tests on tens of
virtual machines with
Jenkins, Vagrant and
our Vagrant wrapper
MDBCI
Open Source
and Linux Lab
OSLL

2. • Test task
• How to prepare test environment
• Upper layer: CI tool
• Our additions

3. Our setup: product – DB proxy server Maxscale
• Product to build and test: Maxscale – database proxy
• lots of plugins (routers, filters, protocols, etc, etc, etc) = many configurations
• Integration tests, performance tests, “scientific experiments”
• Test environment:
– at least 9 VM (1 Maxscale server, 4 Master/Slave servers, 4-servers Galera cluster)
– libvirt/qemu
– some tests are heavy – powerful AWS virtual machines, but their life is short
– need to configure backend for every test run
– need to carefully clean up

4. Our setup: why is it so complex to run test?
• dependency on VM provider: it can easily FAIL
• 12 Linux distributions, 4 major MariaDB version, 3 major MySQL version, a lot of
Maxscale plugins – any combination should be testable
INFO: ==> galera_003: Starting domain.
ERROR: There was an error talking to Libvirt.
The error message is shown
ERROR: below:
ERROR:
ERROR: Call to virDomainCreateWithFlags
failed: internal error: process exited while
connecting to monitor: warning: host doesn't
support requested feature: CPUID.01H:EDX.ds
[bit 21]
ERROR: warning: host doesn't support
requested feature: CPUID.01H:EDX.acpi [bit 22]
ERROR: warning: host doesn't support
requested feature: CPUID.01H:EDX.ht [bit 28]
...

5. What does it mean for test management system?
• Several VMs are needed for every test round
• Lots of parameters for every test job
• Cascades of jobs: e.g. prepare backend, install Maxscale, run test, cleanup
• Error processing (e.g. backend VMs preparation failure) – how to do cleanup in this
case?
• Matrix jobs: test a lot of combinations of distros, backend MariaDB version, backend
parameters
• very dynamic setup

6. VM management: Vagrant
• libvirt, AWS, Docker, LXC, DigitalOcean, OpenStack, ...
• write Vagrantfile and 'vagrant up' ...
• but
– it can fail: restart logic is needed
– Vagrantfile is not the simplest thing to write
• Provisioning:
– lots of tools, let's just take simplest – Puppet
– many version of MariaDB/MySQL – need a tool to configure paths to binary repos

7. MDBCI
• simple human-readable descriptions of VMs
• VM up and provisioning processes monitoring
and restart logic
"node_000" :
{
"hostname" : "node0",
"box" : "centos_7_libvirt",
"product" : {
"name": "mariadb",
"version": "10.0",
"cnf_template" : "server1.cnf",
}
},
"node_001" :
{
"hostname" : "node1",
"box" : "ubuntu_xenial_libvirt",
"product" : {
"name": "mariadb",
"version": "10.2",
"cnf_template" : "server2.cnf",
}
},
"maxscale" :
{
"hostname" : "maxscale",
"box" : "ubuntu_xenial_libvirt",
"product" : {
"name": "maxscale",
"version": "2.1.7"
}
}
simple_test_config.json
Infrastructure as Code!

8. MDBCI
• a set of binary repositories descriptions for
MariaDB, MySQL, Maxscale
• a set of Vagrant boxes descriptions
[{
"product": "maxscale",
"version": "default",
"repo": "https://downloads.mariadb.com/MaxScale//2.1.6/centos/7/$basearch",
"repo_key": "https://downloads.mariadb.com/MaxScale/MariaDB-MaxScale-GPG-KEY",
"platform": "centos",
"platform_version": "7"
}]
"ubuntu_xenial_libvirt": {
"provider": "libvirt",
"box": "tnmt/xenial-alpha1",
"platform": "ubuntu",
"platform_version": "xenial"
},
"centos_7_libvirt": {
"provider": "libvirt",
"box": "centos/7",
"platform": "centos",
"platform_version": "7"
},
….
boxes_libvirt.json
"centos_7_aws_large" : {
"provider": "aws",
"ami": "ami-1c2e8b6b",
"user": "ec2-user",
"default_instance_type": "m1.large",
"platform": "centos",
"platform_version": "7"
},
"rhel_7_aws" : {
"provider": "aws",
"ami": "ami-25158352",
"user": "ec2-user",
"default_instance_type": "m3.medium",
"platform": "rhel",
"platform_version": "7"
},
….
boxes_aws.json
repository-config/maxscale-2.1.7-centos_7.json

9. MDBCI vs Vagrantfile
• separate
– high-level test setup description
– VM box description
– Repositories description
– Chef recipes
– Credentials
• hide
– low-level provider-dependent stuff
### Import AWS Provider access config ###
require 'yaml'
aws_config = YAML.load_file('/home/vagrant/mdbci/aws-config.yml')['aws']
## of import AWS Provider access config
### Vagrant configuration block ###
####################################
Vagrant.configure(2) do |config|
config.omnibus.chef_version = '12.9.38'
### AWS Provider config block ###
###########################################################
config.vm.box = "dummy"
config.vm.provider :aws do |aws, override|
aws.keypair_name = "max-tst-02.mariadb.com_daily_test-382-2.1_1505122513"
aws.region = aws_config["region"]
aws.security_groups = aws_config["security_groups"]
aws.user_data = aws_config["user_data"]
override.ssh.private_key_path = "/home/vagrant/vms/daily_test-382-2.1/maxscale.pem"
override.nfs.functional = false
aws.aws_profile = "mdbci"
end ## of AWS Provider config block
# --> Begin definition for machine: node_000
config.vm.define :node_000 do |node_000|
node_000.vm.provider :aws do |aws,override|
aws.ami = "ami-1c2e8b6b"
aws.tags = { "hostname" => "max-tst-02.mariadb.com", "username" => "vagrant", "full_config_path
aws.instance_type = "t1.micro"
override.ssh.username = "ec2-user"
end
node_000.vm.synced_folder "~/build-scripts/test-setup-scripts/cnf", "/home/vagrant/cnf_templates", type: "r
node_000.vm.provision 'shell', inline: 'curl -L https://omnitruck.chef.io/install.sh | sudo bash -s -- -v 12.9.38'
node_000.vm.provision "chef_solo" do |chef|
chef.cookbooks_path = "/home/vagrant/mdbci/recipes/cookbooks/"
chef.roles_path = "."
chef.add_role "node_000"
chef.synced_folder_type = "rsync"
end #<-- end of chef binding
end # <-- End AWS definition for machine: node_000

10. Upper layer: BuildBot, Jenkins
• Buildbot
– has been in use from very beginning (since middle of 90th
)
– do we want different tools for every our product?
– there are many useful scripts for BuildBot
• Jenkins
– popular tool
– easy to start
– let's try for Maxscale

11. Jenkins: Job Builder
• A tool from OpenStack developers https://docs.openstack.org/infra/jenkins-job-builder/
• YAML
• Pipelines did not exist when we started
• our jobs: https://github.com/mariadb-corporation/maxscale-jenkins-jobs

12. What can we get from JJB?
• All our jobs (>60) are represented as a code (branches, forks, review, pull requests, …)
• Parameters values are in one place for all jobs (easy maintenance of parameters defaults
and lists)
• Automatic validation of jobs takes place at every push to jobs repo
• very easy jobs deployment
• at the beginning: app. 20 jobs, we spent 1 week to convert to JJB code (1.5 persons)

13. Our “run_test” job: thanks to Parametrized Trigger plugin
Build
Setup backend
Install Maxscale
Run tests
Extract logs, coredumps
cleanup VMs (or not)
Parse logs
Send report email
Put result to DB
Pipelines? may be...

14. - job:
name: run_test
description: 'This job perform integration testing of maxscale'
parameters:
- !include: './maxscale_jobs/include/boxes_all_incl.yaml'
- !include: './maxscale_jobs/include/products_incl.yaml'
- !include: './maxscale_jobs/include/versions_incl.yaml'
....
- !include: './maxscale_jobs/include/test_branch.yaml'
- !include: './maxscale_jobs/include/slave.yaml'
scm:
- git:
# TODO parametrize this url
url: https://github.com/mariadb-corporation/maxscale-system-test.git
branches:
- $test_branch
builders:
- !include: './maxscale_jobs/include/build_parser/create_env_vars.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_initial_env.yaml'
- !include: './maxscale_jobs/include/build_parser/run_test_and_collect.yaml'
- !include: './maxscale_jobs/include/build_parser/parse_build_log.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_build_results.yaml'
- !include: './maxscale_jobs/include/build_parser/create_env_coredumps.yaml'
- !include: './maxscale_jobs/include/build_parser/inject_coredumps_var.yaml'
- !include: './maxscale_jobs/include/build_parser/write_build_results.yaml'
publishers:
- !include: './maxscale_jobs/include/build_parser_mail_subject_with_name.yaml'
- !include: './maxscale_jobs/include/call_cleanup.yaml'
wrappers:
- !include: './maxscale_jobs/include/workspace-cleanup.yaml'
- !include: './maxscale_jobs/include/timeout.yaml'
concurrent: true
choice:
name: box
choices:
!include: './maxscale_jobs/include/boxes_all.yaml'
description: 'Virtual machine OS (name of Vagrant box)'
- centos_7_libvirt
- centos_6_libvirt
- centos_5_libvirt
- ubuntu_wily_libvirt
- ubuntu_wily_aws
…
- suse_13_libvirt
boxes_all_incl.yaml
boxes_all_incl.yaml
shell:
'$HOME/build-
scripts/test/run_test.sh | tee
$WORKSPACE/build_log_$BUI
LD_ID; echo ${PIPESTATUS[0]}
> result_$BUILD_ID'
boxes_all_incl.yaml
email-ext:
recipients: !include-raw: "./maxscale_jobs/include/mail_recipients.yaml"
reply-to: $DEFAULT_REPLYTO
content-type: default
subject: $DEFAULT_SUBJECT ($name)
body: !include-raw: "./maxscale_jobs/include/build_parser_email_body"
attach-build-log: false
...
boxes_all_incl.yaml

15. Jenkins: results visualization problem
• Jenkins: all “run_test” jobs are the
same in the list
• any search engine in Jenkins?
• our solution:
– report email
– test script puts results into …
MariaDB, developers query DB
by writing SQL manually

16. Our additions
• simple wrapper to load jobs https://github.com/OSLL/jjg.git
– adds timestamps and git commit ID to job description
– easier way to manage Jenkins creds
– manage Views ('description' contains reference to 'View' to put the job)
• Matrix job killer groovy script
https://github.com/mariadb-corporation/maxscale-jenkins-jobs/tree/master/maxscale_jobs/inc
• Job Visualizer (not ready yet, but working)
https://github.com/OSLL/jenkins_job_builder_visualization

17. Job Visualizer: Call Graph
●
Do you call all the jobs
correctly?
●
Do you have cyclic
dependencies?

18. Job Visualizer: Include Graph
If you change one file,
which jobs you can break?

19. Jenkins job description pre-processing
description: 'View: test. This job perform integration testing of maxscale'
description: 'View: build. This job builds Maxscale'

20. Plans
• full automatic deployment
• move Jenkins to VM or container
• try JobDSL (via JJB?)
• BuildBot is still in the view
• results visualization: plugin? 3rd
party tool? something own?

21. Results
• easy to create any number of VMs (described as easy JSON)
• MDBCI does all dirty job
• Jenkins is good for scheduling and logs storing
• setup works!
• results visualization is bad (Buildbot?)
• Pipelines or JobDSL – worth to try

22. MariaDB Maxscale
Continuous Integration
Open Source
and Linux Lab
OSLL
in cooperation with
Open Source
and Linux Lab
Open Innovations Association FRUCT
and
http://fruct.org/
https://osllblog.wordpress.com/
https://mariadb.com/products/mariadb-maxscale
timofey.turenko@mariadb.com
Mark Zaslavskiy <mark.zaslavskiy@gmail.com>
Sergey Balandin <Sergey.Balandin@fruct.org>