How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions
How MariaDB in Debian uses
Ensure smooth upgrades and avoid regressions
In Miniauditório at Tue 23rd July 2019 14:30-15:15
Making use of new Debian infrastructure
● Debian’s own Gitlab instance
● First prototype in 2017,
rolled-out big scale in 2018
● See docs at https://wiki.debian.org/Salsa
● Run by Debian Salsa-Admin team
● MariaDB/MySQL packaging
repositories moved from
salsa.debian.org in January
● Debian packaging specific
● First version in summer 2018,
very active development all the
● See README at
● Run by Debian Salsa-CI team (I
am a former member of it)
● Repositories for mariadb-10.1,
adopted it in August 2018
If you are a
start using it now!
Ensures that on every commit/push:
● Sources build and Debian
packages are generated
● blhc: Build logs don’t have
● reprotest: Builds are
reproducible (no randomness)
● autopkgtests: binaries can run
● lintian: static code analysis
● piuparts: packages install,
uninstall and upgrade
● Stop wasting your time on stupid mistakes and
unnecessary prepare-upload-wait-check cycles.
● Stop wasting Debian archive resources with
● Give conﬁdence to your contributors who can
run CI on their merge requests.
● Give yourself a peace of mind, reduce stress and
free up time to contribute more to Debian!
● Go from 10x engineer to 20x engineer!
Multiple upstream versions in maintenance in parallel in
two distros each with multiple releases...
The conﬁguration and data needs to persist and work over each upgrade...
Lot’s of direct users: 30 % of Debian popcon submitters
Hundreds of indirect users and dependencies that need to continue working...
Packages that depend on MariaDB server or client directly
apt-cache rdepends 'default-mysql-s*' 'mariadb-s*' | wc -l
apt-cache rdepends 'default-mysql-c*' 'mariadb-c*' | wc -l
Packages that depend on MariaDB client libraries
apt-cache rdepends 'default-libmysql*' 'libmariadb*' | wc -l
build mariadbclient consumer Python-MySQLdb
- apt-get install -y pkg-config ./libmariadb-dev*.deb ./libmariadb3_*.deb
- pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with
- apt-get install -y python3-pip
- pip3 install mysqlclient # Compiles module against libmysqlclient
- apt-get purge -y libmariadb-dev # Not needed for run-time
- python3 -c "import MySQLdb; print(MySQLdb.get_client_info())"
Ensures we have a chance to notice if other software that used (legacy)
libmysqlclient and are now automatically using libmariadb3 still works.
Priceless: This caught upstream bugs what otherwise would have gone
unnoticed for a long time since the “damage” is so subtle and invisible.
Explore it yourself!
Gitlab, Salsa-CI and MariaDB evolves constantly
Challenges and peculiarities
● Custom gitlab-ci.yml has been
developed now for a little over
a year and grown to 550+ lines.
● Tries to cover all relevant
● Docker images don’t have
systemd running by default,
must activate with
○ sed -i "s/101/0/g" -i
● Build time max 2h
○ Slowness was tackled with
ccache (and also upstreamed
● Build log max 4 MB
○ build command runs with
■ | tail -n 10000
○ before build starts a a
progress indicator is spawned
■ while true; do sleep 600;
echo "10 minutes passed"
>&2; done &
● Fix Jessie to latest upgrade
test that regressied in early
May 2019 with:
○ Bug in Jessie or Jessie Docker
image, not MariaDB
● Making RocksDB and TokuDB
plugins build reproducibly
○ Can you help?
● Extend testing of MariaDB
plugins and high-level features
not covered by upstream MariaDB
● Manipulate config files and
database contents before
starting upgrade to test how
the maintainer scripts can
recover from customized
situations and heterogeneous
● Custom runner guarantee it is
Reminder of next session:
Salsa and Salsa-CI BoF
Time: Jul 26 (Fri), 14:30