Successfully reported this slideshow.
Your SlideShare is downloading. ×

How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions

Loading in …3

Check these out next

1 of 30 Ad

More Related Content

Slideshows for you (17)

Similar to How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions (20)


More from Otto Kekäläinen (20)

Recently uploaded (20)


How MariaDB packaging uses Salsa-CI to ensure smooth upgrades and avoid regressions

  1. 1. How MariaDB in Debian uses Salsa-CI/Gitlab-CI Ensure smooth upgrades and avoid regressions In Miniauditório at Tue 23rd July 2019 14:30-15:15 Follow @ottokekalainen to get link to slides!
  2. 2. Making use of new Debian infrastructure ● Debian’s own Gitlab instance ● First prototype in 2017, rolled-out big scale in 2018 ● See docs at ● Run by Debian Salsa-Admin team ● MariaDB/MySQL packaging repositories moved from to in January 2018 Salsa-CI ● Debian packaging specific Gitlab-CI pipeline ● First version in summer 2018, very active development all the time ● See README at ● Run by Debian Salsa-CI team (I am a former member of it) ● Repositories for mariadb-10.1, mariadb-connector-c, galera-3 adopted it in August 2018
  3. 3. Salsa-CI is amazing If you are a Debian Developer, start using it now! Ensures that on every commit/push: ● Sources build and Debian packages are generated ● blhc: Build logs don’t have errors ● reprotest: Builds are reproducible (no randomness) ● autopkgtests: binaries can run ● lintian: static code analysis ● piuparts: packages install, uninstall and upgrade
  4. 4. ● Stop wasting your time on stupid mistakes and unnecessary prepare-upload-wait-check cycles. ● Stop wasting Debian archive resources with excess uploads. ● Give confidence to your contributors who can run CI on their merge requests. ● Give yourself a peace of mind, reduce stress and free up time to contribute more to Debian! ● Go from 10x engineer to 20x engineer!
  5. 5. Example: debian/gitlab-ci.yml contents in galera-3 Branch: master include: - - Branch: buster include: - - variables: RELEASE: 'buster'
  6. 6. Example: check Gitlab-CI config path and runner settings
  7. 7. Example: Gitlab-CI offers real-time logs, graphs, artifacts download… Everything you need!
  8. 8. Example: Commit history and Merge Request statuses
  9. 9. Example: Notification email
  10. 10. Salsa-CI is very easy for a simple package See also Augustin Henze’s talk:
  11. 11. ..but MariaDB far from simple
  12. 12. 13 million lines of code...
  13. 13. One source package generates 25 binary packages... libmariadb-dev libmariadbclient-dev libmariadb-dev-compat libmariadb3 libmariadbd19 libmariadbd-dev mariadb-common mariadb-client-core-10.3 mariadb-client-10.3 mariadb-server-core-10.3 mariadb-server-10.3 mariadb-server mariadb-client mariadb-backup mariadb-plugin-connect mariadb-plugin-rocksdb mariadb-plugin-oqgraph mariadb-plugin-tokudb mariadb-plugin-mroonga mariadb-plugin-spider mariadb-plugin-gssapi-server mariadb-plugin-gssapi-client mariadb-plugin-cracklib-password-check mariadb-test mariadb-test-data
  14. 14. hon=names&keywords=mariadb-server chon=names&keywords=mariadb-server Multiple upstream versions in maintenance in parallel in two distros each with multiple releases...
  15. 15. The configuration and data needs to persist and work over each upgrade... Default configuration /etc/mysql/ /etc/mysql/conf.d /etc/mysql/conf.d/mysql.cnf /etc/mysql/conf.d/mysqldump.cnf /etc/mysql/mariadb.conf.d /etc/mysql/mariadb.conf.d/50-client.cnf /etc/mysql/mariadb.conf.d/50-mysql-clients.cnf /etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf /etc/mysql/mariadb.conf.d/50-server.cnf /etc/mysql/my.cnf.fallback /etc/mysql/my.cnf /etc/mysql/mariadb.cnf /etc/mysql/debian-start /etc/mysql/debian.cnf Default data /var/lib/mysql/debian-10.1.flag /var/lib/mysql/mysql /var/lib/mysql/mysql/db.frm /var/lib/mysql/mysql/db.MYI /var/lib/mysql/mysql/db.MYD /var/lib/mysql/mysql/host.frm /var/lib/mysql/mysql/host.MYI /var/lib/mysql/mysql/host.MYD /var/lib/mysql/mysql/user.MYI /var/lib/mysql/mysql/user.MYD … /var/lib/mysql/mysql/user.frm /var/lib/mysql/aria_log_control /var/lib/mysql/aria_log.00000001 /var/lib/mysql/ibdata1 /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile0 /var/lib/mysql/performance_schema /var/lib/mysql/performance_schema/db.opt
  16. 16. Lot’s of direct users: 30 % of Debian popcon submitters
  17. 17. Hundreds of indirect users and dependencies that need to continue working... Packages that depend on MariaDB server or client directly apt-cache rdepends 'default-mysql-s*' 'mariadb-s*' | wc -l 96 apt-cache rdepends 'default-mysql-c*' 'mariadb-c*' | wc -l 73 Packages that depend on MariaDB client libraries apt-cache rdepends 'default-libmysql*' 'libmariadb*' | wc -l 194
  18. 18. MariaDB packaging custom Gitlab-CI pipeline
  19. 19. 18 jobs in 5 stages
  20. 20. Stage 1: build Build in target release plus one older (ensuring backporting is still possible).
  21. 21. Stage 2: test quality of built packages Quickly detect if the software malfunctions in an obvious way.
  22. 22. Stage 3: upgrade from current version in same release E.g. what will happen to users when the built version is uploaded to unstable (or stable if maintenance branch).
  23. 23. Stage 4: upgrade from the version in (old) stable releases E.g. what will happen when users run dist-upgrade from the previous stable or oldstable to the version being built now.
  24. 24. Stage 5: additional tests E.g. what will happen if users upgrade from MySQL 5.7 (which would not normally happen in a Debian dist-upgrade). Also lesser priority tests that are allowed to fail.
  25. 25. libmysql* to libmariadb* upgrade # Install all libmysql* available in Debian unstable - apt-get install -y pkg-config libmysqld-dev libmysqlclient-dev - pkg-config --list-all - pkg-config --cflags mysqlclient # mysqlclient.pc from original package - apt-get install -y ./libmariadb3_*.deb ./mariadb-common_*.deb - pkg-config --list-all - apt-get install -y ./libmariadb-dev_*.deb - pkg-config --list-all ... - apt-get install -y default-libmysqlclient-dev default-libmysqld-dev - ldconfig -p # | grep -e mariadb -e mysql - pkg-config --list-all - pkg-config --cflags --libs mysqlclient - pkg-config --cflags --libs libmariadb - pkg-config --cflags --libs mariadb
  26. 26. build mariadbclient consumer Python-MySQLdb - apt-get install -y pkg-config ./libmariadb-dev*.deb ./libmariadb3_*.deb ./mariadb-common*.deb - pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with - apt-get install -y python3-pip - pip3 install mysqlclient # Compiles module against libmysqlclient - apt-get purge -y libmariadb-dev # Not needed for run-time - python3 -c "import MySQLdb; print(MySQLdb.get_client_info())" Ensures we have a chance to notice if other software that used (legacy) libmysqlclient and are now automatically using libmariadb3 still works. Priceless: This caught upstream bugs what otherwise would have gone unnoticed for a long time since the “damage” is so subtle and invisible.
  27. 27. Explore it yourself! Gitlab, Salsa-CI and MariaDB evolves constantly
  28. 28. Challenges and peculiarities ● Custom gitlab-ci.yml has been developed now for a little over a year and grown to 550+ lines. ● Tries to cover all relevant install/upgrade scenarios. ● Docker images don’t have systemd running by default, must activate with ○ sed -i "s/101/0/g" -i /usr/sbin/policy-rc.d ● Build time max 2h ○ Slowness was tackled with ccache (and also upstreamed to Salsa-CI ● Build log max 4 MB ○ build command runs with ■ | tail -n 10000 ○ before build starts a a progress indicator is spawned ■ while true; do sleep 600; echo "10 minutes passed" >&2; done &
  29. 29. Next additions? ● Fix Jessie to latest upgrade test that regressied in early May 2019 with: ○ systemd-udevdstart-stop-daemon: unrecognized option '--notify-await' ○ Bug in Jessie or Jessie Docker image, not MariaDB ● Making RocksDB and TokuDB plugins build reproducibly ○ Can you help? ● Extend testing of MariaDB plugins and high-level features not covered by upstream MariaDB test suite ● Manipulate config files and database contents before starting upgrade to test how the maintainer scripts can recover from customized situations and heterogeneous environments ● Custom runner guarantee it is fast?
  30. 30. Thank you! @ottokekalainen Reminder of next session: Salsa and Salsa-CI BoF Room: Miniauditório Time: Jul 26 (Fri), 14:30