MariaDB, MySQL and Ansible: automating database infrastructures
1.
MariaDB, MySQL and Ansible:
automating database infrastructures
Federico Razzoli
2.
$ whoami
Hi, I’m Federico Razzoli from Vettabase Ltd
Database consultant, open source supporter,
long time MariaDB and MySQL user
● vettabase.com
● Federico-Razzoli.com
3.
This talk
● General Ansible concepts
● Opinionated code examples, automating MariaDB/MySQL
● Good and bad practices
9.
Ansible hosts
[pmm]
pmm-1 ansible_host=123.123.123.123
[mariadb_main]
mariadb-main-1 ansible_host=123.0.0.1
mariadb-main-2 ansible_host=123.0.0.2
mariadb-main-3 ansible_host=123.0.0.3
[mariadb_main:vars]
pmm_server_host=123.123.123.123
pmm_server_port=80
● Sometimes we cannot use meaningful hostnames
10.
Hierarchical groups
[mariadb_master]
mariadb-master-1
mariadb-master-2
[mariadb_replica]
mariadb-replica-1
mariadb-replica-2
[mariadb:children]
mariadb_master
mariadb_replica
● We have different groups for masters and replicas, and a supergroup that
includes them both
11.
Groups goals
● You assign each group some roles
● You can deploy one or more group
14.
Play example
---
- hosts: mariadb_main
roles:
- linux-for-mysql
- mariadb
- hosts: mysql_main
roles:
- linux-for-mysql
- mysql
● You assign roles to groups (or individual hosts)
15.
Applying all roles to a certain host
Call Ansible specifying an inventory and a play:
% ansible-playbook -i staging -l mariadb_main main.yml
16.
Play example
Here is where it can be convenient to have multiple inventories:
% ansible-playbook -i staging -l mariadb_main main.yml
% ansible-playbook -i production -l mariadb_main main.yml
18.
Example
- name: Create script dir
file:
name: "{{ script_dir }}"
owner: root
group: root
mode: 755
state: directory
● This is a task, but the term is misleading
● A role is a list of tasks
● The name appears on screen
● "file" is the module
● script_dir is a variable
20.
Notes on variables
● The same variable can be used by multiple roles
● Roles from Ansible Galaxy can’t use this opportunity
● This is a good reason to make your own roles
● Document what each variable does in README.md
21.
Copying a configuration file
- name: Copy my.cnf
copy:
src: ./files/my.cnf
dest: "/etc/mysql/{{ inventory_hostname }}.cnf"
● my.cnf is copied to /etc/mysql
22.
Using a template
- name: Copy my.cnf
template:
src: ./templates/my.cnf .j2
dest: /etc/mysql/my.cnf
● This time my.cnf is a Jinja 2 template
23.
Template example
[mysqld]
user = mysql
datadir = {{ mysql_data_dir }}
innodb_buffer_pool_size = {{ innodb_buffer_pool_size }}
● This time my.cnf is a Jinja 2 template
24.
Problems
● If we want to set a variable, we should be able to add it in one place
○ host_vars or group_vars or role's defaults
● We can make a template more dynamic
25.
Dynamic template example
In group_vars we create a list of dictionaries:
(arrays of objects, if you prefer)
group_mysql_variables:
- { name: 'innodb_buffer_pool_size', value: '50G' }
- { name: 'innodb_log_file_size', value: '50G' }
In host_vars we optionally create another list:
host_mysql_variables:
- { name: 'innodb_buffer_pool_size', value: '80G' }
26.
Dynamic template example
We loop over both the lists in the template:
{% for var in group_mysql_variables %}
{{ var.name }} = {{ var.value }}
{% endfor %}
{% if host_mysql_variables is defined %}
{% for var in host_mysql_variables %}
{{ var.name }} = {{ var.value }}
{% endfor %}
{% endif %}
27.
Notes
● We cannot use the same list in group_vars and host_vars, because the
one in host_vars would overwrite the whole list
● Some variables may appear twice in the configuration file
● This is fine: the last occurrence of a variable will override the previous one
28.
Another use for lists
Hosts of a cluster can be a list too:
# print the IPs separated by commas
wsrep_cluster_address = gcomm://{{ private_ips|join(',') }}
# first node is the donor
wsrep_sst_donor = {{ cluster_hosts[0].node_name }}
# Only the first node imports a backup when the cluster is created
- name: Import backup
...
when: cluster_hosts[0] == hostvars[inventory_hostname]['ansible_default_ipv4']['address']
29.
Validate variables
Validate variables for complex roles:
- name: Validate cluster size
assert:
that: cluster_hosts|length > 0
fail_msg: cluster_hosts must contain at least 1 element
success_msg: "cluster_hosts size: {{ cluster_hosts|length }}"
- name: Validate cluster IPs
assert:
that: "'{{ item.public_ip }}'|ipaddr"
fail_msg: "{{ item.public_ip }} is not a valid IP"
success_msg: "{{ item.public_ip }}: OK"
with_items: "{{ cluster_hosts }}"
31.
Conditional tasks
● Certain tasks should only be run for certain servers:
- name: Make server read only
mysql_variables:
variable: read_only
value: 1
when:
is_replica is sameas true
● Sometimes this avoids the need to create multiple roles which are very similar
● Instead of creating mariadb and mariadb_replica roles, we may create
mariadb only, with some conditional tasks
○ Sometimes it makes sense, sometimes it doesn't
32.
Conditional tasks
● We can group optional tasks into separate files:
- name: Make server read only
include: replica.yml
when:
is_replica is sameas true
33.
Idempotency of Tasks
● Most modules are idempotent
● This means that you can run a task like this multiple times safely:
- name: Start MariaDB
service:
state: started
● If the service is already running, nothing happens
34.
Idempotency of Tasks
● Sometimes you make non-idempotent tasks on purpose:
- Copy my.cnf
...
- name: Restart MariaDB
service:
state: restarted
● If the service is already running, nothing happens
35.
Idempotency of Tasks
● Sometimes you'd like a task to be idempotent, but it can't be:
- name: Run a system command
shell: >
rm "{{ mysql_data_dir }}"/*
- name: Run a script
shell: >
"{{ mysql_script_dir }}"/my_script.py
- name: Run some SQL
mysql_query:
query: CREATE OR REPLACE TABLE db.table ( ... );
● Ansible doesn't understand system commands, scripts or queries, so it has not
way to check if they were already run. It runs them every time.
36.
Tags
● Ansible supports tags
● Each task can be assigned a list of tags:
- name: (Re)create users
...
- name: Update timezone info
tags: [ tzinfo-update ]
...
- name: Do something else
...
ansible-playbook -i production_mysql --tag tzinfo-update production.yml
37.
Tags
● Typical tasks I want to have:
- name: Do something
- name: Copy my.cnf
tags: [ mysql-config- static-update ]
- name: Restart MySQL
tags: [ mysql-config- static-update ]
- name: Set variables at runtime
tags: [ mysql-config- dynamic-update ]
- name: Do something else
38.
Tags
● So there is a way to selectively run certain tasks
● But what if we want to selectively exclude certain tasks?
● There is not built-in way, but we can do something like this:
- name: Do something
- name: Restart MySQL
service:
state: restarted
when: {{ mysql_restart }} is defined and {{ mysql_restart }} is sameas true
- name: Do something else mething else
ansible-playbook -i production_mysql
--tag mysql-config-dynamic-update -e 'mysql_no_restart=1' production.yml
41.
Example
CREATE OR REPLACE TABLE ticket (
id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
state ENUM('OPEN', 'VERIFIED', 'FIXED', 'INVALID') NOT NULL
DEFAULT 'OPEN',
summary VARCHAR(200) NOT NULL,
description TEXT NOT NULL
)
ENGINE InnoDB
;
● We want to start to track changes to bugs over time