OSMC 2019 | Use Cloud services & features in your redundant Icinga2 Environment by Marcel Weinberg

Icinga2Icinga2
A highly available cloud setup deployed at DigitalOceanA highly available cloud setup deployed at DigitalOcean

Who am I? | Short tales from the past
Solution | Result - know your why!
incl. the database & Galera cluster configuration
Highlights and critical sections
Don't do the same mistakes as I did
Agenda
> Quick Introduction
> High level overview
> Installation of all components
> Configuration deep-dive
> Pitfalls & Experienced limitations

Who am I?
Marcel Weinberg
Living in Hamburg
Working at CoreMedia AG (since 09/2019)
Doing O(o)ps things for more then ten years now
With a great passion for
Automation, Monitoring, Open Source
& Mountainbiking
Quick Introduction
@winem_
winem

+ extensive documentation
+ well defined & tested DR* procedure
- single instance with many SPOFs
- limited redundancy on the uplink
Quick Introducation - Tales from the past
Previous Setup
It was a single instance in the server room:
* DR = Disaster Recovery

And we
still have
alarming!
High Level Overview
Solution
Result

High Level Overview
Solution
Result - Know your why
vs.

Installation Of All Components
Always refer to the public available documentation for up-to-date installation and upgrade
instructions.
> PHP7.3 - ppa:ondrej/php
> icinga2, icingacli, icingaweb2 - packages.icinga.com
apt install icinga2 icinga2-ido-mysql icingaweb2 icingacli vim-icinga2 vim-addon-manager php7.3-gd
> haproxy - ppa:vbernat/haproxy-1.9
> keepalived from the standard repositories

Always refer to the public available documentation for up-to-date installation and upgrade
instructions.
> PHP7.3 - ppa:ondrej/php
> icinga2, icingacli, icingaweb2 - packages.icinga.com
apt install icinga2 icinga2-ido-mysql icingaweb2 icingacli vim-icinga2 vim-addon-manager php7.3-gd
> haproxy - ppa:vbernat/haproxy-1.9
> keepalived from the standard repositories
keepalived_script - a dedicated user to run check and notification scripts:
sudo adduser --no-create-home --disabled-login keepalived_script

MariaDB
Secure your installation and don't start mariadb automatically:
sudo mysql_secure_installation
sudo systemctl disable mariadb
Link: https://downloads.mariadb.org/mariadb/repositories/

Installation of all components
MariaDB Con guration Files
Use mysqld --help --verbose to see the loaded configuration files and sections:
agent@m-do-ffm-galera-01:~$ mysql --help --verbose | head -n 10
mysql Ver 15.1 Distrib 10.4.8-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Usage: mysql [OPTIONS] [database]
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf
The following groups are read: mysql mariadb-client client client-server client-mariadb

Galera Cluster Con guration
/etc/mysql/mariadb.conf.d/galera.cnf
[mysqld]
bind-address=10.135.209.4
default_storage_engine=InnoDB
binlog_format=row
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
# Galera cluster configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://10.135.209.4,10.135.207.79,10.135.207.111"
wsrep_cluster_name="mariadb-galera-backend"
wsrep_slave_threads=8
wsrep_sst_method=rsync
# Cluster node configuration
wsrep_node_address="10.135.209.4"
wsrep_node_name="galera-01"
> Highlighted lines show those values that are unique per node.

Galera Cluster Con guration
/etc/mysql/mariadb.conf.d/galera.cnf
[mysqld]
bind-address=10.135.209.4
default_storage_engine=InnoDB
binlog_format=row
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
# Galera cluster configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://10.135.209.4,10.135.207.79,10.135.207.111"
wsrep_cluster_name="mariadb-galera-backend"
wsrep_slave_threads=8
wsrep_sst_method=rsync
# Cluster node configuration
wsrep_node_address="10.135.209.4"
wsrep_node_name="galera-01"
> Set wsrep_slave_threads to the number of CPUs * 2.
> innodb_flush_log_at_trx_commit = 0 means to write the caches to disk every second.

Start the first cluster node:
sudo galera_new_cluster
Start mariadb on the remaining nodes:
sudo service mariadb start
MariaDB Con guration Files |GaleraClusterCon guration
Bootstrap the cluster
Wait until all nodes joined the cluster and restart mariadb on the first node
MariaDB [(none)]> show global status where Variable_name in
('wsrep_cluster_size', 'wsrep_local_state_comment', 'wsrep_ready', 'wsrep_connected');
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| wsrep_local_state_comment | Synced |
| wsrep_cluster_size | 3 |
| wsrep_connected | ON |
| wsrep_ready | ON |
+---------------------------+--------+
4 rows in set (0.002 sec)

Database cluster, configuration & accounts
icinga2
CREATE DATABASE icinga2;
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON
icinga2.* TO 'icinga2'@'10.135.201.194' IDENTIFIED BY 'maa3EeHah0Ea';
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON
icinga2.* TO 'icinga2'@'10.135.221.140' IDENTIFIED BY 'maa3EeHah0Ea';
Copy the mysql.sql file to any of the galera nodes and import it:
mysql -u root icinga2 < /tmp/mysql.sql
Icingaweb2
CREATE DATABASE icingaweb2;
GRANT ALL ON icingaweb2.* TO icingaweb2@10.135.201.194 IDENTIFIED BY 'ievachiYie4ooz5a';
GRANT ALL ON icingaweb2.* TO icingaweb2@10.135.221.140 IDENTIFIED BY 'ievachiYie4ooz5a';
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, INDEX, EXECUTE, CREATE VIEW
ON icinga2.* TO 'icingaweb2'@'10.135.201.194';
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, INDEX, EXECUTE, CREATE VIEW
ON icinga2.* TO 'icingaweb2'@'10.135.221.140';

HAProxy
CREATE USER 'haproxy_checks'@'10.135.201.194';
CREATE USER 'haproxy_checks'@'10.135.221.140';

Keepalived
GRANT SELECT ON icingaweb2.* TO 'keepalived_check'@'10.135.221.140'
IDENTIFIED BY 'iiw8ahthe1Ch';
GRANT SELECT ON icingaweb2.* TO 'keepalived_check'@'10.135.221.249'
IDENTIFIED BY 'iiw8ahthe1Ch';
my.cnf file for the keepalived healthches from the icinga2 nodes:
sudo cat > /etc/keepalived/.my.cnf << EOF
[client]
host=127.0.0.1
password=iiw8ahthe1Ch
user=keepalived_check
EOF
Set proper permissions:
sudo chown keepalived_script:root /etc/keepalived/.my.cnf
sudo chmod 600 /etc/keepalived/.my.cnf

agent@m-do-ffm-plugins-01:~$ sudo icinga2 node setup
--master
--cn m-do-ffm-plugins-01.mydomain.com
--disable-confd
--accept-config
--accept-commands
--zone do-ffm-masters
--master
--disable-confd
--accept-config
--accept-commands
Deep-dive into the configuration
Icinga2 - Node Setup
Icinga2 configuration using the node setup utility

--master
--disable-confd
--accept-config
--accept-commands
--master
--disable-confd
--accept-config
--accept-commands
Icinga2 - Node Setup
Icinga2 configuration using the node setup utility
/etc/icinga2/constants.conf
[...]
const NodeName = "m-do-ffm-plugins-01.mydomain.com"
/* Our local zone name. */
const ZoneName = "do-ffm-masters"
/* Secret key for remote node tickets */
const TicketSalt = "689eb691e3d5cbf8e3a11fcb50089610"
[...]

Icinga2 - Zone & endpoint con guration
/etc/icinga2/zones.conf
object Endpoint "m-do-ffm-plugins-01.mydomain.com" {
host = "10.135.201.194"
}
object Endpoint "m-do-ffm-plugins-02.mydomain.com" {
host = "10.135.221.140"
}
object Zone "do-ffm-masters" {
endpoints = [ "m-do-ffm-plugins-01.mydomain.com", "m-do-ffm-plugins-02.mydomain.com" ]
}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}

Icinga2 - ido-mysql feature
The configuration is on both nodes the same. /etc/icinga2/features-available/ido-mysql.conf
library "db_ido_mysql"
object IdoMysqlConnection "ido-mysql" {
user = "icinga2",
password = "maa3EeHah0Ea",
host = "127.0.0.1",
database = "icinga2"
enable_ha = true
cleanup = {
contactnotifications_age = 30d
downtimehistory_age = 60d
statehistory_age = 60d
externalcommands_age = 2d
}
}
agent@m-do-ffm-plugins-01:~$ icinga2 feature enable ido-mysql
agent@m-do-ffm-plugins-01:~$ sudo icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite
influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql mainlog notification

Icinga2 - API users
/etc/icinga2/zones.d/do-ffm-masters/api-users.conf
/**
* The ApiUser objects are used for authentication against the API.
*/
object ApiUser "api-healthchecks" {
password = "eiy8aiGh0aog"
permissions = [ "actions/process-check-result" ]
}
object ApiUser "icingaweb2" {
password = "Eila5iRaed2o"
permissions = [ "status/query", "actions/*", "objects/modify/*", "objects/query/*" ]
}
object ApiUser "keepalived" {
password = "paigohLe8soh"
permissions = [ "status/query" ]
}

Icinga2 - Valdiate & apply the adjusted con guration
Validate the new configuration
agent@m-do-ffm-plugins-01:~$ sudo icinga2 daemon -C
[2019-09-28 23:29:05 +0000] information/cli: Icinga application loader (version: r2.11.0-1)
[2019-09-28 23:29:05 +0000] information/cli: Loading configuration file(s).
[2019-09-28 23:29:06 +0000] information/ConfigItem: Committing config item(s).
[2019-09-28 23:29:06 +0000] information/ApiListener: My API identity: m-do-ffm-plugins-01.mydomain.com
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 3 Zones.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 3 ApiUser.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2019-09-28 23:29:06 +0000] information/ConfigItem: Instantiated 235 CheckCommands.
[2019-09-28 23:29:06 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-09-28 23:29:06 +0000] information/cli: Finished validating the configuration file(s).

Icinga2
HAProxy - Global and default con guration for both nodes
global
log /dev/log
local0
log /dev/log
local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd
defaults
log global
option httplog
option dontlognulltimeout
timeout connect 5000
timeout client 5000
timeout server 5000
retries 2

Icinga2
HAProxy - General listener con guration & statistics endpoint
listeners
stats timeout 30s
user haproxy
group haproxy
daemon
listen ha-stats
bind 127.0.0.1:8026
mode http
stats enable
stats uri /
stats realm Strictly Privatestats auth icinga_monitor:chong4gohCe8
> The ha-stats endpoint exposes statistics about haproxy, the configured listeners and backends.
> Useful for debugging, testing and monitoring.
> Keep it secure! (see the last line and setup firewall rules)

Icinga2
HAProxy
Special thanks to Carsten!
@Mikeschova
https://community.icinga.com/t/galera-mysql-cluster-with-vips-and-haproxy-for-ido-mysql-and-more/

Icinga2
HAProxy - 3 di erent listeners for the MariaDB backend
> use galera-01 and galera-02 as prefered endpoint for write-intensive access (port 3306)
> balance read queries round robin across all 3 nodes; individual weight of each galera node varies
between the two icinga2 nodes (port 3307)
> the director has probably an equal distribution of read and write queries (port 3308)

Icinga2
HAProxy
Listener to provide database access to for icinga2
# 1st node
listen icinga2_ido
bind 127.0.0.1:3306
mode tcp
option mysql-check user haproxy_checks
balance source
server do-ffm-galera01 10.135.209.4:3306 check weight 1 inter 5s rise 2 fall 2
server do-ffm-galera02 10.135.207.79:3306 check weight 2 inter 5s rise 2 fall 2 backup
# 2nd node
listen icinga2_ido
bind 127.0.0.1:3306
mode tcp
balance source

Icinga2
HAProxy
Listener to provide database access for icingaweb2 and handle mostly read queries
# 1st node
listen mysqlread
bind 127.0.0.1:3308
mode tcp
balance roundrobin
# 2nd node
listen mysqlread
bind 127.0.0.1:3308
mode tcp
balance roundrobin

Icinga2
HAProxy
Listener to provide access to the icingaweb2 director database backend
listen director
bind 127.0.0.1:3307
mode tcp
balance source
> This is the same for both nodes again.

Icinga2|HAProxy
Apache2 & PHP
Virtual host configuration:
> set proper date.timezone in php.ini
> Enforce encryption and redirect HTTP requests to HTTPS:
Redirect permanent / https://moni.mydomain.com
> Rewrite empty URIs to /icingaweb2:
RewriteRule ^(.*) http://%{HTTP_HOST}/icingaweb2
> Enable the required modules and the new site:
sudo a2enmod proxy_http rewrite ssl
sudo a2ensite moni.mydomain.com

Icinga2|HAProxy|Apache2&PHP
Icingaweb2 - https://moni.mydomain.com/Setup
> Protected custom variables (pw, pass, password, hmac, community)

Icinga2|HAProxy|Apache2&PHP
Icingaweb2
Add a dedicated user to sync /etc/icingaweb2/ between both icinga2 nodes.
agent@m-do-ffm-plugins-01:~$ sudo adduser --disabled-password icingaweb_sync
agent@m-do-ffm-plugins-01:~$ sudo su - icingaweb_sync
icingaweb_sync@m-do-ffm-plugins-01:~$ ssh-keygen -t rsa -b 4096
Add the public key to the authorized_keys file.
Setup the crontab on both nodes
agent@m-do-ffm-plugins-01:~$ sudo cat /etc/cron.d/icingaweb_sync
MAILTO=""
* * * * * icingaweb_sync <ip-of-the-peer-node>:/etc/icingaweb2 rsync -rugopl /etc/icingaweb2/

> the system user keepalived_script
> the .my.cnf file in /etc/keepalived
> the icinga2 API user keepalived to perform healthchecks
> the icinga2 API user api_healthchecks to send the check results to icinga
> the database user keepalived_checks to test the connection to the DB
Icinga2|HAProxy|Apache2&PHP|Icingaweb2
Keepalived
Remember what we already have:

vrrp_script check_apache2 {
script "/usr/bin/killall -0 apache2"
interval 2
fall 1
rise 5
user root
}
vrrp_script check_icinga2_api {
script "/usr/local/bin/check_icinga_api"
interval 2
fall 3
rise 3
}
Keepalived - /etc/keepalived/keepalived.conf (Healthchecks)
vrrp_script check_mysql_conn {
script "mysql --defaults-file=/etc/keepalived/.my.cnf -e 'select count(*) from icingaweb2.icingaweb_user'"
interval 2
fall 1
rise 3
}

Keepalived - Icinga2 API Healthcheck
/usr/local/bin/check_icinga_api - a simple healthcheck
#!/usr/bin/env bash
icinga_api_user='keepalived'
icinga_api_pass='paigohLe8soh'
icinga_api_timeout=1
api_rc=$(curl -k -s -o /dev/null -w %{http_code} --max-time $icinga_api_timeout
-u $icinga_api_user:$icinga_api_pass https://localhost:5665/v1/status/ApiListener)
if [ $api_rc -eq 200 ]; then
exit 0
else
exit 1
fi

global_defs {
enable_script_security
}
vrrp_instance icinga_fe {
state BACKUP
nopreempt
interface eth1
virtual_router_id 10
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass CaemooXah1Ke
}
unicast_src_ip 10.135.201.194
unicast_peer {
10.135.221.140
}
global_defs {
enable_script_security
}
vrrp_instance icinga_fe {
state BACKUP
nopreempt
interface eth1
virtual_router_id 10
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass CaemooXah1Ke
}
unicast_src_ip 10.135.201.140
unicast_peer {
10.135.221.194
}
Keepalived - /etc/keepalived/keepalived.conf (global_defs & vrrp_instance)

Keepalived - /etc/keepalived/keepalived.conf (track & notify scripts)
track_script {
check_apache2
check_mysql_conn
check_icinga2_api
}
notify_backup "/usr/local/bin/keepalived-alarming BACKUP"
notify_fault "/usr/local/bin/keepalived-alarming FAULT"
notify_master /etc/keepalived/master.sh
> /etc/keepalived/master.sh triggers keepalived-alarming MASTER

Keepalived - Script to assign the oating IP to the new master
#!/usr/bin/env bash
floating_ip='68.183.240.112'
droplet_id=$(curl -s http://169.254.169.254/metadata/v1/id)
has_floating_ip=$(curl -s http://169.254.169.254/metadata/v1/floating_ip/ipv4/active)
export DO_TOKEN='ohjajeelaex8oongiequuxiethae4ing1jooch2ezieP2Shil6ohei8ierei3oizi'
if [ $has_floating_ip = "false" ]; then
n=0
while [ $n -lt 5 ]; do
python3 /usr/local/bin/assign-ip $floating_ip $droplet_id && break
n=$((n+1))
sleep 3
done
fi
keepalived-alarming MASTER
Monitor floating IP action via https://api.digitalocean.com/v2/floating_ips/<floating-ip>/actions?page=1&per_page=1 .
Possible states: completed, in-progress or failed.

MASTER & BACKUP = 0
FAULT = 2
Keepalived - Noti cation Script (Snippets)
/usr/local/bin/keepalived-alarming
#/usr/bin/env bash
# $1 = target state of transition ("MASTER", "BACKUP", "FAULT")
t_state=$1
chk_message="Keepalived entered $t_state state."
icinga_chk_host=$(hostname)
# 1st node endpoints
icinga_endpoints=("127.0.0.1" "10.135.221.140")
Use a case statement to translate t_state as following:

Keepalived - Noti cation Script (Snippets)
Construct the JSON for the icinga2 API call:
icinga_post_data()
{
cat <<EOF
{
"service": "${icinga_chk_host}!${icinga_service}",
"exit_status": "$e_status",
"plugin_output": "$chk_message",
"performance_data": [
"entered_keepalived_state=$new_state;;2;0;2",
]
}
EOF
}

Pitfalls & Don't Dos
> Don't stop all galera nodes at the same time! If that happens, you have to bootstrap the cluster
again.
> Don't forget to set enable_ha = true for the ido-mysql feature
> Don't create the rsync user without --disabled-password
> Don't rely on the volatile instance / droplet IPs. Use floating IPs
> Don't forget backups / snapshots: droplets, database content, configuration & setup
> Don't miss to migrate to the director if you have the chance to
> Setup a local firewall if the cloud providers firewall does not support VRRP2

Firewall Rules
> Communication between the Icinga2 masters
TCP/5665, VRRP2
> Communication from the satellites and agents to the master nodes
TCP/5665
> Icinga2 nodes to the galera nodes
TCP/3306 , TCP/3307 , TCP/3308
> Communication between the galera cluster nodes
TCP/4567, TCP/4568, UDP/4567
> Restrictive ipset & iptables How-To:
https://community.icinga.com/t/firewall-setup-at-a-icinga2-cluster-with-iptables-ipset/2156

THANK YOU!THANK YOU!
@winem_@winem_

OSMC 2019 | Use Cloud services & features in your redundant Icinga2 Environment by Marcel Weinberg

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to OSMC 2019 | Use Cloud services & features in your redundant Icinga2 Environment by Marcel Weinberg

Similar to OSMC 2019 | Use Cloud services & features in your redundant Icinga2 Environment by Marcel Weinberg (20)

Recently uploaded

Recently uploaded (20)

OSMC 2019 | Use Cloud services & features in your redundant Icinga2 Environment by Marcel Weinberg