OSMC 2015:The road to lazy monitoring with Icinga 2 and Puppet by Tom de Vylder

The road to lazy monitoringThe road to lazy monitoring
with Icinga2 & Puppetwith Icinga2 & Puppet
Tom De Vylder
@tomdevylder

whoiswhois
● Ops guyOps guy
● Rusty, old, Bavarian vehicles enthousiastRusty, old, Bavarian vehicles enthousiast
● The one who always gets the patdown at checkpointsThe one who always gets the patdown at checkpoints
● Open Source Consultant & Enthousiast at InuitsOpen Source Consultant & Enthousiast at Inuits
● Team Packages & Tools at IcingaTeam Packages & Tools at Icinga

Monitoring!Monitoring!
Image credits: http://mta.hu/data/cikk/13/00/73/cikk_130073/20120611-144546.jpgImage credits: http://mta.hu/data/cikk/13/00/73/cikk_130073/20120611-144546.jpg

Monitoring is...Monitoring is...
● BoringBoring
● TediousTedious
● RepetitiveRepetitive
● Production =Production = deploymentdeployment + monitoring + backup+ monitoring + backup
● Often neglectedOften neglected

Why can'tWhy can't
someone else do it?someone else do it?

Someone else!Someone else!
● TediousTedious
● People make mistakesPeople make mistakes
● Nobody likes Nagios configsNobody likes Nagios configs

Someone else!Someone else!
● TediousTedious
● People make mistakesPeople make mistakes
● Nobody likes Nagios configsNobody likes Nagios configs
● Monitor servers not peopleMonitor servers not people
Image credits: https://openclipart.org/image/2400px/svg_to_png/219371/You-Are-Being-Monitored.pngImage credits: https://openclipart.org/image/2400px/svg_to_png/219371/You-Are-Being-Monitored.png

Single source of truthSingle source of truth

Config management!Config management!

Config management!Config management!
PuppetPuppet RudderRudder
JujuJuju
ChefChef
AnsibleAnsible
SaltstackSaltstack
CFEngineCFEngine
TrueConfigTrueConfig

PuppetPuppet
● Config management toolConfig management tool
● Exported resourcesExported resources
● PuppetDBPuppetDB
● Nagios_* {} resource typesNagios_* {} resource types

node “web.example.org” {
@@nagios_host { $::fqdn:
ensure => present,
alias => $::hostname,
address => $::ipaddress,
use => "generic-host",
}
@@nagios_service { "check_ping_${::fqdn}":
check_command => "check_ping",
use => "generic-service",
host_name => $::fqdn",
notification_period => "24x7",
service_description => "${::fqdn}_check_ping",
}
}

Nagios_* {} resource typesNagios_* {} resource types
node icinga.example.org {
Nagios_host <<||>>
Nagios_service <<||>>
}

@@nagios_host { $::fqdn:
ensure => absent,
}
@@nagios_service { "check_ping_${::fqdn}":
ensure => absent,
}

Nagios_host <<||>>
Nagios_service <<||>>

/etc/nagios/

PuppetPuppet
● Nagios_* resource typesNagios_* resource types

PuppetPuppet
● Nagios_* resource typesNagios_* resource types
● https://github.com/Icinga/puppet-icinga2https://github.com/Icinga/puppet-icinga2
● https://github.com/Icinga/puppet-icingaweb2https://github.com/Icinga/puppet-icingaweb2

Puppet-Icinga2Puppet-Icinga2
puppet-icinga2 (develop) ☃ git log | tail -4
Author: Nick Chappell
Date: Fri May 2 19:36:32 2014 -0700
Initial commit.

Other tools...Other tools...
● https://github.com/Icinga/puppet-icinga2https://github.com/Icinga/puppet-icinga2
● https://github.com/Icinga/puppet-icingaweb2https://github.com/Icinga/puppet-icingaweb2
● https://github.com/Icinga/icinga2-ansiblehttps://github.com/Icinga/icinga2-ansible
● https://github.com/Icinga/docker-icinga2https://github.com/Icinga/docker-icinga2
● https://github.com/Icinga/chef-icinga2https://github.com/Icinga/chef-icinga2

Puppet asPuppet as
central source of truthcentral source of truth

Exported resourcesExported resources

Collected resourcesCollected resources

Icinga2Icinga2
● ApplyApply rulesrules
● Assign rulesAssign rules

template Service "generic-service" {
max_check_attempts = 3
check_interval = 5min
retry_interval = 5min
enable_perfdata = true
}
apply Service "ping4" {
import "generic-service"
check_command = "ping4"
assign where host.address
}
apply Service "ping6" {
check_command = "ping6
assign where host.address6
}
Icinga2Icinga2

apply Service "ssh" {
check_command = "ssh"
assign where host.address && host.vars.os == "Linux"
}
Icinga2Icinga2

object HostGroup "database" {
display_name = "Database Servers"
assign where match("mysql*", host.name)
assign where match("psql*", host.name)
assign where match("db*", host.name)
}
Icinga2Icinga2

Icinga2Icinga2
● ApplyApply rulesrules
● Assign rulesAssign rules

node 'web.example.org' {
@@::icinga2::object::host { $::fqdn:
groups => ['puppet', $::operatingsystem],
target_dir => '/etc/icinga2/objects/hosts',
target_file_name => "${::fqdn}.conf",
template_to_import => 'linux-host',
vars => {
distro => $::operatingsystem,
distro_version => $::operatingsystemrelease,
virtual_machine => $::is_virtual,
puppet_version => $::puppetversion,
puppet_environment => $::environment,
},
}
}

node 'web.example.org' {
vars => {
puppet_classes => $::puppet_classes,
puppet_modules => $::puppet_modules,
},
}
}

classes_file = '/var/lib/puppet/classes.txt'
classes_hash = {}
modules_array = []
File.foreach(classes_file) do |l|
modules_array << l.chomp.gsub(/::.*/, '')
end
modules_array = modules_array.sort.uniq
modules_array.each do |i|
classes_array = []
classes_array << i
File.foreach(classes_file) do |l|
classes_array << l.chomp if l =~ /^#{i}/
classes_array = classes_array.sort.uniq
end
classes_hash[i] = classes_array
end

Facter.add(:puppet_modules, :timeout => 10) do
confine :kernel => 'Linux'
setcode do
modules_array.sort.uniq.join(', ').to_s
end
end
Facter.add(:puppet_classes, :timeout => 10) do
confine :kernel => 'Linux'
setcode do
classes_hash.map { |_k, v| v }.sort.uniq.join(', ').to_s
end
end

# facter -p puppet_modules --yaml
---
puppet_modules:
- apache
- collectd
- concat
- dnsclient
- firewall
- grafana
- icinga2
- logrotate
- mysql
- postfix
- ssh
- stdlib
- sudo
- sysctl
- ulimit

# facter -p puppet_classes --yaml
---
puppet_classes:
mysql:
- mysql
- "mysql::client"
- "mysql::client::install"
- "mysql::server"
- "mysql::server::config"
- "mysql::server::install"
- "mysql::server::installdb"
- "mysql::server::service"
icinga2:
- icinga2
- "icinga2::server"
- "icinga2::server::config"
- "icinga2::server::features"
- "icinga2::server::install"
- "icinga2::server::service"

vars => {
puppet_classes => inline_template(
'<%= puppet_classes.collect { |k, v| v }.sort.join(" ") %>'
),
puppet_modules => inline_template(
'<%= puppet_modules.sort.join(" ") %>'
),
},
}

icinga2::object::apply_service_to_host { 'mysql_server':
check_command => 'by_ssh',
display_name => 'MySQL',
target_dir => '/etc/icinga2/objects/applys',
assign_where => '"match(*mysql::server*", host.vars.puppet_classes)"',
vars => {
by_ssh_command => "check_tcp -H localhost -p 3306",
by_ssh_logname => 'root',
by_ssh_quiet => true,
by_ssh_timeout => 60,
},
}

There's a better wayThere's a better way

curl
-s
-X GET
http://puppetdb:8080/pdb/query/v4/resources
--data-urlencode
'query=["extract", "certname",
["and",
["=", "type", "Class"],
["~", "title", "^Settings"]
]
]'
Query PuppetDBQuery PuppetDB

$nodes = query_nodes('Class[Settings]', 'fqdn')
PuppetDBqueryPuppetDBquery

notify { "${nodes}": }
# puppet agent --test –noop
Info: Applying configuration version '1447591034'
Notice: /Stage[main]/Main/Notify[web.example.org,
db.example.org, icinga2.example.org,
puppet.example.org]/message: current_value absent,
should be [web.example.org, db.example.org,
icinga2.example.org, puppet.example.org] (noop)
Notice: Applied catalog in 0.21 seconds

$nodes = query_nodes('Class[Mysql::Server]', 'fqdn')
notify { "${nodes}": }
# puppet agent --test –noop
Info: Applying configuration version '1447591391'
Notice:
/Stage[main]/Main/Notify[db.example.org]/message:
current_value absent, should be db.example.org (noop)
Notice: Applied catalog in 0.21 seconds

define monitor::host ($host_name) {
$foo.each |Integer $index, String $host| {
::icinga2::object::host { $host:
target_file_name => "${host}.conf",
}
}
}
::monitor::host { 'all':
host_name => $nodes,
}

SummarySummary
● No more exported resourcesNo more exported resources
● Automagically worksAutomagically works
● Production = deployment +Production = deployment + monitoringmonitoring + backup+ backup
● Purge decommissioned nodesPurge decommissioned nodes

ContactContact
Tom De VylderTom De Vylder
@tomdevylder@tomdevylder
tomdv@inuits.eutomdv@inuits.eu
INUITS bvbaINUITS bvba
Heidestraat 257Heidestraat 257
2070 Zwijndrecht2070 Zwijndrecht
BelgiumBelgium
+32 475 961221+32 475 961221
info@inuits.euinfo@inuits.eu

OSMC 2015:The road to lazy monitoring with Icinga 2 and Puppet by Tom de Vylder

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Viewers also liked

Viewers also liked (16)

Similar to OSMC 2015:The road to lazy monitoring with Icinga 2 and Puppet by Tom de Vylder

Similar to OSMC 2015:The road to lazy monitoring with Icinga 2 and Puppet by Tom de Vylder (20)

Recently uploaded

Recently uploaded (20)

OSMC 2015:The road to lazy monitoring with Icinga 2 and Puppet by Tom de Vylder