Distributed monitoring at Hyves- Puppet

Welcome Jeﬀrey Lensen System Engineer 1

Hyves Infrastructure 3000+ Gentoo servers 190 func;ongroups/types 3 datacenters Database for server management 2

Using Puppet Since: January 2007 Puppetmasters: 3 Loadbalanced, 1 for CA and development Version: 2.6.1 MySQL backend for (thin_)storeconﬁgs Nginx + 8 Mongrel instances per server 100+ modules nodes.rb uses management database Puppet run every morning on every server 3

Nagios 8 Nagios hosts in distributed setup 2500 hosts 1 Nagios master for web and aler;ng Scripts to generate conﬁgura;on Management database for informa;on Templates for service checks Died during large fallouts 4

Icinga Switched to Icinga November 2010 Distributed Icinga setup doesn’t require centralized host Very fast standalone Icinga-‐web interface Uses database backend REST API Switching was easy due to similar conﬁgura;on 5

Current monitoring setup Monitoring hosts: 12 (4 per DC) Services: over 83.000 Hosts: nearly 3.500 Average check interval: every 5 min NOC monitoring host: 1 Overview checks using API Commandline interface 6

Problems with monitoring Adding new checks meant manually edi;ng a lot of templates Things that should be monitored aren’t Won’t realize it un;l it’s too late No monitoring makes it harder to ﬁnd the problem 7

Using Puppet to configure Icinga Puppet knows it all so why not use that informa;on? Exported resources from Naginator to define monitoring checks Include the monitoring defini;ons in profiles Running Puppet defines all necessary monitoring checks for that host 8

Example modules/monitoring/manifests/init.pp: class monitoring { service { "nrpe": ensure => running, enable => true } Appending $hostname in @@nagios_host { "$hostname": nagios_service deﬁnition to address => $ip prevent duplicate deﬁnitions } on monitoring hosts @@nagios_service { "NRPE $hostname": service_description => "NRPE", check_command => "check_nrpe_scripts", } } 9

Example Nginx Automa;cally create HTTP checks when including Nginx modules/nginx/manifests/init.pp: class nginx { service { "nginx": ensure => running, enable => true } @@nagios_service { "HTTP $hostname": service_description => "HTTP", check_command => "check_http", event_handler => "service_restart!nginx”, contact_groups => “admins_email, admins_sms” } } 10

Predefining and distributingmanifests/defines.pp:$__notifications_enabled = $systemstatus ? { operational => "1", fail => "0"}Nagios_host { Nagios_service { ensure => present, ensure => present, host_name => $hostname.$domain, host_name => $hostname.$domain, hostgroups => $role, use => "generic-service", #our standard template use => "generic-host", #our standard template notifications_enabled => $__notifications_enabled, alias => $hostname, target => "/etc/icinga/puppetgenerated/services/ notifications_enabled => $__notifications_enabled, $hostname.cfg", target => "/etc/icinga/puppetgenerated/hosts/ notes => $monitoringhost$hostname.cfg", } notes => $monitoringhost} 11

Retrieving exported resources modules/icinga/manifests/init.pp: class icingacollect { Nagios_host <<| notes == "$hostname" |>> { require => File["/etc/icinga/puppetgenerated/hosts"] } Nagios_service <<| notes == "$hostname" |>> { require => File["/etc/icinga/puppetgenerated/services"] } } 12

Why not Tags? Using “notes” to assign monitoring host Tagging caused problems when seing require in Nagios_host and Nagios_service Tagging meant redeﬁning, it’s not inherited Solu;on: stages (?) 13

Fail-safes modules/icinga/manifests/init.pp: class icinga { include icingacollect exec { "verify new cfg": command => "/usr/bin/icinga -v /etc/icinga/verify-puppetgenerated.cfg", require => Class["icingacollect"] } exec { "mv cfgs": command => "rm -rf /etc/icinga/puppet/* ; mv /etc/icinga/puppetgenerated/* /etc/icinga/puppet/", require => Exec["verify new cfg"] } exec { "restart icinga": command => ""/usr/bin/printf [] RESTART_PROGRAMn > /var/icinga/rw/icinga.cmd"", require => [ Exec["mv cfgs"], Service["icinga"] ] } } 14

Deploying monitoring Deploy script starts Puppet run on all monitoring hosts Threaded with small sleep in between start to prevent thundering herd on Puppet masters Waits for all puppet runs to ﬁnish and reports whether they were successful or not 15

Downsides Puppet run on Icinga hosts takes about 20 minutes (using separate conﬁg ﬁles for each host helps) Modifying a servicecheck requires a puppet run on all hosts with that servicecheck (solu;on: use -‐-‐noop) Cleaning up old resources 16

Cleaning up $fqdn = $host_to_be_removed.$domain puppet apply --certname $fqdn --node_name facter --thin_storeconﬁgs $dbsettings --execute resources { ["nagios_service","nagios_host"]: purge => true } 17

What if something isn’t running Puppet? Conﬁgcheck check Compares management database with Icinga API 18

Other cool stuff Genera;ng daemon checks modules/role/lib/facter/customfacters.rb: Facter.add("hyves_daemons") do daemons = ["None"] if File::exists?( "/<path_to_conﬁg>/daemons.conf" ) daemons = [] daemonarray = [] daemonconf = %x{grep name /<path_to_conﬁg>/daemons.conf} for daemon in daemonconf daemon.sub!(/.** name:/, ) daemonarray.push(daemon.chomp) end end setcode do daemonarray.uniq end end 19

Other cool stuff Genera;ng daemon checks modules/daemons/manifests/init.pp: class daemons { deﬁne add_daemon_check { @@nagios_service { "$name Daemon $hostname": use => "Daemon-check", service_description => "$name Daemon", check_command => "check_daemon!$name" } } add_daemon_check { $hyves_daemons: } } 20

Other cool stuff Genera;ng overview daemon checksrequire net/httpmodule Puppet::Parser::Functions newfunction(:get_daemons, :type => :rvalue, :docs => " This function returns an array of all current hyves_autodaemons, based on the Icinga API ") do |args| domain = "<domain_of_icinga_web>" url = "/icinga-web/web/api/service/ﬁlter[AND(SERVICE_NAME%7Clike%7C*Daemon)]/columns[SERVICE_NAME]/order[SERVICE_NAME;ASC]/authkey=<api_key>/json" response = Net::HTTP.get_response(domain, url) data = response.body results = PSON.parse(data) daemons = Array.new results.each { |result| daemon = result[SERVICE_NAME] daemon.sub!(/ Daemon/, ) daemons << daemon } daemons.uniq endend 21

Other cool stuff Genera;ng overview daemon checks modules/icinga/manifests/noc.pp: $__daemons = get_daemons() templatefile { "/etc/icinga/puppetgenerated/other/daemons.cfg": template => template("icinga/daemons.cfg.erb") } hyvesdaemons.cfg.erb: define host{ use generic-host host_name daemons alias daemons address www.hyves.nl } <% __daemons.each do |daemon| -%> define service{ use DaemonOverview-check host_name daemons service_description <%= daemon %> } <% end -%> 22

The End Ques%ons? Remarks? Ideas? Jeffrey Lensen | System Engineer | jeffrey@hyves.nl 23

http://lanyrd.com	409
http://twitter.com	10
https://twitter.com	4
http://a0.twimg.com	3
http://paper.li	1

Distributed monitoring at Hyves- Puppet

by Puppet Labs

Accessibility

Upload Details

Usage Rights

5 Embeds 427

Statistics

Distributed monitoring at Hyves- Puppet Presentation Transcript