OSMC 2010 | Munin & Nagios by Stig Sandbeck Mathisen

Munin and Nagios 2010-10-07 1 / 53
Munin and Nagios
Stig Sandbeck Mathisen
Redpill Linpro AS
2010-10-07
PRODUCTS • DEVELOPMENT • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING

Munin and Nagios Introduction 2010-10-07 2 / 53
Outline
1 Introduction
2 The Munin project
3 Munin master and node
4 Munin plugins
5 Integration with Nagios
6 Example graph set
7 New features in Munin 1.4
9 Speeding up munin
10 The end

About the speaker
System Administrator - 12 years with open source based system
administration. 5 years at Redpill Linpro.
Debian Developer - varnish, munin, puppet, facter

About Redpill Linpro
Leading Nordic provider of professional Open Source services - across the
stack.
Presence in Denmark, Finland, Norway and Sweden
Hosting and training facilities in Oslo and Karlstad.
190 employees in Gothenburg, Helsinki, Karlstad, Oslo, Stavanger and
Stockholm.
More than 300 customers across the Nordic countries, 60% in the
“enterprise tier”
15 years in the Open Source business.

Munin and Nagios The Munin project 2010-10-07 5 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Introduction to Munin
Munin is a networked resource monitoring tool. It gathers data from your
systems, creates lots of graphs, and...
can show you trends
can help you predict bottlenecks
can show you old data for comparison with current numbers
can send events to other systems

History
Originally called LRRD, about the time Nagios was still known as Netsaint.
The old code is still available as the “LRRD” project at SourceForge.
2002: Started by Linpro
2004: 1.0 released, development moved from CVS to Subversion
2005: 1.2 released
2009: 1.4 released

Munin and Nagios Munin master and node 2010-10-07 8 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Master
The munin master runs from cron. It runs four jobs, each with its own
lock. A new cron job can begin while the previous already runs.
munin-update: Contacts each
node, and retrieves plugin
conﬁguration and data
munin-graph: Creates graphs
from RRD ﬁles
munin-limits: Checks for limit
breaches
munin-html: Updates the
HTML documents

Node
The munin node listens for connections on 4949/TCP, and runs plugins on
request. It runs each plugin when the master asks, to retrieve
conﬁguration and values.
The node uses Net::Server
Plugins are commonly written in shell, perl, python, ruby, awk...
SNMP plugins runs on a node, and queries other hosts

Wire protocol
The wire protocol is a simple and in clear text. Keywords are “list”,
“conﬁg” and “fetch”.
Example
# munin node at puppet1.example.org
list
apache_accesses apache_processes [...] uptime users vmstat
config uptime
graph_title Uptime
[...]
.
fetch uptime
uptime.value 148.03
.
quitPRODUCTS • DEVELOPMENT • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING

Munin and Nagios Munin plugins 2010-10-07 12 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

What is a munin plugin?
A plugin is a standalone executable.
It is run by the Munin Node when
the Munin Master connects.
The plugin prints a clear text
key/value list of conﬁguration and
values on STDOUT.
plugins
Most plugins are shell scripts,
but many are made in perl,
python or ruby.

Plugin design
Munin plugins are designed to be
simple to develop.
If run with the argument “conﬁg”, it
displays its conﬁguration.
If run with no arguments, it outputs
its values.
Magic markers inside the plugin lists
other capabilities, like the optional
arguments “autoconf” and
“suggest”.
Magic markers
#%# capabilities=autoconf

Plugin configuration
Each plugin have a sensible default
configuration.
You can configure plugins in
/etc/munin/plugin-conf.d/ on
the munin node.
You can also configure plugins in
munin.conf on the munin master.
Configuration items
user (default is “nobody”)
group (default is “nogroup”)
Environment variables for the
plugin

Example
#!/bin/sh
case "$1" in
config)
echo ’graph_title System Boredom Index’
echo ’graph_vlabel boredom in %’
echo ’time.label Total time’
echo ’bored.value Bored time’
;;
*)
awk ’{printf "time.value %dn"
"bored.value %dn", $1, $2}’ /proc/uptime
;;
esac

Boring example graph
A single line is described as a label,
and given a value.
Several lines can be combined in
interesting ways with CDEF.
Example
time.graph no
bored.graph no
foo.cdef time,100,*,bored,/
foo.label Part time bored silly

Munin and Nagios Integration with Nagios 2010-10-07 18 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Munin events
Munin generates events whenever a value
rises above, or sinks below a predeﬁned limit.
Many plugins support limits, but some do
not set them by default.
Example
[cpu]
env.iowait_warning 5
env.iowait_critical 20

Munin and Nagios events
Munin ships with usable defaults to send
events to Nagios.
Events...
correspond with Nagios events
are processed by a templating system
are sent to a deﬁned contact
Supported events
CRITICAL
WARNING
UNKNOWN
OK

Integration with Nagios
The contact.nagios.text template is
defined by default in munin.
Use the contact.nagios.command
configuration setting to send events.
where?
This is configured in
/etc/munin/munin.conf

Configuration for the munin master
On the munin master, We define a contact for nagios, using send nsca,
and configure send nsca to encrypt the messages.
example munin.conf
contact.nagios.command /usr/sbin/send_nsca
-H nagios.example.com -c /etc/send_nsca.cfg
example send nsca.conf
password="I like cheese!"
encryption_method=8

Conﬁguration for the nagios server
On the nagios side, we set up nsca to receive events from our munin
masters.
Example part of nsca.conf
server_port=5667
nsca_user=nagios
nsca_group=nagios
command_file=/var/run/nagios/nagios.cmd
password="I like cheese!"
decryption_method=8

Conﬁguration for the nagios server
Nagios, by default, do not accept external commands. We need to
conﬁgure this to enable nsca to send the events to Nagios.
Example part of nagios.conf
check_external_commands=1
log_external_commands=1
command_file=/var/run/nagios/nagios.cmd

Nagios service conﬁguration
To accept munin services, you will need a passive service check.
example nagios service
passive_checks_enabled 1
active_checks_enabled 0
max_check_attempts 1

Nagios service conﬁguration
For services we have not heard from in a while, we use the “freshness”
feature to set the service state to UNKNOWN. Munin reports state on all
services every 24 hours by default, so a bit more than this is a sensible
default.
example nagios service
check_command check_passive_timeout
check_freshness 1
freshness_threshold 93600 # 26 hours
normal_check_interval 604800 # 1 week
The check passive timeout command is a dummy script that always
return state UNKNOWN.

Integration with other systems
You can define your own contact text and command to interface with any
system you need. Both the text and the command can use a
Text::Balanced template to get event data from Munin.
Command run defined in contact.example.command
Text for STDIN defined in contact.example.text
Default STDIN text is defined in: contact.default.text

Text::Balanced templates
The command and text templates use Text::Balanced. This provides
variables, tests and loops.
Example
[${var:group};${var:host}] -> ${var:graph_title} ->
warnings: ${loop<,>:wfields ${var:label}=${var:value}} /
criticals: ${loop<,>:cfields ${var:label}=${var:value}}
Example
[example.com;foo] -> HDD temperature -> warnings:
sde=29.00,sda=26.00,sdc=25.00,sdd=26.00,sdb=26.05
/ criticals:

Example: mail
Add the following to /etc/munin/munin.conf:
Example
contact.mail.command /usr/bin/mail -s "[...]" me@example.com
If contact.mail.text is not deﬁned, it will fall back to
contact.default.text.
The “[...]” should be replaced with a Text::Balanced template to
provide a useful subject.
For mail, you do not need to provide a contact.mail.text, since
contact.default.text is designed as a mail template.

Munin and Nagios Example graph set 2010-10-07 30 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Example graph set
Peak in network connections and
traﬃc. We look for corresponding
graphs
Note: logarithmic scale on the
“netstat” graph ensures detail is not
lost to peaks.

Example graph set
Apache HTTPD has a corresponding
peak in traﬃc.
In this case this is a customer
publishing a new version of their
development tool.

Example graph set
We check the server health. Do we
have a bottleneck? It does not look
like it, but there is a strange bump in
I/O latency and utilisation not
related to the network traﬃc. Also,
the periods of 100% utilisation means
we have to talk the customer into
getting faster disks or more RAM.

Example graph set
Found more corresponding graphs:
“MySQL”. We notice no cached
SELECTs? Contact the customer?
No corresponding network traﬃc -
looks like a local job. Probably worth
mentioning to the customer, but
don’t panic.

Munin and Nagios New features in Munin 1.4 2010-10-07 35 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Released in December 2009, after three months of solid development
following four years of having the project in “maintenance mode”.
23 committers, with contributions from many more
1500 changesets
100 new plugins, including JVM proﬁling plugins.
TLS / SSL support
Better SNMP Support
Multigraph plugins
Documentation improvements, including good per-plugin
documentation
Better non-Linux support

Multigraph plugins
“You are in a maze of twisty little
graphs, all alike”. Multigraph plugins
create a tree of graphs, all from one
plugin.
nested graphs
fast plugins
new scaling issues on the master

Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Version numbers
Starting with “munin 2.0”, the project has changed what the version
numbers signify.
Major version - new features
Minor version - bug ﬁxes

Asynchronous proxy node
This node contacts munin-node periodically, and
stores the result. The master connects to the proxy
node, and retrieves stored results.
The master does not have to wait for nodes to
respond.
This will increase peak write loads on the master.
Munin
master
Server
Async
proxy node
Munin node

SSH transport
With Munin 1.4, we have SSH tunnelling. Starting with Munin 2.0, we
have a native SSH transport.
[old-style-host]
address host.example.com
[new-style-host]
address ssh://user@host.example.com:
/path/to/stdio-enabled-node --params
For now, this requires the use of the “asynchronous proxy node”, to limit
the privileges of the SSH node.

Zooming graphs
No longer locked to speciﬁc time periods, you can now drill down in the
graphs to look at interesting time periods.

Multi master
With the “Asynchronous Proxy Node”, we
can also support multi master setups
without conﬂicts.
One master for the customer, one for the
hosting provider.
Munin node
Munin
master
(provider)
Munin
master
(customer)
Munin node
Munin node
Munin node

Unresolved issues in 2.0
There is a fair bit of work left before 2.0 can be released.
Performance and scaling
Functional and pretty HTML (got functional so far)
Whatever we broke in 1.4...

Munin and Nagios Speeding up munin 2010-10-07 45 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Scaling munin
On a 1 CPU (2 threads) system, 65k RRD
ﬁles are...
Updated in 1 minute
Graphed in 40 minutes
The cron job runs every 5 minutes; Houston,
we have a problem.

FastCGI
FastCGI to the rescue!
This gives us graphs on demand. When we have a large number of graphs,
it makes little sense to update them all.
The “munin 2.0” CGI grapher can make zoom-able graphs for any time
period.
We still need a CGI HTML generator to reduce resource usage for large
graph sets. This is not implemented yet.

Storage tuning
Enough RAM for ﬁle system caching
Everything on SSD
OpenSolaris / FreeBSD: ZFS
ZIL/L2ARC
Linux: FlashCache
Turn oﬀ atime
Linux ext3/ext4: mount option
data=journal

Munin and Nagios The end 2010-10-07 49 / 53
Outline
1 Introduction
2 The Munin project
4 Munin plugins
6 Example graph set
9 Speeding up munin
10 The end

Munin links
http://munin-monitoring.org/ - project, documentation, bug
tracker, svn source code access
http://exchange.munin-monitoring.org/ Extra plugins for
Munin contributed by the Munin community
http://munin-monitoring.org/wiki/HowToContactNagios
In-depth conﬁguration examples for Munin / Nagios integration
http://munin-monitoring.org/wiki/MuninAlertVariables
What to put in the contact.example.text when making templates

Munin book
Gabriele Pohl and Michael Renner
have published an entire, thoroughly
written book on Munin in German:
“Munin - Graphisches Netzwerk- und
System-Monitoring” (ISBN
978-3-937514-48-2), published by
Open Source Press in cooperation
with Linpro.

Questions from the audience?
? ? ? ??? ? ?? ??
? ? ? ? ? ? ? ? ?
?? ?? ? ? ??? ?? ??
? ? ?? ? ? ? ?? ?
I was going to write “If I timed this
correctly, we should now have around
10 minutes for questions”.
I decided against it. What if I missed
the time? That would be
embarrassing.

Thank you for listening
...and I hope to see you at the next OSMC.

OSMC 2010 | Munin & Nagios by Stig Sandbeck Mathisen

Recommended

Recommended

More Related Content

Similar to OSMC 2010 | Munin & Nagios by Stig Sandbeck Mathisen

Similar to OSMC 2010 | Munin & Nagios by Stig Sandbeck Mathisen (20)

Recently uploaded

Recently uploaded (20)

OSMC 2010 | Munin & Nagios by Stig Sandbeck Mathisen