Your SlideShare is downloading. ×
  • Like
Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment


Leland Lammert's presentation on Nagios in a Multi-Platform Enviornment. …

Leland Lammert's presentation on Nagios in a Multi-Platform Enviornment.
The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit:

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Nagios in a Multi-Platform Environment Leland V. Lammert, PhD Chief Scientist Omnitec Corporation
  • 2. The Problem Different OSs can require – Different type of connection – Different installation procedure Nagios checks or an agent require a connection to the remote machine (i.e. from the Nagios server to the remote machine) No issue inside the firewall There are issues for machines outside the firewall
  • 3. The Solution – SSH SSH forward connections are a good solution for monitoring inside the firewall An ssh reverse tunnel is a good solution for monitoring outside the firewall Active checks can then be used on all systems – No Agents – No Complex Installation – A shell is always available for troubleshooting
  • 4. 4 Today's Outline Secure Shell Connection Process Credentials Setting up a Reverse Tunnel Using checks Gui's - RDP or X Troubleshooting
  • 5. Secure Shell
  • 6. SSH History SSH [Secure Shell] is a data exchange protocol that uses an encrypted connection between two network devices SSH replaced Telnet and other insecure remote shells which send information, most notably passwords, in plaintext SSH encryption provides confidentiality and integrity of data over an insecure network Connection Types – Normal [Forward] – Tunnel [Reverse]
  • 7. SSH Advantages Secure [ssh2] – Host Identity Verified – User Authorization – Secure Data Transmission Ubiquitous ('Nix, Windows [Cygwin], OSX) Reverse Tunnels require no firewall changes Allows Active Checks No agent to install or configure Simplified testing, GUI still available [X or RDP]
  • 8. SSH Details Authentication is verified with SSH keys [Forward] ssh connections work for hosts behind the firewall [Reverse or Tunnel] ssh connections work for hosts anywhere else The tunnel passes traffic TO the remote machine FROM the Nagios Server (reverse operation) SSH Connections can be used to standardize connections to ALL machines!
  • 9. Forward connections Requires – Static IP – Open port A static IP may not available at external locations An open port for an incoming connection is a BIG security problem Even when an agent is used [NSCA], there is no access to the remote machine for testing and troubleshooting
  • 10. Reverse Connections - Tunnels A Tunnel is packet encapsulation using a network protocol The data payload protocol is then encapsulated in a delivery protocol. – L2TP (layer 2) Tunneling Protocol) – SSH – Socks Reverse Tunnels - an ssh connection from client to server, where data is transmitted from server to client (i.e. reverse)
  • 11. Connection Process Startup – Start Session – Exchange Server key – Generate Session key Authentication – Exchange Client key – If ACK, continue
  • 12. SSH Startup
  • 13. SSH Authentication
  • 14. SSH uses Asymetric Encryption Public/Private Keypair A Keypair verifies identity for: – Machine – User Currently keys are 2048 bits in length, usual RSA Each Key has two parts: – Public and Private – AKA the keypair Keys are created with ssh-keygen
  • 15. Credentials
  • 16. Host and User Credentials Host Key Verifies that the host/server at the other end of the connection is the same one seen previously User Keys Verifies that the user is authorized for the connection Why not Passwords? ● Requires manual entry ● Cannot be automated without storing as plaintext ● Much less secure than key
  • 17. Keys on Unix/Linux Keys: /home/<user>/.ssh Private key: id_rsa Public key:
  • 18. Keys on Windows Typical Cygwin install location C:Program FilesCygwin Home directories: <Cygwin>/home/<user>/.ssh Keys: ~/.ssh Private key: id_rsa Public key:
  • 19. Keys on OSX Keys: /Users/<user>/.ssh Private key: id_rsa Public key:
  • 20. Setting up a Reverse Tunnel
  • 21. Basic ssh command ssh Base command -f Run in background -X Enable X Forwarding ('Nix) -n Prevent reading from STDIN -N No remote command -R Reverse Operation ssh -f -X -n -N -R
  • 22. Ports REMOTE:* REMOTE Port on remote machine [from] localhost LOCAL Port on local machine [to] * Remember, this is being done at the external machine – so the “Remote” port here is actually on the Nagios server!
  • 23. Authentication $USER_NAME@$REMOTE_HOST USER_NAME User ID on remote machine REMOTE _HOST Hostname or IP of remote hostt -p LOGIN_PORT LOGIN_PORT Port for login on remote machine -i IDENTITY_FILE IDENTITY_FILE Private key file to use
  • 24. Three Steps to Create a Tunnel 1. Remote to Nagios Connect remote machine to Nagios server and create Tunnel 2. Make it permanent – cron – autossh – launchctl/launchd 3. Nagios to Remote Setup access from Nagios user or process to remote machine
  • 25. Step 1. Remote to Nagios Connect remote machine to Nagios server and create Tunnel
  • 26. Step 1 – 'Nix Create keypair on remote system and copy to Nagios Server – ssh-keygen – cat ~/.ssh/ ● <highlight> – ssh <nagios server> – vim ~/.ssh/authorized_keys – G o <shift insert> – :x <exit> test
  • 27. Step 1 – Windows Install Cygwin ( – Install in c:program filesCygwin – Add autossh, rsync, bash, perl Create keypair on remote system and copy to Nagios Server [same as 'Nix]
  • 28. Step 1 – OSX Create keypair on remote system and copy to Nagios Server [same as 'Nix]
  • 29. Step 2. Make it permanent
  • 30. Step 2 – 'Nix cron Connection variables REMOTE_HOST="" USER_NAME="________" REMOTE_PORT="____" LOCAL_PORT="____" * LOGIN_PORT="____" * IDENTITY_KEY="/home/nagios/.ssh/id_rsa" * NOTE: The ssh port should be changed for security, do NOT use the standard port 22.
  • 31. Step 2 – 'Nix cron Command to create the link COMMAND="ssh -f -n -N -R $REMOTE_PORT:$LOCAL_PORT $USER_NAME@$REMOTE_HOST -p$LOGIN_PORT -i $IDENTITY_KEY"
  • 32. Step 2 – 'Nix cron Running? If not, start it pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND Working? Login to Nagios and check from the other side ssh -i$IDENTITY_KEY -p$LOGIN_PORT $USER_NAME@$REMOTE_HOST netstat -an | egrep "tcp.*$REMOTE_PORT.*LISTEN" > /dev/null 2>&1 If not working, kill and restart. if [ $? -ne 0 ] ; then pkill -f -x "$COMMAND" $COMMAND fi
  • 33. Step 2 – Windows autossh Install service cygrunsrv -I AutoSSH -f "nagios_link" -p /usr/bin/autossh -a " -M <port + 1000>:<port + 2000> -N -R <port>:<ssh port> <user>@<nagios server>" Use services.msc to set restart options and credentials: – cyg_server – <created Nagios user> – Local Administrator
  • 34. Step 2 – OSX Install autossh with homebrew Command /usr/local/bin/autossh -M <port + 1000>:<port + 2000> -N -R -p 2206 -g 4000: <user>@<nagios server> -tt Launchctl Can install AutoSSH as System Service Autostart at boot
  • 35. Step2 – OSX launchctl Autossh file template ID=username HOST=hostname if [ "X$SSH_AUTH_SOCK" = "X" ]; then eval `ssh-agent -s` ssh-add $HOME/.ssh/id_rsa fi #AUTOSSH_POLL=600 #AUTOSSH_PORT=20000 #AUTOSSH_GATETIME=30 #AUTOSSH_LOGFILE=$HOST.log #AUTOSSH_DEBUG=yes #AUTOSSH_PATH=/usr/local/bin/ssh export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_DEBUG AUTOSSH_PATH AUTOSSH_GATETIME AUTOSSH_PORT autossh -2 -fN -M 20000 -R 2200:localhost:22 ${ID}@${HOST}
  • 36. Step 3 – Nagios to Remote Setup access from Nagios user or process to remote machine
  • 37. Step 3 - 'Nix Copy Nagios user public key to remote Note: Nagios may use different UID than the one for testing – cat ~/.ssh/ – <highlight> – ssh <remote server> – vim .ssh/authorized_keys – G o <shift insert> – :x
  • 38. Step 3 – Windows Same as 'Nix
  • 39. Step 3 – OSX Same as 'Nix
  • 40. What was just created? Take note – at the current time there are working ssh connections to: – Any 'Nix – Any Windows* – OSX These connections can be used for active checks, as well as troubleshooting and setting up an RDP or X session if needed. * There sometimes are authentication issues with Windows Domain servers that must be handled uniquely.
  • 41. Using Checks
  • 42. Normal [forward] checks Local checks define service{ use local-service host_name Nagios service_description Root Partition check_command check_local_disk!20%!10%!/ } Checks via ssh connection define host{ use openbsd-server host_name mx1 alias mail_mx1_server address icon_image envelope.gif statusmap_image envelope.gif check_command check_smtp }
  • 43. Reverse Check Same as other checks, except command format includes additional data define service{ use generic-service host_name hanley service_description SSH Check Proc normal_check_interval 15 retry_check_interval 5 notifications_enabled 1 check_command check_by_ssh_reverse!-p2210!proc } Additional data required in command – ssh Private key – ssh Port – Host address
  • 44. Command Comparison Local command_name check_local_disk command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p } Forward command_name check_by_ssh command_line /usr/bin/ssh -i <nagios user>/.ssh/id_rsa nagios@$HOSTADDRESS$ $ARG1$ 2>/dev/null Reverse command_name check_by_ssh_reverse command_line /usr/bin/ssh $ARG1$ -i <nagios user>/.ssh/id_rsa nagios@$HOSTADDRESS$ $ARG2$ 2>/dev/null
  • 45. Building Plugins Create test machine VMs work great! – Linux – Windows – OSX Compile set of plugins for that platform Maintain platform plugin set
  • 46. Gui's - RDP or X
  • 47. X apps run natively Be sure to include “-X” in tunnel setup Connect to remote with “ssh -X” Launch app Possibly explicit invocation options, a la Firefox: firefox –no-remote Start firefox, but run on the remote machine, forwarding the display to the Nagios server.
  • 48. RDP on demand On remote machine # # Reverse ssh link for RDP tunnel # ssh -f -n -N -R 3389: <user>@<nagios server> -p <ssh port> Tunnel 3389 on remote machine to 3389 on Nagios Server Check for open port netstat -an | grep 3389 Use one connection at a time for sanity!
  • 49. Troubleshooting
  • 50. Common Problems Host Key Verification Failed – The Host Key on the sending side is not the same as seen last time – Possible cause - the remote machine has been rebuilt Troubleshooting checks – $64K Secret – tmux – Very step-by-step – Paremeters often not required Useful aliases
  • 51. ●Hostkey verification failed @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The ECDSA host key for [storm]:2206 has changed, and the key for the corresponding IP address []:2206 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is 6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87. Please contact your system administrator. Add correct host key in /home/lvl/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/lvl/.ssh/known_hosts:5 You can use following command to remove all keys for this IP: ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts Host key verification failed.
  • 52. tmux tmux is a terminal multiplexer A terminal multiplexer permits switching between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. Essential for troubleshooting checks!
  • 53. tmux session
  • 54. Useful Aliases alias nagc='cd /etc/nagios' alias nagh='cd /var/lib/nagios' alias naghup='ps ax|grep /sbin/nagios |grep -v grep |awk "{print $1}" | xargs kill -HUP' alias nagq='netstat -an | grep 127 | grep LISTEN | grep -v 25 | sort --field-separator=''':''' --key=2.1' alias nagv='/usr/sbin/nagios -v /etc/nagios/nagios.cfg'
  • 55. Resources SSH and Authentication SSH Cookbook – Tunnels k/Tunnels
  • 56. Thank you! Leland Lammert