Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment


Published on

Leland Lammert's presentation on Nagios in a Multi-Platform Enviornment.
The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit:

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Nagios Conference 2013 - Leland Lammert - Nagios in a Multi-Platform Enviornment

  1. 1. Nagios in a Multi-Platform Environment Leland V. Lammert, PhD Chief Scientist Omnitec Corporation
  2. 2. The Problem Different OSs can require – Different type of connection – Different installation procedure Nagios checks or an agent require a connection to the remote machine (i.e. from the Nagios server to the remote machine) No issue inside the firewall There are issues for machines outside the firewall
  3. 3. The Solution – SSH SSH forward connections are a good solution for monitoring inside the firewall An ssh reverse tunnel is a good solution for monitoring outside the firewall Active checks can then be used on all systems – No Agents – No Complex Installation – A shell is always available for troubleshooting
  4. 4. 4 Today's Outline Secure Shell Connection Process Credentials Setting up a Reverse Tunnel Using checks Gui's - RDP or X Troubleshooting
  5. 5. Secure Shell
  6. 6. SSH History SSH [Secure Shell] is a data exchange protocol that uses an encrypted connection between two network devices SSH replaced Telnet and other insecure remote shells which send information, most notably passwords, in plaintext SSH encryption provides confidentiality and integrity of data over an insecure network Connection Types – Normal [Forward] – Tunnel [Reverse]
  7. 7. SSH Advantages Secure [ssh2] – Host Identity Verified – User Authorization – Secure Data Transmission Ubiquitous ('Nix, Windows [Cygwin], OSX) Reverse Tunnels require no firewall changes Allows Active Checks No agent to install or configure Simplified testing, GUI still available [X or RDP]
  8. 8. SSH Details Authentication is verified with SSH keys [Forward] ssh connections work for hosts behind the firewall [Reverse or Tunnel] ssh connections work for hosts anywhere else The tunnel passes traffic TO the remote machine FROM the Nagios Server (reverse operation) SSH Connections can be used to standardize connections to ALL machines!
  9. 9. Forward connections Requires – Static IP – Open port A static IP may not available at external locations An open port for an incoming connection is a BIG security problem Even when an agent is used [NSCA], there is no access to the remote machine for testing and troubleshooting
  10. 10. Reverse Connections - Tunnels A Tunnel is packet encapsulation using a network protocol The data payload protocol is then encapsulated in a delivery protocol. – L2TP (layer 2) Tunneling Protocol) – SSH – Socks Reverse Tunnels - an ssh connection from client to server, where data is transmitted from server to client (i.e. reverse)
  11. 11. Connection Process Startup – Start Session – Exchange Server key – Generate Session key Authentication – Exchange Client key – If ACK, continue
  12. 12. SSH Startup
  13. 13. SSH Authentication
  14. 14. SSH uses Asymetric Encryption Public/Private Keypair A Keypair verifies identity for: – Machine – User Currently keys are 2048 bits in length, usual RSA Each Key has two parts: – Public and Private – AKA the keypair Keys are created with ssh-keygen
  15. 15. Credentials
  16. 16. Host and User Credentials Host Key Verifies that the host/server at the other end of the connection is the same one seen previously User Keys Verifies that the user is authorized for the connection Why not Passwords? ● Requires manual entry ● Cannot be automated without storing as plaintext ● Much less secure than key
  17. 17. Keys on Unix/Linux Keys: /home/<user>/.ssh Private key: id_rsa Public key:
  18. 18. Keys on Windows Typical Cygwin install location C:Program FilesCygwin Home directories: <Cygwin>/home/<user>/.ssh Keys: ~/.ssh Private key: id_rsa Public key:
  19. 19. Keys on OSX Keys: /Users/<user>/.ssh Private key: id_rsa Public key:
  20. 20. Setting up a Reverse Tunnel
  21. 21. Basic ssh command ssh Base command -f Run in background -X Enable X Forwarding ('Nix) -n Prevent reading from STDIN -N No remote command -R Reverse Operation ssh -f -X -n -N -R
  22. 22. Ports REMOTE:* REMOTE Port on remote machine [from] localhost LOCAL Port on local machine [to] * Remember, this is being done at the external machine – so the “Remote” port here is actually on the Nagios server!
  23. 23. Authentication $USER_NAME@$REMOTE_HOST USER_NAME User ID on remote machine REMOTE _HOST Hostname or IP of remote hostt -p LOGIN_PORT LOGIN_PORT Port for login on remote machine -i IDENTITY_FILE IDENTITY_FILE Private key file to use
  24. 24. Three Steps to Create a Tunnel 1. Remote to Nagios Connect remote machine to Nagios server and create Tunnel 2. Make it permanent – cron – autossh – launchctl/launchd 3. Nagios to Remote Setup access from Nagios user or process to remote machine
  25. 25. Step 1. Remote to Nagios Connect remote machine to Nagios server and create Tunnel
  26. 26. Step 1 – 'Nix Create keypair on remote system and copy to Nagios Server – ssh-keygen – cat ~/.ssh/ ● <highlight> – ssh <nagios server> – vim ~/.ssh/authorized_keys – G o <shift insert> – :x <exit> test
  27. 27. Step 1 – Windows Install Cygwin ( – Install in c:program filesCygwin – Add autossh, rsync, bash, perl Create keypair on remote system and copy to Nagios Server [same as 'Nix]
  28. 28. Step 1 – OSX Create keypair on remote system and copy to Nagios Server [same as 'Nix]
  29. 29. Step 2. Make it permanent
  30. 30. Step 2 – 'Nix cron Connection variables REMOTE_HOST="" USER_NAME="________" REMOTE_PORT="____" LOCAL_PORT="____" * LOGIN_PORT="____" * IDENTITY_KEY="/home/nagios/.ssh/id_rsa" * NOTE: The ssh port should be changed for security, do NOT use the standard port 22.
  31. 31. Step 2 – 'Nix cron Command to create the link COMMAND="ssh -f -n -N -R $REMOTE_PORT:$LOCAL_PORT $USER_NAME@$REMOTE_HOST -p$LOGIN_PORT -i $IDENTITY_KEY"
  32. 32. Step 2 – 'Nix cron Running? If not, start it pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND Working? Login to Nagios and check from the other side ssh -i$IDENTITY_KEY -p$LOGIN_PORT $USER_NAME@$REMOTE_HOST netstat -an | egrep "tcp.*$REMOTE_PORT.*LISTEN" > /dev/null 2>&1 If not working, kill and restart. if [ $? -ne 0 ] ; then pkill -f -x "$COMMAND" $COMMAND fi
  33. 33. Step 2 – Windows autossh Install service cygrunsrv -I AutoSSH -f "nagios_link" -p /usr/bin/autossh -a " -M <port + 1000>:<port + 2000> -N -R <port>:<ssh port> <user>@<nagios server>" Use services.msc to set restart options and credentials: – cyg_server – <created Nagios user> – Local Administrator
  34. 34. Step 2 – OSX Install autossh with homebrew Command /usr/local/bin/autossh -M <port + 1000>:<port + 2000> -N -R -p 2206 -g 4000: <user>@<nagios server> -tt Launchctl Can install AutoSSH as System Service Autostart at boot
  35. 35. Step2 – OSX launchctl Autossh file template ID=username HOST=hostname if [ "X$SSH_AUTH_SOCK" = "X" ]; then eval `ssh-agent -s` ssh-add $HOME/.ssh/id_rsa fi #AUTOSSH_POLL=600 #AUTOSSH_PORT=20000 #AUTOSSH_GATETIME=30 #AUTOSSH_LOGFILE=$HOST.log #AUTOSSH_DEBUG=yes #AUTOSSH_PATH=/usr/local/bin/ssh export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_DEBUG AUTOSSH_PATH AUTOSSH_GATETIME AUTOSSH_PORT autossh -2 -fN -M 20000 -R 2200:localhost:22 ${ID}@${HOST}
  36. 36. Step 3 – Nagios to Remote Setup access from Nagios user or process to remote machine
  37. 37. Step 3 - 'Nix Copy Nagios user public key to remote Note: Nagios may use different UID than the one for testing – cat ~/.ssh/ – <highlight> – ssh <remote server> – vim .ssh/authorized_keys – G o <shift insert> – :x
  38. 38. Step 3 – Windows Same as 'Nix
  39. 39. Step 3 – OSX Same as 'Nix
  40. 40. What was just created? Take note – at the current time there are working ssh connections to: – Any 'Nix – Any Windows* – OSX These connections can be used for active checks, as well as troubleshooting and setting up an RDP or X session if needed. * There sometimes are authentication issues with Windows Domain servers that must be handled uniquely.
  41. 41. Using Checks
  42. 42. Normal [forward] checks Local checks define service{ use local-service host_name Nagios service_description Root Partition check_command check_local_disk!20%!10%!/ } Checks via ssh connection define host{ use openbsd-server host_name mx1 alias mail_mx1_server address icon_image envelope.gif statusmap_image envelope.gif check_command check_smtp }
  43. 43. Reverse Check Same as other checks, except command format includes additional data define service{ use generic-service host_name hanley service_description SSH Check Proc normal_check_interval 15 retry_check_interval 5 notifications_enabled 1 check_command check_by_ssh_reverse!-p2210!proc } Additional data required in command – ssh Private key – ssh Port – Host address
  44. 44. Command Comparison Local command_name check_local_disk command_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p } Forward command_name check_by_ssh command_line /usr/bin/ssh -i <nagios user>/.ssh/id_rsa nagios@$HOSTADDRESS$ $ARG1$ 2>/dev/null Reverse command_name check_by_ssh_reverse command_line /usr/bin/ssh $ARG1$ -i <nagios user>/.ssh/id_rsa nagios@$HOSTADDRESS$ $ARG2$ 2>/dev/null
  45. 45. Building Plugins Create test machine VMs work great! – Linux – Windows – OSX Compile set of plugins for that platform Maintain platform plugin set
  46. 46. Gui's - RDP or X
  47. 47. X apps run natively Be sure to include “-X” in tunnel setup Connect to remote with “ssh -X” Launch app Possibly explicit invocation options, a la Firefox: firefox –no-remote Start firefox, but run on the remote machine, forwarding the display to the Nagios server.
  48. 48. RDP on demand On remote machine # # Reverse ssh link for RDP tunnel # ssh -f -n -N -R 3389: <user>@<nagios server> -p <ssh port> Tunnel 3389 on remote machine to 3389 on Nagios Server Check for open port netstat -an | grep 3389 Use one connection at a time for sanity!
  49. 49. Troubleshooting
  50. 50. Common Problems Host Key Verification Failed – The Host Key on the sending side is not the same as seen last time – Possible cause - the remote machine has been rebuilt Troubleshooting checks – $64K Secret – tmux – Very step-by-step – Paremeters often not required Useful aliases
  51. 51. ●Hostkey verification failed @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The ECDSA host key for [storm]:2206 has changed, and the key for the corresponding IP address []:2206 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is 6b:da:2e:50:a9:ea:b0:b1:3d:c1:b8:4a:a3:a5:56:87. Please contact your system administrator. Add correct host key in /home/lvl/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/lvl/.ssh/known_hosts:5 You can use following command to remove all keys for this IP: ssh-keygen -R storm -f /home/lvl/.ssh/known_hosts Host key verification failed.
  52. 52. tmux tmux is a terminal multiplexer A terminal multiplexer permits switching between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal. Essential for troubleshooting checks!
  53. 53. tmux session
  54. 54. Useful Aliases alias nagc='cd /etc/nagios' alias nagh='cd /var/lib/nagios' alias naghup='ps ax|grep /sbin/nagios |grep -v grep |awk "{print $1}" | xargs kill -HUP' alias nagq='netstat -an | grep 127 | grep LISTEN | grep -v 25 | sort --field-separator=''':''' --key=2.1' alias nagv='/usr/sbin/nagios -v /etc/nagios/nagios.cfg'
  55. 55. Resources SSH and Authentication SSH Cookbook – Tunnels k/Tunnels
  56. 56. Thank you! Leland Lammert