Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Nagios Conference 2014 - Mike Weber - Expanding NRDS Capabilities on Linux Systems

860 views

Published on

Mike Weber's presentation on Expanding NRDS Capabilities on Linux Systems.
The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Nagios Conference 2014 - Mike Weber - Expanding NRDS Capabilities on Linux Systems

  1. 1. NRDS: Enhancements Mike Weber mweber@spidertools.com
  2. 2. NRDS: Introduction • Basic Functionality Passive checks with Active Config Updates • Multiple Configurations Group Check Categories Create Multiple Check Times • Store­and­Forward Create timestamps to maintain data in disconnects.
  3. 3. Basic Functions • Passive Checks • Active Configuration Updates
  4. 4. Multiple Configurations • Multiple Group Checks Grouping checks based on type provide more efficient management aspects. (Linux_OS, Oracle, Apache, etc.) • Multiple Check Times Multiple check times lessens the load on the server and provides best practice use of resources. (5 min., 30 min.,etc.) • Multiple Download Options NRDS can be used to download, packages, plugins, scripts,etc.
  5. 5. Multiple Configurations Why Multiple Configurations • Reduce Load on Server • Logs Analyzed for Summaries daily, weekly, monthly • Low Tech Skills • Limited Host Access
  6. 6. Multiple Configurations Spreading the Load on a Busy Server command[Calls: Daily Guest]=/usr/lib/nagios/plugins/daily_guest.sh 2am command[Calls: Daily Summary]=/usr/lib/nagios/plugins/daily_summary.sh command[Calls: Daily Voicemail]=/usr/lib/nagios/plugins/daily_voice.sh 2.10am command[Daily Web Consol Use]=/usr/lib/nagios/plugins/check_web_consol command[Calls: Daily Room]=/usr/lib/nagios/plugins/daily_room.sh 2.20am command[Calls: Daily Room­Room]=/ usr/lib/nagios/plugins/daily_room_room.sh 2.30am command[Calls: Daily Admin]=/usr/lib/nagios/plugins/daily_admin.sh 2.35am command[Calls: Daily Admin­Guest]=/ usr/lib/nagios/plugins/daily_admin_guest.sh 2.45am command[Calls: Daily Admin­Room]=/ usr/lib/nagios/plugins/daily_admin_room.sh 2.55am command[Calls: Daily Admin­VoiceM]=/ usr/lib/nagios/plugins/daily_admin_voicemail.sh
  7. 7. Multiple Configurations calls/min/ave/max Calls: Daily Guest 2013­09­05 12:31:46 from_guest:1054/0/18/446 Calls: Daily Room 2013­09­01 20:07:23 from_room:127/0/15/194 Calls: Daily Room­Room 2013­09­01 20:07:23 from_room_to_room:2/0/0/1 Calls: Daily Voicemail 2013­09­01 20:07:23 from_guest_to_voicemail:24/3/15/56 Calls: Monthly Guest 2013­09­05 03:20:26 from_guest:1054/0/18/446 Calls: Monthly Room 2013­09­05 03:41:04 from_room:127/0/15/194 Calls: Monthly Room­Room 2013­09­05 03:51:08 from_room_to_room:2/0/0/1
  8. 8. Multiple Configurations
  9. 9. Multiple Configurations Install Procedure SSH into the hotel pbx and become root. sudo su cd /tmp wget ­­no­check­certificate ­O install.tar.gz "https://192.168.1.1/nrdp/? cmd=nrdsgetclient&token=Gh78TFcD&configname= Unpack the package tar xzvf install.tar.gz Install the package cd clients ./installnrds install 30 Note, this install script will be used for every hotel. It is going to install NRDS and download the package for the rest of the installation. Any host called install in the Unconfigured Objects should be removed,
  10. 10. Multiple Configurations Install the Package cd /tmp tar xvf package64.tar cd package Now you will execute a script that will ask for the name of the hotel server and the IP Address of the hotel server, be sure these are absolutely correct as the script will change all of the files to reflect the options entered. Enter the host when asked. Now execute the install script. ./install.sh
  11. 11. Multiple Configurations Purpose of package64.tar • Download All Plugins • Download All NRDS Groups When the script executes it will detect and setup the proper groups. • Download install.sh
  12. 12. Multiple Configurations
  13. 13. Multiple Configurations
  14. 14. Multiple Configurations Group Configuration Files All Point to One Hostname admin_server.cfg admin_server.pl admin_server_updater.pl app_server.cfg app_server.pl app_server_updater.pl custom.cfg custom.pl custom_server_updater.pl oracle_db_server60.cfg oracle_db_server60.pl oracle_db_server60_updater.pl oracle_db_server.cfg oracle_db_server.pl oracle_db_server_updater.pl
  15. 15. Multiple Configurations Admin_Server.cfg CONFIG_VERSION="0.2" CONFIG_OS="Linux" CONFIG_NAME="oracle_admin_server" URL="http://192.168.5.5/nrdp/" TOKEN="j5pj5adf89" PLUGIN_DIR="/usr/local/nagios/libexec" SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh" TMPDIR="/usr/local/nrdp/clients/tmp" COMMAND_PREFIX="" LOG_FILE="" UPDATE_CONFIG="1" UPDATE_PLUGINS="1"
  16. 16. Multiple Configurations Admin_Server.pl Required Changes to Custom Groups my $nrds_updater = "/usr/local/nrdp/clients/nrds/admin_server_updater.pl"; my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";
  17. 17. Multiple Configurations Admin_Server_updater.pl Required Changes to Custom Groups my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";
  18. 18. Multiple Configurations Install.sh #!/bin/bash CHECK_ORACLE_HEALTH_FILE=/usr/local/nagios/libexec/check_oracle_heal th DEFAULT_CONFIG_FILE="/tmp/package64/nrds/custom.cfg" ##### 64 Bit Nagios Monitoring Install ##### arch=$(uname ­m) if [ ${arch} == 'x86_64' ]; then echo "64_bit Operating System Detected" yum install ­y libaio rpm ­Uvh /tmp/package64/rpms/oracle­instantclient11.2­basic­11.2.0.2.0. x86_64.rpm rpm ­Uvh /tmp/package64/rpms/oracle­instantclient11.2­devel­11.2.0.2.0. x86_64.rpm
  19. 19. Multiple Configurations Install.sh ##### Install Nagios Directories and Plugins ##### useradd nagios mkdir ­p /usr/local/nagios/libexec cd plugins cp check* /usr/local/nagios/libexec/ cp urlize /usr/local/nagios/libexec/ cp negate /usr/local/nagios/libexec/ chown ­R nagios:nagios /usr/local/nagios/libexec/ chmod +x /usr/local/nagios/libexec/*
  20. 20. Multiple Configurations Install.sh ##### Install check_oracle_health ##### cd .. cd support export ORACLE_HOME="/usr/lib/oracle/11.2/client64" export LD_LIBRARY_PATH="$ORACLE_HOME/lib" tar zxvf check_oracle_health­1.7.8.1. tar.gz cd check_oracle_health­1.7.8.1 ./configure make make install mkdir /var/tmp/check_oracle_health chown ­R nagios /var/tmp/check_oracle_health
  21. 21. Multiple Configurations Install.sh ##### VNC Ports ##### config_file=${DEFAULT_CONFIG_FILE} VNC_START_PORT=5900 vnc_ports=$(sudo netstat ­anptu | grep 'vnc[[:space:]]*$'| tr ­s ' '|cut ­d ' ' ­f4| cut ­d ':' ­f2) i=0 for port in ${vnc_ports} ; do : $((i++)) printf "command[Check VNC %02d]=/usr/local/nagios/libexec/check_tcp ­H localhost ­p %d ­w 5 ­c 8 ­e n" ${i} ${port} >> ${config_file} done
  22. 22. Multiple Configurations Install.sh ##### CPU Core Set Up ##### arg0=2 arg1=1.5 arg2=1 arg3=3 arg4=2.5 arg5=0.5 cpu_nb=$(grep ­c ^processor /proc/cpuinfo) arg0=$((cpu_nb * arg0)) arg1=$(echo "scale=1;${cpu_nb} * ${arg1}" | bc) arg1=$(echo ${arg1} | sed ­e 's/.0$//') arg2=$((cpu_nb * arg2))
  23. 23. Multiple Configurations Install.sh ##### Set Hostname ##### host=$(hostname) echo $host echo "#####" echo "Use the hostname above as a base as you type the server name" read name echo "#####" oldString="txb" newString="$name" dpath="/tmp/package64/nrds/*.cfg" sed ­i "s/${oldString}/${newString}/g" /tmp/package64/crontab for i in $dpath; do
  24. 24. Multiple Configurations Install.sh ##### Determine Linux Version ##### version=$(lsb_release ­rs| cut ­f1 ­d.) echo $version if [ $version == '6' ]; then echo "version 6" yum install ­y glibc* perl­YAML perl­Time­HiRes elif [ $version == '5' ]; then echo "version 5" yum install ­y glibc* perl­YAML rpm ­ivh /tmp/package64/rhel5/perl­Time­HiRes­1.9724­1. el4.rfx.x86_64.rpm else
  25. 25. Multiple Configurations Install.sh ##### Manage Nagios Environement ##### echo 'NAGPATH=/usr/local/nagios/libexec; export NAGPATH' >> /home/nagios/.bash_profile echo 'LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib; export LD_LIBRARY_PATH' >> /home/nagios/.bash_profile echo 'ORACLE_HOME=/usr/lib/oracle/11.2/client64; export ORACLE_HOME' >> /home/nagios/.bash_profile echo 'TNS_ADMIN=/home/nagios; export TNS_ADMIN' >> /home/nagios/.bash_profile echo 'PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/nagios/bin: $ORACLE_HOME/bin; export PATH' >> /home/nagios/.bash_profile echo 'HOME=/home/nagios; export HOME' >> /home/nagios/.bash_profile
  26. 26. Multiple Configurations Install.sh ##### Oracle DB Plugins Install ##### cp /tmp/package64/nrds/oracle_db* /usr/local/nrdp/clients/nrds/
  27. 27. Multiple Configurations Install.sh ##### App Server Install ##### APP_SERVER=$(ps ­ef| grep tnslsnr|grep ­v grep|wc ­l) if [ ${APP_SERVER} == '1' ]; then echo "Installing Application Server" cp /tmp/package64/nrds/oracle_app* /usr/local/nrdp/clients/nrds/ sed ­i 's/#*/5/*5/' /tmp/package64/crontab else echo "No Application Server Installed...no plugins will be installed for App Server" fi
  28. 28. Multiple Configurations Install.sh ##### Admin Server Install ##### ADMIN_SERVER=$(ps ­ef| grep FNDLIBR|grep ­v grep|wc ­l) if [ ${ADMIN_SERVER} == '1' ]; then echo "Installing Admin Server" cp /tmp/package64/nrds/oracle_admin* /usr/local/nrdp/clients/nrds/ sed ­i 's/#*/6/*6/' /tmp/package64/crontab else echo "No Admin Server Installed...no plugins will be installed for Admin Server" fi
  29. 29. Multiple Configurations Install.sh ##### Move New Configs ##### cd /tmp/package64/nrds cp nrds* /usr/local/nrdp/clients/nrds/ cp custom* /usr/local/nrdp/clients/nrds/ chown nagios:nagios /usr/local/nrdp/clients/nrds/* ##### Install Cron Jobs ##### /tmp/package64/crontab exit
  30. 30. Multiple Configurations Crontab Script #!/bin/bash crontab ­u nagios ­r crontab ­u nagios ­l | { cat; echo "### Cron Jobs for nagios ### ##### OS Metric Checks and Oracle Server Checks ##### */5 * * * * /usr/local/nrdp/clients/nrds/nrds.pl ­H txb 2>&1 */4 * * * * /usr/local/nrdp/clients/nrds/custom.pl ­H txb 2>&1 */6 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server.pl ­H txb 2>&1 #*/5 * * * * /usr/local/nrdp/clients/nrds/oracle_app_server.pl ­H txb 2>&1 #*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_admin_server.pl ­H txb 2>&1 01 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server60.pl ­H txb 2>&1"; } | crontab ­u nagios ­exit
  31. 31. Store-and-Forward • Basic Problem Network connections broken for multiple days. • Solution: NRDS Timestamps Using timestamps NRDS is able to rebuild data and graphing on the Nagios server.
  32. 32. Store-and-Forward • Example: Broken Connection Network connections breaks 5:17 AM.
  33. 33. Store-and-Forward • Example: Broken Connection Network connections breaks 5:17 AM.
  34. 34. Store-and-Forward • Example: Broken Connection Network connections restored 6:30 AM.
  35. 35. Store-and-Forward • Example: Broken Connection Network connections restored 6:30 AM.
  36. 36. Store-and-Forward • Example: Broken Connection Network connections breaks 2:20 PM.
  37. 37. Store-and-Forward • Example: Broken Connection Network connections breaks 2:20 PM.
  38. 38. Store-and-Forward • Example: Broken Connection Network connections restored 6:26 AM (16 hours).
  39. 39. Store-and-Forward • Example: Broken Connection Network connections restored 6:26 AM (16 hours).
  40. 40. Store-and-Forward NRDS Client Add a cache line in /usr/local/nrdp/clients/nrds/nrds.cfg CACHEDIR="/usr/local/nrdp/clients/cache" Create folder /usr/local/nrdp/clients/cache The cache needs to collect timestamped data in order to rebuild the information later. mkdir /usr/local/nrdp/clients/cache chown nagios.nagios /usr/local/nrdp/clients/cache chmod 0775 /usr/local/nrdp/clients/cache Timestamps Being Created drwxr­xr­x 2 nagios nagios 4096 Sep 20 07:39 mylock ­rw­­­­­­­1 nagios nagios 1742 Sep 20 07:24 nrdp.6pvqRR1IJ ­rw­­­­­­­1 nagios nagios 1742 Sep 20 07:30 nrdp.B6qVekdsa
  41. 41. Store-and-Forward NRDS Client Replace /usr/local/nrdp/clients/send_nrdp.sh original if [ "x$url" == "x" ­o "x$token" == "x" ] echo "Usage: send_nrdp ­u url ­t token" Reference to cache directory if [ "x$url" == "x" ­o "x$token" == "x" ­o "x$directory" == "x" ] echo "Usage: send_nrdp ­u url ­t token ­D directory"
  42. 42. Store-and-Forward NRDS Client Replace /usr/local/nagios/libexec/process_perfdata.pl chown nagios.nagios /usr/local/nagios/libexec/process_perfdata.pl chmod 0755 /usr/local/nagios/libexec/process_perfdata.pl if ( $NAGIOS{PERFDATA} =~ /s*__check_timestamp__=(d+)/ ) { $NAGIOS{TIMET} = $1; print_log( "Found encoded timestamp ($NAGIOS{TIMET}) in $NAGIOS{HOSTNAME} / $NAGIOS{SERVICEDESC} ($NAGIOS{PERFDATA})", 1 ); # remove encoded timestamp $NAGIOS{PERFDATA} =~ s/(s*__check_timestamp__=d+)//; print_log( "Perfdata now is: ($NAGIOS{PERFDATA})", 2 ); 218a231,232
  43. 43. Script Downloads NRDS Modifications: Beginlinuxservers.com/nagiosconf User: nagiosconference Password: 54TBwh9 Only available during conference.
  44. 44. Conclusion Mike Weber mweber@spidertools.com

×