OSMC 2014: Server Hardware Monitoring done right | Werner Fischer

Server Hardware Monitoring
done right!
Werner Fischer, Thomas-Krenn.AG

2
Status quo
_ Überwachen Sie Ihre Server Hardware?
Ja Nein

Nach diesem Vortrag
überwachen Sie sicherer
und umfangreicher
3

Nach diesem Vortrag
überwachen Sie sicherer
und umfangreicher
(hoffe ich zumindest... ;-)
4

5
Status quo
_ Welche Technologien nutzen Sie?
IPMI / SNMP NRPE CAM CAT

7
CATinspection → satification?

8
Agenda
_ IPMI (20')
_ RAM (5')
_ RAID (10')
_ SMART (5')
_ GPU (5')

9
monitor your
IPMI-Sensors!

12
Intelligent Platform Management
Interface

13
 Monitoring
(temp, fans, ...)
 Recovery Control
(on/off/reset)
 Logging
(System Event Log)
Inventory
(FRU information)
Funktionen

FRU Temp.
sensor
…
Chassis board
14
Aufbau
Motherboard
private mgmt. busses
Processor
board
Memory
board
Zugriff mit
Benutzername
& Passwort
Baseboard
Management
Controller
(BMC)
System bus
NVS Storage
SDR
SEL
FRU
Chassis
mgmt.
(Satellite
Controller)
Sensors & Controls
Fan sensor
Temp. sensor
Power control
Reset control
…
FRU
Temp. s.
FRU
IPMB
LAN
interface
Serial
Port
Sharing
M/B
Serial
Controller
BMC
Serial
Controller
Serial/Modem
interface
Serial
Connector
LAN
Connector
PCI mgmt. bus
Network
(LAN)
Controller
Remote Mmgt. Card
(KVM over IP, ...)
Auxillary
IPMB Connector
ICMB
ICMB
bridge
System
interface
Redundant Power
board
FRU
Zugriff mit
root Rechten

15
IPMI Sensor Klassen
Discrete (True/False) Threshold (Schwellwerte)
Mehrere Zustände möglich:
● bis zu 15 Status möglich
● jeder Status = 1 Bit
● mehrere aktive Statusbits möglich
Zustand abhängig von:
● Vergleich analoger Messert mit dem
Schwellwerten (Thresholds)
Liefert:
● allgemeine Zustände
● Sensor-spezifische Zustände
Liefert:
● analogen Messwert
● diskreten Status
Ähnliche Klasse OEM
● Bedeutung der Zustände werden
vom OEM definiert

16
IPMI Sensor Klassen
Discrete Threshold
[root@test ~]# ipmitool sdr get "PS2 Status"
Sensor ID : PS2 Status (0x71)
Entity ID : 10.2 (Power Supply)
Sensor Type (Discrete): Power Supply
States Asserted : Power Supply
[Presence detected]
[Power Supply AC
lost]
Assertion Events : Power Supply
[Presence detected]
[Power Supply AC
lost]
Assertions Enabled : Power Supply
[Presence detected]
[Failure detected]
[Predictive failure]
[Power Supply AC
lost]
[...]
Deassertions Enabled : Power Supply
[...]
[root@test ~]# ipmitool sdr get "Fan 1"
Sensor ID : Fan 1 (0x50)
Entity ID : 29.1 (Fan
Device)
Sensor Type (Analog) : Fan
Sensor Reading : 5719 (+/0)
RPM
Status : ok
Nominal Reading : 6708.000
Normal Minimum : 2451.000
Normal Maximum : 10965.000
Lower critical : 1720.000
Lower noncritical
: 1978.000
Positive Hysteresis : 86.000
Negative Hysteresis : 86.000
Minimum sensor range : Unspecified
Maximum sensor range : Unspecified
Event Message Control : Perthreshold
Readable Thresholds : lcr lnc
Settable Thresholds : lcr lnc
Threshold Read Mask : lcr lnc
Assertion Events :
Assertions Enabled : lnclcrDeassertions
Enabled : lnclcr

$ sudo ipmisensors
outputsensorstate
interpretoemdata
Password:
ID | Name | Type | State | Reading | Units | Event
4 | System Temp | Temperature | Nominal | 27.00 | C | 'OK'
71 | Peripheral Temp | Temperature | Nominal | 35.00 | C | 'OK'
138 | CPU Temp | OEM Reserved | Nominal | N/A | N/A | 'Low'
205 | FAN 1 | Fan | Nominal | 1800.00 | RPM | 'OK'
… 942 | VBAT | Voltage | Nominal | 3.15 | V | 'OK'
1009 | VSB | Voltage | Nominal | 3.34 | V | 'OK'
1076 | AVCC | Voltage | Nominal | 3.38 | V | 'OK'
1143 | Chassis Intru | Physical Security | Critical | N/A | N/A | 'Gen...'
17
IPMI Sensoren OK
Critical

18
IPMI Sensoren (Discrete)
$ cat /etc/freeipmi/freeipmi_interpret_sensor.conf
[…]
## IPMI_Physical_Security
#
# IPMI_Physical_Security_No_Event Nominal
# IPMI_Physical_Security_General_Chassis_Intrusion Critical
# IPMI_Physical_Security_Drive_Bay_Intrusion Critical
[…]
# IPMI_Power_Supply_No_Event Nominal
# IPMI_Power_Supply_Presence_Detected Nominal
# IPMI_Power_Supply_Power_Supply_Failure_Detected Critical
# IPMI_Power_Supply_Predictive_Failure Critical
# IPMI_Power_Supply_Power_Supply_Input_Lost_AC_DC Critical
[…]

$ ./check_ipmi_sensor H
192.168.255.5 f
ipmi.cfg vv
IPMI Status: OK | 'System Temp'=27.00 'Peripheral Temp'=35.00 'FAN
1'=1800.00 'Vcore'=0.98 '3.3VCC'=3.36 '12V'=11.93 'VDIMM'=1.53
'5VCC'=5.09 '12V'=
12.09
'VBAT'=3.15 'VSB'=3.34 'AVCC'=3.38
System Temp = 27.00 (Status: Nominal)
Peripheral Temp = 35.00 (Status: Nominal)
CPU Temp = 'Low' (Status: Nominal)
FAN 1 = 1800.00 (Status: Nominal)
Vcore = 0.98 (Status: Nominal)
3.3VCC = 3.36 (Status: Nominal)
12V = 11.93 (Status: Nominal)
VDIMM = 1.53 (Status: Nominal)
5VCC = 5.09 (Status: Nominal)
12V
= 12.09
(Status: Nominal)
VBAT = 3.15 (Status: Nominal)
VSB = 3.34 (Status: Nominal)
AVCC = 3.38 (Status: Nominal)
Chassis Intru = 'OK' (Status: Nominal)
19
IPMI Plugin

20
IPMI Plugin
#!/usr/bin/perl
# check_ipmi_sensor: Nagios/Icinga plugin to check IPMI sensors
##
Copyright (C) 20092014
ThomasKrenn.
AG,
# additional contributors see changelog.txt
##
This program is free software; you can redistribute it and/or modify it under
[…]
Version 3.5 20141031
* Fix LAN Driver if called on localhost
Version 3.4 20140929
* Fix implicit array warning with split
* Add option to disable LAN protocol version 2.0
Version 3.3 20140606
* Print a warning if ipmisensors
only returned a single output row
* Ignore sudo errors and warnings in IPMI command output
(Thanks to Robert Heinzmann for contributing)
* Use LAN protocol version 2.0 per default
* Print empty output error only if return code was 0
* Exit the plugin with return code 3 if fru command fails
* Added an include list option to only include specific sensors
Version 3.2 20131028
* Added FRU serial number to output

Intelligent? Platform Management
22
Interface

Das Abhörsystem
in ihrem Computer
24
The Eavesdropping System in Your Computer
(Bruce Schneier, Schneier on Security Blog 31.01.2013)

230.000 1HE Server
→ 10.223,5 m Höhe
(Mount Everest 8.848 m)
27

29
IPMI Firmware by ATEN / AMI
_ Mainboard-Hersteller
passen Firmware an
_ OS = Embedded Linux
_ IPMI Firmware Teile
Closed-Source

Wir empfehlen administrative Zugänge
wie IPMI- aber auch etwa SSH-Dienste
nicht offen im Internet zu betreiben,
30
sondern mittels Firewall/VPN den
Zugriff auf solche Dienste
ausschließlich berechtigten Personen
zu ermöglichen.

31
Was wenn doch?
Enable
&DROP

32
IPMI Top 3
Sicherheitstipps

35
#2 – User Management
sjfaiklaz afjhuijoh
Administrator
User

In short, the authentication process for IPMI 2.0 mandates
that the server send a salted SHA1 or MD5 hash of the
requested user's password to the client, prior to the client
authenticating.
36
A Penetration Tester's Guide to IPMI and BMCs (rapid7.com)
msf > use auxiliary/scanner/ipmi/ipmi_dumphashes
msf auxiliary(ipmi_dumphashes) > set RHOSTS 10.1.102.141
RHOSTS => 10.1.102.141
msf auxiliary(ipmi_dumphashes) > set THREADS 128
THREADS => 128
msf auxiliary(ipmi_dumphashes) > run
[+] 10.1.102.141:623 - IPMI - Hash found:
admin:14667523250000004ec525d3852f4fa73c93b674788217fe00000000000000
00000000000000000000000000000000000000000000000000140561646d696e:2c7
6e372d89ac7cd4e3bfecb423962f708d0741c

37
$ ./cudaHashcat64.bin --outfile=ipmi.out -m 7300 hash.txt -a 3 ?lu?
lu?lu?lu?lu?lu
[...]
Session.Name...: cudaHashcat
Status.........: Exhausted
Input.Mode.....: Mask (?lu?lu?lu?lu?lu?lu) [12]
Hash.Target....:
54414378fb2db5ff365e4bc5856adaf4c1b8a2f2153efd1b81fb54dfe1bf56478788
ea7ba154375b40167e34f026e1020010d21d1ea31625040561646d696e:0a0b16023
1e204a6d0bd086e26718002409b35b7
Hash.Type......: IPMI2 RAKP HMAC-SHA1
Time.Started...: Thu Sep 18 10:11:17 2014 (6 secs)
Time.Estimated.: 0 secs
Speed.GPU.#1...: 52732.3 kH/s
Recovered......: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.......: 308915776/308915776 (100.00%)
Skipped........: 0/308915776 (0.00%)
Rejected.......: 0/308915776 (0.00%)
HWMon.GPU.#1...: -1% Util, 41c Temp, 31% Fan

38
20
Komplexe
& lange
Passwörter

42
monitor your RAM!
(it's ECC, isn't it?)

44
3%
min 1 CE/Jahr (DDR2)
Google 2009, Jaguar-Cluster 2012

45
70%
CE's vor UE's
Google 2009

1,3%
46
Server mit UE's/Jahr
Google 2009

root@debiantest:/
sys/devices/system/edac/mc/mc0/csrow0# ls l
total 0
rrr1
root root 4096 Nov 12 09:02 ce_count
rrr1
root root 4096 Nov 12 09:02 ch0_ce_count
rwrr1
root root 4096 Nov 12 09:02 ch0_dimm_label
rrr1
root root 4096 Nov 12 09:02 ch1_ce_count
rwrr1
root root 4096 Nov 12 09:02 ch1_dimm_label
rrr1
root root 4096 Nov 12 09:02 dev_type
rrr1
root root 4096 Nov 12 09:02 edac_mode
rrr1
root root 4096 Nov 12 09:02 mem_type
drwxrxrx
2 root root 0 Nov 12 09:02 power
rrr1
root root 4096 Nov 12 09:02 size_mb
lrwxrwxrwx 1 root root 0 Nov 12 09:02 subsystem >
../../../../../../bus/mc0
rrr1
root root 4096 Nov 12 09:02 ue_count
rwrr1
root root 4096 Nov 12 09:02 uevent
root@debiantest:/
sys/devices/system/edac/mc/mc0/csrow0# cat ce_count
47
0 root@debiantest:/
sys/devices/system/edac/mc/mc0/csrow0# cat ue_count
0
Linux EDAC

Linux EDAC Supportmatrix
Treibermodul CPUs Kernel Unterstützte Architekturen
amd64_edac.c AMD 2.6.31
48
2.6.39
3.10
3.13
3.15
K8 und F10
F15
F16
F15_M30H
F16_M30H
i7core_edac.c Intel Single/Dual 2.6.35 Nehalem/Westmere
ie31200_edac.c Intel Single-CPU 3.17 Sandy & Ivy Bridge
Haswell
sb_edac.c Intel Dual-CPU 3.2
3.13
3.17
Sandy Bridge
Ivy Bridge
Haswell

53
Linux
Software
RAID
LSI / Adaptec
Hardware
RAID

55
root@debiantest:~#
storcli64
Storage Command Line Tool Ver 1.13.06 Sep 03, 2014
(c)Copyright 2014, LSI Corporation, All Rights Reserved.
help lists
all the commands with their usage. E.g. storcli help
<command> help gives
details about a particular command. E.g. storcli add help
List of commands:
Commands Description
add
Adds/creates a new element to controller like VD,Spare..etc
delete Deletes an element like VD,Spare
show Displays information about an element
set Set a particular value to a property
get Get a particular value to a property
compare Compares particular value to a property
start Start background operation
stop Stop background operation
pause Pause background operation
resume Resume background operation
download Downloads file to given device
expand expands size of given drive
insert inserts new drive for missing
transform downgrades the controller
/cx Controller specific commands
/ex Enclosure specific commands
/sx Slot/PD specific commands
/vx Virtual drive specific commands
/dx Disk group specific commands
/fall Foreign configuration specific commands
/px Phy specific commands
/[bbu|cv] Battery Backup Unit, Cachevault commands

$ /usr/lib/nagios/plugins/check_lsi_raid vv
Warning (LD Warn) [c0/v0_Consist = Warning (No)]|
CV_Temperature=22;70;85 ROC_Temperature=57;80;90
c0/e252/s0_Drive_Temperature=21;40;45
c0/e252/s1_Drive_Temperature=21;40;45
Used storcli commands:
/
usr/bin/sudo /usr/sbin/storcli64 /c0 /cv show status
/
usr/bin/sudo /usr/sbin/storcli64 adpallinfo a0
/
usr/bin/sudo /usr/sbin/storcli64 /c0/vall show all
/
usr/bin/sudo /usr/sbin/storcli64 /c0/vall show init
/
usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show all
/
usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show initialization
/
usr/bin/sudo /usr/sbin/storcli64 /c0/eall/sall show rebuild
Warning sensors:
c0/
v0_Consist (No)
56
check_lsi_raid

Warum adpallinfo a0?
„storcli /0 show all …
blocks the whole raid card
i/o for … upto ~4 seconds“
57

Warum adpallinfo a0?
„storcli /0 show all …
blocks the whole raid card
i/o for … upto ~4 seconds“
58

59
check_lsi_raid
$ /usr/lib/nagios/plugins/check_lsi_raid h
check_lsi_raid: Nagios/Icinga plugin to check LSI Raid Controller status
Pulgin version: 2.0
ThomasKrenn.
AG
Current updates available at
http://git.thomaskrenn.
com/check_lsi_raid.git
This Nagios/Icinga Plugin checks LSI RAID controllers for controller,
physical device, logical device, BBU and CV warnings and errors.
In order for this plugin to work properly you need to add the nagios
user to your sudoers file (or create a new one in /etc/sudoers.d/).
Usage:
[ h
| help
]
Display this help page
[ v
| vv
| vvv
| verbose
]
Sets the verbosity level.
No v
is the normal single line output for Nagios/Icinga, v
is a
more detailed version but still usable in Nagios. vv
is a
multiline output for debugging configuration errors or more
detailed information. vvv
is for plugin problem diagnosis.
For further information please visit:
http://nagiosplug.sourceforge.net/developerguidelines.
html#AEN39
[ V
version
]
Displays the plugin and, if available, the version if StorCLI.
[ C
<num> | controller
<num> ]
Specifies a controller number, defaults to 0.
...

61
VMware? → Plugin
check_esxi_hardware.py check_vmware_esx.pl
Hardware VMware allgemein
python-pywbem VMware Perl SDK
Claudio Kuenzler et.al.
Infos:
Martin Fürstenau

VMware? check_esxi_hardware.py
62
#!/usr/bin/python
# *
coding:
UTF8
*

##
Script for checking global health of host running VMware ESX/ESXi
##
Licence : GNU General Public Licence (GPL) http://www.gnu.org/
# This program is free software; you can redistribute it and/or
...
# Copyright (c) 2008 David Ligeret
# Copyright (c) 2009 Joshua Daniel Franklin
# Copyright (c) 2010 Branden Schneider
# Copyright (c) 20102014
Claudio Kuenzler
# Copyright (c) 2010 Samir Ibradzic
# Copyright (c) 2010 Aaron Rogers
# Copyright (c) 2011 Ludovic Hutin
# Copyright (c) 2011 Carsten Schoene
# Copyright (c) 20112012
Phil Randal
# Copyright (c) 2011 Fredrik Aslund
# Copyright (c) 2011 Bertrand Jomin
# Copyright (c) 2011 Ian Chard
# Copyright (c) 2012 Craig Hart
# Copyright (c) 2013 Carl R. Friend

check_adaptec_raid Update
$ ./check_adaptec_raid p
/usr/sbin/arcconf
AACRAID CRITICAL (Ctrl #1): [ZMM critical]
$ ./check_adaptec_raid h
ThomasKrenn
Adaptec Raid Controller Nagios/Icinga Plugin Version: 1.0
ThomasKrenn.
AG
Current updates available via git at:
65
http://git.thomaskrenn.
com/check_adaptec_raid.git
This Nagios/Icinga Plugin checks ADAPTEC RAIDControllers
for Controller,
PhysicalDevice
and Logical Device warnings and errors.
In order for this plugin to work properly you need to add the
nagiosuser
to your sudoers file (or create a new one in /etc/sudoers.d/).
This is required as arcconf must be called with sudo permissions.
Usage:
[ C
<Controller number> ] [ LD
<Logical device number> ]
[ PD
<Physical device number> ] [ T
<Warning Temp., Crit. Temp.> ]
[ h
| help
]
Display this help page
[ v
| vv
| vvv
| verbose
]
Sets the verbosity level
no v
single line output for Nagios/Icinga
v
single line with more details
...
geplant
(2015)

VMware? → CIM Provider erwartet
_ aktuell:
66
_ „CIM Provider“ für remote arcconf
_ Adaptec MSM in einer VM
_ künftig:
_ „echter“ CIM Provider

68
Self-
Monitoring,
Analysis &
Reporting
Technology

69
Standardisiert NICHT standadisiert
Datenformat
Kommandos
Errorlogs
Tests
Attribute
Dokumentation
vom Hersteller
erforderlich
(oft nicht
öffentlich, außer
Intel/Samsung)

70
check_smart_attributes
$ /usr/lib/nagios/plugins/check_smart_attributes
> d
/dev/sda
> dbj
/etc/nagiosplugins/
config/check_smartdb.json
OK (sda) |sda_Media_Wearout_Indicator=098;16;6
sda_Host_Writes_32MiB=575272 sda_Host_Reads_32MiB=723527

/etc/nagiosplugins/
...
"Intel DC S3700" : {
"Device" : ["Intel DC S3700 Series SSDs","INTEL SSDSC2BA100G3",
"ID#" : {
"5" : "RAW_VALUE", # Reallocated
Sector Count
...
"194" : "RAW_VALUE", # Temperature Device
Internal Te
...
"232" : "VALUE", # Available Reserved Space
"233" : "VALUE", # Media Wearout Indicator
"234" : "VALUE", # Thermal Throttle Status
"241" : "RAW_VALUE", # Total LBAs Written (32MiB)
"242" : "RAW_VALUE", # Total LBAs Read (32MiB)
"1024" : "VALUE" # ATA error count (custom)
71
},
"Threshs" : {
"5" : ["20","40"],
...
"232" : ["16:","11:"],
"233" : ["16:","6:"],
"1024" : ["0","10"]
},
"Perfs" : ["194","233","241","242"]
},
...

/etc/nagiosplugins/
72
...

/etc/nagiosplugins/
Ständig neue SSDs&HDDs
73

/etc/nagiosplugins/
Ständig neue SSDs&HDDs
74
Aktualisierungen?

/etc/nagiosplugins/
Git(t) sei
Dank ;-)
75

ja cool, aber was ist mit RAID Controllern?
...
[d|
device
<path to device being checked>]
Specify the device being monitored. If multiple devices should be
checked provide the 'd'
option multiple times.
E.g. 'd
/dev/sda d
/dev/sdb'
For devices behind LSI RAID controllers specify 'megaraid' and then the
device number, e.g. 'd
megaraid6'. Use storcli to find out the
corresponding device numbers.
For devices behind Adaptec RAID controllers specify '/dev/sg<X>' where
<X> is the number for your device. Use e.g. sg_scan to find the device.
You must also use 'O
sat' or 'O
scsi' according to the device
interface. This are extra options only necessary for '/dev/sg<X>'
devices.
76
...

ja cool, aber was ist mit RAID Controllern?
> d
megaraid6
> dbj
/etc/nagiosplugins/
OK (megaraid6) |
megaraid6_Temperature_Internal=26
megaraid6_Media_Wearout_Indicator=100;16;6
megaraid6_Host_Writes_32MiB=70283
megaraid6_Host_Reads_32MiB=1650800
> d
megaraid7
> dbj
/etc/nagiosplugins/
Warning (megaraid7) [megaraid7_CRC_Error_Count = Warning]|
megaraid7_Temperature_Internal=34
megaraid7_Media_Wearout_Indicator=098;16;6
megaraid7_Host_Writes_32MiB=189904
megaraid7_Host_Reads_32MiB=29658
77

79
check_gpu_sensor
$ /usr/lib/nagios/plugins/check_gpu_sensor db
0000:83:00.0
OK Tesla
K20 |ECCL2AggSgl=0;1;2;
ECCTexAggSgl=0;1;2;
memUtilRate=0
PWRUsage=49.81;150;200;
ECCRegAggSgl=0;1;2;
SMClock=705
ECCL1AggSgl=0;1;2;
GPUTemperature=38;85;100;
memClock=2600
usedMemory=0.24;95;99;
fanSpeed=30;80;95;
graphicsClock=705
GPUUtilRate=0
ECCMemAggSgl=0;1;2;

NVIDIA: „angezeigte
Lüfterdrehzahl lässt nicht
darauf schließen, ob sich der
Lüfter tatsächlich dreht.“
80
„es ist jene Drehzahl, mit der der Lüfter-Algorithmus versucht den Lüfter zu betreiben.“
wir empfehlen:
„Temperatursensor“

81
Plugins - Future
_ Überwachung von
FW-Versionen
_ RAID Consistency
Checks
_ Temperatur von
10GBit NICs
(siehe Intel X540 FAQs)

83
Relax ...
_ alle Plugins unter git.thomas-krenn.com
_ alle Plugins erfüllen
Plugin Developer Guidelines (-h für Hilfe)
_ „Plugin Entwicklung für Einsteiger“
von Alexander Wirt heute um 14:15h

84
Relax, start ...
Serverliste
erstellen
IPMI
sicher
konfigurieren
relevante
Plugins
einrichten

85
Relax, start and have fun at

OSMC 2014: Server Hardware Monitoring done right | Werner Fischer

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to OSMC 2014: Server Hardware Monitoring done right | Werner Fischer

Similar to OSMC 2014: Server Hardware Monitoring done right | Werner Fischer (20)

Recently uploaded

Recently uploaded (20)

OSMC 2014: Server Hardware Monitoring done right | Werner Fischer