Information Gathering 2

Information Gathering Michael Johnson SYS-TSC-VSP

Information Gathering BIOS / DMI / POST events - overview IPMI - Generic overview SEL, SP, IPMI, DMI and register data How to view gathered data and analysis of events Types of events available and were to look Normal / expected output Explorer, siga, sysreport, mpsreport, msinfo32 Working with non-service processor platforms

A Briefer on BIOS Execution Stages BIOS = Basic Input Output System BIOS has certain execution phases, namely checkpoints, defined and reported throughout it's execution life These checkpoints are both written out onto on a specific port (80,81), and reported at the bottom of POST screen if possible, via LCD display or SP (sp get port80 vX0z) progress is also available through sundiag - current-port80 on the Galaxy SP

BIOS Output (HT Sync Flood Example)

BIOS Stages Initial jump to Reset Vector At This stage,first CPU (also called Boot Strap Processor, or BSP) has a hard-coded jump instruction to a globally_defined address of F000:FFF0 Boot block execution from Flash ROM Consists of Initial and extremely efficient code for very early bring-up of a minimalist system. During this phase: HyperTransport Devices which are interconnected to CPU0 are detected and linked to with minimum speed/bandwidth Available and working DIMM memory which is attached to each North bridge, is arranged in local memory map registers

BIOS Stages cont Potential Sync Flood is detected and flagged in CMOS Compressed BIOS binary checksum is verified Motherboard initialization: All general chips are initialized Hyper Transport, North Bridges South Bridge, Tunnels, etc Video initialization: Graphic is initialized USB resource configuration: South Bridge USB controller is initialized Attached USB devices are recognized

General Execution Stages of BIOS cont Option ROM initialization: All ROM binaries of available PCI devices are sequentially executed (like SAS, GB LAN, etc.) System boot initiation: Execution is handed over to boot loader of an OS on certain Boot device (Solaris on Disk0) User-initiated system setup: Alternate entry into BIOS Setup Page has happened By pressing F2 during BIOS POST

IPMI Ipmitool is a utility for interfacing with devices that support the Intelligent Platform Management Interface specification. IPMI is an open standard for machine health, inventory, and remote power control. Ipmitool can communicate with IPMI-enabled devices through either a kernel driver such as OpenIPMI or over the RMCP (Remote Management Control Protocol) LAN protocol defined in the IPMI specification. IPMIv2 adds support for encrypted LAN communications and remote Serial-over-LAN functionality. Ipmitool is found in /usr/sfw/bin on Solaris operating systems. Ipmitool has to be installed as an option on Linux and Windows.

IPMI v2.0 Architecture Baseboard System Bus Bridge Controller ICMB Aux. IPMB Remote Mgmt. Card SMBus/PCI Mgmt. Bus Baseboard Mgmt. Controller (BMC) I 2 C/SMBus SDR, SEL, FRU NV Store Mgmt Netwk Ctrlr LAN PCI RS-232 MODEM / Serial IPMB (I 2 C) Chassis FRU SEEPROM System Interface SENSORs & control circuitry I 2 C / SMBus sensors & control circuitry Satellite Mgmt. Controller In Band Out of Band “ side-band” IPMI Messages

IPMI SDR - Sensor Data Repository displays sensor values via i2c SEL - System Event Log displays the SM BIOS log events FRU - Field Replaceable Units displays the contents of a FRU PROM chip built onto the component itself. If the device does not have a FRU PROM chip, its values cannot be displayed URL: http://ipmitool.sourceforge.net/ URL: http://ipmiutil.sourceforge.net/ IPMI Specification Second Generation v2.0 ftp://download.intel.com/design/servers/ipmi/IPMIv2_0rev1_0.pdf Or of course, man ipmitool

Ipmitool Command Options: Commands: raw Send a RAW request and print response i2c Send I2C master write-read command and print response lan Configure LAN channels chassis Get chassis status and set power state power Shortcut to chassis power commands event Send pre-defined events to MC mc Management controller status and global enables sdr Print sensor data repository entries and readings sensor Print detailed sensor information fru Print built-in FRU and scan SDR for FRU locators sel Print system event log (SEL) pef Configure platform event filtering (PEF) sol Configure and connect IPMIv2.0 Serial-over-LAN tsol Configure and connect with Tyan IPMIv1.5 Serial-over-LAN isol Configure IPMIv1.5 Serial-over-LAN user Configure management controller users channel Configure management controller channels session Print session information sunoem OEM commands for sun servers kontronoem OEM Commands for Kontron devices picmg Run a PICMG/ATCA extended cmd fwum Update IPMC using Kontron OEM Firmware Update Mgr exec Run list of commands from file set Set runtime variable for shell and exec

IPMI Command Options ipmitool [-chvV] -I lan -H hostname [-p <port>] [-U <user-name>] [-f <password_file>] [-S <sdrcache>] <command> ipmitool [-chvV] -I lanplus -H hostname [-p <port>] [-U <username>] [-f <password_file>] [-S <sdrcache>] <command> ipmitool [-chvV] [-S <sdrcache>] -I bmc <command> -c Present output in CSV (comma separated variable) format. -f <password_file> -v Increase verbose output level. Each instance increases verbosity -V Display version information.

IPMI Shell Command An alternative to this is to use the provided shell interface to issue repeated commands that will all use the same automatically generated cache. The shell interface is not available with all platforms and sometimes it is more advantageous to use the cache method (SDR), but if you are going to be analyzing the SEL chances are you will want to issue multiple commands and the shell interface makes this much easier with command history and editing. ipmitool -I lanplus -H IPADDR -U root -P changeme shell ipmitool> sel elist 100 | Pre-Init Time-stamp | Entity Presence ps1.prsnt | Device Absent 200 | Pre-Init Time-stamp | Entity Presence io.f0.prsnt | Device Absent 300 | Pre-Init Time-stamp | Power Supply ps0.vinok | State Asserted NOTE: In order to improve readability and avoid repeating useless command line arguments all further examples will assume that the shell interface is being used and that an appropriate session is already established either over LAN interface or using KCS interface and an OS kernel driver.

IPMI Sel Elist for X4450 Typically and uncorrectable or correctable memory error is reported as the following: 0 | Pre-Init Time-stamp | Memory | Correctable Error | Asserted | CPU 0 DIMM(PAIR) 0 0 | Pre-Init Time-stamp | Memory | Uncorrectable Error | Asserted | CPU 0 DIMM(PAIR) 0 Note the updates in firmware to include the term (PAIR) to aid misconception in FRU part. The BIOS is responsible for DIMM ECC handling. When a CE/UE error occurs, the chipset will generate a SMI, the BIOS will detect it and send a SEL event to the BMC. The IPMI spec. defines the format of SEL for DIMM ECC events. In addition, since the UE may cause the system hang or reset, the BIOS checks the UE status bit during early post stage and fires a SEL event to BMC as appropriate. Make sure the system is running the latest firmware. Earlier firmware reported as follows: 19 | 02/14/2008 | 19:53:51 | Memory #0x7a | Correctable ECC | Asserted

Entities An important foundation for sensors and events and FRUs is the concept of entities. In IPMI every sensor is assigned an entity ID and instance, which at its most basic level is a classification system that helps define what device type and number a sensor monitors. These are closely related to the concept of a Field Replaceable Unit in that a particular entity ID and instance can be used to describe a FRU. An entity ID is mapped to a physical device through a table defined in the IPMI specification. For our purposes the following entities are used on the Sun Fire X4000 platform: Their primary usefulness is for grouping and querying sensors, and in fact they come in very handy with ipmitool to do specific sensor queries.

Entities cont For example to see all power supply related sensors: ipmitool> sdr entity 10 ps0.prsnt | 1Ch | ok | 10.0 | Device Present ps0.pwrok | 1Dh | ok | 10.0 | State Deasserted ps0.vinok | 1Eh | ok | 10.0 | State Asserted ps1.prsnt | 1Fh | ok | 10.1 | Device Present ps1.pwrok | 20h | ok | 10.1 | State Deasserted ps1.vinok | 21h | ok | 10.1 | State Deasserted As you can see above the Entity ID and Instance is provided in the sdr elist command output in the form: <entity ID>.<entity instance>.

Entities cont Entities can also be used for entity lookups whereby you can associate a sensor with other sensors by finding all other sensors with the same entity ID and instance as the one you are looking up. This allows you to associate a sensor (and event). Sensor ID : ft0.fm2.f1.speed (0x49) Entity ID : 29.2 Sensor Type (Analog) : Fan Sensor Reading : 7000 (+/- 0) RPM Status : ok Lower Non-Recoverable : 3000.000 Lower Critical : na Lower Non-Critical : na Upper Non-Critical : na Upper Critical : na Upper Non-Recoverable : 22000.000 Assertions Enabled : lnr- unr+ Deassertions Enabled : lnr- unr+ SEL Record ID : 1002 Record Type : 02 Timestamp : 03/21/2007 06:08:09 Generator ID : 0020 EvM Revision : 04 Sensor Type : Fan Sensor Number : 49 Event Type : Threshold Event Direction : Assertion Event Event Data (RAW) : 541a1e Trigger Reading : 2600.000 RPM Trigger Threshold : 3000.000 RPM Description : Lower Non-recoverable going low

Threshold Threshold (or analog) sensors are used for inputs like temperature, voltages, and fan speeds. These are connected to a sensor chip and read by the service processor using I2C. They can have both upper and lower thresholds set and multiple different events can be configured around each of them. There can be zero or more of the following thresholds configured as upper or lower bounds for each threshold sensor: * Non-Critical * Critical * Non-Recoverable These are commonly represented in short form with ipmitool, here is how the short names map to the longer names Each threshold can also have both a direction and a status flag attached to it. The use of a "+" or "-" is used to indicate whether that particular threshold applies to a reading that is going high or going low, respectively. The status flag indicates whether the event is generated is an assertion or a deassertion event. Each of these can be configured independently by setting various bits in the sensor data record.

Threshold cont Example Thresholds:

Discrete Sensors While threshold sensors represent analog readings, discrete sensors can be considered to represent digital readings. There are different types of discrete sensors as well: Generic Discrete: these sensors can have various flags from a generic pre-defined table. None of these sensors are used on this platform. Digital Discrete: similar to the above but are for sensors that only represent one of two possible states. Think of a GPIO pin where the state is either 1 or 0. There are many sensors of this type used on this platform, everything from presence detection sensors to LED state sensors. Sensor-Specific Discrete: similar to the first type these sensor types have any number of flags defined depending on the sensor type. This makes it possible to represent a large number of sensors, each with different states. There is only one of these types of sensor on this platform and it belongs to the Chassis Intrusion sensor.

BIOS Error Handling and Reporting : Correctable Errors – DMI Log

Sync Flooding is a HyperTransport™ method used to stop data propagation in the case of a serious error Device that detects the error initiates sync flood packets All other HT devices cease operation They transmit sync flood packets out of all HT links Packets finally reach the platform South Bridge (AMD8111/NF2200/3400) BIOS has Pre-programmed SB to trigger system RESET signal, when sync flood is detected. System reboots During Boot Block and POST, BIOS analyzes related error bits in all HT Nodes, reports of Sync Flood reasons BIOS detects some of the error sources, but not all of them BIOS Error Handling & Reporting Overview

Error Handling and Reporting Uncorrectable ECC Errors

Rev.F BIOS Error Handling Framework General Conditions: 1- Two lines of SEL Log will be reported for each Node with valid fault and MCA address 2- One line of SEL Log will be reported for each node with evidence of a generic sync flood and no valid address 3- If a node has detected CRC errors on incoming HT links... Link# will be appended after "FEDBADF00D" (indicator of a CRC fed to HT link)... "000201" comes in four least significant nibbles (00[future link4, zero for now][link2][link1][link0]) each nibble content represents: 0 = no CRC Error was received on this link 1 = incoming Byte0 Link received the CRC error 2 = incoming Byte1 Link received the CRC Error 3 = incoming Bytes0+1 Links both saw CRC Errors on incoming links

Data Gathering Tools Windows MPSreports Solaris - Explorer Redhat -sysreport Suse – Siga, supportconfig

Windows MPSreports Microsoft Premier Field Engineering (PFE) Reporting Utility Supported Operating Systems: Windows NT 4.0 Windows 2000 Windows XP Windows Server 2003 x86 Windows Server 2003 x64 (AMD64) Windows Server 2003 for Itanium

Windows MPSreports cont The PFE version of MPS_REPORTS gathers a wide range of diagnostic information from Windows and limited information for server applications installed such as SQL or Exchange. The user running the utility must have administrative privileges on the system. During runtime there are uncompressed text files that could possibly consume 100 MB or more or more of disk space, depending on the size of the event logs. Most of the space is freed when the reporting tool finishes, leaving behind only the tools folder and the .cab file, which can be deleted if no longer needed.

Windows MPSreports Microsoft Product Support Reporting Tool Supported Operating Systems: Windows 2000 Windows NT Windows Server 2003 Windows XP There are 8 specialty versions, one for each of the following support scenario categories: Alliance, Directory Services (not for Windows NT 4.0), Networking, Clustering, SQL, Software Update Services, MDAC and Base/Setup/Storage/Print/Performance. Each version gathers some of the same basic information but there are specific reports unique to each of the support scenario categories. Please read the readme.txt files for more details about each version.

Windows MPSreports cont The Microsoft Product Support Reporting Tool facilitates the gathering of critical system and logging information used in troubleshooting support issues. The reporting tool DOES NOT make any registry changes or modifications to the operating system. Files created are zipped CAB files cabextract can be used on Solaris to extract. Microsoft Product Support Reporting Tool Cabextract website for Solaris and other packages. /usr/local/gnu/bin/cabextract - - help

Windows Utilities “msconfig” The System.ini file The Win.ini file The Boot.ini file Programs that are set to load during the startup process Environment settings International settings Description of Windows XP System Information (Msconfig.exe) Tool

Windows Utilities “msinfo32” Hardware Resources Components Software Environment Applications Internet Explorer Description of Windows XP System Information (Msinfo32.exe) Tool

Windows Event Viewer The Event Logs for Windows include: System Event Log Security Event Log Application Event Log Directory Service File Replication Service

Other Windows Utilities System Monitor - perfmon.exe Ipconfig Dosutil network information Device Manager / Control Panel Network Sniffers Ethereal Wireshark Dumps - three memory dump file types: Complete memory dump Kernel memory dump Small memory dump (64 KB) Overview of memory dump file options for Windows Server 2003 and XP

Windows DRAM Limitations http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx Distribution 32-bit 64-bit Windows Server 2003 SP2, Datacenter 128GB 4TB Windows Server 2003 SP2, Enterprise Edition 8GB 2TB Windows Storage Server 2003, Enterprise Edition 8GB - Windows Storage Server 2003 4GB - Windows Server 2003 R2 Datacenter Edition 128GB 1TB Windows Server 2003 SP1, Datacenter Edition 128GB 1TB Windows Server 2003 R2 Enterprise Edition 64GB 1TB Windows Server 2003 SP1, Enterprise Edition 64GB 1TB Windows Server 2003, Standard Edition 4GB 16GB

Solaris Explorer A new version has been released 5.8 it is available from the sun download site Sun Explorer Data Collector version 5.8 Supported Platforms Solaris 7/8/9/10 SPARC/Solaris 9/10 X86 Platform

Solaris Explorer The following describes the changes made in Sun Explorer 5.8: New Features Explorer 5.8 now collects /usr/sbin/smbios output. (6412712) Modified Features Now supports transport target with HTTP protocol in addition to HTTPS protocol. Because some companies have problems using HTTPS internally, such as in the DMZ net, Explorer supports the use of HTTP from the DMZ to the company's own HTTP endpoint. (6471251) Explorer 5.8 restores the ability to gather system controller data via telnet with the scextended and 1280extended modules. A defect in Explorer 5.7 caused these telnet connections to be blocked unless an ssh connection was made to accept the host key, before running Explorer. (6521236, 6528445 )

Explorer Modules “ilomextended” Explorer -if <ipmiinput.txt-file> -w <module-list> Collects remote Integrated Lights Out Manager (ILOM) Intelligent Platform Management Interface (IPMI) data from Galaxy systems. Commands Collected /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} mc info /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} mc getenables /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} chassis poh /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} chassis power status /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} fru print /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} pef status /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} pef list /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} sdr list full /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} sel info /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} sel elist /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} sensor list /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} user summary /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} user list /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} sunoem led get /usr/sfw/bin/ipmitool -H {host} -p {port} -U {user} -f {pwfile} chassis restart_cause

Explorer Modules “ipmi” Collects local Intelligent Platform Management Interface (IPMI) data on x86 platform. The following commands are collected: /usr/sfw/bin/ipmitool chassis status /usr/sfw/bin/ipmitool chassis poh /user/sfw/bin/ipmitool chassis power status /user/sfw/bin/ipmitool chassis restart_cause /usr/sfw/bin/ipmitool fru /user/sfw/bin/ipmitool fru print /user/sfw/bin/ipmitool mc getenables /user/sfw/bin/ipmitool mc info /usr/sfw/bin/ipmitool pef status /usr/sfw/bin/ipmitool pef list usr/sfw/bin/ipmitool sel info /usr/sfw/bin/ipmitool sel elist /user/sfw/bin/ipmitool sdr enlist full /usr/sfw/bin/ipmitool sdr list all info /user/sfw/bin/ipmitool sensor list /user/sfw/bin/ipmitool sunoem led get

Griffon Knowledge Engine Sun Alerts and FINS checked against Recommended Patches for Solaris 7, 8, 9 and 10 (sparc) (x86) checked against Firmware Patches checked against ASP Group Patches checked against BADPATCH/WITHDRAWN Patches checked against Griffin Knowledge Engine Home page Griffon System Analysis Job Submission Form Features of the Griffon System Analysis Job Submission Form Explorer files may be referenced via a network path. Provides a high-level matrix to highlight critical problem areas. Individual System Analysis Reports Container files of hostids to group results Hostid references utilize the current Explorer file Proactive servers Patch references are linked to external SunSolve.

Sun Gathering Debug Data (Sun GDD) Tools provide the right approach to problem resolution by leveraging proactive actions and best practices to help you gather the required debug data needed for further analysis. For each product covered, GDD tools provide documentation and scripts which detail the relevant data the Sun Technical Support Center requires for analyzing your problem

Sun Gathering Debug Data (Sun GDD) The tools gather 90% of the debug data frequently requested by the Sun Technical Support Center - including data for more common critical situations including memory, start/stop, installation, hang, and crash issues. By collecting this data before you initiate a Service Request, you can substantially reduce the time needed to analyze and resolve the problem. Sun Java System Calendar Server Sun Java System Directory Editor Sun Java System Directory Proxy Server Sun Java System Directory Server Sun Java System Messaging Server Sun Java System Portal Server Sun Java System Web Proxy Server 3.6 Service Pack 9 Sun Java System Web Server External GDD sun website Sun Gathering Debug Data (GDD) (WZT-0253) - Web based

SUSE “supportconfig” Detailed system information and logs are collected and organized in a manner that helps reduce service request resolution times. Private system information can be disclosed when using this tool. If this is a concern, please prune private data from the log files. Several startup options are available to exclude more sensitive information. Refer to the man page to see these options. Output format is better the SIGA scripts, however needs to be installed from the SUSE web site.

Supportconfig Installation The script has been renamed from config.sh to supportconfig. It has been repackaged as an RPM for easy installation and updating. There is now a supportconfig man page. 1. Download supportconfig-2.13-0.4.noarch.rpm 2. Install the RPM rpm -Uvh supportconfig-2.13-0.4.noarch.rpm 3. Run the supportconfig command as root 4. To view the information files, assuming the tarball filename is: /var/log/nts_host1_070406_1300.tar.bz2 cd /var/log tar jxf nts_host1_070406_1300.tar.bz2 cd nts_host1_070406_1300 ls -l

VMware ESX Server includes a script called vm-support, which collects information that VMware support might request and packages that information into one file. To make sure you have version 1.14 or later of the script Version 1.14 (and later) of the vm-support script produces more diagnostic information than previous versions of the script. ESX Server 3.x includes a version of vm-support newer than 1.14. You don't need to check the version of this script or update it. ESX Server 2.x might include a version earlier than 1.14. To see which version is installed on your system, run the command with no options. For example: [user@esx2host]$ cd /tmp [user@esx2host]$ /usr/bin/vm-support VMware ESX Server Support Script 0.94 Preparing Files: [Ctrl+C to cancel] If you have a version earlier than 1.14, follow the instructions later in this presentation.

VMware cont To collect diagnostic information using the script: 1. Log on to the service console as root 2. Confirm what version of VMware ESX Server you are running: [root@esxhost]# vmware -v 3. Change to the directory where you want the output to appear. For example: [root@esxhost]# cd /tmp If you run the script in /usr/bin, the output will appear in that directory and remain there until you delete it 4. Run the script: [root@esxhost]# /usr/bin/vm-support You don't need to power off your virtual machines before running this script For ESX Server performance issues, VMware Technical Support might ask you to collect performance snapshots using the the -s and -S switches. Please refer to: http://kb.vmware.com/kb/1967 for more information 5. When the script finishes, it informs you of the output filename and location

VMware Upgrading “vm-support” Download vm-support script and move it to the /tmp directory on the service console of the ESX Server system Make a backup copy of your existing script: [root@esxhost]# cp /usr/bin/vm-support /usr/bin/vm-support.old Extract the archive to /usr/bin/, replacing the original vm-support script: [root@esxhost]# cd /usr/bin [root@esxhost]# tar xvzf /tmp/653_fvm-support_114.tgz The archive places the vm-support script in the current directory. Note: When running on an older version of ESX Server, the updated script might report errors about missing commands. This is normal. Supports Versions: VMware ESX Server 2.0.x VMware ESX Server 2.1.x VMware ESX Server 2.5.x VMware ESX Server 3.0.x

Various Other Tools Available Dmidecode Available Solaris Linux later versions Biosdecode Available Linux part of dmidecode 2.9 Smbios Solaris nvidia-bug-report.sh Graphics 'X' data gather Kstat Solaris Various information gathered Scanpci Linux Solaris Pcitweak Linux (SuSe) Solaris Setpci Linux Redhat

Working With Non-SP Machines You have no IPMI data You are likely to have no DMI logs either Some systems will pause on boot to indicate syncflood and other serious events Platform messages PCI.exe could gather northbridge registers POST codes POST LEDs What does occur LEDS, fans Graphics

Lab Connecting To Remote Platforms Ultra 24 vi64-u24a-kvm-gmp03.uk X2100/X2200 va64-x2100g-sp-gmp03.uk X4100/X4200 Not available X4150 Should be available by 12pm X4600 va64-x4600a-sp-gmp03.uk Collect ipmi SEL, SDR and FRU data in/out of band

Thank you Michael Johnson [email_address]

Information Gathering 2

More Related Content

Viewers also liked

Similar to Information Gathering 2

More from Aero Plane

Information Gathering 2

Editor's Notes