Reliability, Availability
and Serviceability on
Linux
Mauro Carvalho Chehab
Linux Kernel Expert
Samsung Open Source Group
...
What is RAS (1)
●

Used originally by IBM to measure mainframe robusteness

●

Reliability
–
–

Generally measured as Mean...
What is RAS (2)
●

Serviceability (or maintainability)
–

Simplicity and speed with which a system can be repaired or main...
Improving RAS (1)
●

In order to improve RAS, both IT services and hardware require improvements

●

Examples of hardware ...
Improving RAS (2)
●

Examples of IT measures
–

●

24x7x365 days on-site support; low latency support from their vendors

...
RAS features on Linux (1)
●

Storage errors reported is supported since early versions, as RAID/SAS/SAN
controllers/driver...
RAS features on Linux (2)
●

EDAC (Error Detection and Correction) subsystem
–

Provides a way to report errors detected b...
RAS features on Linux (3)
●

PCIe AER (Advanced Error Reporting)
–
–

AER logs data at console;

–
●

Some PCIe hardware p...
Typical D-RAM implementation
Bank 7
Bank 6
Bank 5
Bank 4
Bank 3
Bank 2
Bank 1
Bank 0

Rank 1

DRAM Memory Matrix

Bank n

...
Memory Arrangements on PC

IBM PC original architecture

Classic server (most common) architecture

NOTE: Nehalem-EX has a...
Evolution of RAS on Kernel (1)
●

Before Kernel 2.6.32
–
–

On EDAC, memory controllers are assumed to be Rank-based

–
●
...
Evolution of RAS on Kernel (2)
●

Kernel 3.9
–
–

●

Added firmware first EDAC driver (ghes_edac);
Added trace events for ...
Firmware First x Hardware First
●

Hardware-first approach
–

Errors come directly from the hardware

–

BIOS doesn't hand...
RASDAEMON (1)
●

It is a new tool
–

Currently, provided on Fedora 18, Fedora 19 and rawhide

–

Has:
●

●

A tool to conf...
RASDAEMON (2)
$ ras-mc-ctl --print-labels
LOCATION
mc0 channel 0 slot 0

mc1 channel 0 slot 0

CONFIGURED LABEL
DIMM_A1
DI...
RASDAEMON (3)
$ util/ras-mc-ctl --print-labels
LOCATION
mc0 channel 0 slot 0

mc1 channel 0 slot 0

Open Source Group – Si...
RASDAEMON(4)
# rasdaemon -r -f
overriding event (931) ras:mc_event with new print handler
rasdaemon: ras:mc_event event en...
Thank you.
Questions?

Open Source Group – Silicon Valley

© 2013 SAMSUNG Electronics
Co.
Upcoming SlideShare
Loading in …5
×

Reliability, Availability and Serviceability on Linux

902 views

Published on

Linux Kernel Multmedia Subsystem maintainer and Samsung OSG team member Mauro Chehab presents his thoughts on how to apply RAS to Linux platforms.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
902
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Reliability, Availability and Serviceability on Linux

  1. 1. Reliability, Availability and Serviceability on Linux Mauro Carvalho Chehab Linux Kernel Expert Samsung Open Source Group Sep 16, 2013 Open Source Group – Silicon Valley Not to be used for commercial purpose without getting permission All information, opinions and ideas herein are exclusively the author's own opinion © 2013 SAMSUNG Electronics Co.
  2. 2. What is RAS (1) ● Used originally by IBM to measure mainframe robusteness ● Reliability – – Generally measured as Mean Time Between Failures (MTBF) – ● Probability that a system will produce correct outputs Enhanced by features that help to avoid, detect and repair hardware faults Availability – Probability that a system is operational at a given time – Generally measured as a percentage of downtime per a period of time ● Examples: – – – – Open Source Group – Silicon Valley 99.9% (“three nines”) means 3.65 days unavailable per year 99.999% (“five nines”) means 5.26 minutes of downtime per year Minimal down-time for service and repair. Detect and correct hardware faults as opposed to detect and repair © 2013 SAMSUNG Electronics Co.
  3. 3. What is RAS (2) ● Serviceability (or maintainability) – Simplicity and speed with which a system can be repaired or maintained – Generally measured on Mean Time Between Repair – Can be increased with redundant parts, and higher support grade (24/7/365) Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  4. 4. Improving RAS (1) ● In order to improve RAS, both IT services and hardware require improvements ● Examples of hardware measures – CPU – to detect errors at instruction execution and L1/L2/L3 caches; – Memory – add error correction logic (ECC) to detect and correct errors; – I/O – add CRC checksums for tranfered data (PCIe has such feature); – Storage – RAID, journal file systems, checksums; – Power/cooling – component duplication, over-design, surge protector, UPS – System – hot swap of components, predictive failure analysis, partitioning of system components, virtual machines running on redundant servers, clustering, dynamic software update, independent CPU for RAS – RAS servers have features to hot add/replace/remove I/O cards that reduce – down-time for adding new hardware. Replacing failing I/O cards based on – PCIe AER features. – memory mirroring and active/ative and active/standby comfigurations that reduce down-time Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  5. 5. Improving RAS (2) ● Examples of IT measures – ● 24x7x365 days on-site support; low latency support from their vendors Usage of Virtual Machines – – ● vm migration with minimal application down-time Cloud computing Predictive analysis – Hardware/OS should provide data to detect systems/components degradation – It should have tools to analyze and (hot)replace those degraded components Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  6. 6. RAS features on Linux (1) ● Storage errors reported is supported since early versions, as RAID/SAS/SAN controllers/drivers offer measurements. – ● There are also userspace tools to manage it. Machine Check Architecture – MCA – CPU errors are provided on x86 machines since Pentium 4 – Depending on the processor, it can also provide memory and bus errors – Kernel implements it at mcelog subsystem ● – Fatal errors produce panic() and are reported at console; At userspace, the mcelog tool reads the corrected/non-fatal error data from time to time and reports at console ● The Kernel-userspace API is obscure: userspace receives a dump of a series of registers; – – – – Open Source Group – Silicon Valley Decoding those errors are CPU-specific; Kernel decodes those errors for fatal errors; The userspace tool decodes those errors for non-fatal/corrected ones Errors are reported also via a kernel trace event; © 2013 SAMSUNG Electronics Co.
  7. 7. RAS features on Linux (2) ● EDAC (Error Detection and Correction) subsystem – Provides a way to report errors detected by memory controllers to userspace; – Some (old) drivers also report PCI errors via EDAC; – Kernel decodes the error into the DIMM labels affected by an error; ● ● – Errors are reported at console and via a kernel trace event The association between the memory architecture and DIMM is done via some files that are loaded by an userspace tool (edac-utils or rasdaemon) Most drivers talk directly with the memory controller (MC) ● ● That provides a more reliable error report BIOS data is not very reliable: on several cases, the same BIOS is used on different machines The DMI BIOS tables may contain the wrong DIMM labels Race conditions may happen on BIOS that also collect error data – ● – There's one driver (ghes_edac) on Kernel 3.9+ that get errors from BIOS ● “firmware first” mode: BIOS tell OS to not talk with the MC directly Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  8. 8. RAS features on Linux (3) ● PCIe AER (Advanced Error Reporting) – – AER logs data at console; – ● Some PCIe hardware provide ways to get AER error reports on OS; It also reports error via a kernel trace event Userspace tools: – mcelog – collects and decodes MCA error events on x86; – edac-utils – fills DIMM labels data and summarizes memory errors; – rasdaemon ● collects errors via kernel trace events from several sources: MCA, EDAC and PCIe AER events Fills DIMM labels data – ● ● Store error data into a persistent database (sqllite3) ● Allow to latter query/summarize errors ● Use new resources available on Kernel 3.10 Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  9. 9. Typical D-RAM implementation Bank 7 Bank 6 Bank 5 Bank 4 Bank 3 Bank 2 Bank 1 Bank 0 Rank 1 DRAM Memory Matrix Bank n Row Decoder Column Decoder DRAM Memory Matrix Source: http://lwn.net/Articles/250967/ A DIMM can have 1, 2 or 4 ranks Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  10. 10. Memory Arrangements on PC IBM PC original architecture Classic server (most common) architecture NOTE: Nehalem-EX has an additional buffer chip between the RAM and CPU, called Intel SMB (Intel Scalable Memory Buffer) ● ● AMD-64 and newer Intel CPU architecture (Nehalem, Sandy Bridge and upcoming) ● It means that the CPU memory controller doesn't see the DIMMs directly This is to improve performance when there are lots of CPU sockets (-EX machines) Only BIOS knows how the memory is organized on Nehalem-EX Images took from: http://lwn.net/Articles/250967/ Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  11. 11. Evolution of RAS on Kernel (1) ● Before Kernel 2.6.32 – – On EDAC, memory controllers are assumed to be Rank-based – ● EDAC reports errors via dmesg mcelog reports error via its own interface only Kernel 2.6.32 – ● Added kernel trace events on MCE; Kernel 3.5 – EDAC/HERM patches added support for modern memory architectures ● Modern Intel CPUs/MCs proper support (2002 and upper Intel systems) Memory controllers are DIMM-based, – Memory controllers can be grouped in branches (FB-DIMM) Added kernel trace events for EDAC; – ● Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  12. 12. Evolution of RAS on Kernel (2) ● Kernel 3.9 – – ● Added firmware first EDAC driver (ghes_edac); Added trace events for PCIe AER; Kernel 3.10 brings a series of new features useful for RAS events tracing: – – Added blocking functionality to trace_pipe_raw; – ● Allow to create independent tracing facility for each process using traces; Added “uptime” clock reference for tracing events; While the rasdaemon tool works with kernels below 3.10, it is optimized to use those new features found on Kernel 3.10. Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  13. 13. Firmware First x Hardware First ● Hardware-first approach – Errors come directly from the hardware – BIOS doesn't handle it ● ● – ● It is faster Can help to avoid long SMI interrupts Require a deep knowledge on the hardware Firmware-first approach – BIOS and/or dedicated CPUs collect errors – OS doesn't need to know deeply the hardware – BIOS can mask/group errors, apply proprietary algorithms, avoid spurious report – Unfortunately, current ACPI API doesn't expose the memory slot label, with makes harder to be used by the system admin Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  14. 14. RASDAEMON (1) ● It is a new tool – Currently, provided on Fedora 18, Fedora 19 and rawhide – Has: ● ● A tool to configure DIMMs and do RAS reports (ras-mc-ctl) ● – A daemon that waits for kernel trace events (rasdaemon) Some contrib tools to test EDAC and to fake inject errors Example: ● Dell T620 with 2 Sandy Bridge-EP Xeon CPUs (E5-2670) ● 2 8GB dual-rank DIMMs (Samsung M393B1K70DH0-YK0) ● Driver: sb-edac ras-mc-ctl –-layout +-----------------------------------------------------------------------------------------------+ | mc0 | mc1 | | channel0 | channel1 | channel2 | channel3 | channel0 | channel1 | channel2 | channel3 | -------+-----------------------------------------------------------------------------------------------+ slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | slot0: | 8192 MB | 0 MB | 0 MB | 0 MB | 8192 MB | 0 MB | 0 MB | 0 MB | -------+-----------------------------------------------------------------------------------------------+ Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  15. 15. RASDAEMON (2) $ ras-mc-ctl --print-labels LOCATION mc0 channel 0 slot 0 mc1 channel 0 slot 0 CONFIGURED LABEL DIMM_A1 DIMM_A2 DIMM_A3 DIMM_A4 DIMM_A5 DIMM_A6 DIMM_A7 DIMM_A8 DIMM_A9 DIMM_A10 DIMM_A11 DIMM_A12 DIMM_B1 DIMM_B2 DIMM_B3 DIMM_B4 DIMM_B5 DIMM_B6 DIMM_B7 DIMM_B8 DIMM_B9 DIMM_B10 DIMM_B11 DIMM_B12 SYSFS CONTENTS CPU_SrcID#0_Channel#0_DIMM#0 0:0:1 missing 0:0:2 missing 0:0:3 missing 0:1:0 missing 0:1:1 missing 0:1:2 missing 0:1:3 missing 0:2:0 missing 0:2:1 missing 0:2:2 missing 0:2:3 missing CPU_SrcID#1_Channel#0_DIMM#0 1:0:1 missing 1:0:2 missing 1:0:3 missing 1:1:0 missing 1:1:1 missing 1:1:2 missing 1:1:3 missing 1:2:0 missing 1:2:1 missing 1:2:2 missing 1:2:3 missing # ras-mc-ctl --register-labels Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  16. 16. RASDAEMON (3) $ util/ras-mc-ctl --print-labels LOCATION mc0 channel 0 slot 0 mc1 channel 0 slot 0 Open Source Group – Silicon Valley CONFIGURED LABEL DIMM_A1 DIMM_A2 DIMM_A3 DIMM_A4 DIMM_A5 DIMM_A6 DIMM_A7 DIMM_A8 DIMM_A9 DIMM_A10 DIMM_A11 DIMM_A12 DIMM_B1 DIMM_B2 DIMM_B3 DIMM_B4 DIMM_B5 DIMM_B6 DIMM_B7 DIMM_B8 DIMM_B9 DIMM_B10 DIMM_B11 DIMM_B12 SYSFS CONTENTS DIMM_A1 0:0:1 missing 0:0:2 missing 0:0:3 missing 0:1:0 missing 0:1:1 missing 0:1:2 missing 0:1:3 missing 0:2:0 missing 0:2:1 missing 0:2:2 missing 0:2:3 missing DIMM_B1 1:0:1 missing 1:0:2 missing 1:0:3 missing 1:1:0 missing 1:1:1 missing 1:1:2 missing 1:1:3 missing 1:2:0 missing 1:2:1 missing 1:2:2 missing 1:2:3 missing © 2013 SAMSUNG Electronics Co.
  17. 17. RASDAEMON(4) # rasdaemon -r -f overriding event (931) ras:mc_event with new print handler rasdaemon: ras:mc_event event enabled rasdaemon: Enabled event ras:mc_event overriding event (847) ras:aer_event with new print handler rasdaemon: ras:aer_event event enabled rasdaemon: Enabled event ras:aer_event overriding event (56) mce:mce_record with new print handler rasdaemon: mce:mce_record event enabled rasdaemon: Enabled event mce:mce_record rasdaemon: Listening to events for cpus 0 to 31 Calling ras_mc_event_opendb() rasdaemon: cpu 0: Recording events at /var/lib/rasdaemon/ras-mc_event.db cpu 12:rasdaemon: mc_event store: 0x19c6968 rasdaemon: register inserted at db <...>-2742 [732433178] 2507.782000: mc_event: 2013-08-15 19:58:50 -0300 1 Corrected error: FAKE ERROR on DIMM_A1 (mc: 0 location: 0:0:0 grain: 7 for EDAC testing only) cpu 12:rasdaemon: mc_event store: 0x19c6968 <...>-2742 [732433178] 2507.864000: mc_event: 2013-08-15 19:58:50 -0300 1 Corrected error: FAKE ERROR on DIMM_B1 (mc: 1 location: 0:0:0 grain: 7 for EDAC testing only) cpu 12:rasdaemon: mc_event store: 0x19c6968 # ras-mc-ctl --summary Memory controller events summary: Corrected on DIMM Label(s): 'DIMM_A1' location: 0:0:0:0 errors: 1 Corrected on DIMM Label(s): 'DIMM_B1' location: 1:0:0:0 errors: 1 Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.
  18. 18. Thank you. Questions? Open Source Group – Silicon Valley © 2013 SAMSUNG Electronics Co.

×