Successfully reported this slideshow.

Hardware Errors and the OS

510 views

Published on

Lecture by Bjoern Doebel for Summer Systems School'13 (http://sss.ksyslabs.org)

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Hardware Errors and the OS

  1. 1. Introduction Hardware Errors and the OS Bj¨orn D¨obel () OS Resilience 06.08.2013 30 / 58
  2. 2. Introduction Hardware Errors in Theory Bulk Substrate Source – – Drain – – Gate + ++ Oxide Layer Bj¨orn D¨obel () OS Resilience 06.08.2013 31 / 58
  3. 3. Introduction Hardware Errors in Theory Bulk Substrate Source – – Drain – – Gate + ++ Oxide Layer Radiation-induced errors Cosmic radiation Alpha particles emitted by packaging Thermal stress Aging of circuitry Electromigration Hot Carrier Injection Negative-Bias Temperature Instability Bj¨orn D¨obel () OS Resilience 06.08.2013 31 / 58
  4. 4. Introduction Hardware Errors in the Real World Several studies investigated manifestation of hardware errors in software: Saggese, 2005 85% of hardware errors masked Error outcome depends on a↵ected HW unit Li, 2008, focus on permanent errors Permanent errors mainly lead to crashes / HW exceptions 65% of errors corrupt OS state before crashing Bj¨orn D¨obel () OS Resilience 06.08.2013 32 / 58
  5. 5. Introduction Hardware Errors in the Real World Several studies investigated manifestation of hardware errors in software: Saggese, 2005 85% of hardware errors masked Error outcome depends on a↵ected HW unit Li, 2008, focus on permanent errors Permanent errors mainly lead to crashes / HW exceptions 65% of errors corrupt OS state before crashing Arlat 2002, Chorus and LynxOS microkernels Significant amount (30%) of ”no change” errors Some OS components are more error-prone than others Wang, 2003, focus on branching errors Several cases (up to 40%) where taking di↵erent branch does not change program result Bj¨orn D¨obel () OS Resilience 06.08.2013 32 / 58
  6. 6. Introduction Challenges and Opportunities Challenge: detect and correct hardware errors in software Bj¨orn D¨obel () OS Resilience 06.08.2013 33 / 58
  7. 7. Introduction Challenges and Opportunities Challenge: detect and correct hardware errors in software Optimization Potential: don’t track harmless errors Bj¨orn D¨obel () OS Resilience 06.08.2013 33 / 58
  8. 8. Introduction Challenges and Opportunities Challenge: detect and correct hardware errors in software Optimization Potential: don’t track harmless errors Challenge: Binary applications Bj¨orn D¨obel () OS Resilience 06.08.2013 33 / 58
  9. 9. Introduction Challenges and Opportunities Challenge: detect and correct hardware errors in software Optimization Potential: don’t track harmless errors Challenge: Binary applications Optimization Potential: Hardware-Level Concurrency Bj¨orn D¨obel () OS Resilience 06.08.2013 33 / 58
  10. 10. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  11. 11. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors RAD-hard CPUs Redundant Multithr. Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  12. 12. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors RAD-hard CPUs Redundant Multithr. HP NonStop IBM z/OS Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  13. 13. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors RAD-hard CPUs Redundant Multithr. HP NonStop IBM z/OS SeL4 Minix3 Carburizer Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  14. 14. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors RAD-hard CPUs Redundant Multithr. HP NonStop IBM z/OS SeL4 Minix3 Carburizer SWIFT Encoded Processing Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  15. 15. Introduction Fault Tolerance: State of the Union non- COTS COTS Hardware errors Software errors RAD-hard CPUs Redundant Multithr. HP NonStop IBM z/OS SeL4 Minix3 Carburizer SWIFT Encoded Processing Romain Bj¨orn D¨obel () OS Resilience 06.08.2013 34 / 58
  16. 16. Introduction CS 101 Compute Application Inputs Outputs Determinism property Bj¨orn D¨obel () OS Resilience 06.08.2013 35 / 58
  17. 17. Introduction Redundant execution App App’ App” Bj¨orn D¨obel () OS Resilience 06.08.2013 36 / 58
  18. 18. Introduction Redundant execution Collected Inputs App App’ App” Bj¨orn D¨obel () OS Resilience 06.08.2013 36 / 58
  19. 19. Introduction Redundant execution Collected Inputs App App’ App” Bj¨orn D¨obel () OS Resilience 06.08.2013 36 / 58
  20. 20. Introduction Redundant execution Collected Inputs App App’ App” = Bj¨orn D¨obel () OS Resilience 06.08.2013 36 / 58
  21. 21. Introduction Redundant execution Collected Inputs App App’ App” = Bj¨orn D¨obel () OS Resilience 06.08.2013 36 / 58
  22. 22. Introduction Inputs and Outputs Inputs Outputs System Calls System Calls Shared Memory Shared Memory I/O Memory I/O Memory Special Instructions (e.g., rdtsc) Hardware Interrupts Hardware Exceptions (e.g., page faults) Bj¨orn D¨obel () OS Resilience 06.08.2013 37 / 58
  23. 23. Introduction Process-Level Redundancy [Shye 2007] Binary recompilation Complex, unprotected compiler Architecture-dependent System calls for replica synchronization Virtual memory fault isolation Restricted to Linux user-level programs Bj¨orn D¨obel () OS Resilience 06.08.2013 38 / 58
  24. 24. Introduction Process-Level Redundancy [Shye 2007] Binary recompilation Complex, unprotected compiler Architecture-dependent Reuse OS mechanisms System calls for replica synchronization Additional synchronization events Virtual memory fault isolation Restricted to Linux user-level programs Microkernel-based Bj¨orn D¨obel () OS Resilience 06.08.2013 38 / 58
  25. 25. Introduction Transparent Replication as OS Service Application L4 Runtime Environment L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 39 / 58
  26. 26. Introduction Transparent Replication as OS Service Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 39 / 58
  27. 27. Introduction Transparent Replication as OS Service Unreplicated Application Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 39 / 58
  28. 28. Introduction Transparent Replication as OS Service Replicated Driver Unreplicated Application Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 39 / 58
  29. 29. Introduction Transparent Replication as OS Service Reliable Computing Base Replicated Driver Unreplicated Application Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 39 / 58
  30. 30. Introduction Romain: Structure Master Bj¨orn D¨obel () OS Resilience 06.08.2013 40 / 58
  31. 31. Introduction Romain: Structure Replica Replica Replica Master Bj¨orn D¨obel () OS Resilience 06.08.2013 40 / 58
  32. 32. Introduction Romain: Structure Replica Replica Replica Master = Bj¨orn D¨obel () OS Resilience 06.08.2013 40 / 58
  33. 33. Introduction Romain: Structure Replica Replica Replica Master System Call Proxy Resource Manager = Bj¨orn D¨obel () OS Resilience 06.08.2013 40 / 58
  34. 34. Introduction Resource Management: Capabilities 1 22 3 4 5 6 Replica 1 Bj¨orn D¨obel () OS Resilience 06.08.2013 41 / 58
  35. 35. Introduction Resource Management: Capabilities 1 22 3 4 5 6 Replica 1 1 22 3 4 5 6 Replica 2 Bj¨orn D¨obel () OS Resilience 06.08.2013 41 / 58
  36. 36. Introduction Resource Management: Capabilities 1 22 3 4 5 6 Replica 1 1 22 3 4 5 6 Replica 2 1 2 3 4 5 6 Master Bj¨orn D¨obel () OS Resilience 06.08.2013 41 / 58
  37. 37. Introduction Partitioned Capability Tables 1 2 3 4 5 6 Replica 1 1 2 3 4 5 6 Replica 2 1 2 3 4 5 6 Master Marked used Master private Bj¨orn D¨obel () OS Resilience 06.08.2013 42 / 58
  38. 38. Introduction Replica Memory Management Replica 1 rw ro ro Replica 2 rw ro ro Master Bj¨orn D¨obel () OS Resilience 06.08.2013 43 / 58
  39. 39. Introduction Replica Memory Management Replica 1 rw ro ro Replica 2 rw ro ro Master Bj¨orn D¨obel () OS Resilience 06.08.2013 43 / 58
  40. 40. Introduction Replica Memory Management Replica 1 rw ro ro Replica 2 rw ro ro Master Bj¨orn D¨obel () OS Resilience 06.08.2013 43 / 58
  41. 41. Introduction Shared Memory Not in complete control of master Standard technique: trap&emulate Execution overhead (x100 - x1000) Adds complexity to RCB Disassembler 6,000 LoC Tiny emulator 500 LoC Our implementation: copy & execute Bj¨orn D¨obel () OS Resilience 06.08.2013 44 / 58
  42. 42. Introduction Copy&Execute Master Replica Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  43. 43. Introduction Copy&Execute Master Replica mov eax, [ebx] X Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  44. 44. Introduction Copy&Execute Master Replica mov eax, [ebx] Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  45. 45. Introduction Copy&Execute Master Replica mov eax, [ebx] load repl. state NOP; NOP; ...; NOP restore master state Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  46. 46. Introduction Copy&Execute Master Replica mov eax, [ebx]mov eax, [ebx] load repl. state NOP; NOP; ...; NOP restore master state Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  47. 47. Introduction Copy&Execute Master Replica mov eax, [ebx] load repl. state NOP; NOP; ...; NOP restore master state mov eax, [ebx] Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  48. 48. Introduction Copy&Execute Master Replica mov eax, [ebx] load repl. state NOP; NOP; ...; NOP restore master state mov eax, [ebx] Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  49. 49. Introduction Copy&Execute Master Replica mov eax, [ebx] load repl. state NOP; NOP; ...; NOP restore master state mov eax, [ebx] Bj¨orn D¨obel () OS Resilience 06.08.2013 45 / 58
  50. 50. Introduction Runtime Overhead SPEC INT 2006 400 perl 401 bzip2 403 gcc 429 mcf 445 gobmk 456 hm- mer 458 sjeng 462 lib quan- tum 464 h264ref 471 om- net++ 473 as- tar 1 1.05 1.1 1.15 1.2 1.25 1.3 Runtimenormalized vs.nativeexecu- tion Single DMR TMR 1.45 1.95 Bj¨orn D¨obel () OS Resilience 06.08.2013 46 / 58
  51. 51. Introduction Replica-Core Placement Matters 429 mcf 429 mcf adj 462 lib quan- tum 462 lib quan- tum adj 471 om- net++ 471 om- net++ adj 1 1.05 1.1 1.15 1.2 1.25 1.3 Runtimenormalized vs.nativeexecu- tion Bj¨orn D¨obel () OS Resilience 06.08.2013 47 / 58
  52. 52. Introduction Romain Lines of Code Base code (main, logging, locking) 325 Application loader 375 Replica manager 628 Redundancy 153 Memory manager 445 System call proxy 311 Shared memory 281 Total 2,518 Fault injector 668 GDB server stub 1,304 Bj¨orn D¨obel () OS Resilience 06.08.2013 48 / 58
  53. 53. Introduction User land is covered! Replicated Driver Unreplicated Application Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 49 / 58
  54. 54. Introduction User land is covered! Reliable Computing Base Replicated Driver Unreplicated Application Replicated Application L4 Runtime Environment Romain L4/Fiasco.OC microkernel Bj¨orn D¨obel () OS Resilience 06.08.2013 49 / 58
  55. 55. Introduction Minimizing the RCB What to minimize? Lines of Code (as in TCB)? Bj¨orn D¨obel () OS Resilience 06.08.2013 50 / 58
  56. 56. Introduction Minimizing the RCB What to minimize? Lines of Code (as in TCB)? Time spent executing RCB code? Bj¨orn D¨obel () OS Resilience 06.08.2013 50 / 58
  57. 57. Introduction Minimizing the RCB What to minimize? Lines of Code (as in TCB)? Time spent executing RCB code? More likely: runtime ⇥ vulnerability Bj¨orn D¨obel () OS Resilience 06.08.2013 50 / 58
  58. 58. Introduction Hardening the RCB We need: Dedicated mechanisms to protect the RCB (HW or SW) We have: Full control over software RAD-hardened hardware? Too expensive Embrace heterogeneity! IBM Cell ARM big.LITTLE Bj¨orn D¨obel () OS Resilience 06.08.2013 51 / 58
  59. 59. Introduction Hardening the RCB We need: Dedicated mechanisms to protect the RCB (HW or SW) We have: Full control over software RAD-hardened hardware? Too expensive Embrace heterogeneity! IBM Cell ARM big.LITTLE Our proposal: Split HW into ResCores and NonRes-Cores ResCore NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core NonRes Core Bj¨orn D¨obel () OS Resilience 06.08.2013 51 / 58
  60. 60. Introduction Signaling Performance 10 20 30 40 50 60 Overheadin% Overhead by notification method Local Faults Migration Sync IPC Shared Mem susan CRC32 DMR susan CRC32 TMR Bj¨orn D¨obel () OS Resilience 06.08.2013 52 / 58
  61. 61. Introduction Signaling Performance 10 20 30 40 50 60 Overheadin% Overhead by notification method Local Faults Migration Sync IPC Shared Mem susan CRC32 DMR susan CRC32 TMR Fast shared-memory message passing would be good ! Intel SCC / Knights Corner RCB/Non-RCB boundary is vulnerable Messaging / Exceptions need to function Must not overwrite other data Bj¨orn D¨obel () OS Resilience 06.08.2013 52 / 58
  62. 62. Introduction Is software-level protection feasible? We have full source of the RCB. Compiler support for fault tolerance (SWIFT1 , AN-Encoded Processing2 ) may help. Hasn’t been done for kernel code yet. Bj¨orn D¨obel () OS Resilience 06.08.2013 53 / 58
  63. 63. Introduction Is software-level protection feasible? We have full source of the RCB. Compiler support for fault tolerance (SWIFT1 , AN-Encoded Processing2 ) may help. Hasn’t been done for kernel code yet. Gedankenexperiment: We know how much RCB-related execution is added due to replication. We know average overheads for SWIFT (9.5%) and AN encoding(390%)3 Bj¨orn D¨obel () OS Resilience 06.08.2013 53 / 58
  64. 64. Introduction Modeling software-level RCB protection Application Code tapp Kernel: System Calls tkern Romain Master Code tmaster Additional Kernel Invocations t0 kern Hardware Stalls (e.g., caching) thw Native execution time Replication overhead T = tnat + trep = tapp + tkern + tmaster + t0 kern + thw Tprot = tapp + C ⇥ (tkern + tmaster + t0 kern + thw ) tkern = t0 kern = thw = 0 Tprot = tapp + C ⇥ tmaster Bj¨orn D¨obel () OS Resilience 06.08.2013 54 / 58
  65. 65. Introduction Estimating RCB protection runtime 400 perl 401 bzip2 429 mcf 445 gobmk 456 hm- mer 458 sjeng 462 lib quan- tum 464 h264ref 471 om- net++ 473 as- tar 1 1.05 1.1 1.15 1.2 1.25 1.3 1.4 1.5 1.6 Runtimenormalized vs.nativeexecu- tion Romain only Romain+SWIFT Romain+ANBD Bj¨orn D¨obel () OS Resilience 06.08.2013 55 / 58
  66. 66. Introduction Summary OS-level techniques to tolerate SW and HW faults Address-space isolation Microreboots Various ways of handling session state Replication against hardware errors Special care needed to protect Reliable Computing Base Bj¨orn D¨obel () OS Resilience 06.08.2013 56 / 58
  67. 67. Introduction Further Reading Minix3: Jorrit Herder, Ben Gras,, Philip Homburg, Andrew S. Tanenbaum: Fault Isolation for Device Drivers, DSN 2009 CuriOS: Francis M. David, Ellick M. Chan, Je↵rey C. Carlyle and Roy H. Campbell CuriOS: Improving Reliability through Operating System Structure, OSDI 2008 L4ReAnimator: Dirk Vogt, Bj¨orn D¨obel, Adam Lackorzynski: Stay strong, stay safe: Enhancing Reliability of a Secure Operating System, IIDS 2010 Bj¨orn D¨obel () OS Resilience 06.08.2013 57 / 58
  68. 68. Introduction Further Reading Reliability Analysis: Saggese et al.: An Experimental Study of Soft Errors in Microprocessors, IEEE Micro 2005 Li et al.: Understanding the Propagation of Hard Errors to Software and Implications for Resilient System Design, ASPLOS 2008 Arlat et al.: Dependability of COTS Microkernel-Based Systems, IEEE ToCS 2002 Wang et al.: Y-Branches: When you come to a Fork in the Road: Take it!, PACT 2003 PLR: Alex Shye, Tipp Moseley, Vijay Janapa Reddi, Joseh Blomsted, Ramesh Peri: Using Process-Level Redundancy to Exploit Multiple Cores for Transient Fault Tolerance, DSN 2007 Romain: Bj¨orn D¨obel, Hermann H¨artig, Michael Engel: Operating System Support for Redundant Multithreading, EMSOFT 2012 Bj¨orn D¨obel, Hermann H¨artig: Who watches the watchmen? – Protecting Operating System Reliability Mechanisms, HotDep 2012 Bj¨orn D¨obel () OS Resilience 06.08.2013 58 / 58

×