2. Dependable Systems Course PT 2014
Dependability
ā¢ Umbrella term for operational requirements on a system
ā¢ IFIP WG 10.4: "[..] the trustworthiness of a computing system which allows reliance
to be justiļ¬ably placed on the service it delivers [..]"
ā¢ IEC IEV: "dependability (is) the collective term used to describe the availability
performance and its inļ¬uencing factorsĀ : reliability performance, maintainability
performance and maintenance support performance"
ā¢ Laprie: ā Trustworthiness of a computer system such that āØ
reliance can be placed on the service it delivers to the user ā
ā¢ Adds a third dimension to system quality
ā¢ General question: How to deal with unexpected events ?
ā¢ In German: ,VerlƤsslichkeitā vs. ,ZuverlƤssigkeitāāØ
2
6. Dependable Systems Course PT 2014
Dependability Stakeholders
ā¢ System - Entity with function, behavior, and structure
ā¢ A number of components or subsystems, which interact under the control of a
design [Robinson]
ā¢ Service - System behavior abstraction, as perceived by the user
ā¢ User - Human or physical system that interacts with the systems service
ā¢ Speciļ¬cation - Deļ¬nition of expected service and delivery conditions
ā¢ On diļ¬erent levels, can lead to speciļ¬cation fault
ā¢ Reliance demands assessment of non-functional dependability attributes
ā¢ Provide ability for trustworthy service delivery by dependability means
ā¢ Undesired (maybe expected) circumstances form dependability threats
6
7. Dependable Systems Course PT 2014
2 - Dependability Threats
ā¢ System failure - ,Ausfallā
ā¢ Event that occurs when the service no longer complies with the speciļ¬cation /
deviates from the correct service.
ā¢ System error - ,Fehler(zustand)ā
ā¢ Part of system state that can lead to subsequent failure
ā¢ Some sources deļ¬ne errors as active faults - not in this course ...
ā¢ System fault - ,Fehler(ursache)ā
ā¢ Adjudged or hypothesized cause of an error
ā¢ Failure occurs when error state alters the provided service
ā¢ Systems are build from connected components, which are again systems
ā¢ Fault is the consequence of a failure of some other system to deliver its service
7
Fault
Error
Failure
8. Dependable Systems Course PT 2014
Chain of Dependability Threats (Avizienis)
8
Fault
Error
Failure
Activation
Propagation
Causation
Fault
9. Dependable Systems Course PT 2014
Faults
ā¢ High diversity in possible sources and types
ā¢ Fault nature
ā¢ Accidental faults (,Zufallsfehlerā) vs. intentional faults (,Absichtsfehlerā)
ā¢ Intentional faults are created deliberately, presumably malevolently
ā¢ Fault origin viewpoints (not exclusive)
ā¢ Phenomenological causes: Physical / natural faults vs. human-made faults
ā¢ System boundaries: Internal faults (part of system state that produces an error)
vs. external faults (interference with the environment)
ā¢ Phase of creation: Design faults vs. operational faults
ā¢ Temporal persistence
ā¢ Permanent faults vs. temporary faults
9
10. Dependable Systems Course PT 2014
System-Level Fault Model
ā¢ Fault model idea originates from hardware
ā¢ How many faults of diļ¬erent classes can occur ? āØ
What do I tolerate ?
ā¢ Timing of faults: Fault delay, repeat time, recovery time, ...
ā¢ Also mappable to software or even complete systems
ā¢ Activities as black box, only look on input and output messages
ā¢ Link faults are mapped to the āØ
participating components
ā¢ Every participating component āØ
would need a fault model - āØ
pick the most urgent ones
10
12. Dependable Systems Course PT 2014
Error Message Occurrence (Hansen & Siewiorek)
ā¢ Same fault can lead to diļ¬erent (detected or undetected) errors
ā¢ Errors become detected by error detection mechanism
ā¢ Some undetected errors are detected by several detectors
ā¢ Some detectors report several undetected errors as one
ā¢ Some undetected errors are never uncovered
ā¢ Detected errors might not be logged, if the system stops too fast
12
13. Dependable Systems Course PT 2014
Failures
ā¢ Non-compliance with the speciļ¬cation - arbitrary failure (āwillkĆ¼rlicher Ausfallā)
ā¢ System failures can be further categorized in failure modes
ā¢ Fail-silent / crash failure mode - incorrect results are not delivered
ā¢ Fail-stop mode - constant value is delivered
ā¢ Failure mode domain - what is inļ¬uenced
ā¢ Service result - value failures
ā¢ Service timeliness - timing failures
ā¢ Service availability - stopping failures
ā¢ User perception in the mode - consistent / inconsistent for all users
ā¢ Failure mode consequences for ranking the identiļ¬ed issues
13
14. Dependable Systems Course PT 2014
Failure Severity (,Schweregrad des Ausfallsā)
ā¢ Denotes consequences of failure
ā¢ Benign failures (,unkritische AusfƤlleā)
ā¢ Failure costs and operational beneļ¬ts are similar
ā¢ Sometimes also umbrella term for failures only detected by inspection
ā¢ A system with only such failures is fail-safe
ā¢ Catastrophic failures (,kritische AusfƤlleā)
ā¢ Costs of failure consequences are much larger than service beneļ¬t
ā¢ Signiļ¬cant / serious failures - Intermediate steps expressing reduced service
ā¢ Grading of failure consequences on overall system depends on application
ā¢ Flying airplane - Catastrophic stopping failure, Train - Benign stopping failure
ā¢ Criticality - Highest severity of possible failure modes in the system
14
16. Dependable Systems Course PT 2014
3 - Means for Dependability
ā¢ Fault prevention - Prevent fault occurrence or introduction
ā¢ Fault tolerance - Provide service matching the speciļ¬cation under faults
ā¢ Fault removal - How to reduce the presence of faults
ā¢ Fault forecasting- Estimate the present number, future incidence, and the
consequences of faults
ā¢ Combined utilization
16
17. Dependable Systems Course PT 2014
Fault Tolerance
ā¢ Fault tolerance is the ability of a system to operate correctly in presence of faults.
or
ā¢ A system S is called k-fault-tolerant with respect to a set of āØ
algorithms {A1, A2, ... , Ap} and a set of faults {F1, F2, ... , Fp} āØ
if for every k-fault F in S, Ai is executable by a subsystem of system S with k faults.
(Hayes, 9/76)
or
ā¢ Fault tolerance is the use of redundancy (time or space) to achieve the desired level
of system dependability - costs !
ā¢ Accepts that an implemented system will not be fault-free
ā¢ Implements automatic recovery from errors
ā¢ Is a recursive concept (voter replication, self-checking checkers, stable memory)
17
18. Error Processing
Dependable Systems Course PT 2014
Phases of Fault Tolerance (Hanmer)
18
Latent
Fault
Error
Normal
OperationFault
Activation
Error Recovery
Error Mitigation
Error
Detection
Fault
Treatment
19. Dependable Systems Course PT 2014
Fault Tolerance - Error Processing Through Recovery
ā¢ Forward error recovery
ā¢ Error is masked to reach again a consistent state (fault compensation)
ā¢ Corrective actions need detailled knowledge (damage assessment)
ā¢ New state is typically computed in another way
ā¢ Examples with compensation: Error correcting codes, non-journaling ļ¬le
system check, advanced exception handlers, voters
ā¢ Backward error recovery
ā¢ Roll back to previous consistent state (recovery point / checkpoint)
ā¢ Very suitable for transient faults
ā¢ Computation can be re-done with same components (retry), āØ
with alternate components (reconļ¬gure), or can be ignored (skip frame)
19
20. Dependable Systems Course PT 2014
āFail-Fastā
ā¢ A common concept from system engineering, company management, ...
ā¢ āReport failure and stop immediately without further actionā
ā¢ Discussed by Jim Gray in 1985 as part of his famous articleāØ
āWhy do computers stop and what can be done about it ?ā
ā¢ Useful when beneļ¬t from recovery is not good enough for its costs, āØ
or if error propagation is highly probable
ā¢ Single units of a redundant set
ā¢ Deeply interwired IT system components
ā¢ Components under heavy request load
ā¢ And ... crappy start-up companies
20
21. Dependable Systems Course PT 2014
4 - Fault Tolerance Patterns
ā¢ Architectural patterns
ā¢ Considerations that cut across all parts of the system
ā¢ Need to be applied in early design phase
ā¢ Detection patterns
ā¢ Detect the presence of root faults, error states, and failures
ā¢ Errors vs. failures -> a-priori knowledge vs. comparison of redundant elements
ā¢ Error Recovery Patterns
ā¢ Methods to continue execution in a new error-free state
ā¢ Undoing the error eļ¬ects + creating the new state
21
22. Dependable Systems Course PT 2014
4 - Fault Tolerance Patterns
ā¢ Error Mitigation Patterns
ā¢ Do not change application or system state, āØ
but mask the error and compensate for the eļ¬ects
ā¢ Typical strategies for timing or performance faults
ā¢ Fault Treatment Patterns
ā¢ Prevent the error from reoccurring by repairing the fault
ā¢ System veriļ¬cation
ā¢ Diagnosis of fault location and nature
ā¢ Correction of the system and / or the procedures
22
23. Dependable Systems Course PT 2014
5 - Attributes of Dependability
ā¢ Non-functional attributes such as reliability and maintainability
ā¢ Complementary nature of viewpoints in the area of dependability
ā¢ In comparison to functional properties
ā¢ ... hard to deļ¬ne
ā¢ ... hard to abstract
ā¢ ... ,Divide and conquerā does not work as good
ā¢ ... diļ¬cult interrelationships
ā¢ ... often probabilistic dependencies
23
24. Dependable Systems Course PT 2014
In Detail
ā¢ Reliability - Function R(t)
ā¢ Probability that a system is functioning properly and constantly over time period t
ā¢ Assumes that system was fully operational at t=0
ā¢ Denotes failure-free interval of operation
ā¢ Availability - Statement if a system is operational at a point in time / fraction of time
ā¢ Describe system behavior in presence of error treatment mechanisms
ā¢ Instantaneous availability (at t) - Probability that a system is performing
correctly at time t; equal to reliability for non-repairable systems: Ai(t) = R(t)
ā¢ Steady-state availability - Probability that a system will be operational at any
random point of time, expressed as the fraction of time a system is operational
during its expected lifetime: As = Uptime / Lifetime
24
25. Dependable Systems Course PT 2014
Why Exponential ?
ā¢ Distribution function that models the memoryless property of the Poisson process
ā¢ P(T > t + s|T > t) = P(T > s), e.g. PFailure(5 years|T > 2 years) = PFailure(3 years)
ā¢ Failure is not the result of wear-out
ā¢ Models ,intrinsic failureā behavior, assumed for the majority of hardware life time
ā¢ Weibull distribution as alternative, can also model tear-in and wear-out
ā¢ Some natural phenomena have constant failure rate (e.g. cosmic ray particles)
25
26. Dependable Systems Course PT 2014
Variable Failure Rate in Real World
26
Burn in Use Wear out Integration
& Test
Use Obsolete
Hardware Software
ā¢ Failure rate is treated as constant parameter of the exponential distribution
ā¢ (maybe invalid) simpliļ¬cation, mostly combined solution in practice:
ā¢ Exponential distribution when failure rate is constant
ā¢ Weibull distribution when failure rate is monotonic decreasing / increasing
27. ā¢ Mean time to failure (MTTF) - āØ
Average time it takes to failāØ
-> average uptime
ā¢ Mean time to recover / repair (MTTR) -
Average time it takes to recover
ā¢ Mean time between failures (MTBF) -
Average time between two successive failures
ā¢ Availability = Uptime / LifetimeāØ
= MTTF / MTBF
Dependable Systems Course PT 2014
Steady-State Availability
27
up down up
MTBF
down up
up
C1
up
C3
up
C2
MTTF
28. Dependable Systems Course PT 2014
Steady-State Availability
28
Availability Downtime per year Downtime per week
90.0 % (1 nine) 36.5 days 16.8 hours
99.0 % (2 nines) 3.65 days 1.68 hours
99.9 % (3 nines) 8.76 hours 10.1 min
99.99 % (4 nines) 52.6 min 1.01 min
99.999 % (5 nines) 5.26 min 6.05 s
99.9999 % (6 nines) 31.5 s 0.605 s
99.99999 % (7 nines) 0.3 s 6 ms
A = Uptime
Uptime+Downtime = MT T F
MT T F +MT T R
29. Dependable Systems Course PT 2014
MTTR << MTTF [Fox]
ā¢ Armando Fox on ,Recovery-Oriented Computingā
ā¢ A = MTTF / (MTTF + MTTR)
ā¢ 10x decrease of MTTR as good as 10x increase of MTTF ?
ā¢ MTTFās are not claimable, but MTTR claims are veriļ¬able
ā¢ Proving MTTF numbers demands system-years of observation and experience
ā¢ Lowering MTTR directly improves user experience of one speciļ¬c outage, āØ
since MTTF is typically longer than one user session
ā¢ HCI factor of failed system
ā¢ Miller, 1968: >1sec āsluggishā, >10sec ādistractedā (user moves away)
ā¢ 2001 Web user study: ~5sec āacceptableā, ~10sec āexcessively slowā
29
30. Dependable Systems Course PT 2014
6 - Dependability Modeling
ā¢ Default approach: Utilize a formalism to model system dependability
ā¢ Quantify the availability of components, calculate system availability based on this
data and a set of assumptions (the availability model)
ā¢ Most models expose the same expressiveness
ā¢ Each formalism allows to focus on certain aspects
ā¢ Structure-based models: Reliability block diagram, fault tree
ā¢ State-based models: Markov chain, petri net
ā¢ System understanding evolved from hardware to software to IT infrastructures
ā¢ Example: Organization management inļ¬uence on business service reliability
ā¢ Information Technology Infrastructure Library (ITIL)
ā¢ CoBiT(Control Objectives for Information and related Technology)
30
31. Dependable Systems Course PT 2014
Dependability Modeling
ā¢ The Failure Space-Success Space concept
ā¢ Often easier to agree on what constitutes a system failure
ā¢ Success tends to be associated with system eļ¬ciency, which makes it harder to
formulate events in the model (āThe car drives fast.ā, āThe car stops driving.ā)
ā¢ In practice, there are more ways to success than to failure
31
Fault Tree Handbook with Aerospace Applications Version 1.1
2. System Logical Modeling Approaches
2.1 Success vs. Failure Approaches
The operation of a system can be considered from two standpoints: the various ways for system
success can be enumerated or the various ways for system failure can be enumerated. Such an
enumeration would include completely successful system operation and total system failure, as
well as intermediate conditions such as minimum acceptable success. Figure 2-1 depicts the
Failure/Success space concept.
Figure 2-1. The Failure Space-Success Space Concept
32. Dependable Systems Course PT 2014
Dependability Modeling
ā¢ System analysis approaches
ā¢ Inductive methods - Reasoning from speciļ¬c cases to a general conclusion
ā¢ Postulate a particular fault or initiating event, ļ¬nd out system eļ¬ect
ā¢ Determine what system (failure) states are possible
ā¢ Trivial approach: āparts countā method
ā¢ Examples: Failure Mode and Eļ¬ect Analysis (FMEA), Preliminary Hazards
Analysis (PHA), Event Tree Analysis, Reliability Block Diagrams (RBD), ...
ā¢ Deductive methods - Postulate a system failure, ļ¬nd out what system modes or
component behaviors contribute to this failure
ā¢ Determine how a particular system state can occur
ā¢ Examples: Fault Tree Analysis (FTA)
32
33. S = cLB (cW S1 ā„ cW S2) (cDB1 ā„ cDB2)
Asite = aLB ā„ AW Sset ā„ ADBset
= aLB ā„ [1 (1 aW S)nW S
] ā„ [1 (1 aDB)nDB
]
Dependable Systems Course PT 2014
Examples
33
aWS
aWS
aDB
aDB
aLB
34. Dependable Systems Course PT 2014
Reliability Block Diagrams (RBD)
ā¢ Model logical interaction for success-oriented analysis of system reliability
ā¢ Building blocks: series structure, parallel structure, k-out-of-n structure
ā¢ System is available only if there is a path between s and t
ā¢ Granularity based on data and lowest actionable item concept
ā¢ Structure formula can be obtained from RBD by identifying the āØ
subset of nodes that disconnects s from t if removed
34
s t
A1
A2
A3
A4
B
Spare
C
D
E
2/3
C
C
D
D
E
E
35. Dependable Systems Course PT 2014
Deductive Analysis - Fault Trees
ā¢ Structure analysis eļ¬ort grows exponentially with the number of components
ā¢ Fault Trees
ā¢ Invented 1961 by H. Watson (Bell Telephone Laboratories)
ā¢ Facilitate analysis of the launch control system of the intercontinental
Minuteman missile
ā¢ Used by Boeing since 1966, meanwhile adopted by diļ¬erent industries
ā¢ Root cause analysis, risk assessment, safety assessment
ā¢ Basic idea
ā¢ Technique for describing the possible ways in which an āØ
undesired system state can occur
ā¢ Complex system failures are broken down into basic events
35
36. Example: 2-of-3 System
State-Based Modeling | Dependable Systems 2014 24
The complexity of the petri net does not
depend on the number of components!
Dependable Systems Course PT 2014
7 - State-Based Modeling
ā¢ Structural vs. state-based modeling
ā¢ State Transition Diagrams
ā¢ Markov Chains
ā¢ Petri Nets
36
37. 37
State-Based Modeling | Dependable Systems 2014 34
Dependable Systems Course
Real World
Systems
Model
Solution
Technique
Evaluations
Input
Parameters
Modeling Errors
ā¢ Structural Errors: Initial state,
missing or extra states / transitions
ā¢ Error Propagation Model
ā¢ Parametric Models: Failure and repair rates,
coverage parameters
ā¢ Errors due to non-independence
Solution Errors
ā¢ Approximation Errors: System partition, state aggregation
ā¢ Numerical Errors: Truncation, Round-off
ā¢ Programming Errors
ā¢ Estimation Errors: Stochastic estimators, not enough data
Parametric Errors
ā¢ Different component parameter sources
ā¢ Projected stress factors assume unrealistic
operational conditions
38. Dependable Systems Course PT 2014
8 - Qualitative Dependability Investigation
ā¢ Diļ¬erent approaches that focus on structural (qualitative) system evaluation
ā¢ Root cause analysis
ā¢ Broad research / industrial topic targeting error diagnosis
ā¢ Specialized topic in quality methodologies
ā¢ Development process investigation
ā¢ Procedures for ensuring industry quality in production
ā¢ Software development process
ā¢ Organizational investigation
ā¢ Non-technical inļ¬uence factors on system reliability
38
39. Dependable Systems Course PT 2014
FMEA
ā¢ Failure Mode and Eļ¬ects Analysis
ā¢ Engineering quality method for early concept phase - identify and tackle weak points
ā¢ Introduced in the late 1940s for military usage (MIL-P-1629)
ā¢ Later also used for aerospace program, automotive industry, āØ
semiconductor processing, software development, healthcare, ...
ā¢ Main goal is to identify and prevent critical failures
ā¢ Performed by cross-functional team of subject matter experts
ā¢ Most important task in many reliability programs
ā¢ Six Sigma certiļ¬cation, reliability-centered maintenance (RCM) approach
ā¢ Automotive industry (ISO16949, SAE J1739)
ā¢ Medical devices
39
40. Dependable Systems Course PT 2014
Hazard & Operability Studies (HAZOPS)
ā¢ Process for identiļ¬cation of potential hazard & operability problems,āØ
caused by deviations from the design intend
ā¢ Diļ¬erence between deviation (failure) and its cause (fault)
ā¢ Conduct intended functionality in the safest and most eļ¬ective manner
ā¢ Initially developed to investigate chemical production processes, āØ
meanwhile for petroleum, food, and water industries
ā¢ Extended for complex (software) systems
ā¢ Qualitative technique
ā¢ Take full description of process and systematically question every part of it
ā¢ Assess possible deviations and their consequences
ā¢ Based on guide-words and multi-disciplinary meetings
40
41. Dependable Systems Course PT 2014
Root Cause Analysis
ā¢ What caused the fault ? - Starting point of dependability chain
ā¢ Peeling back the layers
ā¢ Must be performed systematically as an investigation
ā¢ Establish sequence of events / timeline
41
(C) Avizienis
42. Dependable Systems Course PT 2014
Reliability Models for IT Infrastructures
ā¢ System reliability in a commercial environment is determined by many factors:
ā¢ Software and hardware reliability
ā¢ Training of maintenance personnel
ā¢ āBusiness processesā how maintenance is handled
ā¢ The way the IT department is organized
ā¢ ...
ā¢ Impact of management organization on reliability is an emerging research ļ¬eld
ā¢ Standards for IT organization, based on best practices
ā¢ Describe which processes have to be established in an IT department
ā¢ Provide reference models for organization of IT department
42
44. Dependable Systems Course PT 2014
ITIL
ā¢ Information Technology Infrastructure Library (ITIL), latest version v3
ā¢ Started as set of recommendations by the UK Government
ā¢ Concepts and guides for IT service management
ā¢ Supports to deliver business-oriented āØ
quality IT services
ā¢ Core publications: Service Strategy, āØ
Service Design, Service Transition, āØ
Service Operation, āØ
Continual Service Improvement
ā¢ Broad tool support from vendors
ā¢ High costs for certiļ¬cation and training
ā¢ Methodology sometimes over-respected at āØ
the expense of pragmatism
44
45. Dependable Systems Course PT 2014
9 - Predicting System Reliability
ā¢ Feasibility evaluation for given model, identiļ¬cation of potential problem sources
ā¢ Comparison of competing designs
ā¢ Identiļ¬cation of potential reliability problems - low quality, over-stressed parts
ā¢ Input to other reliability-related tasks
ā¢ Maintainability analysis, testability evaluation
ā¢ Failure modes and eļ¬ects analysis (FMEA)
ā¢ Ultimate goal is the prediction of system failures by a reliability methodology
ā¢ Approach depends on nature of component (electrical, electronic, mechanical)
45
46. Dependable Systems Course PT 2014
Reliability Data
ā¢ Field (operational) failure data
ā¢ Meaningful source of information, experience from real world operation
ā¢ Operational and environmental conditions may be not fully known
ā¢ Service life data with / without failure
ā¢ Helpful in assessing time characteristics of reliability issues
ā¢ Data from engineering tests
ā¢ Example: Accelerated life tests
ā¢ Results from controlled environment
ā¢ Trustworthy for analysis purposes
ā¢ Lack of failure information is the ,greatest deļ¬ciencyā in reliability research [Misra]
46
47. Dependable Systems Course PT 2014
Reliability Prediction Models
ā¢ Economic requirements aļ¬ect ļ¬eld-data availability
ā¢ Customers do not need generic ļ¬eld data, so collection is extra eļ¬ort
ā¢ Numerical reliability prediction models available for speciļ¬c areas
ā¢ Hardware-oriented reliability modeling
ā¢ MIL-HDBK-217, Bellcore Reliability Prediction Program, ...
ā¢ Software-oriented reliability modeling
ā¢ Jelinski-Moranda model, Basic Execution model, software metrics, ...
ā¢ Prediction models look for corrective factors, so they are always focused
ā¢ Demands conļ¬dence in understanding of environmental conditions
ā¢ Reconciliation is often needed between the diļ¬erent values of failure data from
various sources
47
48. Dependable Systems Course PT 2014
Software Reliability Assessment
ā¢ Software bugs are permanent faults, but behave as transient faults
ā¢ Activation depends on software usage pattern and input values
ā¢ Timing eļ¬ects with parallel or distributed execution
ā¢ Software aging eļ¬ects
ā¢ Fraction of system failures reasoned by software increased in the last decades
ā¢ Possibilities for software reliability assessment
ā¢ Black box models - No details known, observations from testing or operation
ā¢ Reliability growth models
ā¢ Software metric models - Derive knowledge from static code properties
48
49. Dependable Systems Course PT 2014
Halstead Metric
ā¢ Statistical approach - Complexity is related to number of operators and operands
ā¢ Only deļ¬ned on method level, OO code analysis must consider additional structure
ā¢ Program length N = Number of operators NOP + total number of operands NOD
ā¢ Vocabulary size n = Number of unique operators nOP + number of unique operands nOD
ā¢ Program volume V = N * log2(n)
ā¢ Describes size of an implementation by vocabulary and (mainly) by program length
ā¢ In-place changes are ok ...
ā¢ Diļ¬culty level D = (nOP / 2) * (NOD / nOD), program level L = 1 / D
ā¢ Error proneness is proportional to number of unique operators āØ
-> since most languages only oļ¬er a few, this should be small
ā¢ Also proportional to the level of operand reuse through the code
49
x = x + 1;
NOP=nOP=3
NOD=3
nOD=2
50. Dependable Systems Course PT 2014
10 - Distributed Systems Theory
ā¢ Fallacies of Distributed Computing
ā¢ The network is reliable, Latency is zero, Bandwidth is inļ¬nite, The network is
secure, Topology doesnāt change, There is one administrator, Transport cost is
zero, The network is homogeneous
ā¢ Timing Models
ā¢ Logical time, Lamport clocks
ā¢ Fault Models
ā¢ Consensus Problems
ā¢ 2PC
ā¢ Paxos
50
Asynchronous*
Par-ally*synchronous*
Synchronous*
53. Dependable Systems Course PT 2014
13 - Hardware Diagnosis
ā¢ Contains of fault detection (what ?) and fault location (where ?)
ā¢ Fault detection
ā¢ Replication checks - Test operation / execution against alternate implementation
ā¢ Timing checks - Test operation / execution against timing constraints
ā¢ Reversal checks - Make use of operation reversibility
ā¢ Coding checks - Utilize redundant (but diļ¬erent) representation of data
ā¢ Reasonableness checks - Check against known system / data properties
ā¢ Structural checks - Ensure consistent structure of data, diagnostic tests, ...
ā¢ Diagnostic checks - Use set of inputs for which the outputs are known
ā¢ Algorithmic checks - Check invariants of an algorithm
53
54. Dependable Systems Course PT 2014
14 - Hardware Redundancy
ā¢ Redundancy for error detection and forward error recovery
ā¢ Redundancy types: spatial, temporal, informational (presentation, version)
ā¢ Redundant not mean identical functionality, just perform the same work
ā¢ Static redundancy implements error mitigation
ā¢ Fault does not show up, since it is transparently removed
ā¢ Examples: Voting, error-correcting codes, N-modular redundancy
ā¢ Dynamic redundancy implements error processing
ā¢ After fault detection, the system is reconļ¬gured to avoid a failure
ā¢ Examples: Back-up sparing, duplex and share, pair and spare
ā¢ Hybrid approaches
54
55. Dependable Systems Course PT 2014
Sphere of Replication
ā¢ Components outside the
sphere must be protected by
other means
ā¢ Level of output comparison
decides upon fault coverage
ā¢ Larger sphere tends to
decrease the required
bandwidth on input and output
ā¢ More state changing
happens just inside the
sphere
ā¢ Vendor might be restricted on
choice of sphere size
55
HDD HDD
Input Replication
Output
Comparison
I/O Bus
HDD HDD
Input Replication
Output
Comparison
HDD HDD
Controll
er
Controll
er
RAID1
RAID10
56. Dependable Systems Course PT 2014
Masking / Static Redundancy: Voting
ā¢ Exact voting: Only one correct result possible
ā¢ Majority vote for uneven module numbers
ā¢ Generalized median voting - Select result that is the median, āØ
by iteratively removing extremes
ā¢ Formalized plurality voting - Divide results in partitions, āØ
choose random member from the largest partition
ā¢ Inexact voting: Comparison at high level might lead to multiple correct results
ā¢ Non-adaptive voting - Use allowable result discrepancy, put boundary on
discrepancy minimum or maximum (e.g. 1,4 = 1,3)
ā¢ Adaptive voting - Rank results based on past experience with module results
ā¢ Compute the correct value based on ātrustā in modules from experience
ā¢ Example: Weighted sum R=W1*R1 + W2*R2 + W3*R3 with W1+W2+W3=1
56
Voter
A B C
57. Dependable Systems Course PT 2014
N-Modular Redundancy (with perfect voter)
57
Module 1
Module 2
Module 3 Voter
...
Module N
Input Output
RNMR =
mX
i=0
ā
N
i
ā
(1 R)i
RN i
ā
n
k
ā
=
n!
k!(n k)!
R2 of 3 =
ā
3
0
ā
(1 R)0
R3
+
ā
3
1
ā
(1 R)R2
R2 of 3 = R3
+ 3(1 R)R2
R3 of 5 = ...
58. ā¢ Special cases for combination of duplex (with comparator) and sparing (with switch)
ā¢ Pair and spare - Multiple duplex pairs, connected as standby sparing setup
ā¢ Two replicated modules operate as duplex pair (lockstep execution), āØ
connected by comparator as voting circuit
ā¢ Same setting again as spare unit, āØ
spare units connected by switch
ā¢ On module output mismatch, comparators āØ
signal switch to perform failover
ā¢ Commercially used, e.g. Stratus XA/R Series 300
Dependable Systems Course PT 2014
Pair and Spare
58
MODULES
1
2
3
OUTPUTINPUT
COMPARATOR
4 COMPARATOR
SWITCH/COMPARATOR
59. Dependable Systems Course PT 2014
Hybrid Approaches
ā¢ N-modular redundancy āØ
with spares
ā¢ Also called hybrid redundancy
ā¢ System has basic NMR āØ
conļ¬guration
ā¢ Disagreement detector replacesāØ
modules with spares if their āØ
output is not matchingāØ
the voting result
ā¢ Reliability as long as the spare pool is not exhausted
ā¢ Improves fault masking capability of NMR
ā¢ Can tolerate two faults with one spare, while classic NMR would need 5
modules with majority voting to tolerate two faults
59
Adds fault
localization
61. Dependable Systems Course PT 2014
Memory Redundancy
ā¢ Standard technology in DRAMs
ā¢ Bit-per-byte parity, check on read access
ā¢ Implemented by additional parity memory chip
ā¢ ECC with Hamming codes - 7 check bits for 32 bit data words, 8 bit for 64 bit
ā¢ Leads to 72 bit data bus between DIMM and chipset
ā¢ Computed by memory controller on write, checked on read
ā¢ Study by IBM: ECC memory achieves R=0.91 over three years
ā¢ Can correct single bit errors and detect double bit errors
ā¢ Hewlett Packard Advanced ECC (1996)
ā¢ Can detect and correct single bit and double bit errors
61
62. Dependable Systems Course PT 2014
RAID
ā¢ Redundant Array of Independent Disks (RAID) [Patterson et al. 1988]
ā¢ Improve I/O performance and / or reliability by building raid groups
ā¢ Replication for information reconstruction on disk failure (degrading)
ā¢ Requires computational eļ¬ort (dedicated controller vs. software)
ā¢ Assumes failure independence
62
64. Dependable Systems Course PT 2014
15 - Software Dependability
ā¢ Fault elimination
ā¢ Reduce number of dormant faults at development time
ā¢ Fault-tolerant software
ā¢ Techniques to achieve fault tolerance for software faults
ā¢ Application of redundancy idea to software modules
ā¢ Software fault tolerance
ā¢ Techniques to achieve fault tolerance by software mechanisms
ā¢ Typically for hardware failures on lower levels in the system stack
ā¢ Redundancy managed by operating system, cluster framework, application code
64
65. Dependable Systems Course PT 2014
Fault Elimination through Software Testing
65
(C) Ian Sommerville
Responsibility of
the component
developer, based
on experience
Responsibility of an
independent testing
team, based on
speciļ¬cation
Black-Box
Tests
Access to source
code, tested while
components are
integrated
Steadily
increasing
load
Excess
maximum
design load
66. Dependable Systems Course PT 2014
Testing Through Software Fault Injection
ā¢ Intentionally trigger erroneous behavior of the execution environment
ā¢ Typical approach for communicating software entities
ā¢ Orientation towards implementation details - program state, functional behavior
ā¢ Several non-intrusive implementation techniques, such as AOP
ā¢ Injection can be done as part of execution under real conditions
ā¢ Huge variety of fault classes, including SWIFI fault classes
ā¢ New approaches support remote fault injection (e.g. fuzzing)
ā¢ Compile-time injection: Leads to erroneous image being executed
ā¢ Run-time injection: Demands some altering of application state during runtime
ā¢ Typical triggers: Time-out, exception, debugging trap, code insertion
66
67. Dependable Systems Course PT 2014
Software Fault Model [Goloubeva]
ā¢ Example: Control loop writing computation result to a variable
ā¢ Temporary fault: Write NULL to result variable after end of usage
ā¢ Permanent fault: Compute and store incorrect output from input data
ā¢ Single vs. multiple faults depends on granularity level of investigation
ā¢ On source code level
ā¢ Program: Structured collection of features with syntax and semantic
ā¢ Syntax allows to describe fault model
ā¢ Negation of semantic properties deļ¬nes a fault model
ā¢ Examples: Calling non-existent functions, Functions not returning a value, values
out of their type range, missing input parameters
ā¢ On executable level, e.g. stack overļ¬ow due to recursion
67
68. Dependable Systems Course PT 2014
Fault-Tolerant Software -
Another categorization [Lyu 95]
ā¢ Single version techniques
ā¢ Add mechanisms for detection, containment, and handling of errors to the
software component itself
ā¢ Examples: Software structure and actions approaches, error detection, āØ
exception handling, checkpointing and restart, process pairs, data diversity
ā¢ Multi version techniques
ā¢ Rely on structured utilization of variants of the same software
ā¢ Examples: Recovery blocks, N-version programming
ā¢ Principles can be applied to any software layer
ā¢ Identify source of most design faults
ā¢ Typically no problem with parallel application, beside cost factor
68
69. Dependable Systems Course PT 2014
Single-Version Approaches -
Wrapper
ā¢ Piece of software that encapsulates a given program when it is being executed
ā¢ Typical approach for operating systems and middleware stacks
ā¢ Structure: Wrapper software and wrapped entity
ā¢ Inputs and outputs are checked by the wrapper
ā¢ Examples:
ā¢ Dealing with buļ¬er overļ¬ow, checking scheduler correctness (e.g., EDF),
bypassing known bugs, checking output correctness
ā¢ When pre- or postconditions are violated, usually an exception is being raised
ā¢ Wrapper forms an acceptance test in the dependability rings
ā¢ Good approach for ļ¬xing issues in the operational phase of software
69
70. ā¢ Save application state data at recovery points
ā¢ Can be reloaded on crash or any other kind āØ
of data loss
ā¢ Possible on diļ¬erent levels: local per process, āØ
partial, complete, distributed
ā¢ Optimum checkpointing interval
ā¢ Checkpointing too frequent: Majority of time spent for data saving
ā¢ Checkpointing too rare: May take long time to recover
ā¢ Several specialized solutions for C / C++ language, easier with reļ¬ection support
ā¢ Popular approach in clusters / high-performance computing
ā¢ Latest trend: In-memory checkpointing
Dependable Systems Course PT 2014
Single Version Approaches -
Checkpointing
70
taken from āØ
Software Fault Tolerance: A Tutorial
71. Dependable Systems Course PT 2014
Single Version Approaches -
Data Diversity [Ammann 88]
71
takenfromāØ
SoftwareFaultTolerance:ATutorial
72. Dependable Systems Course PT 2014
Single Version Approaches -
High-Level Instruction Duplication [Goloubeva]
ā¢ Introduce data and code redundancy through
high-level transformation
ā¢ Duplicate every variable
ā¢ Perform every write operation on both copies
of the variable
ā¢ After each read operation, the copies must be
checked for consistency
ā¢ Should be close to read operation, āØ
in order to avoid error propagation
ā¢ Includes also expression evaluation
ā¢ Procedure parameters treated as variables
ā¢ Independent from underlying hardware, āØ
targets cache / main memory faults āØ
72
a=b;
.. becomes ...
a0=b0;
a1=b1;
if (b0 != b1)
error();
...
a=b+c;
... becomes ...
a0=b0+c0;
a1=b1+c1;
if ((b0!=b1)||(c0!=c1))
error();
73. Dependable Systems Course PT 2014
Control Flow Error
ā¢ Control ļ¬ow error (CFE) leads to unexpected instruction
execution
ā¢ Terminology:
ā¢ Basic block (BB) - branch-free serial code fragment
ā¢ Branch / jump instruction is modeled as last instruction
of a basic block
ā¢ Control ļ¬ow graph (CFG) - one node per basic block
ā¢ Illegal branch - Node transition is not part of the CFG
ā¢ Wrong branch - Node transition is already part of the CFG
ā¢ Inter-block error - Erroneous branch to diļ¬erent block
ā¢ Intra-block errors - Erroneous branch inside a block
73
i=0;
while(i<n) {
------------------
if (a[i] < b[i])
------------------
x[i]=a[i];
------------------
else
x[i]=b[i];
------------------
i++;
}
------------------
BB 0
BB 1
BB 2 BB 3
B 4
BB 5
74. Dependable Systems Course PT 2014
Multi-Version Approaches
Recovery Blocks
ā¢ Redundant system
implementations are
typically used
simultaneously, āØ
best answer is picked i.e.
by voting
ā¢ Alternative way:
Sequential execution of
recovery blocks
ā¢ Introduced in 1974 by
Horning et. al.
ā¢ Dynamic fault tolerance
approach, related to
stand-by sparing in
hardware
74
establish Checkpoint
Primary Module
Acceptance Test, else
load Checkpoint
Alternative Module 1
Acceptance Test, else
load Checkpoint
Alternative Module 2
...
else Failure Exception
75. Dependable Systems Course PT 2014
Multi-Version Approaches -
N-Version Programming
ā¢ Common mode errors are only catchable by design diversity
ā¢ Design diversity is a complex issue
ā¢ Design philosophies, software tools, programming languages, test philosophies
ā¢ Typical approaches try to utilize randomness - separate teams on diļ¬erent locations
ā¢ N-Version Programming
ā¢ Suggested by Elmendorf in 1972, developed by Avizienis & Chen in 1977
ā¢ Static approach, combination of decision mechanism and forward recovery
ā¢ At least two independently designed and functionally equivalent variants
ā¢ Variants are executed in parallel, decision mechanism selects the ābestā result
ā¢ Can support reliability, but also system security against malicious logic
75
76. Dependable Systems Course PT 2014
Simplex
ā¢ Idea: Using simplicity to control complexity
ā¢ Forward recovery approach, based on feedback loop
ā¢ HAC subsystem - Simple construction, formal methods, reliable hardware
ā¢ HPC subsystem - Complex technology, advanced features
ā¢ HPC can use HAC output, but not vice versa
ā¢ Decision logic based on control loop output
ā¢ Typically performance degredation with HAC
ā¢ Example: Boeing 777 primary and secondary āØ
ļ¬ight controller
76