Peter Tröger, profile picture
Uploaded byPeter Tröger
PDF, PPTX1,406 views

Dependable Systems -Fault Tolerance Patterns (4/16)

The document discusses various patterns for achieving fault tolerance in dependable systems. It covers architectural patterns like units of mitigation and error containment barriers. It also discusses detection patterns such as fault correlation, system monitoring, acknowledgments, voting, and audits. Finally, it discusses error recovery patterns like quarantine, concentrated recovery, and checkpointing to avoid data loss during recovery. The patterns provide reusable solutions for commonly occurring problems in building fault tolerant systems.

Embed presentation

Download as PDF, PPTX
Dependable Systems ! Fault Tolerance Patterns Dr. Peter Tröger ! Source: ! Hanmer, Robert S.: Patterns for Fault Tolerant Software. 
 Wiley, 2007. !
Error Processing Dependable Systems Course PT 2014 Phases of Fault Tolerance (Hanmer) 2 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
Dependable Systems Course PT 2014 Design Pattern • Definition from software engineering:
 „A general reusable solution to a commonly occurring problem“ • No finished / directly applicable solution, but a template • On the level of components and interactions • Popular approach in computer science (Gang of Four, Portland Pattern Repository) • Fault tolerance patterns • ... might be suited for stateless / stateful / both kinds of system • ... are based on observers and monitors (humans / computers) • ... work orthogonal to primary function • Note: ,Hamner‘-Book is about software fault tolerance, 
 but the patterns are generic (enough) 3
Dependable Systems Course PT 2014 Fault Tolerance Patterns • Architectural patterns • Considerations that cut across all parts of the system • Need to be applied in early design phase • Detection patterns • Detect the presence of root faults, error states, and failures • Errors vs. failures -> a-priori knowledge vs. comparison of redundant elements • Error Recovery Patterns • Methods to continue execution in a new error-free state • Undoing the error effects + creating the new state 4
Dependable Systems Course PT 2014 Fault Tolerance Patterns • Error Mitigation Patterns • Do not change application or system state, 
 but mask the error and compensate for the effects • Typical strategies for timing or performance faults • Fault Treatment Patterns • Prevent the error from reoccurring by repairing the fault • System verification • Diagnosis of fault location and nature • Correction of the system and / or the procedures 5
Architectural Patterns
Dependable Systems Course PT 2014 Architectural Patterns - Units of Mitigation • Only parts of the system should potentially get into error state • Design units of mitigation that contain errors and their error recovery mechanism • Component size vs. bookkeeping overhead vs. fault tolerance options • Should contain independent atomic actions without communication focus • Hints for granularity • Architectural style (n-tier) • Functional and resource (memory, CPU) boundaries • Choice of recovery action (e.g. restart) • Units without any recovery / mitigation are too small • Should perform self checks and fail silently, act as barrier to an error state 7
Dependable Systems Course PT 2014 Architectural Patterns - Error Containment Barrier • Errors spread through several mechanisms - messages, memory, follow-up actions • Unit of mitigation boundary implemented by error containment barrier • Treated as separate system component • Barrier must encapsulate error state, should trigger recovery / mitigation • In best case, perform detection close to the fault 
 (structural proximity / time) • Example: Babbling idiot problem • Suspicious nodes should never be in control of the 
 communication bus • Bus guardian as barrier implementation 8 [fromWikipedia]
Dependable Systems Course PT 2014 Guardian Example: Temporal Firewall in the Time-Triggered Architecture (TTA) 9 (C) Kopetz et al., TU Wien
Dependable Systems Course PT 2014 Architectural Patterns - Correcting Audits • Data element corruption can occur on hardware level (external physical faults) and software level (data types, currencies, pointers, ...) • Auditing demands correctness criteria • Structural properties of the data structure (linked lists, pointer boundaries, ...) • Known correlations (multiple locations, known conversion factors, cross linkage) • Sanity checks (value boundaries, checksums) • Direct comparison (duplication, mostly of static data) • Automatic correction is often easy, but must consider item consistency • Actions: Correction, logging, resume execution • Errors from faulty data easily propagate, common audit infrastructure helps 10
Dependable Systems Course PT 2014 Architectural Patterns - Redundancy • Improving availability by reducing MTTR is the easiest way • Error recovery phase makes the effect undone, but must be short • Idea: Resume execution before bad effects are undone, by using identical copy • Accomplish the same work on different hardware / software • Does not mean identical functionality, just perform the same work • Quick activation of redundant feature needed • Redundancy types: spatial, temporal, informational (presentation, version) • Special issues with software redundancy regarding deterministic behavior • Redundancy for performance improvement, availability then by excess capacity • Example: Checkpointing vs. fail-over 11
Dependable Systems Course PT 2014 Example: VAX Spatial Hardware Redundancy 12
Dependable Systems Course PT 2014 Example: Persistency in Redundant Systems 13 CPUCPU Memory Network CPUCPU Memory CPUCPU Memory Network CPUCPU Memory SAN Shared NothingShared Disk
Dependable Systems Course PT 2014 Spatial Redundancy with Replication • Replication: Process of ensuring consistency between redundant resources • Mostly applied for data replication • Active (synchronous) replication performs the same activity on every replica • First introduced by Leslie Lamport as state machine replication • Demands a deterministic processing of activities • Passive replication performs activity on one replica, and transmits the delta • Primary server vs. backup servers • Delayed response in failover case • Works also for non-deterministic processes • Example: Master-Slave vs. Master-Master replication setup 14
Dependable Systems Course PT 2014 Example: PostgreSQL 9 Redundancy Options • Shared-Disk setup • Avoids synchronization overhead, but demands network storage resp. file system • Mutual access exclusion from active / passive node must be ensured • Shared-Nothing setup • Block-device replication - Operating system can mirror file system modifications (e.g. GFS, DRBD) • Point-In-Time Recovery (PITR) - Passive nodes receive stream of write-ahead log (WAL) records, after each transaction commit • Master-Slave / Multimaster Replication - Batch updates on table granularity • Statement-Based Replication Middleware - SQL is sent to all nodes 15
Dependable Systems Course PT 2014 Example: PostgreSQL 9 Redundancy Options 16
Dependable Systems Course PT 2014 Architectural Patterns - Humans • Minimize Human Interaction • Error state root causes: Hardware, Software, Procedural / Operational • Humans are bad in: Long series of steps, routine tasks, operation, response time • Reduce failure risk due to mistakes in error treatment 
 -> process errors automatically • Operational staff should be able to monitor, but not be required for the solution • Maximize Human Participation • System should support experts in contributing to an error solution • Humans are good in drawing meaning from sequence of unrelated events • Examples: Reporting prioritization, context information (timestamp etc.) • Safe mode: Wait for human participation 17
Dependable Systems Course PT 2014 Architectural Patterns - Maintenance Interface • Making maintenance task visible to the outside world • Additional form of input • Separated interfaces and handling needed • Overload defense should not affect operator • Intermixed interfaces might bring security problems • Not a hidden trap door, but well well-protected dedicated path into the system • Prevent application workload from using it • Special functionality, e.g. log information fetching 18
Dependable Systems Course PT 2014 Architectural Patterns - Someone in Charge • Anything can go wrong, even during error processing • If something does not work, some entity must be able to restart processing action • For any fault tolerance activity, there must be a clearly identifiable responsible • Example: Active / Passive standby • Single component in charge means single failure point, also increases complexity • Examples: Initialization module, cluster management node • Multiple fault tolerance activities may be needed at the same time • Component must monitor progress and might initiate alternative actions • Dual masters problem (also with voting) 19
Dependable Systems Course PT 2014 Architectural Patterns - Escalation • Endless recovery attempts might be valid in some cases (transient faults) • But error processing becomes stalled when: • Correcting audits remain unsuccessful • Rollback / roll-forward remain unsuccessful • Still human intervention should be minimized • Escalation of the processing makes the error less local and more drastic • Demands understanding of faults and failure modes • Some options: Resume partial operation, perform partial service degradation 20 [https://www.flickr.com/photos/mwichary/3894637275/]
Dependable Systems Course PT 2014 Architectural Patterns - Examples for Pattern Relation 21 Units of MitigationRedundancy Recovery Blocks Maintenance Interface Correcting Audits Escalation Someone In Charge Software Update Fault Observer Minimize Human Interaction Maximize Human Participation Guides Example Supported By Makes use of Can serve as Makes use ofProgress Reports If not working Makes use of Example Triggers If not working Example Benefits from Benefits from Intervention Accessible Through Reports to
Detection Patterns
Error Processing Dependable Systems Course PT 2014 Phases of Fault Tolerance (Hanmer) 23 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
Dependable Systems Course PT 2014 Detection Patterns - Fault Correlation • Prerequisite: Early fault removal uncovered common error types • Look at unique signature of an error to identify an according fault category • Enables the activation of a well-known matching error processing • Examples: • Many off-by-one errors found in testing, prepare system for this • On data errors, related data to be checked should be known beforehand • Multiple errors can happen close in time - useful to triangulate the fault location • fault - error - error chain • In best case, take care of the initial fault that started the error chain 24
Dependable Systems Course PT 2014 Detection Patterns - System Monitor / Heartbeat • System Monitor • How can one part keep track that another part is functioning ? • Monitor for system (or system parts) behavior • Might be part of fault observer or someone in charge, or separate element • Location of the monitor is highly application-dependent • Heartbeat • How does system monitor knows that a task is still working ? • Send health reports at regular intervals (cost / benefit tradeoff) • Ping-alike messages, heartbeat function, push / pull approach 25
Dependable Systems Course PT 2014 Detection Patterns - Acknowledgment / Watchdog • Acknowledgment • Typical part of protocol definitions • Alternative for heartbeat, does not demand additional messaging • Piggybacking - Add acknowledgment information to response data frame • Prominent approach in bidirectional networking protocols • Watchdog • Watch visible effects of the monitored task, without adding complexity to it • Ensure that a task is alive, without messaging / processing overhead • Strategies: Timers, peepholes, hardware test points 26
Dependable Systems Course PT 2014 Detection Patterns - Realistic Threshold • How much time should elapse before the system monitor takes action ? • Message latency (e.g. heartbeat interval) vs. 
 detection latency (e.g. number of missed heartbeat messages) • Balance between short intervals (hypersensitive monitoring) and long intervals 
 (possibility for silent failures) • Influenced by communication round trip time and severity of undetected errors • Message latency is typically worst case communication time + processing time • Maximum unavailability > message latency + detection latency + repair time • System can automatically adjust thresholds based on experience • Example: Voyager spacecraft • One heartbeat to command computer every 2s, failure when one is skipped 27
Dependable Systems Course PT 2014 Detection Patterns - Realistic Threshold - Example • Heartbeat message, latency of one message • Roundtrip time (50ms - 100ms) • Preparation on monitor task (20ms) • Incoming message processing on either task (15ms) • Scenarios • Messaging latency = 50ms : All true failures reported, but many false errors • Messaging latency = 100ms : All true failures reported, but long reporting delay 28 Monitoring Task Monitored Task t
Dependable Systems Course PT 2014 Detection Patterns - Voting • Redundancy in space provides multiple answers - devise a voting strategy • Exact voting: Decision leads to correct result or uncertainty state notification • Inexact voting: Comparison might lead to multiple correct results • Non-adaptive voting: Use allowable result discrepancy, put boundary on discrepancy minimum or maximum • Adaptive voting: Rank results based on past experience • Predict what the correct value should be and take the closest result • Example: Weighted sum of the different results
 R=W1*R1 + W2*R2 + W3*R3 with W1+W2+W3=1 • Different optimizations for large answers (e.g. compare only checksum) • Communication latency shall not influence voter operation 29
Dependable Systems Course PT 2014 Detection Patterns - Voting • Selection in case of multiple events: • Majority vote 
 (uneven node number) • Generalized median voting - select result that is the median, 
 by iteratively removing extremes • Formalized plurality voting - divide results in partitions, choose random member from the largest partition • Weighted average technique • Components that disagree 
 (to some extend) with the vote are marked as erroneous 30 Triple Modular Redundancy (TMR) Cascaded TMR
Dependable Systems Course PT 2014 Detection Patterns - Maintenance and Exercises • Routine Maintenance • Through operator on the maintenance interface, or built in • Typical strategy in operating systems for idle processors • Relies on concept of checkable resources - connections, memory allocations, ... • Routine Exercises • Make sure that redundant spare components truly work in the failover case • Identify latent faults by checks during light workload - typical in hardware • Reproducible error is still better than the failure case on high workload 31
Dependable Systems Course PT 2014 Detection Patterns - Routine Audits / Checksums • Routine Audits (,scrubbing‘) • Find data errors in a controlled way, usually by low priority maintenance task • Logging is important for causal analysis - high possibility of related data errors • Identifies latent faults • Checksums • Detect incorrect data by storing aggregate information along with the value • Example: Space shuttle counts number of integers in a data structure • Many options - parity bits, hashing • Checksums are only for detection, recovery through error correcting codes 32
Dependable Systems Course PT 2014 Detection Patterns - Leaky Bucket Counter • Distinguish between transient and intermittent repeating faults • Assign a leaky bucket counter 
 to each unit of mitigation • Increment for each event / fault • Decrement periodically until initial value -> 
 fault events are periodically leaked • Exceeding the pre-defined upper limit of the bucket identifies a permanent fault • Examples • Faulty messages filling a buffer • Correctable memory errors 33
Error Recovery Patterns
Error Processing Dependable Systems Course PT 2014 Phases of Fault Tolerance (Hanmer) 35 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
Dependable Systems Course PT 2014 Error Recovery Patterns - Quarantine / Concentrated Recovery • Quarantine • Activate the prevention of error spreading and work contribution • Relies on units of mitigation in the architecture • Activate barrier around the component • Example: State indicator from voting unit • Concentrated Recovery • Minimize unavailability by focusing all resources on recovery activity • Inform fault observer about recovery activity, stay inside unit of mitigation • Establish quarantine around recovery activity • Well established in systems with high survivability demands (e.g. telco industry) 36
Dependable Systems Course PT 2014 Error Recovery Patterns - Checkpoint • Avoid loss of results during recovery by saving global state information • Focus on long duration data that is hard to achieve • Checkpoint data consistency and checkpointing interval are relevant • The „snapshot“ problem - how to achieve global (distributed) consistency ? • Global state == local states + messages • Snapshot algorithms: Determine past, consistent, global state • Chandy & Lamport (1985) landmark paper • Relies on flushing principle of FIFO communication channels • Control messages ,push out‘ pending messages 37
Dependable Systems Course PT 2014 Error Recovery Patterns - Individuals Decide Timing / Data Reset • Individuals Decide Timing • Independent checkpoints: Opposite approach to global checkpoints • Each process takes a dynamic local snapshot when it needs to • Consistency establishment overhead at recovery time vs. 
 global checkpoint overhead during operation • Data Reset • Recover from an uncorrectable data error by taking / computing initial values and approximate value • Relationship to return to reference point pattern - data reset is often a correlated activity 38
Dependable Systems Course PT 2014 Error Recovery Patterns - Rollback / Roll-Forward • How to resume processing after error recovery / error handler execution ? • Rollback • Timing of the checkpoint / last requests decides about the rollback point • Consider side effects of repeated work • Errors might re-occur, so limit retries • Roll-Forward • Resynchronization of systems tasks might be faster • Especially useful for event-driven stateless services • Demands proper damage mitigation and containment 39
Dependable Systems Course PT 2014 Error Recovery Patterns - Remote Storage • Storage location for checkpoints is relevant in failover / rollback case • Should not be the single point of failure • Pattern is good decision point for level of redundancy needed • Real-world application: iSCSI Multi-Homing 40 iSCSI SANs Server Server Switch Switch Server Server iSCSI SANs
Dependable Systems Course PT 2014 Error Recovery Patterns - Restart / Limit Retries • Restart • Way to resume execution when recovery / escalation is not possible • cold / warm restart - skip some of the initial checks, hardware vs. software restart • Supported by checkpoints • Limit Retries • Scenario: Faults are deterministic (latent fault -> same stimuli -> activation) • Rollback might not solve the problem when the error activation reason remains • Example: ,Killer messages‘ marked as being unprocessed, faulty checkpoints • Problem: Propagation of error within itself, must be stopped by limiting retries • Solution: Safeguarding and roll-forward 41
• Restoring of error-free operation in active element did not succeed • Switch to redundant resource, based on replication • Important factors are failover time and common data access • Establish someone in charge for steering • Needs proper quarantine for the faulty system part Dependable Systems Course PT 2014 Error Recovery Patterns - Failover 42 A A' A A' Active / Active Active / Passive N+1 (,roaming spare‘)N-to-1 (,dedicated spare‘) N + M A / Spare B / Spare Spare A / Spare B / Spare Spare 1 ... Spare M A B Spare
Dependable Systems Course PT 2014 Redundancy Configurations for Failover • N-to-1 and N+1 are special cases of Active / Passive with multiple services • Active / Active has no downtime, but leads to degraded system performance in failover case and might demand specialized data redundancy • N-to-1 demands a fail-back step, which is not needed with N+1 • Hot standby: No ramp-up needed on failover, no service failure for the user • Natural property of Active / Active setups • Possible even with Active / Passive setting through continuous replication, stateless services or static data • Warm standby / log shipping: Synchronize data block-wise on spare • Failover is typically used as synonym for Active / Passive • Orthogonal: Shared Nothing vs. Shared Disk data management 43
Dependable Systems Course PT 2014 Failover - Dual Master Problem • Current active element might not relinquish control - dual master problem • Typical problem in high-availability clusters • Split brain - Cluster interconnect is broken, several sub-cluster partitions start up • Establish resource fencing to let only one sub-group of the cluster work • Amnesia - Cluster restart with outdated configuration information • Quorum - „The number (as a majority) of officers or members of a body that when duly assembled is legally competent to transact business“ [Merriam-Webster] • ,Transact business‘ in the sense of ,provide service‘ - only one side should operate • Quorum allows fencing the other sub-cluster without communication • Loss of quorum should lead to node suicide, if possible 44
Dependable Systems Course PT 2014 Failover - Quorum Approaches • Central arbitration - Manual quorum, centralized server / admin sets master • Simple majority - More than the half of the nodes must form a group • Weighted majority - Votes for each node, group with higher vote count wins • Group decision is based on static data (nr. of votes, majority needed) • Tie-breaker - Lightweight resolving strategy before decision inside the sub-group • Example: Ping response from common upstream router • Whenever node connectivity changes, quorum decision should happen again • Split brain has different faces • Example in DRBD file system: Multiple replication masters by human error, 
 or temporary connectivity lost, leads to data merging issues 45
Dependable Systems Course PT 2014 Failover - Weighted Majority with Quorum Device • With even node number, provide additional external vote through quorum device • Number of votes by the quorum devices should be less than node votes • Allows cluster to operate with failed quorum device • Connection scheme of the quorum device decides upon valid cases of partitioning • Quorum device is typically a shared disk • Only used when communication with other nodes fails • Implemented by SCSI RESERVE,
 Fibre Channel, or iSCSI 46 (C) Sun
Dependable Systems Course PT 2014 Example - Windows File Server 47 (C) Microsoft • Quorum case: Heartbeat line broken, Node 1 itself still alive • Demands utilization of cluster storage as quorum device
Dependable Systems Course PT 2014 SCSI Quorum Device • Only one SCSI device can use the bus at a time - arbitration process • LUN acts as priority, so host bust adapters typically have the highest one • SCSI commands RESERVE and RELEASE allow to lock one SCSI device for exclusive usage by another device • Automated release on device / bus reset • Periodical renewing of reservation by driver, or persistent reservation feature • Example MS Windows Cluster Server • Master node acts as defender, renews reservation every 3 seconds • One node communication loss, challenger nodes resets the bus, waits for 7 seconds, and tries to get the reservation again 48
Dependable Systems Course PT 2014 Example - Quorum in Clusters 49 (C) Sun
Dependable Systems Course PT 2014 Example - Windows Server 2008 Failover Cluster • Voting elements: Nodes, disk witness, file share witness (= tie-breaker) • Quorum modes • Node majority, works with odd node number • Node and disk majority, for even node number with shared storage • Node and file share majority, for even node number in multi-site cluster • No majority: disk only, disk-based quorum as in Windows Server 2003 • File share / disk contains information about most recent cluster configuration (amnesia prevention) • Disk mode: Hardware must offer persistent arbitration (e.g. SCSI reserve and release) • File share mode: Active node keeps open file lock on the share (SMB feature) 50
Dependable Systems Course PT 2014 Example - Windows Server 2008 Failover Cluster • Permanent point-to-point heartbeat surveillance on each node • Process of achieving quorum • As the node comes up, determine if other cluster members can be contacted • Members compare their membership view on the cluster and agree on one
 (group communication) • Member collection determines if it has quorum • Without enough votes, it is waited for more members to appear • With quorum attended, resources and applications are brought into service 51
Dependable Systems Course PT 2014 Example - Windows Multi-Site Clustered File Server 52 NormalConditions MainSiteGone MainSiteBack CommunicationLost
 SplitBrain (C) Microsoft
Dependable Systems Course PT 2014 Example - Exchange 2007 Clustering • Hub Transport Server allows messaging about the witness file share • Witness share is checked ... • ... when a cluster node comes up and only one cluster node is available • ... when a previously reachable node is gone • ... when a node leaves the cluster (release lock) • ... periodically for validation purposes 53
Dependable Systems Course PT 2014 Example - VMWare HA Split Brain Situation • Even number of hosts run virtual machine images, stored on iSCSI / NFS • Virtual machine image is protected by file lock with timeout • Single host running a VM looses overall network connectivity • Other hosts restart the VM (due to lost external reachability) • Need to prevent the case that the VM will continue to run on primary host • Primary host gets connectivity back • Takes back the virtual machine image file 
 since it has the according processes 54
Error Mitigation (= mostly Overload Handling) Patterns
Error Processing Dependable Systems Course PT 2014 Phases of Fault Tolerance (Hanmer) 56 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
Dependable Systems Course PT 2014 Error Mitigation Patterns - Marked Data • Data error detected, but no recovery option available, error mitigation is acceptable • Data should be quarantined - do not use it, do not derive actions from it • Example: IEEE ,Not a Number‘ (NaN) • Result of division by zero, square root of -1, ... • IEEE 754-1985: Standard representation for binary floating point numbers • Rules for computation when operand is NaN - typically result is again NaN • Options: Assume default value, skip operation, mark result as erroneous 57
Dependable Systems Course PT 2014 Error Mitigation Patterns - Overload Toolboxes • Handle overload situation with too many requests for the system • Each resource class needs dedicated overload treatment • Memory: Exhaustion hinders new request from entering the system • CPU: Overload slows overall processing down • Patterns: Fresh work before stale, share the load, shed load • Tangible resources: Processing demands exclusive system resources • Network ports, shared storage, devices, ... • Patterns: Queue for resources, equitable resource allocation • Consider user demands • Patterns: Fresh work before stale, finish work in progress 58
Dependable Systems Course PT 2014 Error Mitigation Patterns - Shed Load • Throw away a minority of requests to serve the majority • As early as possible, to minimize resource consumption • Rejection method to be considered, e.g. do not send acknowledgements • Example: ICMP • Type 3: Destination Unreachable - no time-out on client host • Type 4: Source Quench - typically only between routers, also used by mail servers • Type 11: Time Exceeded - through congestion (or circular packets) • Example: HTPP 5XX error codes • 503: Service Unavailable • Specialized case: Shed work at periphery 59
Dependable Systems Course PT 2014 Error Mitigation Patterns - Finish Work in Progress / Fresh Work Before Stale • Finish Work in Progress • What to process, what to reject ? • Best case is labeling of requests: „new“ vs. „continuation“ • Distinguish ,continuation‘ processes based on their resource consumption • Try aggressively to get rid of resource hogs • Can lead to oscillation when system is „starving“ for new requests after cleanup • Solution: Let small portion of new requests through • Fresh Work Before Stale • If requester gives up, his retry eats up even more resources • Perform LIFO queue handling or non-queueing for premium requesters 60
Dependable Systems Course PT 2014 Error Mitigation Patterns - Slow It Down • Handle overload cases and avoid saturation by multi-step escalation • Restrict request processing with increasing severity per level • Goal: Slow things down until the system can catch up with the load • Feedback system, demands dedicated resources for the controller part • Add hysteresis effect to prevent oscillation for level changes • System output behavior depends on past behavior, not only on input • Implement by different trigger values to 
 enter / leave an escalation level • If the system stays in-between, the state 
 remains the same • Example: Thermostat 61 Thresholds OverloadState
Dependable Systems Course PT 2014 Error Mitigation Patterns - Deferrable Work / Equitable Resource Allocation • Deferrable Work • High load: Shed incoming work vs. shed routine maintenance workload • Make routine work (only relevant in error case) deferrable • Equitable Resource Allocation • Scenario: Handling of many requests for a set of resources, some of them are rare • Request-level handling would render some resources unnecessarily idle • Solution: Pool similar requests, allocate resources to pools • Additional bookkeeping needed for managing the requests and their related resource demands • Might lead to priority-inversion scenario 62
Dependable Systems Course PT 2014 Error Mitigation Patterns - Expansive / Protective Automatic Controls • Expansive Automatic Controls • Design some system parts for only being used in case of overload • Example: No 100% CPU utilization in normal operation of HA clusters • Example: Dynamic Offloaded Work - Cloud Computing • Increases request processing overhead, so take only as temporary solution • Protective Automatic Controls • Overload options: Shed internal work, shed incoming load, do nothing • Put restrictions on how much work the system accepts while still functioning • System throughput can drop due to contention, but should not drop to zero 63
Fault Treatment Patterns
Error Processing Dependable Systems Course PT 2014 Phases of Fault Tolerance (Hanmer) 65 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
Dependable Systems Course PT 2014 Fault Treatment Patterns - Let Sleeping Dogs Lie / Reintegration • Let Sleeping Dogs Lie • Treating faults by system change can introduce new faults • Known latent fault: Risk of reoccurrence, damage assessment possible • Potential new fault: Additional risk of miss-applied correction, no damage assessment possible for accidentally added faults • Reintegration • Different steps needed to reintegrate repaired component • Take off riding over transient and isolation lists • Watch new component for a while: hardening / soaking / trailing • Follow deterministic procedure, use as standby if possible 66
Dependable Systems Course PT 2014 Fault Treatment Patterns - Reproducible Error / Small Patches / Revise Procedure • Reproducible Error • Apply stimuli again under quarantine in order to prove fix • Can be automated (regression test) • Compare system output with golden unit output • Small Patches • Design system update as small as possible • Revise Procedure • When predetermined procedures contributed to failure duration, fix them 67

More Related Content

PPTX
CSDM - SLIDE JOSH.pptx
byJoshCn2
 
DOCX
Interview questions for manual testing technology.
byVinay Agnihotri
 
PDF
Dependable Systems -Dependability Threats (2/16)
byPeter Tröger
 
PDF
Dependable Systems -Reliability Prediction (9/16)
byPeter Tröger
 
PPT
Ch24
byphanleson
 
PDF
Dependable Systems -Software Dependability (15/16)
byPeter Tröger
 
PPT
Reliability integration across the product life cycle
byAccendo Reliability
 
PPT
Cofidential data storage and deletion 2003 (2)
bymaheshh1947
 
CSDM - SLIDE JOSH.pptx
byJoshCn2
 
Interview questions for manual testing technology.
byVinay Agnihotri
 
Dependable Systems -Dependability Threats (2/16)
byPeter Tröger
 
Dependable Systems -Reliability Prediction (9/16)
byPeter Tröger
 
Ch24
byphanleson
 
Dependable Systems -Software Dependability (15/16)
byPeter Tröger
 
Reliability integration across the product life cycle
byAccendo Reliability
 
Cofidential data storage and deletion 2003 (2)
bymaheshh1947
 

Similar to Dependable Systems -Fault Tolerance Patterns (4/16)

PDF
Dependable Systems - Summary (16/16)
byPeter Tröger
 
PDF
Dependable Systems -Dependability Means (3/16)
byPeter Tröger
 
PDF
Dependable Systems - Introduction (1/16)
byPeter Tröger
 
PDF
Dependable Systems - Hardware Dependability with Redundancy (14/16)
byPeter Tröger
 
PDF
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
byPeter Tröger
 
PPT
Lecture07_FaultTolerance in parallel and distributing
bysameerkumar56473
 
PPT
Lecture07_FaultTolerance in parallel and distributed
bysameerkumar56473
 
PDF
Separation of concerns is a design concept [Dij82] that suggests that any com...
bypremsridev11
 
PPTX
INTRODUCTIONTONFAULTTOLERANTSYSTEMS.pptx
byyormeezhub
 
PPTX
Ch10 - Dependable Systems
byHarsh Verdhan Raj
 
PPTX
Ch10 dependable systems
bysoftware-engineering-book
 
PDF
Developing fault tolerance integrity protocol for distributed real time systems
byDr Amira Bibo
 
PPTX
RTS fault tolerance, Reliability evaluation
by4132lenin6497ram
 
PPTX
real time systems fault tolerance, Redundancy
by4132lenin6497ram
 
PPT
Chapter- Five fault powers poin lecture
byborchala1
 
PPTX
Fault Tolerance System
byEhsan Ilahi
 
PPTX
Fault tol final ppt.pptx
bySamanShaheen5
 
PDF
CSL Seminar presented by Cassiano Campes - 17-03-13
byCassiano Campes
 
PDF
We hear you like papers
byInes Sombra
 
PPTX
The resident season 3 is a bit of a triangle
byABDULRAHMANSANI3
 
Dependable Systems - Summary (16/16)
byPeter Tröger
 
Dependable Systems -Dependability Means (3/16)
byPeter Tröger
 
Dependable Systems - Introduction (1/16)
byPeter Tröger
 
Dependable Systems - Hardware Dependability with Redundancy (14/16)
byPeter Tröger
 
Dependable Systems - Structure-Based Dependabiilty Modeling (6/16)
byPeter Tröger
 
Lecture07_FaultTolerance in parallel and distributing
bysameerkumar56473
 
Lecture07_FaultTolerance in parallel and distributed
bysameerkumar56473
 
Separation of concerns is a design concept [Dij82] that suggests that any com...
bypremsridev11
 
INTRODUCTIONTONFAULTTOLERANTSYSTEMS.pptx
byyormeezhub
 
Ch10 - Dependable Systems
byHarsh Verdhan Raj
 
Ch10 dependable systems
bysoftware-engineering-book
 
Developing fault tolerance integrity protocol for distributed real time systems
byDr Amira Bibo
 
RTS fault tolerance, Reliability evaluation
by4132lenin6497ram
 
real time systems fault tolerance, Redundancy
by4132lenin6497ram
 
Chapter- Five fault powers poin lecture
byborchala1
 
Fault Tolerance System
byEhsan Ilahi
 
Fault tol final ppt.pptx
bySamanShaheen5
 
CSL Seminar presented by Cassiano Campes - 17-03-13
byCassiano Campes
 
We hear you like papers
byInes Sombra
 
The resident season 3 is a bit of a triangle
byABDULRAHMANSANI3
 

More from Peter Tröger

PDF
WannaCry - An OS course perspective
byPeter Tröger
 
PDF
Cloud Standards and Virtualization
byPeter Tröger
 
PDF
Distributed Resource Management Application API (DRMAA) Version 2
byPeter Tröger
 
PDF
OpenSubmit - How to grade 1200 code submissions
byPeter Tröger
 
PDF
Design of Software for Embedded Systems
byPeter Tröger
 
PDF
Humans should not write XML.
byPeter Tröger
 
PDF
What activates a bug? A refinement of the Laprie terminology model.
byPeter Tröger
 
PDF
Dependable Systems - System Dependability Evaluation (8/16)
byPeter Tröger
 
PDF
Dependable Systems - Hardware Dependability with Diagnosis (13/16)
byPeter Tröger
 
PDF
Dependable Systems -Dependability Attributes (5/16)
byPeter Tröger
 
PDF
Verteilte Software-Systeme im Kontext von Industrie 4.0
byPeter Tröger
 
PDF
Operating Systems 1 (12/12) - Summary
byPeter Tröger
 
PDF
Operating Systems 1 (8/12) - Concurrency
byPeter Tröger
 
PDF
Operating Systems 1 (9/12) - Memory Management Concepts
byPeter Tröger
 
PDF
Operating Systems 1 (7/12) - Threads
byPeter Tröger
 
PDF
Operating Systems 1 (1/12) - History
byPeter Tröger
 
PDF
Operating Systems 1 (2/12) - Hardware Basics
byPeter Tröger
 
PDF
Operating Systems 1 (11/12) - Input / Output
byPeter Tröger
 
PDF
Operating Systems 1 (3/12) - Architectures
byPeter Tröger
 
PDF
Operating Systems 1 (6/12) - Processes
byPeter Tröger
 
WannaCry - An OS course perspective
byPeter Tröger
 
Cloud Standards and Virtualization
byPeter Tröger
 
Distributed Resource Management Application API (DRMAA) Version 2
byPeter Tröger
 
OpenSubmit - How to grade 1200 code submissions
byPeter Tröger
 
Design of Software for Embedded Systems
byPeter Tröger
 
Humans should not write XML.
byPeter Tröger
 
What activates a bug? A refinement of the Laprie terminology model.
byPeter Tröger
 
Dependable Systems - System Dependability Evaluation (8/16)
byPeter Tröger
 
Dependable Systems - Hardware Dependability with Diagnosis (13/16)
byPeter Tröger
 
Dependable Systems -Dependability Attributes (5/16)
byPeter Tröger
 
Verteilte Software-Systeme im Kontext von Industrie 4.0
byPeter Tröger
 
Operating Systems 1 (12/12) - Summary
byPeter Tröger
 
Operating Systems 1 (8/12) - Concurrency
byPeter Tröger
 
Operating Systems 1 (9/12) - Memory Management Concepts
byPeter Tröger
 
Operating Systems 1 (7/12) - Threads
byPeter Tröger
 
Operating Systems 1 (1/12) - History
byPeter Tröger
 
Operating Systems 1 (2/12) - Hardware Basics
byPeter Tröger
 
Operating Systems 1 (11/12) - Input / Output
byPeter Tröger
 
Operating Systems 1 (3/12) - Architectures
byPeter Tröger
 
Operating Systems 1 (6/12) - Processes
byPeter Tröger
 

Recently uploaded

PPTX
AN EXTREMELY BORING GENERAL QUIZ FOR UG.pptx
bySriramG47
 
PPTX
THEORIES OF GROWTH AND DEVELOPMENT..pptx
byAneetaSharma15
 
PPTX
Expected Revenue Report In Odoo 18 CRM
byCeline George
 
PPTX
Overview of how to Create a Model in Odoo 18
byCeline George
 
PDF
The voice of the rain poem class 11th.pdf
byReeta Baral
 
PPTX
4 G8_Q3_L4 (Evaluating opinion editorials for textual evidence and quality).pptx
byNorafeSahagun
 
PDF
Toward Massive, Ultrareliable, and Low-Latency Wireless Communication With Sh...
byMan_Ebook
 
PPTX
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 
PDF
Blood Group Incompatibility: Rh Factor and Erythroblastosis Fetalis
bySudhandraKarthi
 
PDF
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
PPTX
3 G8_Q3_L3_ (Cartoon as Representation in Opinion Editorial Article).pptx
byNorafeSahagun
 
PPTX
Repeat Orders_ Use Odoo Blanket Agreement
byCeline George
 
PPTX
Access partner report in Odoo 18.2 _ Odoo 19
byCeline George
 
PPTX
Pain. definition, causes, factor influencing pain & pain assessment.pptx
byPompi Begum
 
PPTX
Details of Epithelial and Connective Tissue.pptx
byAshish Umale
 
PDF
Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System.pdf
byMan_Ebook
 
PPTX
Pig- piggy bank in Big Data Analytics.ppt.pptx
byVarshini M
 
PDF
Projecte de la porta de la classe de primer A: Mar i cel.
byeduardrubio1
 
PPTX
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
PDF
Cultivating Greatness Pune's Best Preschools and Schools.pdf
byWellington College
 
AN EXTREMELY BORING GENERAL QUIZ FOR UG.pptx
bySriramG47
 
THEORIES OF GROWTH AND DEVELOPMENT..pptx
byAneetaSharma15
 
Expected Revenue Report In Odoo 18 CRM
byCeline George
 
Overview of how to Create a Model in Odoo 18
byCeline George
 
The voice of the rain poem class 11th.pdf
byReeta Baral
 
4 G8_Q3_L4 (Evaluating opinion editorials for textual evidence and quality).pptx
byNorafeSahagun
 
Toward Massive, Ultrareliable, and Low-Latency Wireless Communication With Sh...
byMan_Ebook
 
The Cell & Cell Cycle-detailed structure and function of organelles.pptx
byAshish Umale
 
Blood Group Incompatibility: Rh Factor and Erythroblastosis Fetalis
bySudhandraKarthi
 
Unit-III pdf (Basic listening Skill, Effective Writing Communication & Writin...
bySayyadTarannum
 
3 G8_Q3_L3_ (Cartoon as Representation in Opinion Editorial Article).pptx
byNorafeSahagun
 
Repeat Orders_ Use Odoo Blanket Agreement
byCeline George
 
Access partner report in Odoo 18.2 _ Odoo 19
byCeline George
 
Pain. definition, causes, factor influencing pain & pain assessment.pptx
byPompi Begum
 
Details of Epithelial and Connective Tissue.pptx
byAshish Umale
 
Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System.pdf
byMan_Ebook
 
Pig- piggy bank in Big Data Analytics.ppt.pptx
byVarshini M
 
Projecte de la porta de la classe de primer A: Mar i cel.
byeduardrubio1
 
10-12-2025 Francois Staring How can Researchers and Initial Teacher Educators...
byEduSkills OECD
 
Cultivating Greatness Pune's Best Preschools and Schools.pdf
byWellington College
 
In this document
Powered by AI

Introduction to dependable systems and fault tolerance patterns, authored by Dr. Peter Tröger.

Illustration of fault tolerance phases: latent fault, normal operation, fault activation, error recovery, error detection, and fault treatment.

Definition and importance of design patterns in software engineering, emphasizing reusable solutions, and their application in fault tolerance.

Types of fault tolerance patterns including architectural patterns, detection patterns, error recovery patterns, etc.

Detailed overview of error mitigation and fault treatment patterns, including error recovery methods and system verification.

A general introduction to architectural patterns in fault tolerance.

Details on designing mitigation units in systems to contain errors, covering aspects like architectural style and recovery actions.

Implementation of error containment barriers to prevent error state propagation and techniques for recovery.

Example of a temporal firewall used in time-triggered architecture, illustrating the guardian principle.

Implementing audits to ensure data integrity and error correction; addressing structural properties and sanity checks.

Defining redundancy types and their importance in error recovery and system availability.

Examples of spatial hardware redundancy and persistence in redundant systems.

Overview of data replication techniques, including active and passive replication, emphasizing consistency.

Discusses various redundancy options for PostgreSQL, including shared-disk and shared-nothing configurations.

Addressing human interaction in fault tolerance processes, emphasizing areas to minimize or maximize human involvement.

Importance of maintaining clear interfaces to assist in system operations without compromising security.

Highlighting the importance of having a designated responsible entity for fault tolerance activities.

Discussing error escalation strategies to handle failures effectively and understanding failure modes.

Exploration of how different architectural patterns relate and support each other within fault tolerance.

Introduction to detection patterns necessary for effective fault tolerance.

Techniques to correlate faults with specific error types for early detection and mitigation.

Importance of system monitoring mechanisms and heartbeat checks for tracking component functionality.

Techniques for acknowledgment and using watchdog mechanisms to ensure system task status.

Setting realistic thresholds for detection mechanisms to balance sensitivity against latencies.

Voting methods employed for redundancy effectiveness to ensure accurate fault detection.

Routine maintenance and exercise checks to guarantee the reliability of system redundancy.

Utilization of audits and checksums to identify and manage data errors within systems.

Using leaky bucket counters to distinguish between different types of faults for effective management.

Introduction to various error recovery patterns necessary for maintaining system operation.

Defining quarantine strategies to prevent error propagation and concentrating resources for recovery.

Importance of checkpoints in preventing data loss during recovery and ensuring data consistency.

Flexible approaches to timing when deciding for data resets and establishing local snapshots.

Methods to resume processing after errors, including rollback and roll-forward techniques.

Importance of utilizing remote storage in error recovery processes to avoid single point failures.

Strategies for restarting operations when failures occur and the importance of limiting retries.

Describing failover strategies that utilize replication to maintain system performance post-failure.

Different configurations for implementing redundancy during failover situations.

Discussing the dual master problem in failover situations and approaches for resolving it.

Fine-tuning quorum approaches to handle failover and ensure proper operational continuity.

Utilizing quorum devices during failover processes to manage resource access effectively.

Real-world example of quorum implementation in Windows File Server for failover management.

Detailed examples illustrating quorum management in Windows Server 2008 failover clusters.

Illustrating clustering strategies in Exchange Server 2007, focusing on witness file shared validation.

Discussion about managing split-brain scenarios in VMWare high availability setups.

Introduction to error mitigation patterns crucial for managing overload scenarios.

Strategies for managing situations when data errors occur without recovery options.

Techniques to manage system overload through dedicated treatment for each resource class.

Methods for shedding load to prioritize essential requests during overload situations.

Techniques to prioritize fresh work before stale work in overloaded conditions.

Managing system overload by gradually slowing down processing with NLP escalations.

Strategies for handling deferrable work and equitable resource allocation in overdrive.

Designing systems that automatically adapt controls according to workload conditions.

Introduction to fault treatment patterns that aid in managing errors effectively.

Cautionary approaches concerning system changes in the presence of known latent faults.

Strategies for testing and updating the fault management procedures for system reliability.

Dependable Systems -Fault Tolerance Patterns (4/16)

  • 1.
    Dependable Systems ! Fault TolerancePatterns Dr. Peter Tröger ! Source: ! Hanmer, Robert S.: Patterns for Fault Tolerant Software. 
 Wiley, 2007. !
  • 2.
    Error Processing Dependable SystemsCourse PT 2014 Phases of Fault Tolerance (Hanmer) 2 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
  • 3.
    Dependable Systems CoursePT 2014 Design Pattern • Definition from software engineering:
 „A general reusable solution to a commonly occurring problem“ • No finished / directly applicable solution, but a template • On the level of components and interactions • Popular approach in computer science (Gang of Four, Portland Pattern Repository) • Fault tolerance patterns • ... might be suited for stateless / stateful / both kinds of system • ... are based on observers and monitors (humans / computers) • ... work orthogonal to primary function • Note: ,Hamner‘-Book is about software fault tolerance, 
 but the patterns are generic (enough) 3
  • 4.
    Dependable Systems CoursePT 2014 Fault Tolerance Patterns • Architectural patterns • Considerations that cut across all parts of the system • Need to be applied in early design phase • Detection patterns • Detect the presence of root faults, error states, and failures • Errors vs. failures -> a-priori knowledge vs. comparison of redundant elements • Error Recovery Patterns • Methods to continue execution in a new error-free state • Undoing the error effects + creating the new state 4
  • 5.
    Dependable Systems CoursePT 2014 Fault Tolerance Patterns • Error Mitigation Patterns • Do not change application or system state, 
 but mask the error and compensate for the effects • Typical strategies for timing or performance faults • Fault Treatment Patterns • Prevent the error from reoccurring by repairing the fault • System verification • Diagnosis of fault location and nature • Correction of the system and / or the procedures 5
  • 6.
  • 7.
    Dependable Systems CoursePT 2014 Architectural Patterns - Units of Mitigation • Only parts of the system should potentially get into error state • Design units of mitigation that contain errors and their error recovery mechanism • Component size vs. bookkeeping overhead vs. fault tolerance options • Should contain independent atomic actions without communication focus • Hints for granularity • Architectural style (n-tier) • Functional and resource (memory, CPU) boundaries • Choice of recovery action (e.g. restart) • Units without any recovery / mitigation are too small • Should perform self checks and fail silently, act as barrier to an error state 7
  • 8.
    Dependable Systems CoursePT 2014 Architectural Patterns - Error Containment Barrier • Errors spread through several mechanisms - messages, memory, follow-up actions • Unit of mitigation boundary implemented by error containment barrier • Treated as separate system component • Barrier must encapsulate error state, should trigger recovery / mitigation • In best case, perform detection close to the fault 
 (structural proximity / time) • Example: Babbling idiot problem • Suspicious nodes should never be in control of the 
 communication bus • Bus guardian as barrier implementation 8 [fromWikipedia]
  • 9.
    Dependable Systems CoursePT 2014 Guardian Example: Temporal Firewall in the Time-Triggered Architecture (TTA) 9 (C) Kopetz et al., TU Wien
  • 10.
    Dependable Systems CoursePT 2014 Architectural Patterns - Correcting Audits • Data element corruption can occur on hardware level (external physical faults) and software level (data types, currencies, pointers, ...) • Auditing demands correctness criteria • Structural properties of the data structure (linked lists, pointer boundaries, ...) • Known correlations (multiple locations, known conversion factors, cross linkage) • Sanity checks (value boundaries, checksums) • Direct comparison (duplication, mostly of static data) • Automatic correction is often easy, but must consider item consistency • Actions: Correction, logging, resume execution • Errors from faulty data easily propagate, common audit infrastructure helps 10
  • 11.
    Dependable Systems CoursePT 2014 Architectural Patterns - Redundancy • Improving availability by reducing MTTR is the easiest way • Error recovery phase makes the effect undone, but must be short • Idea: Resume execution before bad effects are undone, by using identical copy • Accomplish the same work on different hardware / software • Does not mean identical functionality, just perform the same work • Quick activation of redundant feature needed • Redundancy types: spatial, temporal, informational (presentation, version) • Special issues with software redundancy regarding deterministic behavior • Redundancy for performance improvement, availability then by excess capacity • Example: Checkpointing vs. fail-over 11
  • 12.
    Dependable Systems CoursePT 2014 Example: VAX Spatial Hardware Redundancy 12
  • 13.
    Dependable Systems CoursePT 2014 Example: Persistency in Redundant Systems 13 CPUCPU Memory Network CPUCPU Memory CPUCPU Memory Network CPUCPU Memory SAN Shared NothingShared Disk
  • 14.
    Dependable Systems CoursePT 2014 Spatial Redundancy with Replication • Replication: Process of ensuring consistency between redundant resources • Mostly applied for data replication • Active (synchronous) replication performs the same activity on every replica • First introduced by Leslie Lamport as state machine replication • Demands a deterministic processing of activities • Passive replication performs activity on one replica, and transmits the delta • Primary server vs. backup servers • Delayed response in failover case • Works also for non-deterministic processes • Example: Master-Slave vs. Master-Master replication setup 14
  • 15.
    Dependable Systems CoursePT 2014 Example: PostgreSQL 9 Redundancy Options • Shared-Disk setup • Avoids synchronization overhead, but demands network storage resp. file system • Mutual access exclusion from active / passive node must be ensured • Shared-Nothing setup • Block-device replication - Operating system can mirror file system modifications (e.g. GFS, DRBD) • Point-In-Time Recovery (PITR) - Passive nodes receive stream of write-ahead log (WAL) records, after each transaction commit • Master-Slave / Multimaster Replication - Batch updates on table granularity • Statement-Based Replication Middleware - SQL is sent to all nodes 15
  • 16.
    Dependable Systems CoursePT 2014 Example: PostgreSQL 9 Redundancy Options 16
  • 17.
    Dependable Systems CoursePT 2014 Architectural Patterns - Humans • Minimize Human Interaction • Error state root causes: Hardware, Software, Procedural / Operational • Humans are bad in: Long series of steps, routine tasks, operation, response time • Reduce failure risk due to mistakes in error treatment 
 -> process errors automatically • Operational staff should be able to monitor, but not be required for the solution • Maximize Human Participation • System should support experts in contributing to an error solution • Humans are good in drawing meaning from sequence of unrelated events • Examples: Reporting prioritization, context information (timestamp etc.) • Safe mode: Wait for human participation 17
  • 18.
    Dependable Systems CoursePT 2014 Architectural Patterns - Maintenance Interface • Making maintenance task visible to the outside world • Additional form of input • Separated interfaces and handling needed • Overload defense should not affect operator • Intermixed interfaces might bring security problems • Not a hidden trap door, but well well-protected dedicated path into the system • Prevent application workload from using it • Special functionality, e.g. log information fetching 18
  • 19.
    Dependable Systems CoursePT 2014 Architectural Patterns - Someone in Charge • Anything can go wrong, even during error processing • If something does not work, some entity must be able to restart processing action • For any fault tolerance activity, there must be a clearly identifiable responsible • Example: Active / Passive standby • Single component in charge means single failure point, also increases complexity • Examples: Initialization module, cluster management node • Multiple fault tolerance activities may be needed at the same time • Component must monitor progress and might initiate alternative actions • Dual masters problem (also with voting) 19
  • 20.
    Dependable Systems CoursePT 2014 Architectural Patterns - Escalation • Endless recovery attempts might be valid in some cases (transient faults) • But error processing becomes stalled when: • Correcting audits remain unsuccessful • Rollback / roll-forward remain unsuccessful • Still human intervention should be minimized • Escalation of the processing makes the error less local and more drastic • Demands understanding of faults and failure modes • Some options: Resume partial operation, perform partial service degradation 20 [https://www.flickr.com/photos/mwichary/3894637275/]
  • 21.
    Dependable Systems CoursePT 2014 Architectural Patterns - Examples for Pattern Relation 21 Units of MitigationRedundancy Recovery Blocks Maintenance Interface Correcting Audits Escalation Someone In Charge Software Update Fault Observer Minimize Human Interaction Maximize Human Participation Guides Example Supported By Makes use of Can serve as Makes use ofProgress Reports If not working Makes use of Example Triggers If not working Example Benefits from Benefits from Intervention Accessible Through Reports to
  • 22.
  • 23.
    Error Processing Dependable SystemsCourse PT 2014 Phases of Fault Tolerance (Hanmer) 23 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
  • 24.
    Dependable Systems CoursePT 2014 Detection Patterns - Fault Correlation • Prerequisite: Early fault removal uncovered common error types • Look at unique signature of an error to identify an according fault category • Enables the activation of a well-known matching error processing • Examples: • Many off-by-one errors found in testing, prepare system for this • On data errors, related data to be checked should be known beforehand • Multiple errors can happen close in time - useful to triangulate the fault location • fault - error - error chain • In best case, take care of the initial fault that started the error chain 24
  • 25.
    Dependable Systems CoursePT 2014 Detection Patterns - System Monitor / Heartbeat • System Monitor • How can one part keep track that another part is functioning ? • Monitor for system (or system parts) behavior • Might be part of fault observer or someone in charge, or separate element • Location of the monitor is highly application-dependent • Heartbeat • How does system monitor knows that a task is still working ? • Send health reports at regular intervals (cost / benefit tradeoff) • Ping-alike messages, heartbeat function, push / pull approach 25
  • 26.
    Dependable Systems CoursePT 2014 Detection Patterns - Acknowledgment / Watchdog • Acknowledgment • Typical part of protocol definitions • Alternative for heartbeat, does not demand additional messaging • Piggybacking - Add acknowledgment information to response data frame • Prominent approach in bidirectional networking protocols • Watchdog • Watch visible effects of the monitored task, without adding complexity to it • Ensure that a task is alive, without messaging / processing overhead • Strategies: Timers, peepholes, hardware test points 26
  • 27.
    Dependable Systems CoursePT 2014 Detection Patterns - Realistic Threshold • How much time should elapse before the system monitor takes action ? • Message latency (e.g. heartbeat interval) vs. 
 detection latency (e.g. number of missed heartbeat messages) • Balance between short intervals (hypersensitive monitoring) and long intervals 
 (possibility for silent failures) • Influenced by communication round trip time and severity of undetected errors • Message latency is typically worst case communication time + processing time • Maximum unavailability > message latency + detection latency + repair time • System can automatically adjust thresholds based on experience • Example: Voyager spacecraft • One heartbeat to command computer every 2s, failure when one is skipped 27
  • 28.
    Dependable Systems CoursePT 2014 Detection Patterns - Realistic Threshold - Example • Heartbeat message, latency of one message • Roundtrip time (50ms - 100ms) • Preparation on monitor task (20ms) • Incoming message processing on either task (15ms) • Scenarios • Messaging latency = 50ms : All true failures reported, but many false errors • Messaging latency = 100ms : All true failures reported, but long reporting delay 28 Monitoring Task Monitored Task t
  • 29.
    Dependable Systems CoursePT 2014 Detection Patterns - Voting • Redundancy in space provides multiple answers - devise a voting strategy • Exact voting: Decision leads to correct result or uncertainty state notification • Inexact voting: Comparison might lead to multiple correct results • Non-adaptive voting: Use allowable result discrepancy, put boundary on discrepancy minimum or maximum • Adaptive voting: Rank results based on past experience • Predict what the correct value should be and take the closest result • Example: Weighted sum of the different results
 R=W1*R1 + W2*R2 + W3*R3 with W1+W2+W3=1 • Different optimizations for large answers (e.g. compare only checksum) • Communication latency shall not influence voter operation 29
  • 30.
    Dependable Systems CoursePT 2014 Detection Patterns - Voting • Selection in case of multiple events: • Majority vote 
 (uneven node number) • Generalized median voting - select result that is the median, 
 by iteratively removing extremes • Formalized plurality voting - divide results in partitions, choose random member from the largest partition • Weighted average technique • Components that disagree 
 (to some extend) with the vote are marked as erroneous 30 Triple Modular Redundancy (TMR) Cascaded TMR
  • 31.
    Dependable Systems CoursePT 2014 Detection Patterns - Maintenance and Exercises • Routine Maintenance • Through operator on the maintenance interface, or built in • Typical strategy in operating systems for idle processors • Relies on concept of checkable resources - connections, memory allocations, ... • Routine Exercises • Make sure that redundant spare components truly work in the failover case • Identify latent faults by checks during light workload - typical in hardware • Reproducible error is still better than the failure case on high workload 31
  • 32.
    Dependable Systems CoursePT 2014 Detection Patterns - Routine Audits / Checksums • Routine Audits (,scrubbing‘) • Find data errors in a controlled way, usually by low priority maintenance task • Logging is important for causal analysis - high possibility of related data errors • Identifies latent faults • Checksums • Detect incorrect data by storing aggregate information along with the value • Example: Space shuttle counts number of integers in a data structure • Many options - parity bits, hashing • Checksums are only for detection, recovery through error correcting codes 32
  • 33.
    Dependable Systems CoursePT 2014 Detection Patterns - Leaky Bucket Counter • Distinguish between transient and intermittent repeating faults • Assign a leaky bucket counter 
 to each unit of mitigation • Increment for each event / fault • Decrement periodically until initial value -> 
 fault events are periodically leaked • Exceeding the pre-defined upper limit of the bucket identifies a permanent fault • Examples • Faulty messages filling a buffer • Correctable memory errors 33
  • 34.
  • 35.
    Error Processing Dependable SystemsCourse PT 2014 Phases of Fault Tolerance (Hanmer) 35 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
  • 36.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Quarantine / Concentrated Recovery • Quarantine • Activate the prevention of error spreading and work contribution • Relies on units of mitigation in the architecture • Activate barrier around the component • Example: State indicator from voting unit • Concentrated Recovery • Minimize unavailability by focusing all resources on recovery activity • Inform fault observer about recovery activity, stay inside unit of mitigation • Establish quarantine around recovery activity • Well established in systems with high survivability demands (e.g. telco industry) 36
  • 37.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Checkpoint • Avoid loss of results during recovery by saving global state information • Focus on long duration data that is hard to achieve • Checkpoint data consistency and checkpointing interval are relevant • The „snapshot“ problem - how to achieve global (distributed) consistency ? • Global state == local states + messages • Snapshot algorithms: Determine past, consistent, global state • Chandy & Lamport (1985) landmark paper • Relies on flushing principle of FIFO communication channels • Control messages ,push out‘ pending messages 37
  • 38.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Individuals Decide Timing / Data Reset • Individuals Decide Timing • Independent checkpoints: Opposite approach to global checkpoints • Each process takes a dynamic local snapshot when it needs to • Consistency establishment overhead at recovery time vs. 
 global checkpoint overhead during operation • Data Reset • Recover from an uncorrectable data error by taking / computing initial values and approximate value • Relationship to return to reference point pattern - data reset is often a correlated activity 38
  • 39.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Rollback / Roll-Forward • How to resume processing after error recovery / error handler execution ? • Rollback • Timing of the checkpoint / last requests decides about the rollback point • Consider side effects of repeated work • Errors might re-occur, so limit retries • Roll-Forward • Resynchronization of systems tasks might be faster • Especially useful for event-driven stateless services • Demands proper damage mitigation and containment 39
  • 40.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Remote Storage • Storage location for checkpoints is relevant in failover / rollback case • Should not be the single point of failure • Pattern is good decision point for level of redundancy needed • Real-world application: iSCSI Multi-Homing 40 iSCSI SANs Server Server Switch Switch Server Server iSCSI SANs
  • 41.
    Dependable Systems CoursePT 2014 Error Recovery Patterns - Restart / Limit Retries • Restart • Way to resume execution when recovery / escalation is not possible • cold / warm restart - skip some of the initial checks, hardware vs. software restart • Supported by checkpoints • Limit Retries • Scenario: Faults are deterministic (latent fault -> same stimuli -> activation) • Rollback might not solve the problem when the error activation reason remains • Example: ,Killer messages‘ marked as being unprocessed, faulty checkpoints • Problem: Propagation of error within itself, must be stopped by limiting retries • Solution: Safeguarding and roll-forward 41
  • 42.
    • Restoring oferror-free operation in active element did not succeed • Switch to redundant resource, based on replication • Important factors are failover time and common data access • Establish someone in charge for steering • Needs proper quarantine for the faulty system part Dependable Systems Course PT 2014 Error Recovery Patterns - Failover 42 A A' A A' Active / Active Active / Passive N+1 (,roaming spare‘)N-to-1 (,dedicated spare‘) N + M A / Spare B / Spare Spare A / Spare B / Spare Spare 1 ... Spare M A B Spare
  • 43.
    Dependable Systems CoursePT 2014 Redundancy Configurations for Failover • N-to-1 and N+1 are special cases of Active / Passive with multiple services • Active / Active has no downtime, but leads to degraded system performance in failover case and might demand specialized data redundancy • N-to-1 demands a fail-back step, which is not needed with N+1 • Hot standby: No ramp-up needed on failover, no service failure for the user • Natural property of Active / Active setups • Possible even with Active / Passive setting through continuous replication, stateless services or static data • Warm standby / log shipping: Synchronize data block-wise on spare • Failover is typically used as synonym for Active / Passive • Orthogonal: Shared Nothing vs. Shared Disk data management 43
  • 44.
    Dependable Systems CoursePT 2014 Failover - Dual Master Problem • Current active element might not relinquish control - dual master problem • Typical problem in high-availability clusters • Split brain - Cluster interconnect is broken, several sub-cluster partitions start up • Establish resource fencing to let only one sub-group of the cluster work • Amnesia - Cluster restart with outdated configuration information • Quorum - „The number (as a majority) of officers or members of a body that when duly assembled is legally competent to transact business“ [Merriam-Webster] • ,Transact business‘ in the sense of ,provide service‘ - only one side should operate • Quorum allows fencing the other sub-cluster without communication • Loss of quorum should lead to node suicide, if possible 44
  • 45.
    Dependable Systems CoursePT 2014 Failover - Quorum Approaches • Central arbitration - Manual quorum, centralized server / admin sets master • Simple majority - More than the half of the nodes must form a group • Weighted majority - Votes for each node, group with higher vote count wins • Group decision is based on static data (nr. of votes, majority needed) • Tie-breaker - Lightweight resolving strategy before decision inside the sub-group • Example: Ping response from common upstream router • Whenever node connectivity changes, quorum decision should happen again • Split brain has different faces • Example in DRBD file system: Multiple replication masters by human error, 
 or temporary connectivity lost, leads to data merging issues 45
  • 46.
    Dependable Systems CoursePT 2014 Failover - Weighted Majority with Quorum Device • With even node number, provide additional external vote through quorum device • Number of votes by the quorum devices should be less than node votes • Allows cluster to operate with failed quorum device • Connection scheme of the quorum device decides upon valid cases of partitioning • Quorum device is typically a shared disk • Only used when communication with other nodes fails • Implemented by SCSI RESERVE,
 Fibre Channel, or iSCSI 46 (C) Sun
  • 47.
    Dependable Systems CoursePT 2014 Example - Windows File Server 47 (C) Microsoft • Quorum case: Heartbeat line broken, Node 1 itself still alive • Demands utilization of cluster storage as quorum device
  • 48.
    Dependable Systems CoursePT 2014 SCSI Quorum Device • Only one SCSI device can use the bus at a time - arbitration process • LUN acts as priority, so host bust adapters typically have the highest one • SCSI commands RESERVE and RELEASE allow to lock one SCSI device for exclusive usage by another device • Automated release on device / bus reset • Periodical renewing of reservation by driver, or persistent reservation feature • Example MS Windows Cluster Server • Master node acts as defender, renews reservation every 3 seconds • One node communication loss, challenger nodes resets the bus, waits for 7 seconds, and tries to get the reservation again 48
  • 49.
    Dependable Systems CoursePT 2014 Example - Quorum in Clusters 49 (C) Sun
  • 50.
    Dependable Systems CoursePT 2014 Example - Windows Server 2008 Failover Cluster • Voting elements: Nodes, disk witness, file share witness (= tie-breaker) • Quorum modes • Node majority, works with odd node number • Node and disk majority, for even node number with shared storage • Node and file share majority, for even node number in multi-site cluster • No majority: disk only, disk-based quorum as in Windows Server 2003 • File share / disk contains information about most recent cluster configuration (amnesia prevention) • Disk mode: Hardware must offer persistent arbitration (e.g. SCSI reserve and release) • File share mode: Active node keeps open file lock on the share (SMB feature) 50
  • 51.
    Dependable Systems CoursePT 2014 Example - Windows Server 2008 Failover Cluster • Permanent point-to-point heartbeat surveillance on each node • Process of achieving quorum • As the node comes up, determine if other cluster members can be contacted • Members compare their membership view on the cluster and agree on one
 (group communication) • Member collection determines if it has quorum • Without enough votes, it is waited for more members to appear • With quorum attended, resources and applications are brought into service 51
  • 52.
    Dependable Systems CoursePT 2014 Example - Windows Multi-Site Clustered File Server 52 NormalConditions MainSiteGone MainSiteBack CommunicationLost
 SplitBrain (C) Microsoft
  • 53.
    Dependable Systems CoursePT 2014 Example - Exchange 2007 Clustering • Hub Transport Server allows messaging about the witness file share • Witness share is checked ... • ... when a cluster node comes up and only one cluster node is available • ... when a previously reachable node is gone • ... when a node leaves the cluster (release lock) • ... periodically for validation purposes 53
  • 54.
    Dependable Systems CoursePT 2014 Example - VMWare HA Split Brain Situation • Even number of hosts run virtual machine images, stored on iSCSI / NFS • Virtual machine image is protected by file lock with timeout • Single host running a VM looses overall network connectivity • Other hosts restart the VM (due to lost external reachability) • Need to prevent the case that the VM will continue to run on primary host • Primary host gets connectivity back • Takes back the virtual machine image file 
 since it has the according processes 54
  • 55.
    Error Mitigation (=mostly Overload Handling) Patterns
  • 56.
    Error Processing Dependable SystemsCourse PT 2014 Phases of Fault Tolerance (Hanmer) 56 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
  • 57.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Marked Data • Data error detected, but no recovery option available, error mitigation is acceptable • Data should be quarantined - do not use it, do not derive actions from it • Example: IEEE ,Not a Number‘ (NaN) • Result of division by zero, square root of -1, ... • IEEE 754-1985: Standard representation for binary floating point numbers • Rules for computation when operand is NaN - typically result is again NaN • Options: Assume default value, skip operation, mark result as erroneous 57
  • 58.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Overload Toolboxes • Handle overload situation with too many requests for the system • Each resource class needs dedicated overload treatment • Memory: Exhaustion hinders new request from entering the system • CPU: Overload slows overall processing down • Patterns: Fresh work before stale, share the load, shed load • Tangible resources: Processing demands exclusive system resources • Network ports, shared storage, devices, ... • Patterns: Queue for resources, equitable resource allocation • Consider user demands • Patterns: Fresh work before stale, finish work in progress 58
  • 59.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Shed Load • Throw away a minority of requests to serve the majority • As early as possible, to minimize resource consumption • Rejection method to be considered, e.g. do not send acknowledgements • Example: ICMP • Type 3: Destination Unreachable - no time-out on client host • Type 4: Source Quench - typically only between routers, also used by mail servers • Type 11: Time Exceeded - through congestion (or circular packets) • Example: HTPP 5XX error codes • 503: Service Unavailable • Specialized case: Shed work at periphery 59
  • 60.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Finish Work in Progress / Fresh Work Before Stale • Finish Work in Progress • What to process, what to reject ? • Best case is labeling of requests: „new“ vs. „continuation“ • Distinguish ,continuation‘ processes based on their resource consumption • Try aggressively to get rid of resource hogs • Can lead to oscillation when system is „starving“ for new requests after cleanup • Solution: Let small portion of new requests through • Fresh Work Before Stale • If requester gives up, his retry eats up even more resources • Perform LIFO queue handling or non-queueing for premium requesters 60
  • 61.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Slow It Down • Handle overload cases and avoid saturation by multi-step escalation • Restrict request processing with increasing severity per level • Goal: Slow things down until the system can catch up with the load • Feedback system, demands dedicated resources for the controller part • Add hysteresis effect to prevent oscillation for level changes • System output behavior depends on past behavior, not only on input • Implement by different trigger values to 
 enter / leave an escalation level • If the system stays in-between, the state 
 remains the same • Example: Thermostat 61 Thresholds OverloadState
  • 62.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Deferrable Work / Equitable Resource Allocation • Deferrable Work • High load: Shed incoming work vs. shed routine maintenance workload • Make routine work (only relevant in error case) deferrable • Equitable Resource Allocation • Scenario: Handling of many requests for a set of resources, some of them are rare • Request-level handling would render some resources unnecessarily idle • Solution: Pool similar requests, allocate resources to pools • Additional bookkeeping needed for managing the requests and their related resource demands • Might lead to priority-inversion scenario 62
  • 63.
    Dependable Systems CoursePT 2014 Error Mitigation Patterns - Expansive / Protective Automatic Controls • Expansive Automatic Controls • Design some system parts for only being used in case of overload • Example: No 100% CPU utilization in normal operation of HA clusters • Example: Dynamic Offloaded Work - Cloud Computing • Increases request processing overhead, so take only as temporary solution • Protective Automatic Controls • Overload options: Shed internal work, shed incoming load, do nothing • Put restrictions on how much work the system accepts while still functioning • System throughput can drop due to contention, but should not drop to zero 63
  • 64.
  • 65.
    Error Processing Dependable SystemsCourse PT 2014 Phases of Fault Tolerance (Hanmer) 65 Latent Fault Error Normal OperationFault Activation Error Recovery Error Mitigation Error Detection Fault Treatment
  • 66.
    Dependable Systems CoursePT 2014 Fault Treatment Patterns - Let Sleeping Dogs Lie / Reintegration • Let Sleeping Dogs Lie • Treating faults by system change can introduce new faults • Known latent fault: Risk of reoccurrence, damage assessment possible • Potential new fault: Additional risk of miss-applied correction, no damage assessment possible for accidentally added faults • Reintegration • Different steps needed to reintegrate repaired component • Take off riding over transient and isolation lists • Watch new component for a while: hardening / soaking / trailing • Follow deterministic procedure, use as standby if possible 66
  • 67.
    Dependable Systems CoursePT 2014 Fault Treatment Patterns - Reproducible Error / Small Patches / Revise Procedure • Reproducible Error • Apply stimuli again under quarantine in order to prove fix • Can be automated (regression test) • Compare system output with golden unit output • Small Patches • Design system update as small as possible • Revise Procedure • When predetermined procedures contributed to failure duration, fix them 67