Resilence patternsreloaded kr
•
0 likes•471 views
This document presents various patterns for building resilience into systems. It discusses core architectural patterns like isolation, loose coupling, and latency control. It also covers detection/treatment patterns like monitoring, circuit breakers, and retries. Prevention patterns include redundancy, idempotency, and error handling. Recovery/mitigation patterns include rollback, failover, read repair, and shedding load. The document provides examples and explanations of these patterns.
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Resilence patternsreloaded kr
Similar to Resilence patternsreloaded kr (20)
Recently uploaded
Recently uploaded (20)
Resilence patternsreloaded kr
- 1. RESILIENCE RELOADED MORE RESILIENCE PATTERNS Jisung, Ahn
- 4. !!
- 5. ( ) ( ) ( )
- 6. = / ( + ) (Availability) (MTTF): Mean Time To Failure (MTTR): Mean Time To Recovery
- 7. = / ( + )
- 8. ( )
- 9. 8 ▸ 1. 2. 3. 4. 5. 6. . 7. 8.
- 10. . Leslie Lamport
- 12. “ ” ▸ Cloud-based systems ▸ Microservices ▸ Zero Downtime ▸ IoT & Mobile ▸ Social ->
- 14. Resilience = / ( + )
- 15. Resilience(IT) ▸ ▸ ( ) ▸ ( )
- 16. Isolation Bulkheads Loose Coupling Complete Parameter Checking Shed Load Supervision Latency Control Bounded Queue Fan out & quickest reply Circuit Breaker Fail Fast Timeout Self-Containment Idempotency Relaxed Temporal Constraints Asynchronous Communication Event-Driven Location Transparency Stateless Error Handler Monitor Escalation
- 17. …. ▸ (Recovery) & (Mitigation) ▸ Supervision ▸ ▸ (Anti-Fragility) ▸ &
- 18. RESILIENCE RELOADED MORE RESILIENCE PATTERNS
- 21. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Isolation Loose Coupling Latency Control Supervison Node Level System Level
- 22. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Isolation Redundancy Communication Paradigm Supporting Patterns Bulkhead
- 23. Bulkheads( ) ▸ ( “ (failure units)”,” (Unit of mitigation)”) ▸ ( ) ▸ . (uService, actor, scs,…) ▸
- 24. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Isolation Redundancy Communication Paradigm Supporting Patterns
- 25. Communication Paradigm ▸ Request-response <-> messaging <-> events ▸ , ▸ ▸
- 26. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Isolation Redundancy Communication Paradigm Supporting Patterns
- 30. (Crash) (Omission) (Timing) (Response) (Byzantine) Note: leaky bucket http://wooner2.tistory.com/284
- 31. ▸ Leaky bucket: , bucket 30 . 15 . 10 5 30 15 . bucket 15 .... ..... ? bucket . 30 15 5 (Be) 20 bucket . 30 15 . interval 20 ... 30 20 . 30 . 1 120 .
- 32. (Crash) (Omission) (Timing) (Response) (Byzantine) Note: leaky bucket http://wooner2.tistory.com/284
- 34. ▸ Voting: . (DMR) . . (TMR) . 2 : 1 . , .
- 37. ▸ Majority based quorum: Adaptive weighted sum: 3 , . , , . 50%, 80%, 30% . 100 ! . ∑ = *0.5 + *0.8 + *0.3 60, 70, 80 110 = 60*0.5 + 70*0.8 + 8*0.3 100 .
- 40. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Isolation Redundancy Communication Paradigm Supporting Patterns Stateless Zero downtime deployment Relaxed temporal contstraints Location transparency Esclation Structural Behavioral Idempotency
- 41. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Node Level Supporting Patterns System level Checksum Complete parameter checking Circuit breaker Timeout
- 42. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Node Level Supporting Patterns System level Monitor Heartbeat Acknowledgement Watchdog
- 43. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Node Level Supporting Patterns System level Voting Routing checks Health check Synthetic transaction Leaky bucket Fail fast
- 46. Retry example // doAction returns true if successful, false otherwise boolean doAction(...) { ... } // General pattern boolean success = false int tries = 0; while (!success && (tries < MAX_TRIES)) { success = doAction(...); tries++; } // Alternative one-retry-only variant success = doAction(...) || doAction(...);
- 47. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint
- 48. Rollback ▸ ▸ ▸ check point, safe point ▸ (Recurring error)
- 49. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward
- 50. Roll-forward ▸ ▸ escalation ▸ ▸ Roll-fowrard check point, safe point
- 51. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward Reset Reconnect Restart Data reset Startup consistency
- 52. Reset ▸ ▸ - ▸ Reset ▸ ( )
- 53. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward Reset Reconnect Restart Data reset Startup consistency Failover
- 54. Failover ▸ ▸ - ▸ , ▸ -
- 55. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward Reset Reconnect Restart Data reset Startup consistency Failover Read repair
- 56. Read repair ▸ ▸ - ▸ ▸ NoSQL Note: (digest) , Digest old
- 57. Read repair example public class FooResolver implements ConflictResolver<Foo> { @Override public Foo resolve(List<Foo> siblings) { // Insert your sibling resolution logic here } } public class Buddy { public String name; public Set<String> nicknames; public Buddy(String name, Set<String> nicknames) { this.name = name; this.nicknames = nicknames; } } public class BuddyResolver implements ConflictResolver<Buddy> { @Override public Buddy resolve(List<Buddy> siblings) { if (siblings.size == 0) { return null; } else if (siblings.size == 1) { return siblings.get(0); } else { // Name is also used as key. Thus, all siblings have the same name String name = siblings.get(0). name; Set<String> mergedNicknames = new HashSet<String>(); for (Buddy buddy : siblings) { mergedNicknames.addAll(buddy.nicknames); } return new Buddy(name, mergedNicknames); } }
- 58. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward Reset Reconnect Restart Data reset Startup consistency Failover Read repair Error Handler
- 59. Error Handler ▸ ▸ ( ) ▸ ▸ - escalation
- 60. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Retry Limited retries Rollback Checkpoint Safepoint Roll-forward Reset Reconnect Restart Data reset Startup consistency Failover Read repair Error Handler
- 61. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action
- 62. Fallback ▸ , . ▸ ▸ - ▸ -
- 63. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress
- 64. Queue for resources ▸ ▸ ▸ - ▸ -
- 65. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress Shed load
- 66. Shed Load( ) ▸ ▸ ▸ - ▸ ( )
- 67. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress Shed load Statically Dynamically Share load
- 68. Share load ▸ ▸ ( ) ▸ ▸ ( )
- 69. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress Shed load Statically Dynamically Share load Deferrable work
- 70. Deferrable work( ) ▸ ▸ ▸ ▸ ( ) .
- 71. Deferrable work example // Do or wait variant <init batch> while(<more to process>) { int load = getLoad(); if (load > THRESHOLD) { waitFixedDuration(); } else { <process next batch of work> } } void waitFixedDuration() { Thread.sleep(DELAY); // try-catch left out for better readability } // Adaptive load variant <init batch> while(<more to process>) { waitLoadBased(); <process next batch of work> } void waitLoadBased() { int load = getLoad(); long delay = calcDelay(load); Thread.sleep(delay); // try-catch left out for better readability } long calcDelay(int load) { // Simple example implementation if (load < THRESHOLD) { return 0L; } return (load – THRESHOLD) * DELAY_FACTOR; }
- 72. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress Shed load Statically Dynamically Share load Deferrable work Marked data
- 74. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Fallback Fail silently Default value Alternative action Queue for resource Bounded queue Fresh work before stale Finish work in progress Shed load Statically Dynamically Share load Deferrable work Marked data
- 75. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Let sleeping dogs lie Hot deployments Small releases
- 77. Routine maintenance ▸ . ▸ . ▸ ▸ ,
- 79. Spread News ▸ ▸ (gossip) (epidemic) . ▸ (Reconciliation) . ▸ ,
- 80. ▸ Gossip Protocol https://en.wikipedia.org/wiki/Gossip_protocol . . . ▸ Epidemic Protocol gossip protocol virus
- 82. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Anti-entropy Routine maintanace Spread news Backup request
- 83. Backup request ▸ ( ). ▸ . ▸ ( ) ▸ -
- 84. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Anti-entropy Routine maintanace Spread news Backup request Anti-Fragility Diversity Jitter
- 86. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Anti-entropy Routine maintanace Spread news Backup request Anti-Fragility Diversity Jitter Error injection
- 88. • Chaos Monkey • Chaos Gorilla • Chaos Kong • Latency Monkey • Compliance Monkey • Security Monkey • Janitor Monkey
- 89. Core (Architectural) Detection Treatment Prevention Recovery Mitigation Anti-entropy Routine maintanace Spread news Backup request Anti-Fragility Diversity Jitter Error injection
- 91. ▸ ▸ ▸ ▸ ( ) ▸ & ▸ / ▸ ▸ ▸ ▸ ▸
- 94. . !