This document discusses various patterns and approaches for building resilient systems. It covers two main approaches to coding - "defensive coding" versus "let it crash". It then lists several resilience patterns including exceptions handling, timeouts, circuit breakers, health checks, and more. It analyzes these patterns from an infrastructure, operations, and application viewpoint. It also discusses Netflix's use of patterns like circuit breakers, retries, and failovers. Finally, it covers chaos engineering principles like injecting failures to test systems and verify they can withstand unpredictable conditions.

1. Enterprise resilience
patterns
@seva_dolgopolov

2. Disaster Semantics

3. 2 Approaches
“Defensive coding” vs. “Let it crash”

4. Disaster Math
Defensive coding
Let it crash

5. Patterns
Exceptions, Timeout, Circuit Breaker, Handshaking, Bulkhead,
Health checks, Heartbeat, Retry, Rollback, Reset, Failover,
Fallback, Backpressure, Bounded Queue, Load Balancing,
Dead Letter, Supervision, Governor, ...

6. Infrastructure View
Hardware
|
Process
|
Network
Exceptions, Health checks, Heartbeat, Dead Letter,
Retry, Rollback, Reset, Governor, Supervision
Timeout, Handshaking, Backpressure,
Fallback, Circuit Breaker
Load Balancing, Bulkhead, Failover

7. Employer View
Ops
|
Dev
Failover, Load balancing, Supervisor, Health checks
Fallback, Bulkhead, Timeout, Circuit Breaker,
Handshaking, Backpressure, Retry, Rollback,
Reset,Supervisor, Bounded Queue, Dead Letter

8. Application View
Failover, Load balancing, Supervisor, Bulkhead
Health checks, Timeout, Handshaking, Supervisor, Dead
Letter, Heartbeat
Fallback, Circuit Breaker, Backpressure, Retry, Rollback,
Reset, Bounded Queue
Deployment
|
Detection
|
Repair

9. What makes a
perfect mix

10. Akka
Deployment
- Supervisor
- Bulkhead(as Actor)
-> Detection -> Repair
- Heartbeat
- Dead letters
- Timeouts
- Restart
- Fallback
- Backpressure (akka-stream)
- Failover (akka-cluster)

11. Netflix
Deployment
- Bulkhead(as
Microservice)
-> Detection -> Repair
- Heartbeat
- Timeouts
- Circuit Breaker (Hystrix)
- Fallback
- Retry (Ribbon)
- Failover (Eureka)

12. That’s it?

13. A few things
1. Resilience will shape the way you implement your business logic
2. And get another level of complexity

14. objectives
- Staging env will never be full sized replica of production
- Safety is not composable property

15. Chaos
Engineering
http://principlesofchaos.org/

16. Netflix “Chaos monkey”
If something hurts do it more often.

17. Enrolling Chaos
Opt Out vs. Opt In

18. Targeting chaos
Random vs. Prespecified

19. Playing Chaos
1. Define a “Steady State”
2. Make a hypothesis that state will not change
“Clustered services should be unaffected by instance failures”
“The application is responsive even under high latency conditions”
3. Inject “Chaos”
4. Verify your hypothesis

20. Chaos Automation Platform

21. Chaos Experiment

22. Takeaways
- There is no silver bullet to achieve safe system
- Implementing Resilience brings complexity and you need
to manage it.
- Only test in Production make you confident

23. thx