Designing Fault Tolerant Microservices
•
3 likes•408 views
While developing distributed apps, most teams are focused in delivery of business value. Sometimes, after production deployment, a few moments later, we realize that exceptions arise, time-outs blow. The system need more fault tolerance. Presentations overviews a few patterns and principles of fault and latency tolerance for such systems.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Designing Fault Tolerant Microservices
Similar to Designing Fault Tolerant Microservices (20)
More from Orkhan Gasimov
More from Orkhan Gasimov (11)
Recently uploaded
Recently uploaded (20)
Designing Fault Tolerant Microservices
- 2. Speaker Orkhan Gasimov Solution design & implementation. 15 years of software engineering; Teaching training courses. Architecture, Java, JavaScript / TypeScript. Author of training courses. Microservice, Spring Cloud, Akka for Java. 2
- 3. Agenda Forces & Solutions. Patterns & Approaches. 3
- 4. Forces Network, high-load, RPC mechanics. DBService API DBService DBService DBService Web API Mobile API 4
- 5. Possible Issues Network & services will eventually fail. DBService API DBService DBService DBService Web API Mobile API 5
- 7. Service Coordination Alternate Routes: Consumer Producer Producer Producer 7
- 8. Service Coordination Service Discovery: • Dynamic service locations. Service Registry Consumer Producer 8
- 9. Service Coordination Service Discovery: • Dynamic service locations. • Scale horizontally on demand. Service Registry Consumer Producer Producer Producer 9
- 10. Service Coordination Service Discovery: • Dynamic service locations. • Scale horizontally on demand. • Client-side load balancing on the fly. Service Registry Consumer Producer Producer Producer LB 10
- 11. Service Coordination Service Discovery: • Dynamic service locations. • Scale horizontally on demand. • Client-side load balancing on the fly. • Discovery Cluster. Service Registry Consumer Producer Producer Producer LB Service Registry Service Registry 11
- 12. High Load
- 13. High Load Loaded services may get slower. Common solutions: Horizontal scalability and load-balancing. DBService API DBService DBService DBService Web API Mobile API 13
- 14. High Load Scale manually by deploying more instances. DB Service API DBService DBService DB Web API Mobile API Service Service Service 14
- 15. High Load Scale manually by deploying more instances. Drawbacks: • Deployment requires time and resources. Not a real-time solution. • Lack of automation. Down-scale? Who? When? 15
- 16. High Load Scale dynamically, using an auto-scaler service. DBService API DBService DBService DB Web API Mobile API Service Service Service 16
- 17. High Load Scale dynamically, using an auto-scaler service. Drawbacks: • What happens when resource limit is reached? • Auto-scaler requires support. May fail to down-scale / up-scale. 17
- 18. RPC Mechanics
- 19. RPC Mechanics Cascading call stack deliver the final result. Service ServiceService Service Service ServiceService 19
- 20. RPC Mechanics If network or service fails, clients may get errors instead of meaningful results. Service ServiceService Service Service ServiceService 20
- 21. RPC Mechanics Sometimes you have to wait, to get an error. Service ServiceService Service Service 21
- 22. RPC Mechanics Sometimes you have to wait, to get an error. Users should wait for results, not errors! Service ServiceService Service Service 22
- 23. RPC Mechanics System needs time to free-up resources. DBService API DBService DBService DBService Web API Mobile API 23
- 24. RPC Mechanics Possible solution: • Time-outs – consider slow calls erroneous. • Threshold – e.g. error rate per interval. • Fallback value/function (no network calls). • Recovery time. 24
- 25. Circuit Breaker
- 26. Circuit Breaker Closed – watch requests, keep track of errors. 26 Closed Open Half-Open Success Too many fails Fast Fail Try one request Fail Success
- 27. Circuit Breaker Closed – watch requests, keep track of errors. Open – if there are too many errors, fast fail to a default value. 27 Closed Open Half-Open Success Too many fails Fast Fail Try one request Fail Success
- 28. Circuit Breaker Closed – watch requests, keep track of errors. Open – if there are too many errors, fast fail to a default value. Half-Open – try one request some time later. 28 Closed Open Half-Open Success Too many fails Fast Fail Try one request Fail Success
- 29. Circuit Breaker Track Exceptions & watch for time-outs. 29
- 30. Circuit Breaker On time-out throw a Timeout Exception. Track only Exceptions! 30
- 31. Circuit Breaker Usually applied at caller side. DBService API DBService DBService DBService Web API Mobile APICB CB 31
- 35. N-Modular Redundancy Process faster if possible
- 36. N-Modular Redundancy Send N parallel calls, vote for the first acceptable result. Caller Executor 1 Executor 2 Executor N Voter Receiver
- 37. N-Modular Redundancy Faster processing & lower probability of errors. Remote calls should be idempotent! Executor 1 Executor 2 Executor N ReceiverCaller & Voter
- 38. Example The fastest wins. Maps 1 Maps 2 Address Validation
- 39. Example 1st is ok. 2nd waits for 1st. Maps 1 Maps 2 Address Validation
- 40. Recovery Blocks Slow down if necessary
- 41. Fast Path Protect from time-outs!
- 42. Recovery Blocks Work faster when everything is fine. Slow down to protect when necessary. ReceiverCaller Request Response Result Basic Recovery Fast Path Advanced Recovery
- 43. Recovery Blocks Synchronous Block N ReceiverBlock 2 Block 1 Caller Request Fail Fail Response Result
- 44. Recovery Blocks Asynchronous Block N ReceiverBlock 2 Block 1 Caller Request Fail Fail Response
- 45. Example Prefer the fastest. Switch to next if prior won’t work. Maps 1 Maps 2 Address Validation Maps 1Cached
- 47. Actors Actors: • Communicate by sending messages. • Form into a hierarchy. Tasks: • Split tasks to sub-tasks. • Delegate sub-tasks to subordinates.
- 48. Actors Error handling: • Actors may fail while executing a task. • Fails are reported to parent as an error message. • Parent actors supervise and monitor subordinates. • Parents may resume, restart, stop subordinates or escalate the error.
- 49. Error Kernel Error Kernel Pattern: • The root actor is the Error Kernel of the whole actor system. • Each parent actor is the local Error Kernel of it’s hierarchy.
- 50. Error Kernel • Split the task to sub-tasks and delegate the execution. • Aggregate results, supervise and monitor faults. • Do not execute tasks at root, avoid errors. • If root fails, the whole task fails.
- 51. Instance Healers
- 52. Instance Healers • Services are scaled by running several instances. • Service instances may fail. • Need a mechanism to watch for failed instances.
- 53. Instance Healers • Watch for the service instances using health checks. • If necessary, heal instances (restart or spawn). • May be the Error Kernel of the system. Healer Dead NewHealed
- 55. Instance Healers Possible Issues: • Services may be in the middle of processing. • Stateful services may loose data.
- 56. Instance Healers Event Sourcing + CQRS: • Keep track of state-changing events, rebuild state from events. • Take snapshots for faster rebuilds. • Services may re-read events after healing. • Processing should be idempotent! Command Service Query Service Event Store Data Store Event Processor
- 57. Summary
- 58. Summary • Forces: • Network, Load, RPC. • Solutions: • Service Coordination, Horizontal Scalability & Load Balancing. • Patterns: • Circuit Breaker, N-Modular Redundancy, Recovery Blocks. • Approaches: • Error Kernel & Instance Healer. 58
Editor's Notes
- Network – fails and heals; High load – rush hours; PRC mechanics – algorithms, code;
- Up-scale and down-scale services on demand.
- Up-scale and down-scale services on demand.
- 1. Some executors may perform faster, while others may be overloaded. 2. More chances to get an acceptable response (in a timely manner).
- 1. Some executors may perform faster, while others may be overloaded. 2. More chances to get an acceptable response (in a timely manner).
- 1. Some executors may perform faster, while others may be overloaded. 2. More chances to get an acceptable response (in a timely manner).
- 1. Recover from errors if possible (more attempts, longer processing). 2. Sequential or asynchronous services, layered over the network. 3. Sequentially layered methods.
- 1. Caller may stream to first block; 2. On error, processing is passed to next block; 3. The first acceptable result is sent to Receiver;
- 1. Some executors may perform faster, while others may be overloaded. 2. More chances to get an acceptable response (in a timely manner).
- Prefer stateless processing, keep state in an external storage.