Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with Kafka | Asaf Halili, Oribi
•
0 likes•527 views
Handling failures is important, but it’s a must have when a product handles sensitive data. It also becomes exponentially harder in the world of microservices, since a failure can happen in any of the services and even in their dependencies. One of the go-to solutions when handling errors is to simply “Retry”. However, in a complex system, A “Retrying” mechanism must be Smart, Customisable, Pluggable, and Persistent. In this talk, we'll discuss what capabilities a ""GOOD"" retrying mechanism should have, how I implemented a Kafka-based retrying solution that answers those capabilities, can handle a large and diverse set of errors and why Kafka is a good match for such a solution.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with Kafka | Asaf Halili, Oribi
Similar to Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with Kafka | Asaf Halili, Oribi (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with Kafka | Asaf Halili, Oribi
- 1. Error Resilient Design Building Scalable & Fault-Tolerant Microservices with Kafka
- 2. About Me Hi, My Name is Asaf Halili
- 3. About Me Hi, My Name is Asaf Halili I’m a Software Developer Hello World!
- 4. About Me Hi, My Name is Asaf Halili I’m a Software Developer I work @ Oribi, a web analytics startup Hello World!
- 5. About Me Hi, My Name is Asaf Halili I’m a Software Developer I work @ Oribi, a web analytics startup I’m a technology enthusiast and a hobbyist photographer Hello World!
- 6. Why Fault-Tolerance is Challenging
- 7. Why Fault-Tolerance is Challenging Complex Software Architectures
- 8. Why Fault-Tolerance is Challenging Complex Software Architectures Vast Amounts of Data from Different Sources
- 10. The Common Solutions The Go-To Solution: Retry
- 11. The Common Solutions The Go-To Solution: Retry Circuit Breaker
- 12. The Common Solutions The Go-To Solution: Retry Circuit Breaker Fallback
- 13. How an Ideal Solution Looks Like
- 14. How an Ideal Solution Looks Like Smart
- 15. How an Ideal Solution Looks Like Smart ● Act upon failures automatically ● Allow sufficient time for failures to repair
- 16. How an Ideal Solution Looks Like Smart ● Act upon failures automatically ● Allow sufficient time for failures to repair Persistent
- 17. How an Ideal Solution Looks Like Smart ● Act upon failures automatically ● Allow sufficient time for failures to repair Persistent ● In the era of cloud & orchestration platforms, applications can be volatile.
- 18. How an Ideal Solution Looks Like Customizable
- 19. How an Ideal Solution Looks Like Customizable ● Every microservice has different requirements
- 20. How an Ideal Solution Looks Like Customizable ● Every microservice has different requirements Pluggable
- 21. How an Ideal Solution Looks Like Customizable ● Every microservice has different requirements Pluggable ● Integrate easily into any microservice
- 25. OOPS! An exception was thrown 😨
- 26. Saving the Method Context Retry1 Kafka Topic Publish the method context As a kafka message: { “fullName”: “Asaf Halili”, “mathGrade”: 85 } Kafka Producer
- 27. Invoking 1st Retry Kafka Consumer Retry1 Kafka Topic Polls every 2 seconds saveStudent(“Asaf Halili”, 85) Invokes
- 28. Oh No! The DB is still unavailable 😢
- 29. Saving the Method Context, 2nd time Retry2 Kafka Topic Publish the method context As a kafka message: { “fullName”: “Asaf Halili”, “mathGrade”: 85 } Kafka Producer
- 30. Why Do We Need Another Topic?
- 31. ● Decoupling ○ Separate the first retries from the second retries and so forth. It allows us to handle them differently. ■ Observability ■ Retry Pace ■ etc. Why Do We Need Another Topic?
- 32. Invoking 2nd Retry Kafka Consumer Retry2 Kafka Topic Polls every 1 minute saveStudent(“Asaf Halili”, 85) Invokes
- 33. Hey :-) The DB is available 😃
- 34. The student saved successfully in the DB! 😃
- 35. Students Service Retry 1 Producer Consumer Retry 2 Producer Consumer Retry2 Kafka Topic Retry1 Kafka Topic saveStudent
- 36. but what would happened If the DB Was unavailable again?
- 37. Saving the Method Context to a DLQ topic DLQ Kafka Topic Publish the method context As a kafka message: { “fullName”: “Asaf Halili”, “mathGrade”: 85 } Kafka Producer
- 38. A DLQ Topic
- 39. A DLQ Topic ● DLQ is a Dead Letter Queue DLQ Topic Producer Consumer X
- 40. A DLQ Topic ● DLQ is a Dead Letter Queue ● It’s purpose is to save messages (in our case, method contexts) that can’t be handled automatically DLQ Topic Producer Consumer X
- 41. A DLQ Topic ● DLQ is a Dead Letter Queue ● It’s purpose is to save messages (in our case, method contexts) that can’t be handled automatically ● The messages in this topic will be analyzed manually. DLQ Topic Producer Consumer X
- 42. High Level Architecture Students Service Retry 1 Producer Consumer Retry 2 Producer Consumer DLQ Producer DLQ Kafka Topic Retry2 Kafka Topic Retry1 Kafka Topic saveStudent
- 45. The Solution Capabilities Smart ● Multiple retries with exponential backoff to allow failures enough time to repair
- 46. The Solution Capabilities Smart ● Multiple retries with exponential backoff to allow failures enough time to repair ● DLQ topic for manual analysis
- 47. The Solution Capabilities Smart ● Multiple retries with exponential backoff to allow failures enough time to repair ● DLQ topic for manual analysis Persistent
- 48. The Solution Capabilities Smart ● Multiple retries with exponential backoff to allow failures enough time to repair ● DLQ topic for manual analysis Persistent ● We use Kafka as our persistence layer
- 50. The Solution Capabilities Customizable ○ The entire configuration is customizable: ■ Number of retries ■ Polling time ■ Exception types to be retried etc.
- 52. The Solution Capabilities Pluggable ● Implemented as a method annotation @OribiKafkaRetrying public void methodToRetryOnFailure(String name) { System.out.println("Hello " + name); }
- 53. Helpful Resources ● Uber’s Reliable Reprocessing - https://eng.uber.com/reliable-reprocessing/ The solution I described is based on Uber’s article. ● Handle Failures In A Complex Microservices Architecture - https://techblog.oribi.io/tech-blog/handle-failures-in-co mplex-microservices-architecture
- 54. Thanks! Feel free to contact me :-) asaf.halili@gmail.com linkedin.com/in/asafhalili @asafhalili