Download as PDF, PPTX
Uploaded byIccha Sethi
Resilient Event Driven Systems With Kafka
The document discusses the implementation of a notifications feature using event-driven architecture and Kafka, focusing on technical requirements such as asynchronous scaling and graceful degradation. It emphasizes the importance of resiliency, observability, and handling errors effectively in the system to ensure notifications behave correctly under various scenarios. Key takeaways include the need for extensibility, the impact of resiliency on product behavior, and the application of queuing theory for performance measurement.
More Related Content
Similar to Resilient Event Driven Systems With Kafka
Resilient Event Driven Systems With Kafka
- 1. ICCHA SETHI |PRINCIPAL DEVELOPER, ATLASSIAN | @ICCHASETHI From Requirements To Resilient Event Driven Systems With Kafka
- 2.
- 3. It all beginswith a feature spec… STORY OF NOTIFICATIONS @IcchaSethi
- 4.
- 7. What did weknow about the technical requirements of Notifications feature?
- 8. What did weknow? Asynchronous
- 9. Scale What did weknow? Asynchronous
- 10. Scale What did weknow? Graceful degradationAsynchronous @IcchaSethi
- 11. Event-driven architecture, is a softwarearchitecture pattern promoting the production, detection, consumption of, and reaction to events. Wikipedia, 2018
- 12. “The event-driven architecturepattern is a popular distributed asynchronous architecture pattern used to produce highly scalable applications” Software Architecture Patterns, Mark Richards
- 13. Kafka 0 1 23 4 5 6 7 8 9 Partition 0 0 1 2 3 4 5 Partition 3 Topic Foo Producer Consumer 0 Consumer 1 Consumer Group B Consumer 0 Consumer 1 Consumer Group A Consumer Group B Consumer Group A
- 14.
- 15.
- 16.
- 17.
- 18. MICRO SERVICES MEETEVENT DRIVEN ARCHITECTURE
- 19. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker DependencyA Dependency B Send notification
- 20. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker DependencyA Dependency B Send notification
- 22. Direct Messages Worker Send Notification Worker Group Messages Worker Check User Online Worker Sendnotification Another Rule Worker
- 23.
- 24. Build for extensibilityand flexibility to account for evolving requirements! TAKEAWAY @IcchaSethi
- 25. Good job Team! Weare done!
- 26.
- 27. How do wewant notifications to behave when things go wrong?
- 28.
- 29. Low latency is ofutmost importance Ordering of Events is of utmost important ORDERING LATENCY
- 30.
- 31.
- 32.
- 33.
- 34. Deadletter using Kafka 01 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B
- 35. Deadletter using Kafka 01 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B 0 1 2 3 4 5 Topic Foo.deadletter
- 36. Deadletter using Kafka 01 2 3 4 5 6 7 8 9 Topic Foo Consumer 0 Consumer 1 Consumer Group B 0 1 2 3 4 5 Topic Foo.deadletter
- 37.
- 38.
- 39. Use when orderdoes not matter Do not deadletter un- recoverable errors Utilize metadata to track attempts Deadletter with Kafka
- 40.
- 41. RESILIENCY REQUIREMENTS Ordering HTTP Retries Latency Deadletterqueues Timeouts Circuit breakers
- 42. RESILIENCY REQUIREMENTS Ordering HTTP Retries Latency Deadletterqueues Timeouts Circuit breakers @IcchaSethi
- 43.
- 44. SCENARIO ONE -WHAT TOOLS WOULD YOU USE? Order doesn’t matter! Send notifications ASAP!
- 45. SCENARIO TWO -WHAT TOOLS WOULD YOU USE? I guess upto 1 second delay is acceptable to preserve order. Else order doesn’t matter.
- 46. Always ASK whatif things go wrong? TAKEAWAY @IcchaSethi
- 47. HOW DO YOUSCALE RESILIENCY?
- 48. HOW DO YOUSCALE RESILIENCY?
- 49. Make it low-costto opt in for resiliency and have sensible default values TAKEAWAY
- 51. Why didn’t the customer getthe notification?
- 52. For any product Observability isan unwritten requirement ME
- 53. Observability Stats Logs Tracing What should we measure? •Data for Current state • Data to predict future state
- 54. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service
- 55. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service Arrival Rate
- 56. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service Arrival Rate Queue Length
- 57. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Residence Time Service Time Queue Length
- 58. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Residence Time Service Time Utilization = Arrival Rate * Service Time Queue Length
- 59. THE ANSWER LIESIN QUEUING THEORY Arrivals Queue Service Arrival Rate Wait Time Arrival Rate Service Time Utilization Queue Length
- 60. UTILIZATION = RATEOF ARRIVAL*SERVICE TIME Hockey Stick Residence Time vs Utilization
- 61.
- 62. UTILIZATION = RATEOF ARRIVAL * SERVICE TIME Direct Messages Worker Consumer Lag Residence Time Service Time Broker
- 63. UTILIZATION = RATEOF ARRIVAL * SERVICE TIME Direct Messages Worker Broker Counter for messages produced
- 64. UTILIZATION = RATEOF ARRIVAL * SERVICE TIME
- 66. Observability Stats Logs Tracing What should we measure? •Data for Current state • Data to predict future state • When will HOT partitions in Kafka occur? • Will scaling my consumers help?
- 67. Observability is notonly for measure of internal state of system but can also be used to predict its future state TAKEAWAY @IcchaSethi
- 68. Observability Stats Logs Tracing Logs • All producer/consumersuccess and failure handlers are wrapped with consistent log fields. • topic • partition • offset • Kafka timestamp
- 69.
- 70. Observability Stats Logs Tracing “… one ofthe deficiencies of the Event Notification pattern is the lack of visibility into the flow of the system, which makes it hard to debug.” - Martin Fowler
- 71.
- 72. Observability Stats Logs Tracing Kafka Event Metadata •id • producer of the event • source • tags • timestamp • version
- 73. Observability Stats Logs Tracing "bitbucket_org-hipchat-schemas-platform- ZipkinTrace": { "properties": { "parent_id":{ "type": "string" }, "sampled": { "type": "boolean" }, "span_id": { "type": "string" }, "trace_id": { "type": "string" } } },
- 74.
- 75.
- 78.
- 79.
- 80. Product Requirements • Buildfor evolving requirements - extensibility and flexibility • Always ask WHAT if things go wrong Resiliency • Resiliency affects product behavior • Make it a low cost option to adopt resiliency with sensible default values Observability Observability is not only the measure of the internal state but can also be used to predict future state Takeaways @IcchaSethi
- 81.