Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019
•
7 likes•2,633 views
Nubank is leading financial technology in Latin America with a 100% digital banking experience, being recognized as the fastest growing digital banking outside of Asia. Our business aims at fighting the complexity we see in Brazilian banking and empowering people towards their money once again. To successfully deliver an amazing experience for more than 5 million credit card customers and 2.5 million checking account customers, we created a software platform composed by more than a hundred microsservices that are fast and reliable, even when facing unpredictable failures. Everyday we accomplish this goal with Apache Kafka as our communication backbone. This talk will detail how we are able to successfully run our platform by applying different patterns and development techniques to create a consistent event-driven design, capable of correcting data processing failures as fast as our business needs to be. We’ll show how patterns like dead letter queue, circuit breakers and back off are applied into the architecture to ensure that failures can be handled as consistently and transparently as possible by engineers across the company . Finally, the talk will also show the set of tools that were created on this architecture to address the concerns about quick fixes of failed events, such as a home grown CLI capable of inspecting failed events and reprocessing them as needed, all built on top of Apache Kafka.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019
Similar to Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Maintaining Consistency for a Financial Event-Driven Architecture (Iago Borges, Nubank) Kafka Summit NYC 2019
- 1. Iago Borges
- 3. Who are we?
- 4. Active credit card customers Active digital checking accounts Employees Engineers
- 6. Why is it so important?
- 7. That's why
- 8. Auditing
- 10. Whenever an event happens, the entire world must be eventually in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 11. Good eventual consistency is good enough
- 12. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 13. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 14. Impact of performance on consistency of asynchronous data flows
- 15. Whenever an event happens, the entire world must eventually be in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 16. "Anything that can go wrong will go wrong"
- 17. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 18. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance
- 19. Impact of availability on consistency of asynchronous data flows
- 20. Maybe we just retry it until it works? ● Processing will be stuck. ● Hard to configure properly. ● Very hard observability.
- 21. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3FTP Payment file arrived NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance Dead letter queue
- 22. Anti-corruption layer Brazilian payment system Payments manager Line ItemsBalance sheet Financing HTTP S3 Kafka AWS S3 FTP PAYMENT FILE ARRIVED NEW PAYMENT Writes payments file Fetches payments file Gets purchases, payments and etc Posts requesting a financing Gets balance How about this? Dead letter queue
- 23. Dead letters queue!! ● Fluid processing, regardless of failures. ● Great observability. ● Simpler configuration. ● One solution to rule them all.
- 24. How do we dead letter?
- 25. How do we dead letter? Random Service 2 Random Service 1 RANDOM EVENT TOPIC Mortician Produces to deadletter topic when an error is encountered AWS S3 Database DB references S3 Original message is replayed after problem is fixed HTTP The event is successfully propagated downstream No events lost
- 26. How do we make a good experience out of it?
- 27. How do we make a good experience out of it?
- 28. How do we make a good experience out of it?
- 29. So, would we keep killing letters non stop? Random Service 1 Random Service 2 Dead letter queue Service is failing multiple times Overloaded DB Doing a bunch of requests, overloading the database even more Producing a bunch of dead letters
- 30. Circuit breakers ● Protects your systems from DDOSing each other ● Once a lot of errors start happening it realises something is wrong and stop processing requests ● Probes eventually to know if things got back ● Self recovery
- 31. Circuit breakers Random Service 1 Failing service Dead letter queue Start throwing errors without even processing after multiple errors Overloaded DB Give some rest for the database to recover Stop sending unnecessary dead letters Circuit breaker
- 32. Circuit breakers Random Service 1 Failing service Dead letter queue Start working normally again once errors stop happening Overloaded DB Give some rest for the database to recover No need for deadletters nowCircuit breaker
- 33. Whenever an event happens, the world must be eventually in a state where the event happened exactly once. Not zero times, not more than once, exactly once.
- 34. "The night is dark and full of retries" As it should
- 35. Idempotence "Every event shall be idempotent" Nubank Engineers, circa 2014
- 36. Datomic ● Serial writes ● Immutable ● Fact based. No fact is deleted or changed ● You can force to be in a specific state in time
- 37. Idempotence Random Service 1 Random Service 2 Sends event X Processes and persists event X Query: does it exists? No DB Writes
- 38. Idempotence Random Service 1 Random Service 2 Sends the same event X again Processes the same event X again Query: does it exists? Yes, it has just been written, right?? DB Doesn't write, because the query returned it already exists, right??
- 39. Idempotence Random Service 1 Random Service 2 Sends the same event X again Processes the same event X again Query: does it exists? Yes, it has just been written, right?? DB Doesn't write, because the query returned it already exists, right?? Well… Maybe...
- 40. Idempotence Random Service 1 Random Service 2 Publishes duplicated events with uniquely identifiable ID Processes duplicated event DB Attempts to write duplicate record, encounters unique ID conflict and throws exception Because it's serial!
- 41. Finagle ● Response classification ● Retry behavior ● Timeouts ● Load balancing ● Circuit breakers (fail accrual, fail fast, threshold fail detector) ● Connection pooling
- 42. What we have learned about consistency ● Clients expect companies to consistently take care of their money ● Errors will happen and they're not so bad, embrace and treat them well. ● Everything should be idempotent. ● Keeping things working well is awesome! (ORLY??)