Kafka For Financial Data Processing - The Flipkart Way (Shankar Manian and Rajat Gangwar, Flipkart) Kafka Summit London 2019
At Flipkart, the leading e-commerce company in India, our team processes the financial data for millions of transactions every day. Financial data processing requires exactly-once semantics because of customer sensitivity and financial regulatory requirements. The financial data entities have a long lifecycle for updates and we maintain the idempotency for many months and sometimes years. In addition, we provide near real-time view to our stakeholders on the ‘state’ of the processing. Let’s consider a specific example of a data processing request where we aggregate the Invoices into Payments for our sellers. We have to keep the number of payments as Constant to cope with the varying and increased number of invoices. In addition, the system has to maintain an audit trail of paid invoices versus the corresponding payment indents, to comply with Auditing procedures. The need for scalability and real-time Service License Agreement (SLA) steered us towards adopting a stream processing solution such as Kafka. In this talk, we explain our efforts towards a generic framework using Kafka for all use cases of Financial Data Processing in Flipkart and how we are faring on that challenge. We also explain: – The techniques used to achieve the exactly-once semantics within constraints specific to Financial data processing at Flipkart – Various challenges faced in the process and the use of technologies such as Storm Trident and MongoDB to overcome them. In addition, we also describe how this success has emboldened us to rethink our entire architecture on top of streaming infrastructure.
Recommended
Recommended
More Related Content
Similar to Kafka For Financial Data Processing - The Flipkart Way (Shankar Manian and Rajat Gangwar, Flipkart) Kafka Summit London 2019
Similar to Kafka For Financial Data Processing - The Flipkart Way (Shankar Manian and Rajat Gangwar, Flipkart) Kafka Summit London 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Kafka For Financial Data Processing - The Flipkart Way (Shankar Manian and Rajat Gangwar, Flipkart) Kafka Summit London 2019
- 1. Kafka for Financial Data - The Flipkart way Shankar Manian & Rajat Gangwar
- 2. 2 Agenda • Flipkart - A Background • Financial Data - Requirements and Challenges • Case Study • Future ahead
- 3. 3 Company Intro • Founded in 2007, a multi billion dollar company • Pioneered Indian eCommerce with Innovations like COD • 200+M Registered users, 50+M Active users • Market leader in Fashion, Electronics and many more • 100k+ active sellers
- 4. 4 What we do - Record Transactions ● Diverse set of sources ● Many parties involved ● Spread over time
- 5. 5 What we do - Provide Insights ● Power Balance sheet, P & L statement ● Real time trend analytics and forecasts ● Revenue assurance and fraud detection
- 6. 6 What we do - Be Compliant ● Financial data is highly regulated ● eCommerce is further regulated ● We also provide tax compliance
- 7. 7 Challenges - Data Quality ● Financial data is highly sensitive ● Legal Implications and Social media backlash ● Need to have real time visibility
- 8. 8 Challenges - Complete Audit Trail ● Not enough to be 100% correct, be able to prove to be 100% correct ● Need to be able to “Follow the Money” ● Need to have verifiable, repeatable processing.
- 9. 9 Challenges - Risk Averse Customer ● Primary customers are accountants, auditors and government ● Hard to convince, naturally paranoid ● Credibility is paramount
- 10. Case Study
- 11. 11 Payout flow - A case study ● Overview ● First solution ● Limitations ● Streaming ● Challenges ● Next steps
- 12. 12 Payout flow User Module Seller Module Marketplace Accounting Flipkart Money Order placed Item dispatched Party_from : Flipkart Party_to : Seller Amount : 300
- 13. 13 Payout flow : Seller Dashboard
- 15. 15 Evolution 1 Invoice Payment Disbursement
- 17. 17 Shortcomings - Not real-time - Vicious cycle
- 18. 18 Shortcomings - Not real-time - Vicious cycle Increase batch size of each job Puts extra load on DB Builds up lag
- 19. 19 Shortcomings - Not real-time - Vicious cycle - Throughput saturation Invoice Orches trator Payment 00:00 00:45 00:59 : 1st Job 01:00 : 2nd Job
- 20. 20 Shortcomings - Not real-time - Vicious cycle - Throughput saturation Throughput = Batch size X Number of jobsX a ~ b b (per day) Saturation Point : a
- 22. 22 Enter Stream... Order Stream Invoice Invoice Stream Consumer 30 50 Seller 1 Seller 2 Payout Dashboard
- 23. 23 Architecture Order Stream Invoice Payout Dashboard Invoice Stream Consumer 30 50 Seller 1 Seller 2
- 24. 24 Challenges 1. Avoid Locks 10 500 Bucket Stream Processor 510 20 530 15 201535 153540 Payout Module 530 0
- 25. 25 Challenges 1. Avoid Locks 10 500 Bucket 1 Stream Processor 510 20 530 15 201535 153540 Payout Module Day 1Day 2 530 Bucket 1 0 Bucket 2 15
- 26. 26 Challenges 1. Avoid Locks 2. Exactly-once
- 27. 27 Challenges 1. Avoid Locks 2. Exactly-once B1B2 B1 : T1 Kafka Storm Bucket 1 T1 T1 Bucket 1 T1 Bucket 2
- 28. 28 Challenges 1. Avoid Locks 2. Exactly-once Operation Idempotent X = X + 5 No X = X + 5, T = T1 May be X = 50 Yes Meaning : The operation will produce the same result, no matter how many times it is repeated. Solution : IDEMPOTENCY
- 29. 29 Challenges 1. Avoid Locks 2. Exactly-once 3. Testing / Validation Invoice Payments Buckets Disbursement Old flow New flow
- 30. 30 Summary - Real-time - No extra load on DB - Throughput is not bounded
- 31. 31 Next Steps Stream Invoice Kafka Aggregation D M T Debtor Month-end Tax-filing Stream Stream Stream Stream
- 32. Questions?