Successfully reported this slideshow.
Your SlideShare is downloading. ×

Risk Management in Retail with Stream Processing

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 30 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Risk Management in Retail with Stream Processing (20)

Advertisement

More from confluent (20)

Recently uploaded (20)

Advertisement

Risk Management in Retail with Stream Processing

  1. 1. Risk Management in Retail with Stream Processing Daniel Jagielski
  2. 2. Daniel Jagielski Tech Lead & Manager at VirtusLab Passionate about distributed systems After hours: Contact: djagielski@virtuslab.com kontakt@yagiel.pl @danjagiel
  3. 3. Identity API AUTHENTICATION TOKEN MANAGEMENT RISK ANALYSIS CLUBCARD VERIFICATION AUTHORIZATION
  4. 4. Attack profitability Cost ValueCost Value
  5. 5. Risk engine
  6. 6. Risk engine Analysis of authentication data streams Trigger targeted actions against attackers
  7. 7. Clients Identity API Risk engine Risk based authentication Requests Data feed Orchestration Knowledge
  8. 8. Risk Engine V1
  9. 9. Risk Engine V1 State stored in memory Locks and critical sections Up to 3 DB calls per event No horizontal scalability Slow event processing times No high availability
  10. 10. Session aggregate #DDD #EventSourcing #aggregate #CQRS #projection
  11. 11. Session aggregator
  12. 12. High memory demand CPU spikes Constant rebalancing Data distribution and skew Aggregates reaching up to 1 MB Artificial size limits Scaling up Tuning RocksDB
  13. 13. Session aggregator
  14. 14. Session aggregator
  15. 15. Risk Engine V2 – new principles Let’s not create a session aggregate! Focus on the main goal Single responsibility of each module Independent, parallel pipelines
  16. 16. Risk Engine V2 – core design Calculate IP statistics Block malicious IP Analyse IP statistics
  17. 17. Risk Engine V2 – architecture { "ipAddress": "127.0.0.1", "totalCount": 6, "failureCount": 3, "creationTimestamp": 123456, "eventType": "login", "failureRatio": 0.5 } { "ipAddress": "127.0.0.1", "creationTimestamp": 123459, "violatedRule": "XYZ" }
  18. 18. Risk Engine V2 – single pipeline
  19. 19. Risk Engine V2 – multiple pipelines
  20. 20. Risk Engine V2 – multiple pipelines
  21. 21. Risk Engine V2 - benefits Data locality Scalability No single point of failure Separation of concerns Smaller memory footprint, less CPU usage Faster startup, less state Distributed platform Significant performance improvement
  22. 22. Deployment POD
  23. 23. Observability
  24. 24. Kafka Connect Kafka Connect Integration with external systems
  25. 25. Kafka Connect Kafka Connect Integration with external systems
  26. 26. Summary Identify bottlenecks Know your data! Minimum Value Aggregates Distributed, independent pipelines
  27. 27. Thank you! Q&A

×