The document discusses risk management in retail through stream processing, focusing on the evolution from a first-generation risk engine to a more efficient, scalable risk engine v2. Key improvements include independent, parallel processing pipelines, reduced memory demand, and enhanced performance. The document highlights the importance of understanding data bottlenecks and emphasizes the separation of concerns in architecture design.

1. Risk Management in Retail
with Stream Processing
Daniel Jagielski

2. Daniel Jagielski
Tech Lead & Manager at VirtusLab
Passionate about distributed systems
After hours:
Contact:
djagielski@virtuslab.com
kontakt@yagiel.pl
@danjagiel

3. Identity API
AUTHENTICATION TOKEN
MANAGEMENT
RISK ANALYSIS CLUBCARD
VERIFICATION
AUTHORIZATION

6. Attack profitability
Cost ValueCost Value

7. Risk engine

8. Risk engine
Analysis of authentication
data streams
Trigger targeted actions
against attackers

9. Clients Identity API Risk engine
Risk based authentication
Requests Data feed
Orchestration Knowledge

10. Risk Engine V1

11. Risk Engine V1
State stored
in memory
Locks
and critical
sections
Up to 3 DB
calls per
event
No horizontal
scalability
Slow event
processing
times
No high
availability

13. Session aggregate
#DDD
#EventSourcing
#aggregate
#CQRS
#projection

14. Session aggregator

15. High
memory
demand
CPU spikes
Long
rebalancing
Data
distribution
and skew
Aggregates
reaching up
to 1 MB
Artificial
size limits Scaling up
Tuning
RocksDB

16. Session aggregator

17. Session aggregator

18. Risk Engine V2 – new principles
Let’s not create
a session
aggregate!
Focus on the
main goal
Single
responsibility
of each module
Independent,
parallel pipelines

19. Risk Engine V2 – core design
Calculate IP
statistics
Block
malicious IP
Analyse IP
statistics

20. Risk Engine V2 – architecture
{
"ipAddress": "127.0.0.1",
"totalCount": 6,
"failureCount": 3,
"creationTimestamp": 123456,
"eventType": "login",
"failureRatio": 0.5
}
{
"ipAddress": "127.0.0.1",
"creationTimestamp": 123459,
"violatedRule": "XYZ"
}

21. Risk Engine V2 – single pipeline

22. Risk Engine V2 – multiple pipelines

23. Risk Engine V2 – multiple pipelines

24. Risk Engine V2 - benefits
Data locality
Scalability
No single point of
failure
Separation of
concerns
Smaller memory
footprint, less
CPU usage
Faster startup,
less state
Distributed
platform
Significant
performance
improvement

25. Deployment
POD

26. Observability

27. Kafka Connect Kafka Connect
Integration with external systems

28. Kafka Connect Kafka Connect
Integration with external systems

29. Summary
Identify
bottlenecks
Know your data! Minimum Value
Aggregates
Distributed,
independent
pipelines

30. Thank you!
Q&A