Every 2 seconds, another person becomes a victim of identity theft. The number of online account takeovers is constantly increasing. In this talk we'll show how stream processing was used to combat this for Tesco, one of Europe's largest retailers. The massive scale of e-commerce makes it an interesting target for malicious users. We implemented a risk-management platform built around Kafka and the Confluent Platform to detect and prevent attacks, including those that come through the website's authentication page. We'll present how this project evolved over 2 years to its current state in production, together with some of the challenges we encountered on the way. As the project has had a couple of phases, we will see and compare alternative designs, summarize their pros and cons, and refer them to well known techniques - like Event Sourcing. We'll discuss the architecture and integration with external systems, before moving onto a detailed examination of the stream processors implementation and key internals such as co-partitioning of data. We'll also cover the role of stack components that we used, including Kafka Connect and Schema Registry, as well as the deployment platform, Kubernetes. Over the course of the talk we will put special emphasis on highlighting key factors to take into consideration when designing data pipelines and stream processing platform.

1. Risk Management in Retail
with Stream Processing
Daniel Jagielski

2. Daniel Jagielski
Tech Lead & Manager at VirtusLab
Passionate about distributed systems
After hours:
Contact:
djagielski@virtuslab.com
kontakt@yagiel.pl
@danjagiel

3. Identity API
AUTHENTICATION TOKEN
MANAGEMENT
RISK ANALYSIS CLUBCARD
VERIFICATION
AUTHORIZATION

6. Attack profitability
Cost ValueCost Value

7. Risk engine

8. Risk engine
Analysis of authentication
data streams
Trigger targeted actions
against attackers

9. Clients Identity API Risk engine
Risk based authentication
Requests Data feed
Orchestration Knowledge

10. Risk Engine V1

11. Risk Engine V1
State stored
in memory
Locks
and critical
sections
Up to 3 DB
calls per
event
No horizontal
scalability
Slow event
processing
times
No high
availability

13. Session aggregate
#DDD
#EventSourcing
#aggregate
#CQRS
#projection

14. Session aggregator

15. High
memory
demand
CPU spikes
Long
rebalancing
Data
distribution
and skew
Aggregates
reaching up
to 1 MB
Artificial
size limits Scaling up
Tuning
RocksDB

16. Session aggregator

17. Session aggregator

18. Risk Engine V2 – new principles
Let’s not create
a session
aggregate!
Focus on the
main goal
Single
responsibility
of each module
Independent,
parallel pipelines

19. Risk Engine V2 – core design
Calculate IP
statistics
Block
malicious IP
Analyse IP
statistics

20. Risk Engine V2 – architecture
{
"ipAddress": "127.0.0.1",
"totalCount": 6,
"failureCount": 3,
"creationTimestamp": 123456,
"eventType": "login",
"failureRatio": 0.5
}
{
"ipAddress": "127.0.0.1",
"creationTimestamp": 123459,
"violatedRule": "XYZ"
}

21. Risk Engine V2 – single pipeline

22. Risk Engine V2 – multiple pipelines

23. Risk Engine V2 – multiple pipelines

24. Risk Engine V2 - benefits
Data locality
Scalability
No single point of
failure
Separation of
concerns
Smaller memory
footprint, less
CPU usage
Faster startup,
less state
Distributed
platform
Significant
performance
improvement

25. Deployment
POD

26. Observability

27. Kafka Connect Kafka Connect
Integration with external systems

28. Kafka Connect Kafka Connect
Integration with external systems

29. Summary
Identify
bottlenecks
Know your data! Minimum Value
Aggregates
Distributed,
independent
pipelines

30. Thank you!
Q&A