PostgresQL

1. When Your SUM/COUNT queries are not enough any more
Tarmo Vikat – SR IT Service Engineer (fraud related area)
Nov 2016

2. Agenda

3. Skype Fraud 101
0.10 € 0.06 €
Alex Bob

4. How do we battle the situation?
Find
suspicious
elements
Investigate
Find linked
elements
Block the
fraudsters

5. Data points
Account Order Service
Device ID
Credit Cards
Time and Date
Login Data
Payment Information
IP address
Call Destination
Billing Address
Shopper IP
Client IP
Order IP
Linked Accounts
Issuer Country
E-mail
Caller ID Usage Summary
Profile Information
Contact List

6. What kind of data we use?
Statistics for call by IP
IP Events Cost Duration First Last Users/Blocked users
194.101.26.28 10 500 125,7 10:28:56 2016-09-13 00:15:08 2016-10-11 10:20:00 1178 / 101
Linked users by call IP
Username IP Events Cost Duration First Last
fraudulent_user1 27.11.105.23 273 10,8 02:15:00 2016-09-13 00:15:08 2016-10-11 10:20:00
fraudulent_user1 194.101.26.28 15 2,7 00:45:00 2015-12-01 00:00:08 2015-12-13 00:15:00

7. So what’s the problem?
Big DBs
Batch Jobs SQL
User interface
IP User Creations Callers
194.101.26.28 160/50 35/10

8. Plain SQL vs aggregated data

9. Why not use DWH or materialized views?
Bob
Powerful DWH
USER
IP
PSTN
PostreSQL
materialized views

10. Aggregation types
• Total stats over whole history
• Never expires
• Long-term storage only for non-personalised data
Whole history
• Last X days/hours stats in realtime (ie 30 days)
• Expiration (deaggregation) of old events
• Ability to implement velocity checks / data flow rules
X days back from
current moment
• Hourly/daily statistics
• Destruction of old data
• Ability to monitor data trends
Time series

11. Data structures
LINK
aggr_object
linked_object
count
price
duration
first_event
last_event
PK: aggr_object, linked_object
IDX: linked_object
STATS
aggr_object
count
price
duration
first_event
last_event
linked_object_count
blocked_object_count
PK: aggr_object

12. Basics of PostgreSQL based architecture
½ Data ½ Data

13. Basics of aggregation (IP based)
CALL (raw data)
User IP Duration Cost
testuser1 10.12.35.103 00:10:00 0.27
testuser2 10.12.35.103 00:30:00 1.00
testuser2 20.22.15.100 00:20 2.00
LINK
Object User Count
10.12.35.103 testuser1 1
10.12.35.103 testuser2 1
20.22.15.100 testuser2 1
STATS
Object User count
10.12.35.103 2
STATS
Object User count
20.22.15.100 1

14. Basics of queueing (pgq component)
PostgreSQL
Insert
Update
Delete
EVENT QUEUE
Event type (ins/upd/del)
Event data
Select Script

15. Schematics – whole history aggregation / time series
QUEUE
Aggregator
Worker 1
Config
STATS
LINK
STATS
Aggregator
Worker 2
Config
QUEUE
LINK

16. Dynamic configuration
CALL
User IP Length Cost
testuser1 10.12.35.103 00:10:00 0.27
testuser2 10.12.35.103 00:30:00 1.00
testuser2 20.22.15.100 00:20 2.00
STATS
Object Linked object Duration Price
user ip length cost

17. Schematics – sliding window aggregation
TRACKER
aggr fields
date
Aggregator
worker
STATS LINK
Data
destruction
worker
QUEUE

18. Simplified examples of used functions
new_user_count = 1;
user_count = user_count + <new_user_count>

19. So simple only?

20. Poor man’s SQL streaming (users who bought same product)
PURCHASE_DETAIL
Purchase_ID Product_name
1 Call packet 60 min
2 100 x SMS
QUEUE
PURCHASE
Username IP Cost
Testuser1 10.12.35.103 10.50
DENORMALISED DATA
Product_name Username IP Cost
Call packet 60 min Testuser1 10.12.35.103 10.50
100 x SMS Testuser1 10.12.35.103 10.50
Aggregator
worker

21. Poor man’s SQL streaming
Table 1
Table 2
De-normalised

22. Some caveats

23. Some numbers about performance
USER
IP
PSTN

24. Use cases
0
10
20
30
40
50
60
70
80
Phone number fraudulency
Phone1 Phone2 Phone3

25. Q&A
Tarmo Vikat
tavikat@microsoft.com
Skype: piruelain