0718

1. Introducing VenmoPlus.com
- Explore your Venmo network!
Qingpeng “Q.P.” Zhang, Insight Data Engineering Fellow

3. Features - VenmoPlus.com
● fuzzy searching of user name, with friend list to help
identify users with same name
● labeling the relationship between the payer and receiver
● friend recommendation
● searching transactions in friend circle
● listing friends of the user

4. Features - VenmoPlus.com
● fuzzy searching of user name, with friend list to help
identify users with same name
● labeling the relationship between the payer and receiver
● friend recommendation
● searching transactions in friend circle
● listing friends of the user

5. Demo:
VenmoPlus.com

6. Challenge:
● Find the distance between nodes in dynamic graph in real time

7. Solutions
● Two databases
○ Redis and ElasticSearch
● Algorithm design
○ BFS -> Bidirectional Search
○ Query relationship of a past transaction
● Query/search optimizations

8. Solutions
● Two databases
○ Redis and ElasticSearch
● Algorithm design
○ BFS -> Bidirectional Search
○ Query relationship of a past transaction
● Query/search optimizations

9. Historical
transactions
Real time
transactions
A Tale of Two Databases
API

10. Redis for graph structure
420890 Graham Hadley
1630476 Leon Tang
810029 Harminder Toor
1371353 Ephraim Park
562884 Paul Min
420890 set(14935158, 562884)
1630476 set(1371353)
810029 set(190230,14935158)
1371353 set(810029,971156)
562884 set(196371,1371353)
35 million edges
6 million nodes

11. ElasticSearch for everything

12. ElasticSearch for everything

13. Redis
Elasticsearch

14. Redis + Elasticsearch
=> search transactions in friend circle

15. Breadth First Search -> Bidirectional Search
Shortest distance -> intersection of sets (friend lists)
● A’s 1st degree friends ∩ B’s 1st degree friends
● A’s 2nd degree friends ∩ B’s 1st degree friends
O(N^2) -> O(2*N)
O(N^3) -> O(N + N^2)

16. VenmoPlus.com
m4.xlarge
m4.large
m4.xlarge
m4.large
t2.micro
$29.11/day

17. More optimizations to do
● Only store necessary information in elasticsearch to increase performance
● Labeling the distance of users for historical transaction can be done in
batch job, to reduce the number of the real time queries
● Adjust AWS instances to reduce the cost

18. Qingpeng “Q.P.” Zhang
● Postdoc, Lawrence Berkeley National Lab
○ GeneLearn - to classify DNA sequences
● PhD in Computer Science, Michigan State
○ Khmer - efficient k-mer counting
○ IGS - feature engineering for comparing samples
● BS in Physics, Nanjing University
What I learned from Insight:
Redis, Elasticsearch, Kafka, Spark Streaming, Flask, AngularJS, etc.

19. Query relationship of a past transaction

20. Query relationship of a past transaction
Query distance between vertices in a historic moment in a constantly changing
graph (because we don’t pre-calculate the distance….)
● If there are transactions before that one, distance = 1
● If the transaction is new: distance >1
○ Remove the influence of that specific transaction temporarily
○ Check distance from graph (2, 3, or >3)

23. Pipeline, raw data, in distributed way

24. Query/Search Optimizations
1. Remove aggregation for better performance… (trade-off)
2. Friend recommender:
a. Using Counter to get only 5 users with the most common friends
3. Search message in friend circle
a. Combine query of Elasticsearch and Redis