Download to read offline
Uploaded byQingpeng "Q.P." Zhang
Qingpeng zhang 0713
This document introduces VenmoPlus.com, a service that allows users to explore their Venmo network. It provides features such as fuzzy username searching, labeling relationships between payers and receivers, friend recommendations, searching transactions within a friend circle, and listing friends. It uses Redis to store the graph structure and Elasticsearch to store all other data. Algorithms like breadth-first search and bidirectional search are used to find the distance between nodes in the dynamic graph in real time. The system employs a two database approach with Redis and Elasticsearch to support these features efficiently.
More Related Content
Similar to Qingpeng zhang 0713
Qingpeng zhang 0713
- 1. Introducing VenmoPlus.com - Exploreyour Venmo network! Qingpeng “Q.P.” Zhang, Insight Data Engineering Fellow
- 3. Features - VenmoPlus.com ●fuzzy searching of user name, with friend list to help identify users with same name ● labeling the relationship between the payer and receiver ● friend recommendation ● searching transactions in friend circle ● listing friends of the user
- 4. Features - VenmoPlus.com ●fuzzy searching of user name, with friend list to help identify users with same name ● labeling the relationship between the payer and receiver ● friend recommendation ● searching transactions in friend circle ● listing friends of the user
- 5.
- 6. Challenge: ● Find thedistance between nodes in dynamic graph in real time
- 7. Solutions ● Two databases ○Redis and ElasticSearch ● Algorithm design ○ BFS -> Bidirectional Search ○ Query relationship of a past transaction ● Query/search optimizations
- 8. Solutions ● Two databases ○Redis and ElasticSearch ● Algorithm design ○ BFS -> Bidirectional Search ○ Query relationship of a past transaction ● Query/search optimizations
- 9. Historical transactions Real time transactions A Taleof Two Databases API
- 10. Redis for graphstructure 420890 Graham Hadley 1630476 Leon Tang 810029 Harminder Toor 1371353 Ephraim Park 562884 Paul Min 420890 set(14935158, 562884) 1630476 set(1371353) 810029 set(190230,14935158) 1371353 set(810029,971156) 562884 set(196371,1371353) 35 million edges 6 million nodes
- 11.
- 12.
- 13.
- 14. Redis + Elasticsearch =>search transactions in friend circle
- 15. Breadth First Search-> Bidirectional Search Shortest distance -> intersection of sets (friend lists) ● A’s 1st degree friends ∩ B’s 1st degree friends ● A’s 2nd degree friends ∩ B’s 1st degree friends O(N^2) -> O(2*N) O(N^3) -> O(N + N^2)
- 16.
- 17. More optimization ● Onlystore necessary info in elasticsearch ● Labeling distance of history transaction can be done in batch job, reduce the number the real time queries ● Adjust AWS instances to reduce cost
- 18. Qingpeng “Q.P.” Zhang ●Postdoc ○ Lawrence Berkeley National Lab ● PhD in Computer Science, ○ Michigan State University What I learned from Insight: ● Thinking as data engineer ● Open source tools ○ Redis, Elasticsearch, Kafka, Spark Streaming, Flask, AngularJS, etc.
- 19.
- 20. Query relationship ofa past transaction
- 21. Query relationship ofa past transaction Query distance between vertices in a historic moment in a constantly changing graph (because we don’t pre-calculate the distance….) ● If there are transactions before that one, distance = 1 ● If the transaction is new: distance >1 ○ Remove the influence of that specific transaction temporarily ○ Check distance from graph (2, 3, or >3)
- 24. Pipeline, raw data,in distributed way
- 25. Query/Search Optimizations 1. Removeaggregation for better performance… (trade-off) 2. Friend recommender: a. Using Counter to get only 5 users with the most common friends 3. Search message in friend circle a. Combine query of Elasticsearch and Redis
- 26.