Insight Data Science

1. Sidi Chang
Insight Data Science Data Engineering Fellow
Jul 2016
JustBid

2. Sealed/blind second price auction
Item
Bidder

3. • Demo

4. Data Pipeline
Simulated
Data

5. Data
• 10K bidders
• Nearly 15 million bidding

6. Recommendation—Jaccard Similarity
Jaccard Similarity:
D_i = user_i
C_i = items(user_i)

7. Recommendation
For 𝑵 = 𝟏𝟎 million, it takes more than a year(AWS m4.large cluster)…
Then we will need to use minHash Algorithm which can be easily distributed…
Do an unbiased estimation by Chernoff Bounds and Markov Inequality:
The expected error is

8. MinHash Example
Item Row User 1 User 2 User 3 User 4 x+1 mod
5
3x+1
mod 5
1 0 1 0 0 1 1 1
2 1 0 0 1 0 2 4
3 2 0 1 0 1 3 2
4 3 1 0 1 1 4 0
5 4 0 0 1 0 0 3
U1 U2 U3 U4
Hash 1
Hash 2

9. MinHash Example
Item Row User 1 User 2 User 3 User 4 x+1 mod
5
3x+1
mod 5
1 0 1 0 0 1 1 1
2 1 0 0 1 0 2 4
3 2 0 1 0 1 3 2
4 3 1 0 1 1 4 0
5 4 0 0 1 0 0 3
U1 U2 U3 U4
Hash 1
Hash 2

10. MinHash Example
Item Row User 1 User 2 User 3 User 4 x+1 mod
5
3x+1
mod 5
1 0 1 0 0 1 1 1
2 1 0 0 1 0 2 4
3 2 0 1 0 1 3 2
4 3 1 0 1 1 4 0
5 4 0 0 1 0 0 3
U1 U2 U3 U4
Hash 1
Hash 2

11. MinHash Example
Item Row User 1 User 2 User 3 User 4 x+1 mod
5
3x+1
mod 5
1 0 1 0 0 1 1 1
2 1 0 0 1 0 2 4
3 2 0 1 0 1 3 2
4 3 1 0 1 1 4 0
5 4 0 0 1 0 0 3
U1 U2 U3 U4
Hash 1 1
Hash 2

12. MinHash Example
Item Row User 1 User 2 User 3 User 4 x+1 mod
5
3x+1
mod 5
1 0 1 0 0 1 1 1
2 1 0 0 1 0 2 4
3 2 0 1 0 1 3 2
4 3 1 0 1 1 4 0
5 4 0 0 1 0 0 3
U1 U2 U3 U4
Hash 1 1 3 0 1
Hash 2 0 2 0 0

13. Performance

14. Challenges
• MinHash Algorithm implemented in distributed system
• Jaccard Similarity Tested in distributed system
• Use right data structures to faster computation
• Use both Scala and Python

15. About me
• MS in CS and Operations Research