Minwise hashing (MinHash) has become a standard tool for calculating signatures (fingerprints) of sets that is used in many applications for similarity estimation and nearest neighbor search. Generalizations have been proposed that are able to calculate signatures for weighted sets and allow estimating either the weighted Jaccard similarity or the probability Jaccard similarity. While there are already very fast algorithms for calculating signatures of unweighted sets, until recently there were no such algorithms for weighted sets. In this talk, the basic ideas of the latest weighted minwise hashing algorithms BagMinHash, DartMinHash, TreeMinHash, and ProbMinHash are presented. All of them have been developed only in the last two years and can reduce the computation costs by many orders of magnitude.

2. 2
•
•
•
•
•
•
•
•
•
•
•
•

3. 3

4. 4
• Useful if objects can be represented as sets of features
• and Jaccard similarity is an appropriate similarity measure
coronavirus
hate
the
“I hate the coronavirus!”
I
“I hate lockdowns!”
25 21 18 41 98 12 15 41
25 32 18 11 98 56 33 72
Set representation
lockdowns
hateI
Object Signature Similarity estimation
Minwise hashing
Minwise hashing
used for deduplication of similar web pages

5. 5
I 25 63 98
hate 67 41 18
the 79 34 35
coronavirus 36 21 52
25 21 18
input set
signature
minimum hash value
defines signature component
independent hash functions

6. 6

7. 7

8. 8
hate
the
I
coronavirus

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14
Step 1
Step 2
Step 1
Step 2

15. 15
claims that Ioffe’s algorithm is wrong!

16. 16

17. 17

18. 18

19. 19

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. 26

27. 27

28. 28

29. 29

30. 30

31. 31
“Bagminhash - Minwise hashing algorithm for weighted sets” (Ertl, KDD 2018)

32. 32
“DartMinHash: Fast Sketching for Weighted Sets” (Christiani, 2020)

33. 33
“DartMinHash: Fast Sketching for Weighted Sets” (Christiani, 2020)

34. 34https://github.com/oertl/treeminhash

35. 35https://github.com/oertl/treeminhash

36. 36http://www.nrbook.com/devroye/Devroye_files/chapter_five.pdf

37. 37https://github.com/oertl/treeminhash

38. 38https://github.com/oertl/treeminhash

39. 39https://github.com/oertl/treeminhash

40. 40
DartMinHash performs
best if weights are
normalized
Performance of
DartMinHash depends
on total weight
https://github.com/oertl/treeminhash

41. 41https://github.com/oertl/treeminhash

42. 42

43. 43
“Maximally consistent sampling and the Jaccard index of probability distributions” (Moulton & Jiang, ICDMW 2018)

44. 44
“ProbMinHash–A Class of Locality-Sensitive Hash Algorithms for the (Probability) Jaccard Similarity” (Ertl, TKDE 2020)

45. 45

46. 46

47. 47

48. 48
ProbMinHash4ProbMinHash3
ProbMinHash2ProbMinHash1
with replacement w/o replacement
Label sampling
uncorrelatedcorrelated
Pointsampling

49. 49

50. 50
Correlated point generation of ProbMinHash3/4 may reduce estimation error for small sets!

51. 51

52. 52