This document discusses using polysemous codes to perform large-scale search over visual signatures. Polysemous codes allow product quantization codes to be interpreted as both compact binary codes for efficient Hamming distance search and codes that preserve distance information for accurate nearest neighbor search. The key ideas are to learn an index assignment that maps similar product quantization codes to binary codes with smaller Hamming distance, and to directly optimize this assignment to match the distances between codebook centroids. This allows using a single code representation for both fast Hamming search and precise distance search, without increasing memory requirements. The document provides examples of applying polysemous codes to build a large graph connecting images based on visual similarity.

1. Large-scale search
with polysemous codes
Florent Perronnin
Naver Labs Europe

2. Problem statement
Given a query, find closest match(es) in large database of “entities”
Example entities: image, video, text, post, user, ad, …
Example applications:
• video copy detection (query = video, database = video)
• blog recommendation (query = user, database = blogs)
• ad placement (query = user, database = ads)
→ very large-scale problems
2

3. Problem statement
Visual signatures: compact, fast, accurate
DB
query
3

4. Image signatures
Step 1: embedding in common Eucliean space (approx. 1-10K dim)
Step 2: compression
“the cat”
“the dog”
4

5. Step 1: embedding in common Eucliean space (approx. 1-10K dim)
Step 2: compression → our focus in this talk
“the cat”
“the dog”
Image signatures
5

6. CONTENTS
1. Background: similarity search with compact codes
2. Polysemous codes
3. Application: knn-graph construction
6

7. 1.
Background: large-scale
search with compact codes

8. 1.1 Binary codes
000 001
100
110 111
011
101
010
Idea: design/learn a function mapping the original space into a
compact Hamming space
Neighbors w.r.t Hamming space try to reflect neighbors in original space
Advantages: compact descriptor, fast distance computation
LSH example: random projection + thresholding
[Charikar’02] shows that Hamming distance gives a cosine estimator 8

9. 1.2 Product quantization (PQ)
y = y1 y2 y3 y4
Decompose the feature space as a product space
• use a distinct quantizer in each subspace, typically k-means
• estimate distances by using look-ups and additions only
[Jégou’11]
9

10. 1.3 Binary codes vs PQ
Binary codes (ITQ) [Gong’11] Product quantization
e.g. [01000110…01] e.g. [2 63 27 227]
context-free comparison need quantizer centroids
1,190M comparisons / sec 222M comparisons / sec
precision = 0.143 precision = 0.442
How to get the best of both worlds?
Seen as competing methods in literature
10

11. 2.
Polysemous codes
[Polysemous codes, Douze, Jégou, Perronnin, ECCV’16]

12. 2.1 A naïve approach
q_bin= binary_encode(x) # compute query binary code
d_min = ∞
for i = 1..n # loop over database items
db_bin = db_bin_codes[i] # get binary code for item i
if hamming(q_bin, db_min) < threshold
db_pq = db_pq_codes[i] # get PQ code for item i
d = PQ_distance(x, db_pq)
if d < d_min
nearest_neighbor, d_min = i, d
Encode all DB items with binary and PQ codes:
12

13. 2.2 A naïve approach
Encode all DB items with binary and PQ codes
→ memory increase (x2)
Could we use the same codes for both the Hamming and
PQ distances?
→ polysemous codes
13

14. 2.3 Channel optimized vector quantization
Channel-optimized vector quantizers: “pseudo-Gray coding”
Minimize the overall expected distortion (both from source and channel)
Optimize the index assignment → neighboring codes encode similar info
enc=01001100 dec=01011100
14

15. 2.4 Index assignment optimization
Given a k-means quantizer, learn a permutation of the codes such that the
binary comparison reflects centroid distances
15

16. 2.5 The polysemous approach
q = encode(x) # compute query code
d_min = ∞
for i = 1..n # loop over database items
db = db_codes[i] # get code for item i
if hamming(q, db) < threshold
d = PQ_distance(x, db)
if d < d_min
nearest_neighbor, d_min = i, d
Interpret PQ codes as binary codes:
16

17. 2.5 The polysemous approach
q = encode(x) # compute query code
d_min = ∞
for i = 1..n # loop over database items
db = db_codes[i] # get code for item i
if hamming(q, db) < threshold
d = PQ_distance(x, db)
if d < d_min
nearest_neighbor, d_min = i, d
Interpret PQ codes as binary codes:
→ no memory increase 17

18. 2.6 Objective function
Find a permutation 𝝅() such that
the Hamming distance between permuted indices matches
the distance between centroids
weighting to
favor nearby
centroids
×
optimize permutation
over all pairs
of centroids
monotonous (linear)
function to correct the scale
18

19. 2.6 Objective function
Find a permutation 𝝅() such that
the Hamming distance between permuted indices matches
the distance between centroids
weighting to
favor nearby
centroids
×
optimize permutation
over all pairs
of centroids
monotonous (linear)
function to correct the scale
19

20. 2.6 Objective function
Find a permutation 𝝅() such that
the Hamming distance between permuted indices matches
the distance between centroids
weighting to
favor nearby
centroids
×
optimize permutation
over all pairs
of centroids
monotonous (linear)
function to correct the scale
20

21. 2.6 Objective function
Find a permutation 𝝅() such that
the Hamming distance between permuted indices matches
the distance between centroids
weighting to
favor nearby
centroids
×
optimize permutation
over all pairs
of centroids
monotonous (linear)
function to correct the scale
21

22. 2.7 Optimization
Simulated annealing:
• initialization: random permutation
• swap two entries in the permutation
• converges in approx. 200k iterations (<10s)
22

23. 2.8 Optimization
23

24. 2.9 Results: exhaustive search
1M SIFT vectors: exhaustive comparison (16 bytes)
24

25. 90M CNN descriptors from Flickr100M dataset (16 bytes)
2.10 Results: non-exhaustive search
25
BIGANN academic benchmark (1B vectors) → x2-2.5 speed-up

26. 3.
The knn-graph of a
collection

27. 3.1 Building a graph on images
Testbed: Flickr100M
• public dataset of CC images
• described with AlexNet FC7 features
normalized, PCA to 256D, encoded as 32bytes,
coarse quantizer size 4096
Each image in turn is a query
• compute 100-NN
• build index = 14h, search = 7h
• storage for the graph = 2 x 40 GB RAM

28. 3.2 Graph modes
Graph seen as a Markov model
→ compute stationary distribution [Cho’12]
Sparse matrix – vector multiplication
• 200 iterations (30s / iter)
• mode = local maximum over nodes

29. 3.3 Paths in the graph
Almost all images are connected: find path between pairs of images
→ morphing from one image to another
Which paths?
• shortest path
• minimize sum of distances
• minimize max of distances
29

30. 3.4 Path: flower to flower
30

31. 3.5 Path: lion to panther
31

32. 3.6 Path: dog to cat
32

33. 3.7 Path: 3D morphing
33

34. 3.8 Path: different objects
34

35. 3.9 Path: non intuitive…
35

36. Q & A

37. Thank you