information retrieval

1. Large Scale Image Search

2. The problem
• Large scale image search:
– We have a candidate image
– Want to search a large database to find similar
images
– Search the internet to find similar images
• Fast
• Accurate

3. Large Scale Image Search in Database
• Find similar images in a large database
Kristen Grauman et al

4. Internet Large scale image search
Internet contains billions of images
The Challenge:
– Need way of measuring similarity between images
(distance metric learning)
– Needs to scale to Internet (How?)
Search the internet

5. Large scale image search
• Representation must fit in memory (disk too slow)
• Facebook has ~10 billion images (1010)
• PC has ~10 Gbytes of memory (1011 bits)
 Budget of 101 bits/image
Fergus et al

6. Requirements for image search
• Search must be both fast, accurate and scalable to
large data set
• Fast
– Kd-trees: tree data structure to improve search speed
– Locality Sensitive Hashing: hash tables to improve search speed
– Small code: binary small code (010101101)
• Scalable
– Require very little memory, enabling their use on standard hardware or
even on handheld devices
• Accurate
– Learned distance metric

7. Categorization of existing large scale
image search algorithms
• Tree Based Structure
– Spatial partitions (i.e. kd-tree) and recursive hyper plane
decomposition provide an efficient means to search low-
dimensional vector data exactly.
• Hashing
– Locality-sensitive hashing offers sub-linear time search by
hashing highly similar examples together.
• Binary Small Code
– Compact binary code, with a few hundred bits per image

8. Tree Based Structure
• Kd-tree
– The kd-tree is a binary tree in which every node is
a k-dimensional point
• (No theoretical guarantee!)They are known to
break down in practice for high dimensional
data, and cannot provide better than a worst
case linear query time guarantee.

9. Locality Sensitive Hashing
• Hashing methods to do fast Nearest Neighbor (NN) Search
• Sub-liner time search by hashing highly similar examples
together in a hash table
– Take random projections of data
– Quantize each projection with few bits
– Strong theoretical guarantees
• More detail later

10. Binary Small Code
• 1110101010101010
• Binary?
– 0101010010101010101
– Only use binary code (0/1)
• Small?
– A small number of bits to code each image
– i.e. 32 bits, 256 bits
• How could this kind of small code improve the image
search speed? More detail later.

11. Detail of these algorithms
1. Locality sensitive hashing
– Basic LSH
– LSH for learned metric
2. Small binary code
– Basic small code idea
– Spectral hashing

12. 1. Locality Sensitive Hashing
• The basic idea behind LSH is to project the data
into a low-dimensional binary (Hamming) space;
that is, each data point is mapped to a b-bit
vector, called the hash key.
• Each hash function h must satisfy the locality
sensitive hashing property:
– Where ∈ [0, 1] is the similarity function
of interest
Kristen Grauman et al
Datar, N. Immorlica, P. Indyk, and V. Mirrokni. Locality-Sensitive
Hashing Scheme Based on p-Stable Distributions. In SOCG, 2004.

13. LSH functions for dot products
The hashing function of LSH to produce Hash Code
is a hyperplane separating the space (next page for example)

14. 1. Locality Sensitive Hashing
• Take random projections of data
• Quantize each projection with few bits
0
1
0
1
0
1
101
No learning involved
Feature vector
Fergus et al

15. How to search from hash table?
Q
111101
110111
110101
hr1…rk
Xi
N
hr1…rk
<< N
Q
[Kristen Grauman et al, modified my me]
A set of data points
Hash function
Hash table
New query
Search the hash table
for a small set of images
results

16. Could we improve LSH?
• Could we utilize learned metric to improve LSH?
• How to improve LSH from learned metric?
• Assume we have already learned a distance
metric A from domain knowledge
• XTAX has better quantity than simple metrics such
as Euclidean distance

17. How to learn distance metric?
• First assume we have a set of domain knowledge
• Use the methods described in the last lecture to
learn distance metric A
• As discussed before,
• Thus is a linear embedding function that
embeds the data into a lower dimensional space
• Define G =

18. LSH functions for learned metrics
• Given learned metric with
• G could be viewed as linear parametric function or a
linear embedding function for data x
• Thus the LSH function could be:
• The key idea is first embed the data into a lower
space by G and then do LSH in the lower
dimensional space
Data embedding
Jain, B. Kulis, and K. Grauman. Fast Image Search for
Learned Metrics. In CVPR, 2008

19. Some results for LSH
• Caltech-101 data set
• Goal: Exemplar-based Object Categorization
– Some exemplars
– Want to categorize the whole data set

20. Results: object categorization
Caltech-101 database
[CORR]
ML = metric learning
Kristen Grauman et al

21. Question ?
• Is Hashing fast enough?
• Is sub-linear search time fast enough?
• For retrieving (1 + e) near neighbors is bounded
by O(n1/(1+e) )
• Is it fast enough?
• Is it scalable enough? (adapt to the memory of a
PC?)

22. NO!
• Small binary code could do better.
• Cast an image to a compact binary code, with a few
hundred bits per image.
• Small code is possible to perform real-time searches
with millions from the Internet using a single large PC.
• Within 1 second! (for 80 million data  0.146 sec.)
• 80 million data (~300G)  120M

23. Binary Small code
• First introduced in text search/retrieval
• [3] introduced it for text documents retrieval
• Introduced to computer vision by Antonio
Torralba et al [4].
A. Torralba, R. Fergus, and Y. Weiss. Small Codes and Large
Databases for Recognition. In CVPR, 2008.
Semantic Hashing.
Ruslan Salakhutdinov and Geoffrey Hinton
International Journal of Approximate Reasoning, 2009

24. Semantic Hashing
Address Space
Semantically
similar
images
Query address
Semantic
Hash
Function
Query
Binary
code Images in database
Quite different
to a (conventional)
randomizing hash
Fergus et al
Semantic Hashing.
Ruslan Salakhutdinov and Geoffrey Hinton
International Journal of Approximate
Reasoning, 2009

25. Semantic Hashing
• Similar points are mapped into similar small code
• Then store these code into memory and compute
hamming distance (very fast, carried out by hardware)

26. Overall Query Scheme
Query
Image
generate
small
code
Feature
vector
Binary code
Feature vector
Image 1
Store into memory
Use hard ware to
compute hamming distance
Retrieved images <1ms
~1ms (in Matlab)
<10μs
Fergus et al

27. Searching Framework
• Produce binary code (01010011010)
• Store these binary code into the memory
• Use hardware to compute the hamming distance
(very fast)
• Sort the Hamming distances and get final ranking
results

28. The problem is reduced to how to
learn small binary code
• Simplest method (use median)
• LSH are already able to produce binary code
• Restricted Boltzmann Machines (RBM)
• Optimal small binary code by spectral hashing

29. 1. Simple Binarization Strategy
Set threshold (unsupervised)
- e.g. use median
0
1
0 1
Fergus et al

30. 2. Locality Sensitive Hashing
• LSH is ready to generate binary code (unsupervised
• Take random projections of data
• Quantize each projection with few bits
0
1
0
1
0
1
101
No learning involved
Feature vector
Fergus et al

31. 3. RBM [3] to generate code
• Not going into detail, see [3] for detail
• Use a deep neural network to train small code
• Supervised method
R. R. Salakhutdinov and G. E.
Hinton. Learning a
Nonlinear Embedding by
Preserving Class
Neighbourhood Structure.
In AISTATS, 2007.

32. LabelMe retrieval
• LabelMe is a large database with human
annotated images
• The goal of this experiment is to
– First generate small code
– Use hamming distance to search for similar
images
– Sort the results to produce final ranking
• Gist descriptor: ground truth

33. Examples of LabelMe retrieval
• 12 closest neighbors under different distance metrics
Fergus et al

34. Test set 2: Web images
• 12.9 million images from tiny image data set
• Collected from Internet
• No labels, so use Euclidean distance between Gist
vectors as ground truth distance
• Note: Gist descriptor is a kind of feature widely
used in computer vision for

35. Examples of Web retrieval
• 12 neighbors using different distance metrics
Fergus et al

36. Web images retrieval
Observation: more codes get better performance

37. Retrieval Timings
Fergus et al

38. 4. Spectral hashing
• Closely related to the problem of spectral graph
partitioning
• What makes a good code?
– easily computed for a novel input
– requires a small number of bits to code the full
dataset
– maps similar items to similar binary code words
Y. Weiss, A. Torralba, and R. Fergus. Spectral Hashing. In NIPS, 2008.

39. Spectral Hashing
• To simplify the problem, first assume that the
items have already been embedded in a
Euclidean space
• Try to embed the data into a hamming space
• Hamming space is binary space 010101001…
Fergus et al

40. Some definition
• Let be the list of code words (binary
vectors of length k) for n data points
• is the affinity
matrix characterize similarities between data
points.

41. Objective function
• the average Hamming distance between similar
points is minimal
• What does this objective function mean?

42. Objective of Spectral Hashing
the average Hamming distance between
similar neighbors in the Euclidean space
The code is binary
each bit have 50% to be 0 or 1
the bits to be uncorrelated
(bounding condition for the objective)

43. Graph illustration
Nearby points
Near with each other
Euclidean Space
Hamming space

44. Spectral Relaxation
• We obtain an easy problem whose solutions are
simply the k eigenvectors of D − W with minimal
eigenvalue
• Observation: Similar with spectral graph partition
• Could be solved by computing generalized
eigenvalue problem

45. Problem?
• Only tells us how to compute the code
representation of items in the training set
• How about the testing set?
• Computing the code in the testing set is called
the “out-of-sample extension”

46. Recall the problem
• Compute the eigenvector and eigenvalue of the
graph D − W
• Eigenproblem is computational expensive O(n3)
• Could not handle too large data set
• The solution is use eigenfunction

47. One dimensional eigenfunction
• Multi-dimensional eigenfunction is a difficult
problem
• One dimensional eigenfuntion for simple
distribution is well studied!
• For example: for 1D uniform distribution
• (1)
eigenfunction
eigenvalue

48. Finding independent coordinates
• The problem is reduced to find several
independent 1 dimensional coordinates that
– S1 S2 S3…Sk are single coordinates
– Means that the whole distribution could be
separated
– The same with eigenvectors and eigenvalues

49. The spectral hashing algorithm
• Select a setof n data points
• Find k independent coordinates from data
• For each coordinate, assume the data distribution are uniform and
learn analytical eigenfunction by
• Use analytical eigenfunction to learn the eigenvector and
eigenvalue for whole data set
• Choose top k eigenfunctions from all the eigenvectors learned
• Threshold the analytical eigenfunction to obtain binary codes

50. Results for spectral hashing
• Synthetic results on uniform distribution
• LabelMe retrieval results using spectral
hashing to produce small binary code

51. 2-D uniform Toy Example Comparison
Fergus et al

52. Some results on labelme
Observation: spectral hashing get the best performance
Fergus et al

53. Summary
• Image search should be
– Fast
– Accurate
– Scalable
• Tree based methods
• Locality Sensitive Hashing
• Binary Small Code (state of the art)

54. References/Slide credit
1. M. Datar, N. Immorlica, P. Indyk, and V. Mirrokni. Locality-Sensitive
Hashing Scheme Based on p-Stable Distributions. In SOCG, 2004.
2. P. Jain, B. Kulis, and K. Grauman. Fast Image Search for Learned Metrics.
In CVPR, 2008.
3. Ruslan Salakhutdinov and Geoffrey Hinton. Semantic Hashing.
International Journal of Approximate Reasoning, 2009
4. A. Torralba, R. Fergus, and Y. Weiss. Small Codes and Large Databases for
Recognition. In CVPR, 2008.
5. Y. Weiss, A. Torralba, and R. Fergus. Spectral Hashing. In NIPS, 2008.
• Slide credit: Yunchao Gong, UNC Chapel Hill