The document discusses a novel approach to deep image retrieval using a three-stream siamese network and regional maximum activation of convolutions (r-mac) to effectively encode images into compact feature vectors. It highlights the methodology employed, including the use of triplet loss for optimization, and demonstrates the system's efficiency through experiments using various datasets. The study concludes that this technique offers scalability and improved performance in image retrieval tasks compared to prior methods.

1. Deep Image Retrieval:
Learning global representations for image
search
Albert Gordo, Jon Almazan, Jerome Revaud, Diane Larlus
Slides by Albert Jiménez [GDoc]
Computer Vision Reading Group (10/05/2016)1
[arXiv]

2. 1. Introduction
2

3. 3
Instance Retrieval + Ranking
1.
2.
3.
4.
Image Retrieval
Slide credit: Amaia
Ranking
Image
Query

4. CNN-based retrieval
● CNNs trained for classification tasks
● Features are very robust to intra-class variability
● Lack of robustness to scaling, cropping and image clutter
Related Work
Lamp
We are interested in distinguishing between particular objects from the same class!
4

5. R-MAC
● Regional Maximum Activation of Convolutions
● Compact feature vectors encode image regions
Related Work
Giorgos Tolias, Ronan Sicre, Hervé Jégou, Particular object retrieval with integral max-pooling of CNN
activations (Submitted to ICLR 2016)
5

6. R-MAC
● Regions selected using a rigid grid
● Compute a feature vector per region
● Combine all region feature vectors
○ Dimension → 256 / 512
Related Work
Giorgos Tolias, Ronan Sicre, Hervé Jégou, Particular object retrieval with integral max-pooling of CNN
activations (Submitted to ICLR 2016)
ConvNet
Last
Layer
K feature maps
size = W x H
Different scale
region grids
maximum activation
6

7. 2. Methodology
7

8. 1st Contribution
● Three-stream siamese network
● PCA implemented as a shift + fully connected layer
● Optimize weights (CNN + PCA) from R-MAC representation with a triplet
loss function
8

9. where:
● m is a scalar that controls the margin
● q, d+, d- are the descriptors for the query, positive and negative images
1st Contribution
Ranking Loss Function
9

10. 2nd Contribution
● Localize regions of interest (ROIs)
● Train a Region Proposal Network with bounding boxes (Similar Fast R-CNN,
[arXiv])
In R-MAC → Rigid grid
Replace
Region Proposal Network
10

11. 2nd Contribution
RPN in a nutshell
11
● Predict, for a set of candidate boxes of
various sizes and aspects ratio, and at all
possible image locations, a score
describing how likely each box contains an
object of interest.
● Simultaneously, for each candidate box
perform regression to improve its location.

12. Summary
12
● Able to encode one image into a compact feature vector in a single forward
pass
● Images can be compared using the dot product
● Very efficient at test time

13. 3. Experiments
13

14. Datasets
14
● Training Landmarks dataset: 214k images from 672 landmark sites
● Testing Oxford 5k, Paris 6k, Oxford 105k, Paris 106k, INRIA Holidays
● Remove all images contained in Oxford 5k and Paris 6k datasets
○ Landmarks-full: 200k images from 592 landmarks
● Cleaning Landmarks dataset (Select most relevant images/discard incorrect)
○ SIFT + Hessian Affine keypoint det. → Construct graph of similar images
○ Landmarks-clean: 52k images from 592 landmarks

15. Bounding Box Estimation
15
● RPN trained using automatically estimated bounding box annotations
1. Define initial bounding box: min rectangle that encloses all matched keypoints
2. For a pair (i, j) we predict the bounding box Bj using Bi and an affine transform
Aij
3. Update (Merge using geometrical mean)
4. Iterate until convergence
Bounding box projections Initial vs Final estimations

16. Experimental Details
16
● VGG-16 network pre-trained on ImageNet
● Fine-tune with Landmarks dataset
● Select triplets in an efficient manner
○ Forward pass to obtain image representations
○ Select hard negatives (Large loss)
● Dimension of the feature vector = 512
● Evaluation: mean Average Precision (mAP)
VGG16

17. 1st Experiment
17
Comparison between R-MAC and their implementations
C: Classification Network
R: Ranking (Trained with triplets)

18. 2nd Experiment
18
Comparison between fixed grid vs number of region proposals
16-32 proposals already outperform rigid grid!

19. 2nd Experiment
19
meanAP - Number of triplets Recall - Number of region proposals

20. 2nd Experiment
20
Heatmap vs Bounding Box Estimation

21. Comparison with state of the art
21

22. Comparison with state of the art
22

23. Top Retrieval Results
23

24. 4. Conclusions
24

25. Conclusions
25
● They have proposed an effective and scalable method for image retrieval that
encodes images into compact global signatures that can be compared with the
dot-product.
● Proposal of a siamese network architecture trained for the specific task of
image retrieval using ranking loss function (Triplets).
● Demonstrate the benefit of predicting the ROI of the images when encoding by
using Region Proposal Networks.

26. Thank You!
Any Questions?
26