Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Convolutional Features for
Instance Search
Amaia Salvador
03/05/2016
2
Related Publications
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro,
Bags of Local Conv...
Part I
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro,
Bags of Local Convolutional Featur...
Visual Image Retrieval
4Image Database
Visual Query
“A dog”
Expected outcome:
Visual Instance Retrieval
5Image Database
Visual Query
“This dog”
Expected outcome:
Visual Instance Retrieval
6
Image RepresentationsQuery image
Image
Database
Image Matching Ranking List
Similarity score I...
7
v1
= (v11
, …, v1n
)
vk
= (vk1
, …, vkn
)
...
INVERTED FILE
word Image ID
1 1, 12,
2 1, 30, 102
3 10, 12
4 2,3
6 10
...
...
8
Image Representations
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convoluti...
9
Image Representations
Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014). Neural codes for image retrieval....
10
Image Representations
Babenko, A., & Lempitsky, V. (2015). Aggregating local deep features for image retrieval. ICCV 20...
11
Image Representations
Ng, J., Yang, F., & Davis, L. (2015). Exploiting local features from deep networks for image retr...
12
Motivation
Dataset Complexity
TRECVID Instance Search
464 hours of video content
13
Motivation: Image Representations
High-dimensional & Sparse
Bag of Visual Words
Compact & Dense
(e.g. sum/max pooling c...
Methodology
15
Bag of Words Framework
16
Bag of Words Framework
(336x256)
Resolution
conv5_1 from
VGG16[1]
(42x32)
[1]Simonyan K., Zisserman A., Very Deep Convo...
17
Instance Retrieval
Query Representation
... ... ...
... ... ...
Global Search
(GS)
Local Search
(LS)
18
Spatial Reranking
Image RepresentationsQuery image
Image
Database
Image Matching Ranking List
v = (v1
, …, vn
)
v1
= (v...
19
Spatial Reranking
All window combinations with:
Query Image Target image in top M ranking
...
...
20
Query Expansion
Image RepresentationsQuery image
Image
Database
Image Matching Ranking List
v = (v1
, …, vn
)
v1
= (v11...
Experiments
22
Datasets
Paris Buildings 6k Oxford Buildings 5k
TRECVID Instance Search 2013
(subset of 23k frames)
Philbin, J. , Chum,...
23
Results I: SoA Comparison
24
Results II: TRECVid INS
25
Qualitative Results
26
Conclusion
BoW encoding of convolutional features
• High-dimensional sparse representation suitable for fast retrieval
...
Part II
A. Salvador, X. Giro, F. Marques, S. Satoh,
Faster R-CNN Features for Instance Search
28
Reminder: Spatial Reranking
Query Image Target image in top M ranking
...
...
29
Reminder: Spatial Reranking
Koen E. A. van de Sande, Jasper R. R. Uijlings, Theo Gevers, Arnold W. M. Smeulders. Segmen...
30
Image & Region Representations
“dog”
CNN Architectures
plant, table, dog
CNN
CNN
Image Classification
Object Detection
31
Image & Region Representations
Faster R-CNN
Conv
layers
Region Proposal
Network
FC6
Class probabilities
FC7
FC8
RPN Pro...
32
Image & Region Representations
Faster R-CNN
Conv
layers
Region Proposal
Network
FC6
Class probabilities
FC7
FC8
RPN Pro...
33
Image & Region Representations
Image representation Region Representation
(for reranking)
RoI
Pooling
Conv5_3 RoI
Pooli...
34
Fine tuning for query objects
Faster R-CNN
Conv
layers
Region Proposal
Network
FC6
Class probabilities
FC7
FC8
RPN Prop...
35
Fine tuning for query objects
FT #1: Train FC layers only
Conv
layers
Region Proposal
Network
FC6
Class probabilities
F...
36
Fine tuning for query objects
FT #2: Train all weights after conv2
Conv
layers
Region Proposal
Network
FC6
Class probab...
37
Spatial Reranking Strategies
Class-agnostic Spatial Reranking (CA-SR)
Query Image Database
Image
FC6
Class probabilitie...
38
Results
39Query image Top N retrieved images
40
Conclusion
Faster R-CNN for Instance Search
• Suitable to obtain image and region features in a single forward pass
• F...
41
Thank you for your attention !
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro,
Bags of...
Upcoming SlideShare
Loading in …5
×

Convolutional Features for Instance Search

9,029 views

Published on

Presenter: Amaia Salvador

Related papers:
E. Mohedano, Salvador, A., McGuinness, K., Giró-i-Nieto, X., O'Connor, N., and Marqués, F., “Bags of Local Convolutional Features for Scalable Instance Search”, in ACM International Conference on Multimedia Retrieval (ICMR), New York City, NY; USA. 2016

A. Salvador, Giró-i-Nieto, X., Marqués, F., and Satoh, S. 'ichi, “Faster R-CNN Features for Instance Search”, in CVPR Workshop Deep Vision, Las Vegas, NV, USA. 2016.

Abstract:

Image representations derived from pre-trained Convolutional Neural Networks (CNNs) have become the new state of the art in computer vision tasks such as instance retrieval. This work proposes a simple pipeline for encoding the local activations of a convolutional layer of a pre-trained CNN using the well-known bag of words aggregation scheme (BoW). Assigning each local array of activations in a convolutional layer to a visual word produces an assignment map, a compact representation that relates regions of an image with a visual word. We use the assignment map for fast spatial reranking, obtaining object localizations that are used for query expansion. We further investigate the potential of using convolutional features from an object detection network such as Faster R-CNN, which allows to obtain image- and region- wise features in a single forward pass. We demonstrate the suitability of such representations for image retrieval on the Oxford Buildings 5k, Paris Buildings 6k and a subset of TRECVid Instance Search 2013, achieving competitive results. This talk will review the two publications related to this work, which have been recently accepted at ICMR 2016 and DeepVision CVPRW 2016.

Barcelona, 3 May 2016.

Published in: Technology
  • I pasted a website that might be helpful to you: ⇒ www.WritePaper.info ⇐ Good luck!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • There was more choice at auction than I expected and easier to bid than I thought it would be. Thanks for the confidence to buy this way. ➣➣➣ https://w.url.cn/s/Ayy1McS
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ❤❤❤ http://bit.ly/2F7hN3u ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ♥♥♥ http://bit.ly/2F7hN3u ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Convolutional Features for Instance Search

  1. 1. Convolutional Features for Instance Search Amaia Salvador 03/05/2016
  2. 2. 2 Related Publications E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance Search Accepted at ICMR 2016 A. Salvador, X. Giro, F. Marques, S. Satoh, Faster R-CNN Features for Instance Search Accepted at DeepVision CVPRW 2016
  3. 3. Part I E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance Search
  4. 4. Visual Image Retrieval 4Image Database Visual Query “A dog” Expected outcome:
  5. 5. Visual Instance Retrieval 5Image Database Visual Query “This dog” Expected outcome:
  6. 6. Visual Instance Retrieval 6 Image RepresentationsQuery image Image Database Image Matching Ranking List Similarity score Image ... 0.98 0.97 0.10 0.01 v = (v1 , …, vn ) v1 = (v11 , …, v1n ) vk = (vk1 , …, vkn ) ... Similarity Metric (e.g. cosine similarity) ...
  7. 7. 7 v1 = (v11 , …, v1n ) vk = (vk1 , …, vkn ) ... INVERTED FILE word Image ID 1 1, 12, 2 1, 30, 102 3 10, 12 4 2,3 6 10 ... Local hand-crafted features (e.g. SIFT) Bag of Visual WordsN-Dimensional feature space Image Representations High-dimensional Highly sparse
  8. 8. 8 Image Representations Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105). Convolutional Neural Networks
  9. 9. 9 Image Representations Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014). Neural codes for image retrieval. In ECCV 2014 Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: an astounding baseline for recognition. In DeepVision CVPRW 2014 Convolutional Neural Networks FC layers as global feature representation
  10. 10. 10 Image Representations Babenko, A., & Lempitsky, V. (2015). Aggregating local deep features for image retrieval. ICCV 2015 Tolias, G., Sicre, R., & Jégou, H. (2015). Particular object retrieval with integral max-pooling of CNN activations. ICLR 2015 Kalantidis, Y., Mellina, C., & Osindero, S. (2015). Cross-dimensional Weighting for Aggregated Deep Convolutional Features. arXiv preprint arXiv:1512.04065. Convolutional Neural Networks sum/max pooled conv features as global representation
  11. 11. 11 Image Representations Ng, J., Yang, F., & Davis, L. (2015). Exploiting local features from deep networks for image retrieval. In DeepVision CVPRW 2015 Convolutional Neural Networks conv features encoded with VLAD as global representation
  12. 12. 12 Motivation Dataset Complexity TRECVID Instance Search 464 hours of video content
  13. 13. 13 Motivation: Image Representations High-dimensional & Sparse Bag of Visual Words Compact & Dense (e.g. sum/max pooling conv feats, FC feats) Capacity? High-dimensional & Dense (e.g. VLAD encoding) Scalability?
  14. 14. Methodology
  15. 15. 15 Bag of Words Framework
  16. 16. 16 Bag of Words Framework (336x256) Resolution conv5_1 from VGG16[1] (42x32) [1]Simonyan K., Zisserman A., Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv 2014 25K centroids 25K-D vector
  17. 17. 17 Instance Retrieval Query Representation ... ... ... ... ... ... Global Search (GS) Local Search (LS)
  18. 18. 18 Spatial Reranking Image RepresentationsQuery image Image Database Image Matching Ranking List v = (v1 , …, vn ) v1 = (v11 , …, v1n ) vk = (vk1 , …, vkn ) ... Similarity Metric (cosine similarity) ... Top M images are locally analyzed and reranked (M = 100)
  19. 19. 19 Spatial Reranking All window combinations with: Query Image Target image in top M ranking ... ...
  20. 20. 20 Query Expansion Image RepresentationsQuery image Image Database Image Matching Ranking List v = (v1 , …, vn ) v1 = (v11 , …, v1n ) vk = (vk1 , …, vkn ) ... Similarity Metric (cosine similarity) ... Top N images are added to the query for a new search (N = 5)
  21. 21. Experiments
  22. 22. 22 Datasets Paris Buildings 6k Oxford Buildings 5k TRECVID Instance Search 2013 (subset of 23k frames) Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Object retrieval with large vocabularies and fast spatial matching, CVPR 2007 Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases. CVPR 2008 Smeaton, A. F., Over, P., & Kraaij, W. Evaluation campaigns and TRECVid. ACM MM Multimedia information retrieval Workshop 2006
  23. 23. 23 Results I: SoA Comparison
  24. 24. 24 Results II: TRECVid INS
  25. 25. 25 Qualitative Results
  26. 26. 26 Conclusion BoW encoding of convolutional features • High-dimensional sparse representation suitable for fast retrieval • Competitive results in two image retrieval benchmarks • Well suited and more robust for scenarios where only small number of features are in the target images are relevant to the query (INS).
  27. 27. Part II A. Salvador, X. Giro, F. Marques, S. Satoh, Faster R-CNN Features for Instance Search
  28. 28. 28 Reminder: Spatial Reranking Query Image Target image in top M ranking ... ...
  29. 29. 29 Reminder: Spatial Reranking Koen E. A. van de Sande, Jasper R. R. Uijlings, Theo Gevers, Arnold W. M. Smeulders. Segmentation as Selective Search for Object Recognition, ICCV 2011 Object Proposals
  30. 30. 30 Image & Region Representations “dog” CNN Architectures plant, table, dog CNN CNN Image Classification Object Detection
  31. 31. 31 Image & Region Representations Faster R-CNN Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015
  32. 32. 32 Image & Region Representations Faster R-CNN Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015 Image representation Region Representation
  33. 33. 33 Image & Region Representations Image representation Region Representation (for reranking) RoI Pooling Conv5_3 RoI Pooling sum-pooling max-pooling DD
  34. 34. 34 Fine tuning for query objects Faster R-CNN Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals Train object detector for query instances using query images as training data
  35. 35. 35 Fine tuning for query objects FT #1: Train FC layers only Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals
  36. 36. 36 Fine tuning for query objects FT #2: Train all weights after conv2 Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals
  37. 37. 37 Spatial Reranking Strategies Class-agnostic Spatial Reranking (CA-SR) Query Image Database Image FC6 Class probabilities FC7 FC8 ... Class-specific Spatial Reranking (CS-SR)
  38. 38. 38 Results
  39. 39. 39Query image Top N retrieved images
  40. 40. 40 Conclusion Faster R-CNN for Instance Search • Suitable to obtain image and region features in a single forward pass • Fine tuning as an effective solution to boost retrieval performance (subject to application time constraints) Conv layers Region Proposal Network FC6 Class probabilities FC7 FC8 RPN Proposals RoI Pooling Conv5_3 RPN Proposals Image representation Region Representation
  41. 41. 41 Thank you for your attention ! E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance Search Accepted at ICMR 2016 A. Salvador, X. Giro, F. Marques, S. Satoh, Faster R-CNN Features for Instance Search Accepted at DeepVision CVPRW 2016

×