Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Giuseppe Amato, Fabrizio Falchi, Claudio Gennaro and Fausto Rabitti
fabrizio.falchi@cnr.it
YFCC100M HYBRIDNET FC6 DEEP FEA...
fabrizio.falchi@cnr.it
WHERE WE COME FROM AND MOTIVATIONS
CoPhIR – Content-based Photo Image Retrieval
http://cophir.isti....
fabrizio.falchi@cnr.it
MAJOR RELATED EVENTS
Deep Learning explosion
YFCC100M
The Multimedia Commons Initiative
fabrizio.falchi@cnr.it
CONTRIBUTIONS
• HybridNet fc6 Deep Features for YFCC100M images
multimediacommons.wordpress.com/
• ...
fabrizio.falchi@cnr.it
HYBRIDNET
• Trained on 3.5 million images from 1,183 categories:
o ImageNet-ILSVRC
• about 1 millio...
fabrizio.falchi@cnr.it
WHY HYBRIDNET FC6?
A Practical Guide to CNNs and Fisher Vectors for Image Instance Retrieval
V Chan...
fabrizio.falchi@cnr.it
DEEP FEATURES PROCESSING
• We generated 3 distinct features from the fc6 activations:
o Raw (no ReL...
fabrizio.falchi@cnr.it
fabrizio.falchi@cnr.it
GT RESULTS www.deepfeature.org
fabrizio.falchi@cnr.it
GT RESULTS (SEQUENTIAL SCANNNING)
fabrizio.falchi@cnr.it
GT RESULTS (SEQUENTIAL SCANNNING)
fabrizio.falchi@cnr.it
APPROXIMATE CBIR RESULTS
MI-FileLucene
Quantization
fabrizio.falchi@cnr.it
THE CBIR ONLINE SYSTMES
• MI-File
o Permutation Based method
o Uses Inverted Files
MI-File: using i...
fabrizio.falchi@cnr.it
MI-FILE (INDEXING BINARY FEATURES)
fabrizio.falchi@cnr.it
LUCENE QUANTIZATION (INDEXING RELU L2NORM.)
fabrizio.falchi@cnr.it
MI-FILE (COMPARED TO GT FOR RELU-L2NORM)
ONGOING WORKS
fabrizio.falchi@cnr.it
IMAGE ANNOTATION
fabrizio.falchi@cnr.it
CROSS MEDIA RETRIEVAL (RESULTS ON MS-COCO)
• Text queries are translated in HybridNet fc6 Visual Ve...
fabrizio.falchi@cnr.it
CROSS MEDIA RETRIEVAL (RESULTS ON YFCC100M)
Picture It In Your Mind: Generating High Level Visual R...
fabrizio.falchi@cnr.it
CONCLUSIONS AND FUTURE WORK
Contributions:
• HybridNet fc6 Deep Features
• CBIR Systems for YFCC100...
fabrizio.falchi@cnr.it
THANKS!
Questions are welcomed
Fabrizio Falchi
fabrizio.falchi@cnr.it
Upcoming SlideShare
Loading in …5
×

YFCC100M HybridNet fc6 Deep Features for Content-Based Image Retrieval

6,575 views

Published on

Presentation given at the Multimedia COMMONS 2016, A Workshop of ACM Multimedia 2016, Amsterdam, The Netherlands, October 16

Published in: Science
  • Be the first to comment

  • Be the first to like this

YFCC100M HybridNet fc6 Deep Features for Content-Based Image Retrieval

  1. 1. Giuseppe Amato, Fabrizio Falchi, Claudio Gennaro and Fausto Rabitti fabrizio.falchi@cnr.it YFCC100M HYBRIDNET FC6 DEEP FEATURES FOR CONTENT-BASED IMAGE RETRIEVAL Multimedia COMMONS Workshop at ACM Multimedia 2016 Amsterdam, The Netherlands, October 15-19
  2. 2. fabrizio.falchi@cnr.it WHERE WE COME FROM AND MOTIVATIONS CoPhIR – Content-based Photo Image Retrieval http://cophir.isti.cnr.it • Flickr 106M Photos (not all CC) • title, description, author, tags, comments, notes, and also its GPS, coordinates, the number of views and the number of users considering the photo a favorite • MPEG-7 Visual Features • mainly used by the Similarity Search community (144 citations and about 100 requests) Similarity Search The Metric Space Approach Zezula, Amato, Dohnal, Batko 2008
  3. 3. fabrizio.falchi@cnr.it MAJOR RELATED EVENTS Deep Learning explosion YFCC100M The Multimedia Commons Initiative
  4. 4. fabrizio.falchi@cnr.it CONTRIBUTIONS • HybridNet fc6 Deep Features for YFCC100M images multimediacommons.wordpress.com/ • CBIR Systems on the YFCC100M o MI-File mifile.deepfeatures.org o Lucene Quantization melisandre.deepfeatures.org • Ground-truth Results for evaluating Approximate k-NN (k=10,001) www.deepfeatures.org/ o On 3 types of the neuron activations (features) processing o For subsets of the whole collections at each 1M step
  5. 5. fabrizio.falchi@cnr.it HYBRIDNET • Trained on 3.5 million images from 1,183 categories: o ImageNet-ILSVRC • about 1 million images from 888 categories (removing Places 295 duplicates) o Places 205 • about 2.5 million images from 205 categories Learning Deep Features for Scene Recognition using Places Database Zhou, Lapedriza, Xiao, Torralba, Oliva, NIPS 2014
  6. 6. fabrizio.falchi@cnr.it WHY HYBRIDNET FC6? A Practical Guide to CNNs and Fisher Vectors for Image Instance Retrieval V Chandrasekhar, J Lin, O Morère, H Goh, A Veillard - Signal Processing, 2016 - Elsevier
  7. 7. fabrizio.falchi@cnr.it DEEP FEATURES PROCESSING • We generated 3 distinct features from the fc6 activations: o Raw (no ReLu) + L2Norm. o ReLu + L2Norm. o Binary A simple binarization of deep features was shown to lead to a negligible performance drop for both classification and detection (PASCAL-CLS in particular). 𝑏𝑖 = 1 𝑓𝑖 > 0 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 Analyzing the performance of multilayer neural networks for object recognition. P. Agrawal, R. Girshick, and J. Malik. (ECCV 2014)
  8. 8. fabrizio.falchi@cnr.it
  9. 9. fabrizio.falchi@cnr.it GT RESULTS www.deepfeature.org
  10. 10. fabrizio.falchi@cnr.it GT RESULTS (SEQUENTIAL SCANNNING)
  11. 11. fabrizio.falchi@cnr.it GT RESULTS (SEQUENTIAL SCANNNING)
  12. 12. fabrizio.falchi@cnr.it APPROXIMATE CBIR RESULTS MI-FileLucene Quantization
  13. 13. fabrizio.falchi@cnr.it THE CBIR ONLINE SYSTMES • MI-File o Permutation Based method o Uses Inverted Files MI-File: using inverted files for scalable approximate similarity search G Amato, C Gennaro, P Savino (Multimedia tools and applications) • Lucene Quantization o Exploits the sparsity of deep features (ReLu -> 25% non zeros) o Quantization approach to allow text encoding o Also able to perform text and combined search Large scale indexing and searching deep convolutional neural network features G. Amato, F. Debole, F. Falchi, C. Gennaro, and F. Rabitti (DaWaK 2016)
  14. 14. fabrizio.falchi@cnr.it MI-FILE (INDEXING BINARY FEATURES)
  15. 15. fabrizio.falchi@cnr.it LUCENE QUANTIZATION (INDEXING RELU L2NORM.)
  16. 16. fabrizio.falchi@cnr.it MI-FILE (COMPARED TO GT FOR RELU-L2NORM)
  17. 17. ONGOING WORKS
  18. 18. fabrizio.falchi@cnr.it IMAGE ANNOTATION
  19. 19. fabrizio.falchi@cnr.it CROSS MEDIA RETRIEVAL (RESULTS ON MS-COCO) • Text queries are translated in HybridNet fc6 Visual Vectors by a NN Picture It In Your Mind: Generating High Level Visual Representations From Textual Descriptions Fabio Carrara, Andrea Esuli, Tiziano Fagni, Fabrizio Falchi, Alejandro Moreo Fernández https://arxiv.org/abs/1606.07287
  20. 20. fabrizio.falchi@cnr.it CROSS MEDIA RETRIEVAL (RESULTS ON YFCC100M) Picture It In Your Mind: Generating High Level Visual Representations From Textual Descriptions Fabio Carrara, Andrea Esuli, Tiziano Fagni, Fabrizio Falchi, Alejandro Moreo Fernández https://arxiv.org/abs/1606.07287
  21. 21. fabrizio.falchi@cnr.it CONCLUSIONS AND FUTURE WORK Contributions: • HybridNet fc6 Deep Features • CBIR Systems for YFCC100M: o MI-File mifile.deepfeatures.org o Lucene Quantization melisandre.deepfeatures.org • GT k-NN results for evaluating Approximate Search www.deepfeatures.org/ Ongoing and future works: • HybridNet fc6 PCA256 • Image annotation based on the YFCC100M metadata • Extracting new features, e.g.: Deep Image Retrieval: Learning Global Representations for Image Search Albert Gordo, Xerox Research; Jon Almazan, XRCE; Jerome Revaud, Xerox Research; Diane Larlus, Xerox • Cross-media retrieval Picture It In Your Mind: Generating High Level Visual Representations From Textual Descriptions Fabio Carrara, Andrea Esuli, Tiziano Fagni, Fabrizio Falchi, Alejandro Moreo Fernández https://arxiv.org/abs/1606.07287
  22. 22. fabrizio.falchi@cnr.it THANKS! Questions are welcomed Fabrizio Falchi fabrizio.falchi@cnr.it

×