Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Indexing
Upcoming SlideShare
Loading in...5
×
 

Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Indexing

on

  • 536 views

Paper presentation in ADBIS 2013. ...

Paper presentation in ADBIS 2013.
Abstract: Multimedia data indexing for content-based retrieval has attracted significant attention in recent years due to the commoditization of multimedia capturing equipment and the widespread adoption of social networking platforms as means for sharing media content online. Due to the very large amounts of multimedia content, notably images, produced and shared online by people, a very important requirement for multimedia indexing approaches pertains to their efficiency both in terms of computation and memory usage. A common approach to support query-by-example image search is based on the extraction of visual words from images and their indexing by means of inverted indices, a method proposed and popularized in the field of text retrieval.
The main challenge that visual word indexing systems currently face arises from the fact that it is necessary to build very large visual vocabularies (hundreds of thousands or even millions of words) to support sufficiently precise search. However, when the visual vocabulary is large, the image indexing process becomes computationally expensive due to the fact that the local image descriptors (e.g. SIFT) need to be quantized to the nearest visual words.
To this end, this paper proposes a novel method that significantly decreases the time required for the above quantization process. Instead of using hundreds of thousands of visual words for quantization, the proposed method manages to preserve retrieval quality by using a much
smaller number of words for indexing. This is achieved by the concept of composite words, i.e. assigning multiple words to a local descriptor in ascending order of distance. We evaluate the proposed method in the Oxford and Paris buildings datasets to demonstrate the validity of the
proposed approach.

Statistics

Views

Total Views
536
Views on SlideShare
534
Embed Views
2

Actions

Likes
0
Downloads
7
Comments
0

1 Embed 2

https://twitter.com 2

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Indexing Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Indexing Presentation Transcript

  • ADBIS 2013 Conference Genoa, Italy, Sep 1-4, 2013 Compact and Distinctive Visual Vocabularies for Efficient Multimedia Data Indexing Dimitris Kastrinakis1 , Symeon Papadopoulos2 , Athena Vakali1 1 Aristotle University of Thessaloniki, Department of Informatics (AUTH) 2 Centre for Research and Technology Hellas, Information Technologies Institute (CERTH-ITI)
  • #2 Overview • Problem formulation • Related work • Proposed method • Evaluation • Conclusions
  • #3 Motivation • Multimedia collections are ever-growing – Personal photo collections can easily reach several thousands of photos – Professional photo archives are typically in the range of hundreds of thousands to many millions of photos – Online photos are many billions – It is estimated that a total of 3.5 trillion photos have been captured by people so far • Need for effective and efficient search! – Prevalent paradigm: Content-based image search http://blog.1000memories.com/94-number-of-photos-ever-taken-digital-and-analog-in-shoebox
  • #4 Problem Formulation (1) Content-based image search (CBIR) • Also known as “example-based search”, “similarity- based search” • Given an indexed collection of images and a query image (typically not part of the collection) fetch N images of the collection that are most similar to the query image – similarity is typically computed on the basis of Euclidean distance between feature vectors extracted from the visual content of images
  • #5 Problem Formulation (2) • Recent CBIR systems make use of local descriptors (e.g. SIFT, SURF) extracted from images, according to the BoW retrieval paradigm. – Each image is represented as a set (“bag”) of visual words • Visual words are the result of a learning process (clustering + quantization) on a large set of images – An inverted index structure is used to speed-up retrieval • Such systems achieve good search accuracy, but: – For this to happen, the number of visual words in the vocabulary need to be very large (~105 -106 ) – In case of very large vocabularies, we face two computational problems: (a) creating the vocabularies (offline), and (b) quantizing the local descriptors of a new image to the vocabulary words (online)
  • #6 Overview of BoW indexing Feature extraction (e.g. SIFT, SURF) Feature clustering + quantization Visual vocabulary VOCABULARY LEARNING Feature extraction (e.g. SIFT, SURF) Feature-to- vocabulary mapping TRAINING COLLECTION COLLECTION TO INDEX QUERY IMAGE Feature extraction (e.g. SIFT, SURF) Feature-to- vocabulary mapping Collection index INDEXING RETRIEVAL
  • #7 Our contribution • Propose the concept of Composite Visual Words (CVW) that reduces the need for many visual words – CVW are permutations of plain visual words – Using CVW instead of plain visual words makes it possible to achieve similar search accuracy levels having much fewer visual words in the vocabulary • Experimentally validate our approach in two standard datasets (Oxford and Paris buildings) – With a vocabulary of 200 visual words and the use of CVW we manage to match the retrieval performance of approaches that use two-three orders of magnitude more visual words.
  • #8 Our contribution Feature extraction (e.g. SIFT, SURF) Feature clustering + quantization Visual vocabulary VOCABULARY LEARNING Feature extraction (e.g. SIFT, SURF) Feature-to- vocabulary mapping TRAINING COLLECTION COLLECTION TO INDEX QUERY IMAGE Feature extraction (e.g. SIFT, SURF) Feature-to- vocabulary mapping Collection index INDEXING RETRIEVAL CVW
  • #9 Notation
  • #10 Related Work • First use of BoW approach for image similarity search (Sivic & Zisserman 2003) • Approaches to speed up original method: – Hierarchical vocabulary tree (Nister & Stewenius, 2007) – Approximate k-means clustering (Philbin et al., 2007) • More expressive (and expensive!) representations: – Soft assignment of descriptors to words (Philbin et al., 2008)  requires more index space and time – Visual phrases (Yuan et al., 2007)  does not take into account order of words, expensive mining phase – Words + phrases (Zhang et al., 2009)  high accuracy but computationally intensive learning and still large vocabulary – Vocabulary that maintains spatial layout of words (Zhang et al., 2010), bundled features (Wu et al., 2009)  complex vocabulary generation process + not high coverage of feature space
  • #11 Proposed Approach (1) • Instead of indexing using plain visual words, we index using permutations of visual words based on their relative distance from the image to be indexed PLAIN VISUAL WORDS COMPOSITE VISUAL WORDS
  • #12 Proposed Approach (2) • Important distinction: – Original visual vocabulary (V): Number of visual words that are the result of the clustering and quantization process on the training collection. – Effective visual vocabulary (V’): Number of composite visual words that can be used for indexing. Maximum theoretical size: – For instance, if |V|=100 and B = 3 (the maximum length of a permutation), then |V|’max = 970,200 This is the main way to increase the distinctive capability of the vocabulary without increasing complexity.
  • #13 Proposed Approach (3) • Caveat: If the maximum effective vocabulary size increases a lot (e.g. several millions), the retrieval performance might be harmed since several CVWs will appear very sparsely. • For this, we employ a thresholding strategy to make sure that the resulting CVWs are high-quality. where d(u,f) is the Euclidean distance between local feature f and word u, while max d(u’,f) is the maximum distance between feature f and any word of V. Parameter α controls the “strictness” of the threshold (larger α means stricter thresholding).
  • #14 Algorithmic description of approach
  • #15 Evaluation • Datasets: Oxford and Paris buildings • Implementation: – Local descriptors: SIFT, 2000 features/image (Oxford), 1000 features/image (Paris) – Inverted index: Apache Solr • Evaluation Measure: mean Average Precision (mAP)
  • #16 Results (1) OXFORD PARIS
  • #17 Results (2) EXAMPLE RESULTS (|V|=200) Comparison to SoA
  • #18 Conclusions SUMMARY • Approach to increase distinctive capability of visual vocabulary without harming efficiency. • Validation in standard datasets demonstrates significant reduction in vocabulary size while maintaining state-of-the-art retrieval accuracy. FUTURE WORK • Tests in larger datasets (e.g. using millions of images as distractors) • Alternative thresholding or CVW filtering strategies.
  • #19 References (1) • Classic BoW indexing Sivic, J., Zisserman, A.: Video Google: A text retrieval approach to object matching in videos. Ninth IEEE International Conference on Computer Vision, ICCV (2003) • Efficient BoW indexing Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2. (2006) Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. IEEE Conference on Computer Vision and Pattern Recognition, CVPR’07 (2007)
  • #20 References (2) • Richer BoW-based representations Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization: Improving particular object retrieval in large scale image databases. IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2008) Wu, Z., Ke, Q., Isard, M., Sun, J.: Bundling features for large scale partial- duplicate web image search. IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2009) Yuan, J., Wu, Y., Yang, M.: Discovery of collocation patterns: from visual words to visual phrases. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR’07, 1–8, IEEE (2007) Zhang, S., Tian, Q., Hua, G., Huang, Q., Li, S.: Descriptive visual words and visual phrases for image applications. In Proceedings of the 17th ACM international conference on Multimedia, 75–84, ACM (2009) Zhang, S., Huang, Q., Hua, G., Jiang, S., Gao, W., Tian, Q: Building contextual visual vocabulary for large-scale image applications. In Proceedings of the international conference on Multimedia, 501–510, ACM (2010)
  • #21 Questions Further contact: papadop@iti.gr Acknowledgement: www.socialsensor.eu