European Conference on Operational Research, 2012, Vilinius Lithuania

1. Semantically Relevant Visual Dictionary
Ashish Gupta (CVSSP)
University of Surrey
a.gupta@surrey.ac.uk
July 10,2012
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

2. Contents
Introduction: Visual Category Recognition
Current practice: Visual Dictionary
Problem: inter-mixed feature vectors
Approach: Over-partition + Co-cluster image-word matrix
Solution: Group estimated categorically related partitions
Experiments:
Summary
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

3. Visual Category Recognition
Definition
Detect presence of an instance of a
visual category in an image.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

4. Challenges
Several variations in visual category appearance render category
recognition very difficult.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

5. Visual Dictionary
Visual Word
Representative feature vector
(generally centroid) of each
cluster.
Image Histogram
Histogram of assignments of
image feature vectors to visual
words.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

6. Problems with Visual Dictionary
Inter-mixed
Categorically dissimilar feature vectors inter-mixed in feature space.
Semantic scatter
Feature vectors pertaining to same category part scattered in
feature space.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

7. Inter-mixed Feature Vectors
Categorically equivalent
vectors mapped to naturally
occurring clusters
Easily partitioned to yield
discriminative dictionary
elements
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

8. Inter-mixed Feature Vectors
Categorically dissimilar vectors
inter-mixed
Partitioning yields
non-discriminative dictionary
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

9. Inter-mixed Feature Vectors
Over-partition feature space into tiny clusters.
Build a dictionary using these tiny clusters.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

10. Semantic Scatter
Small variations in instances of object part causes associated
descriptors to get scattered in feature space.
Combine visual words which are related and create a visual
topic.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

11. Hypothesis
Semantically related words can be discovered by analysing
image-word distribution.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

12. Visual Topic Dictionary ← Visual Word Dictionary
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

13. Co-Clustering
Formulate the image-word matrix as a joint probability distribution.
CX : {x1, x2, . . . , xm} → { ˆx1, ˆx2, . . . , ˆxk }
CY : {y1, y2, . . . , yn} → { ˆy1, ˆy2, . . . , ˆyl }
the tuple (CX , CY ) is referred to as co-clustering.
‘re-order’ rows and columns of the matrix, which gives rise to
blocks, referred to as co-clusters.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

14. Co-clustering contd.
Optimal co-clustering minimizes loss in mutual information
I(X; Y ) − I( ˆX; ˆY ), given number of row (k) and column (l)
clusters.
For a (CX , CY ), loss in mutual information can be expressed by
KL-divergence between p(X, Y ) and an approximation q(X, Y ).
I(X; Y ) − I( ˆX; ˆY ) = DKL(p(X, Y ) q(X, Y ))
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

15. Conceptual view
Image histogram feature vectors in high-dimensional visual words
space are projected to lower dimensional visual topic space.
The distance between feature vectors from the same category is
reduced.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

16. Experiment
Feature descriptor
SIFT : Affine co-variant local image patch descriptor.
Data sets
Scene-15; Pascal VOC 2006; VOC 2007; VOC 2010.
Classifier
k-NN : Verify if mutual distance between categorically equivalent
feature vectors is reduced.
Performance metric
F1-score: harmonic mean of precision and recall. Popularly used in
classification and retrieval communities.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

17. Scene-15 Dataset
It has 15 visual categories of natural indoor and outdoor scenes.
Each category has about 200 to 400 images and the entire dataset
has 4485 images.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

18. PASCAL VOC2006 Dataset
It has 10 visual categories with about 175 to 650 images per
category. There are a total of 5304 images.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

19. PASCAL VOC2007 Dataset
It has 20 visual categories. Each category contains images ranging
from 100 to 2000, with 9963 images in all.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

20. PASCAL VOC2010 Dataset
It has 20 visual categories and 300 to 3500 images in each
category. Combines data from VOC2008 and VOC2009.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

21. Dictionary Size
10,000 words → n Topics. Appropriate number of Topics?
Large dictionary becomes category dependent.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

22. Summary
Visual dictionary in limited: unsupervised clustering.
Significant intra-category appearance variation: semantic scatter.
Feature vectors from different visual categories inter-mixed in
feature space.
Visual Topic ← Visual Word: grouping over-partitioned feature
space.
Co-clustering Image-Word distribution: discover optimal grouping
of words with minimal loss in mutual information.
Semantic dimensionality reduction.
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

23. Thank you.
Acknowledgement
Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary