High retrieval precision in content-based image retrieval can be attained by adopting relevance feedback mechanisms. In this paper we propose a weighted similarity measure based on the nearest-neighbor relevance feedback technique that the authors proposed elsewhere. Each image is ranked according to a relevance score depending on nearest-neighbor distances from relevant and non-relevant images. Distances are computed by a weighted measure, the weights being related to the capability of feature spaces of representing relevant images as nearest-neighbors. This approach is proposed to weights individual features, feature subsets, and also to weight relevance scores computed from different feature spaces. Reported results show that the proposed weighting scheme improves the performances with respect to unweighed distances, and to other weighting schemes.

1. PhD Program in Electronic and Computer Engineering
PhD School in Information Engineering
Neighborhood-Based Feature
Weighting for Relevance
Feedback in Content-Based
Retrieval
Luca Piras
luca.piras@diee.unica.it
Giorgio Giacinto
giacinto@diee.unica.it
P R A
Pattern Recognition and Applications Group
Department of Electrical and Electronic Engineering
Pattern Recognition and
Applications Group University of Cagliari, Italy
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 1

2. Outline
• Relevance Feedback
• Image representation
• Weighted similarity measures
• State of the art: Estimation of Feature Relevance
• Neighborhood-Based Feature Weighting
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 2

3. Aim of this work
• Exploiting neighborhood relations to weight
feature sets
• Weight designed to improve Relevance
Feedback based on Distance weighted kth-
Nearest Neighbor
• Dw k-NN estimate the relevance of an image
according to the (non-)relevant one in its
nearest neighborhood
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 3

4. Distance weighted kth-Nearest
Neighbor
pNN (I )
r
I ! NN nr (I )
relevanceNN (I ) = =
p r
NN (I ) + p (I )
nr
NN
I ! NN r (I ) + I ! NN nr (I )
1
where pNN (I ) =
r N
(
V I ! NN r (I ) )
( )
and V I ! NN (I ) " I ! NN (I )
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 4

5. Relevance Feedback
• Query by examples
User
System
Image
Retrieval
Images
database
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 5

6. Relevance Feedback
k best ranked images
User
System
Image
Retrieval
Images
database
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 6

7. Relevance Feedback
image labelling
User
System
Image
Retrieval
Images
database
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 7

8. Image representation
I(F) level
image
F=[ f1 … fi … fF ] feature
color shape texture
f1,1 … f1,i fi,1… fi,j fFi … representation
col. hist. layout color moments co-occurrence texture
f1,1,1 f1,i,1 fF,i,1
. . .
. . .
. . . components
f1,1,j f1,i,j fF,i,j
. . .
. . .
. . .
f1,1,32 f1,i,9 fF,i,16
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 8

9. Image representation
color histogram layout
f1,1 = [g1,1,1, g1,1,2, g1,1,3, g1,1,4 ]
g1,1,1 = [f1,1,1, …, f1,1,8]
g1,1,2 = [f1,1,9, …, f1,1,16]
g1,1,3 = [f1,1,17, …, f1,1,24]
g1,1,4 = [f1,1,25, …, f1,1,32]
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 9

10. Image similarity
1
# &
( ) ( ) ( )
N p
= % " I A fi, j, k ! I B fi, j, k
p
S fi, j ( components (lower)
$ k =1 '
S ( fi ) = ! S fi, j
j
( ) representation
S = ! S ( fi ) feature (higher)
i
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 10

11. Weighted similarity measures
In order to have good performance into images
retrieval systems
• Relevant images should be considered as
neighbors each others.
• Non-relevant images should not be in the
neighborhood of relevant ones.
• Weighted similarity measures.
• Weights related to the capability of feature
spaces of representing relevant images as
nearest-neighbors
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 11

12. Weighted similarity measures
1
# &
( ) ( ) ( )
N p
= % " wi, j, k I A fi, j, k ! I B fi, j, k
p
S fi, j ( components (lower)
$ k =1 '
( )
G
S fi, j = ! wg i d p ( I A , I B )
g
component subset
g =1
j
( )
S ( fi ) = ! wi, j i S fi, j representation
S = ! wi i S ( fi )
i
feature (higher)
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 12

13. State of the art
• Inverse of standard deviation
Rui, Huang, Mehrotra. Int. Conf. on Image Processing , 1997
1
w fj = fj is the j-th feature, !j is its standard deviation
!j
• Probabilistic learning (PFRL)
Peng, Bhanu, Qing. Computer Vision and Image Understanding, 1999
e
(T ir ( z))
fj
w fj = F
rfj(z) is the measure of relevance of the j-th
! e( T irl ( z ))
feature for the query z
l =1
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 13

14. Neighborhood-Based
Feature Weighting
• “Relevance” of different feature space is
estimated in terms of their capability of
representing relevant images as Nearest
Neighbors
• Relevance of an image is estimated according
to the relevant and non-relevant images in its
nearest nieghborhood
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 14

15. Neighborhood-Based
Feature Weighting
" d (I ,N )
fx
wf =
p r
NN (f )
x
= i !R
min i
x
p r
NN (f ) + p (f ) "d
x
nr
NN x
fx
min (I ,R) + " d (I ,N )
i
fx
min i
i !R i !R
1
where p r
(f ) = V
NN x r
NN (f )
x
1
and VNN ( fx ) ! # d min (I i ,R )
r fx
card(R) i "R
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 15

16. Neighborhood-Based
Feature Weighting
• Evaluation of capability to exploit neighborhood
relations in terms of weighted similarity measures
and in terms of weighted relevance score :
– Components level
– Component subset level
( )
S fi, j relevanceNN fi, j ( )
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 16

17. Why better?
• Inverse of standard deviation
– Doesn’t use information about neighborhood of
relevant images
• Probabilistic learning (PFRL)
– It considers only relevant images
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 17

18. Dataset
• Corel 19511 images
• 43 classes (min: 96 - max: 1544 images)
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 18

19. Feature sets
• 4 feature sets
– Co-Occurrence Texture (4x4 subsets)
• 4 directions x 4 values
– Color Moments (3x3 subsets)
• first 3 moments x (H, S, V)
– Color Histogram (4x8 subsets)
• 8 ranges of H x 4 ranges of S
– Color Histogram Layout (4x8 subsets)
• 4 sub-images x 8 color
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 19

20. Experiment setup
• 500 queries
• 9 iterations
• 20 images retrieved each iteration
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 20

21. Legend
Dw 2-NN no weight
* SVM no weight
Dw 2-NN Probabilistic learning
Dw 2-NN Inverse of standard deviation
Dw 2-NN Neighborhood-Based
Dw 2-NN N-Based component subset
Dw 2-NN N-Based Score component subset
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 21

22. Experimental Results
Color Histogram
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 22

23. Experimental Results
Color Histogram
1
F=
1 1
+
2 ! prec 2 ! recall
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 23

24. Experimental Results
Color Histogram (PFRL)
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 24

25. Conclusions
• Reported results show that a weighted measure
improve the performance of the NN technique
• Weighted distance metric based on feature
subset provided the best results
• Neighborhood-Based weights provide similar or
better results with respect to PFRL but without
annoying tuning operations
6-05-2009 Neighborhood-Based Feature Weighting - L. Piras 25