Multimedia and the Semantic Web / European Semantic Web Conference 2005, Heraklion, Crete. 29th May 2005. http://eprints.soton.ac.uk/260954/ In this paper, we propose a model of automatic image annotation based on propagation of keywords. The model works on the premise that visually similar image content is likely to have similar semantic content. Image content is extracted using local descriptors at salient points within the image and quantising the feature-vectors into visual terms. The visual terms for each image are modelled using techniques taken from the information retrieval community. The modelled information from an unlabelled query image is compared to the models of a corpus of labelled images and labels are propagated from the most similar labelled images to the query image

1. Saliency-based Models of Image
Content and their Application to
Auto-Annotation by Semantic
Propagation
Jonathon S. Hare and Paul H. Lewis
Intelligence, Agents, Multimedia Group
Department of Electronics and Computer Science
University of Southampton
{jsh02r, phl}@ecs.soton.ac.uk

2. Introduction
• Image search is much easier if all of the images in a collection
have adequate annotations
• Annotating images manually is a time-consuming and
laborious process
• We propose a simple method for automatic image
annotation based on the idea that visually similar images
should have similar annotations
• Visual similarity is assessed using local descriptors of salient
regions in an information model
• We also address how the quality of the automatic
annotations can be assessed

3. Image Auto-Annotation
Propagating the Semantics (I)
• Simple idea based on propagating semantics between
visually similar images
• Image content is modelled using concepts from
information theory
• Modelling the images in this way allows us to assess
image similarity in a well-founded framework
• We tried two models:
• Vector-Space
• Latent Semantic Indexing

4. Image Auto-Annotation
Propagating the Semantics (II)
• A training corpus of pre-annotated images is created
• An unannotated image is compared to images in the
training corpus using the information models to find
visually similar images
• The annotations from a number of the closest visually
similar images in the training corpus are then applied, or
propagated, to the unannotated image

5. Information Theory
Modelling Textual Information::vector-spaces
• A common approach to modelling documents containing
textual information is to use a vector-space
• Each document is represented by a vector of term-
occurrences
• Each element of the vector is the count of the
number of times the corresponding term occurred in
the document
• Similar documents should have similar vectors
• i.e. the angle between vectors is small

6. Information Theory
Modelling Textual Information::latent semantic indexing
• LSI takes the vector-space model a step further
• LSI attempts to deal with issues of synonymy and
polysemy between terms by using linear algebra
• Term-occurrence vectors are arranged in a term-
document matrix and factored using SVD
• Using the resulting matrices from the SVD it is
possible to create a rank-k estimate of the original
term-document matrix
• Queries can be performed in the k-subspace instead
of the original space

7. Modelling Images
Representing Images using Textual Information Models
• Salient regions are found in
the image from peaks in a
difference-of-Gaussian
pyramid
• Local descriptors (SIFT) are
calculated for each region
• The feature descriptors are
quantised by assigning them to
the closest visual terms in a
predefined vocabulary
• Term-occurrence vectors are
calculated

8. Experimentation
Dataset
• Test dataset consisted of 697
annotated photographic
images
• Original annotations were
processed to remove plurals
and correct mistakes
• 170 processed annotation
terms in total
• Training and test sets were
created by randomly cutting
the dataset into halves

9. Experimentation
Evaluation Technique::considerations (I)
• For any auto-annotation system to be worthwhile it must
perform better than if the annotations were guessed
based on the empirical distribution of keywords in the
training set

10. Experimentation
Evaluation Technique::considerations (II)
• Images in the training set may have been incorrectly
annotated
• For comparative purposes this is not a problem as all
algorithms have to deal with the same data
• However, the reported overall performance is likely to
be less than it would be with correct annotations

11. Experimentation
Evaluation Technique::performance measure
• A reasonable assumption to make of
an annotation algorithm should be
that it will return approximately the
correct number of annotations
• Previous auto-annotation work has
used the normalised score measure
developed by Barnard et al.
• However, this measure does not
sufficiently weight incorrect
guesses, resulting in many more
guessed annotations than correct
annotations when the score is
maximised
r = Number of correctly predicted
annotations
n = Number of actual true annotations
w = Number of wrongly predicted
annotations
N = Number of terms in annotation
vocabulary

12. Experimentation
Evaluation Technique::performance measure
• We have instead adopted the
use of precision and recall to
measure performance
• Unlike in retrieval, we want
both high precision and high
recall
r = Number of correctly predicted
annotations
n = Number of actual true annotations
w = Number of wrongly predicted
annotations

13. Experimentation
Results (I)
• Two models, using
vocabularies of 3000 visual
terms
• Vector-Space
• LSI (K=40)

14. Experimentation
Results (II)
True Annotations Tree, Bush, Sky Temple, Sky
Flower, Bush, Tree,
Building, Sidewalk
Empirical
Annotations
Tree, Building, People,
Bush, Grass
Tree, Building, People,
Bush, Grass
Tree, Building, People,
Bush, Grass
Vector-Space
Annotations
Tree, Bush
Tree, Building, Grass,
Sidewalk, Pole, People,
Clear Sky
Flower, Bush, Tree,
Building, Partially Cloudy
Sky
LSI Annotations
Tree, Bush, Grass,
Sidewalk
Steps, Wall
Flower, Bush, Tree,
Ground

15. Future Work
(I)
• Our current annotation approach is slightly deficient in
that it doesn’t allow us to select individual terms
• This needs to be addressed
• This work used a fixed number of images from which to
draw the annotations
• It is possible that this number could be chosen
dynamically for each unannotated image, based on the
similarity between its vector representation and the
vectors of the training images

16. Future Work
(II)
• The SIFT local descriptor is generated from grey-level
information only
• The addition of other local descriptors may improve
performance
• A local colour descriptor is currently planned

17. Conclusions
• The results show promise for our relatively simple auto-
annotation technique
• The LSI based method marginally outperforms the vector-
space approach
• This result confirms the findings of our work on image
retrieval using the two approaches