Storage and Retrieval Methods and Applications for Multimedia 2005, San Jose, California, USA, 18 - 19 Jan 2005. http://eprints.soton.ac.uk/260419/ This paper presents an investigation into the use of a mobile device as a novel interface to a content-based image retrieval system. The initial development has been based on the concept of using the mobile device in an art gallery for mining data about the exhibits, although a number of other applications are envisaged. The paper presents a novel methodology for performing content-based image retrieval and object recognition from query images that have been degraded by noise and subjected to transformations through the imaging system. The methodology uses techniques inspired from the information retrieval community in order to aid efficient indexing and retrieval. In particular, a vector-space model is used in the efficient indexing of each image, and a two-stage pruning/ranking procedure is used to determine the correct matching image. The retrieval algorithm is shown to outperform a number of existing algorithms when used with query images from the mobile device.

1. Content-based image
retrieval using a mobile
device as a novel
interface
Jonathon S. Hare and Paul H. Lewis
Intelligence,Agents, Multimedia Group
Department of Electronics and Computer Science
University of Southampton
{jsh02r, phl}@ecs.soton.ac.uk

2. Introduction
• This work investigates the use of a mobile
device as a novel interface to a image
retrieval system.
• We develop a new two-stage retrieval
strategy that exhibits impressive retrieval
performance even with the poor imaging
quality of the camera on the mobile device.

3. Content-based Image
Retrieval
• A retrieval system that is robust to the
poor imaging qualities found in low-end
digital cameras.
• Two stage algorithm:
• First stage inspired by Information
Retrieval techniques:Vector-Space model.
• Second stage re-ranking of the first stage
results based on a geometric constraint.

4. Salient Regions for
CBIR
• Retrieval based on
DoG based salient
regions.
• DoG SR’s previous
shown to be robust
to noise, rotation and
other degradation's,
such as those found
in low-end digital
cameras.

5. Local Descriptors
• Lowe’s SIFT Key feature used to describe
image in each salient region.
• Robust to small shifts in salient region
position.
• Robust to illumination changes.
• Doesn’t use colour.

6. Retrieval Using Text
Retrieval Techniques
• We have tried to apply techniques taken
from the information retrieval field:
• Vector-Space Model.
• Documents in the collection are
represented by vectors of term-
frequency.

7. Retrieval Using Text
Retrieval Techniques
• A vocabulary of ‘visual words’ is built, based
on feature vectors from a subset of the
images in the database.
• Each feature vector in the database is
vector quantised to the closest ‘visual
word’.

8. Retrieval Using Text
Retrieval Techniques
• Vector-Space Model.
• Term frequency vectors are weighted:
• TF-IDF.
• WeightedVector is created for the query
image. Documents are ranked by
comparing vectors using cosine similarity.

9. Geometric Consistency
• Second stage of retrieval involves re-
ranking based on geometric constancy of
matching salient regions.
• Use RANSAC to estimate planar
homography between query and target
image, then re-rank on percentage of salient
region pairs fitting the homography.

10. Mobile Client-Server
Implantation
XML-RPC Web Service
Query Engine
Salient
Region &
Feature
Vector
Generator
Feature
Vector and
Metadata
Database
XML-RPC Encoded
query, with embedded
compressed image
XML-RPC Encoded
response, with an
embedded URL

11. Example Application
Viewfinder
Retrieved Metadata

12. Retrieval Performance
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0 20 40 60 80 100
Percentage
Rank
CCV
Colour Histogram
Mono Histogram
PWT
VectorSpace
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90 100
Percentage
N
% Correctly matched
% Incorrectly matched
Performance of retrieval with
second-stage, versus number of
first-stage results considered
Performance of first-stage retrieval,
versus rank of matching image

13. Future Work
• Investigate how to construct an optimal
‘visual’ vocabulary.
• Add a local colour-based descriptor to use
in addition to the SIFT descriptors.
• Investigate the use of “stop-words” and
their effect on retrieval performance.
• Implement an inverted index to aid
efficiency and speed.

14. Conclusions
• We have developed a two-stage image
retrieval algorithm that is able to effectively
retrieve correctly matching images with
query images from low-quality sources,
such as cheap digital cameras.
• The system has been implemented in a
client-server fashion, with a mobile device
used for generating queries and receiving
results.

15. Any Questions?