Indexing Still and Moving Images
Ian Davis
Global Taxonomy Delivery Manager
Dow Jones Consulting Services
© Copyright 2008 Dow Jones and Company | 1

Indexing Still and Moving Images
Still image indexing
Moving image indexing
Two content types can often be treated very
similarly
Two content types also bring unique challenges
© Copyright 2008 Dow Jones and Company | 2

Who uses still images?
 Newspaper and magazine publishers
 Book publishers
 Internet or Intranet sites
 Music labels
 Advertising agencies
 Design houses
 Academics
 Corporates
 TV companies
© Copyright 2008 Dow Jones and Company | 3

What do People Want?
Access to images based on attributes:
e.g. images photographed at a specific time.
e.g. images photographed at a specific place.
e.g. images created using a certain photographic
process.
© Copyright 2008 Dow Jones and Company | 4

What do People Want?
Access to images based on specific depictions:
e.g. 'sun loungers on a tropical beach‘
e.g. 'peacock, full tail display and head’
e.g. 'individuals or couples in modern, clean, bright
kitchens or living rooms - ideally to show blur of
motion.'
© Copyright 2008 Dow Jones and Company | 5

What do People Want?
Access to images based on conceptual terms:
e.g. 'eerie images‘
e.g. 'exciting, speedy images’
e.g. 'images illustrating individuality'
© Copyright 2008 Dow Jones and Company | 6

What do People Want?
Access to images based on basic inherent content:
 Colours
 Textures
 Shapes
© Copyright 2008 Dow Jones and Company | 7

Fitting the system to the users
 Image indexing should focus on the images and
also on the people who will be looking for them.
How images are be indexed should relate closely
to this.
© Copyright 2008 Dow Jones and Company | 8

Fitting the system to the users
Editorial users often appreciate:
 Full captions of two to three sentences
 Detailed factual information
Ad and design users often:
 Care little for concrete facts
 Like to browse for ideas
 Like to search using concepts
© Copyright 2008 Dow Jones and Company | 9

Indexing Strategy
Choose from:
 Manual approach
 Automatic approach
 Combined approach
© Copyright 2008 Dow Jones and Company | 10

Fitting the system to the users
 Understand client needs
 Construct an approach to meet these needs
 Choose from the available techniques and
technologies
 Consider textual or CBIR, or even textual and CBIR
© Copyright 2008 Dow Jones and Company | 11

Methods of Image Classification and Retrieval
Semantic content – textual classification and
retrieval
 Human analysis of images
 Metadata development
 Thesauri or ontology supported classification and
retrieval
Content based image retrieval (CBIR)
 Automatic analysis of low level image attributes
© Copyright 2008 Dow Jones and Company | 12

Manual Image Indexing
Meta-date Coverage – layers of information
 Technical image information
 Basic image attributes
 Image content information
 Image context/aboutness information
© Copyright 2008 Dow Jones and Company | 13

Manual Image Indexing
Technical image information:
 Unique scanning numbers
 Scanner type
 Processor name
 Processing activities
 Batch numbers
 Sizes of image files
© Copyright 2008 Dow Jones and Company | 14

Manual Image Indexing
Basic Image Attributes:
 Photographer name
 Photography Types
 Location photographed
 Date photographed
 Work type
 Points of view
© Copyright 2008 Dow Jones and Company | 15

Manual Image Indexing
Image Content Information:
 Actions and Events
 Animals and Plants
 People: roles, occupations, ethnicity, attributes,
and anatomy
 Topography
 Built works
 Things
© Copyright 2008 Dow Jones and Company | 16

Manual Image Indexing
Image context/aboutness information:
 Emotions
 Abstract concepts
© Copyright 2008 Dow Jones and Company | 17

Automatic Image Indexing
Inherent in image pixels:
 Colours
 Textures
 Simple shapes
© Copyright 2008 Dow Jones and Company | 18

Video Indexing
Layers of information
 Indexing the whole piece
 Indexing the scene
 Indexing the frame
Types of information
 Image information
 Textual information
 Audio information
© Copyright 2008 Dow Jones and Company | 19

Video Indexing
 Dealing with video includes all the challenges of
still image indexing
 Plus the technical issues of capturing textual
data about the content
 And the issue of automatically identifying scenes
and frames
 And the issue of capturing, indexing and linking
audio soundtracks to these scenes and frames.
© Copyright 2008 Dow Jones and Company | 20

Video Indexing
 Much initial textual and audio data can be
captured and indexed automatically
 System can be trained to identify scene breaks
and key frames can be selected
 Focus is often on improving and monitoring the
automated aspects and prioritising what work
needs to be done manually – usually focusing on
key frames/scenes in a piece
© Copyright 2008 Dow Jones and Company | 21

Thank You!
ian.davis@dowjones.com
© Copyright 2008 Dow Jones and Company | 22