Semantics for visual resources: use cases from e-culture

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Semantics for visual resources
Use Cases from E-Culture
Guus Schreiber
Free University Amsterdam
schreiber@cs.vu.nl

2
Purpose
 Analyze a number of use cases from e-culture
domain
– Multimedia plays key role
 Required technology
– Typica...

3
Use case: Asian chairs
User has found an image of an Asian chair
Annotation:
ex:image vra:stylePeriod aat:Guangxu .
How ...

4
AAT info on Guangxu

5
Importance of time and space
information
 Many queries require time/space
knowledge, either absolute or abstracted
 Fo...

8
Sample place information in TGN
<tgn:AdministrativePlace rdf:about="&tgn;1000111"
tgn:standardLatitude="35"
tgn:standard...

9
Issues when searching for
“nearby” Asian chairs
 Close in space:
– Other country in (East) Asia
– Latitude/longitude
 ...

11
Use case: painting style
Find paintings of a similar style
MATISSE, Henri
Le bonheur de vivre (The Joy of Life)
1905-19...

12
How can we find this other Fauve
painting?
DERAIN, Andre
The Turning Road, L'Estaque, 1906
Oil on canvas, 51 x 76 3/4 i...

13
Issues
 Parse annotation to find matches with thesauri
terms
– E.g. match artists to ULAN individuals
 Artists-style ...

16
Search: WordNet patterns that increase recall
without sacrificing precision (Hollink)

17
Issues w.r.t. thesauri
 Public availability!
 RDF/OWL representation
 Learning/specifying term/concept mapping
– owl...

18
Use case: find images with the
same subject
Find another painting which portrays dancing

19
Issues
 Same subjects can be visually very
different
 Subject is often missing from the
annotation
 Mismatch: users ...

20
Conceptual subject descriptions
85% of the user queries:
General Descriptions of generally known items. Only general,
e...

21
Example concepts in image
 Specific
– Fall of the Berlin Wall
 General
– People walking at night
 Abstract
– Fall of...

22
Use of conceptual categories by
people searching for images
Conceptual level: 83%
0%
20%
40%
60%
80%
100%
event time pl...

23
Thesauri for scenes: Iconclass

26
Annotation of image content
 Template for subject description
Agent Action Object Recipient
 Guidelines for manual an...

29
Some forms of image content are
well suited to image analysis
Collection of clothes
Abstract painting

30
The semantic gap
 The distance between Content-Based Image
Retrieval and semantics:
– Smeulders, Worring, Santini, Gup...

31
Example semantic bridge:
microscopic cell images
mpeg7 : StillRegion(region) ^
mpeg7x : Dense(region) ^
mpeg7 : Dominan...

32
Segmentation often requires
user interaction

33
Automatic detection of concepts can be
difficult even in “easy” cases
What is the color
of this ape?

34
Image analysis useful for
collection navigation

35
Bridging the semantic gap:
CBIR and ontologies
Visual WordNet (GE paper)
– Adding knowledge about visual characteristic...

36
Sample visual features and their
mapping to WordNet

37
Experiment: pruning the search
for “conveyance” concepts
6 concepts found
Including taxi cab
12 concepts found
Includin...

38
Bridging the semantic gap:
concept detectors
 Snoek et al., TRECVID2004
– 185 hours of news video
 32 detectors for c...

39
“Concepts” for which visual
detectors were built

40
LSCOM lexicon: 229 - Weather
 Context-specific (i.e.
news broadcast)
interpretation:
“Weather forecast”

41
LSCOM lexicon: 110 – Female Anchor
 Composite concept
 Alignment needed for
semantic search, e.g.
with WordNet

42
Natural-lang proc.
automatic annotation
text stings → concepts
Distributed
cultuurwijzer.nl collections
OAI-based acces...

44
Main observation
A combination of many different techniques
is needed to be able to cope with the
complexity of multime...

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

Share
Email

How the Semantic Web is transformin... by Guus Schreiber 3116 views
Principles for knowledge engineerin... by Guus Schreiber 8161 views
Principles and pragmatics of a Sem... by Guus Schreiber 8219 views
Web Science: the digital heritage case by Guus Schreiber 8229 views
The artof of knowledge engineering,... by Guus Schreiber 8350 views
Ontologies for multimedia: the Sema... by Guus Schreiber 8128 views

Top clipped slide

View on SlideShare

1 of 44 Ad

1 of 44 Ad