Ralf Klamma , Marc Spaniol Mathias Lux RWTH Aachen University University of Klagenfurt http://www.multimedia-metadata.info I-Media 2007 Graz, September 7, 2007 ImageSemantics : User-Generated Metadata, Content-Based Retrieval & Beyond

Agenda Web 2.0 Image Mining Flickr.com Caliph & Emir ImageSemantics Challenges & Requirements Architecture & Algorithms Rule Representation Conclusions and Outlooks

Web 2.0 Image Mining

Flickr.com

Caliph & Emir

ImageSemantics

Challenges & Requirements

Architecture & Algorithms

Rule Representation

Conclusions and Outlooks

Flickr.com Digital Image Retrieval Results for Tag „gardenia“ Tags: gardenia, green Very dífferent results

Digital Image

Retrieval Results for Tag „gardenia“

Caliph & Emir Digital Image: Retrieved images for low-level features Visualization of low-level features Not very similar

Digital Image:

Retrieved images for low-level features

Web 2.0 Image Mining: ImageSemantics Retrieved Images for Low-level Features and Tags

Retrieved Images for Low-level Features and Tags

Problem Definition (Image Mining) Challenges: Image Search based on Human Semantics Classification of Images Strategies: CBIR Web 2.0 Tagging Multimedia semantics as rule-based system }

Challenges:

Image Search based on Human Semantics

Classification of Images

Strategies:

CBIR

Web 2.0 Tagging

Requirements Download & analyse training sets from Flickr.com Rule extraction for building semantics Semantic Web representation of rules Rule execution in multimedia information system

Download & analyse training sets from Flickr.com

Rule extraction for building semantics

Semantic Web representation of rules

Rule execution in multimedia information system

State-of-the-Art MPEG-7 as a multimedia language Metadata V isual descriptors Image Data Mining / Machine Learning K-Means Clustering Hierarchical Clustering Ontologies / Predicate logic RDF OWL Database Support XML database (eXist, IBM DB2 Viper, Oracle 10g/11g)

MPEG-7 as a multimedia language

Metadata

V isual descriptors

Image Data Mining / Machine Learning

K-Means Clustering

Hierarchical Clustering

Ontologies / Predicate logic

RDF

OWL

Database Support

XML database (eXist, IBM DB2 Viper, Oracle 10g/11g)

Rule Component Data Management Maschine Learning OWL Representation XML Database Reasoner Testing

MPEG-7 Feature Descriptors Metadata / Image URI Tags Visual Descriptors Edge Histogram Scalable Color Color Layout

Metadata / Image URI

Tags

Visual Descriptors

Edge Histogram

Scalable Color

Color Layout

Rule Extraction Algorithm Clustering for low-level features Training data Sub rule extraction for low-level features Clustering for tags in the low-level cluster Sub rule extraction for tags Complete rule extraction

OWL Rule Representation Interval_A_B: <owl:Class rdf:about=&quot;Interval_0_100&quot;> <rdfs:subClassOf rdf:resource=&quot;Interval_A_B&quot;/> <Hasminvalue>0<Hasminvalue> <Hasmaxvalue>100</ Hasmaxvalue> </owl:Class> <owl:DatatypeProperty rdf:about=&quot;Hasminvalue&quot;> <rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/> <rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/> </owl:DatatypeProperty> <owl:DatatypeProperty rdf:about=&quot;Hasmaxvalue&quot;> <rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/> <rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/> </owl:DatatypeProperty> 0 100

Interval_A_B:

<owl:Class rdf:about=&quot;Interval_0_100&quot;>

<rdfs:subClassOf rdf:resource=&quot;Interval_A_B&quot;/>

<Hasminvalue>0<Hasminvalue>

<Hasmaxvalue>100</ Hasmaxvalue>

</owl:Class>

<owl:DatatypeProperty rdf:about=&quot;Hasminvalue&quot;>

<rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/>

<rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/> </owl:DatatypeProperty>

<owl:DatatypeProperty rdf:about=&quot;Hasmaxvalue&quot;>

<rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/>

<rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/>

</owl:DatatypeProperty>

OWL Representation of Rules Sub Rule for low-level features: <owl:Class rdf:about=&quot;LowlevelCluster_A&quot;> <Centriod> Values</Centriod> <rdfs:subClassOf> <owl:Restriction> <owl:onProperty> <owl:ObjectProperty rdf:about=&quot;Distance&quot;/> </owl:onProperty> <owl:allValuesFrom> <owl:Class rdf:about=&quot;Interval_A_B&quot;/> </owl:allValuesFrom> </owl:Restriction> </rdfs:subClassOf> centroid Low-level Cluster max. Distance min. Distance

Sub Rule for low-level features:

<owl:Class rdf:about=&quot;LowlevelCluster_A&quot;>

<Centriod> Values</Centriod>

<rdfs:subClassOf>

<owl:Restriction>

<owl:onProperty>

<owl:ObjectProperty rdf:about=&quot;Distance&quot;/>

</owl:onProperty>

<owl:allValuesFrom>

<owl:Class rdf:about=&quot;Interval_A_B&quot;/>

</owl:allValuesFrom>

</owl:Restriction>

</rdfs:subClassOf>

OWL Representation of Rules Complete rule <owl:Class rdf:about=&quot;Lowlevel_A_TagContent_Cluster&quot;> <Hastag>Tagcontent</Hastag> <rdfs:subClassOf rdf:resource=&quot;LowlevelCluster_A&quot;/> </owl:Class> <owl:DatatypeProperty rdf:about=&quot;Hastag&quot;> <rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#string&quot;/> <rdfs:domain rdf:resource=&quot; Lowlevel_A_Tag_Cluster&quot;/> </owl:DatatypeProperty> OWL dataset representation <LowlevelCluster_A rdf:about=&quot;260407965_5c177d3703.mp7.xml&quot;> <rdf:type rdf:resource=&quot;Lowlevel_A_Tag1_Cluster &quot;/> <rdf:type rdf:resource=&quot;Lowlevel_A_Tag2_Cluster&quot;/> </LowlevelCluster_A> Lowlevel_A_TagContent_Cluster

Complete rule

<owl:Class rdf:about=&quot;Lowlevel_A_TagContent_Cluster&quot;>

<Hastag>Tagcontent</Hastag>

<rdfs:subClassOf rdf:resource=&quot;LowlevelCluster_A&quot;/>

</owl:Class>

<owl:DatatypeProperty rdf:about=&quot;Hastag&quot;>

<rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#string&quot;/>

<rdfs:domain rdf:resource=&quot; Lowlevel_A_Tag_Cluster&quot;/> </owl:DatatypeProperty>

OWL dataset representation

<LowlevelCluster_A rdf:about=&quot;260407965_5c177d3703.mp7.xml&quot;>

<rdf:type rdf:resource=&quot;Lowlevel_A_Tag1_Cluster &quot;/>

<rdf:type rdf:resource=&quot;Lowlevel_A_Tag2_Cluster&quot;/> </LowlevelCluster_A>

Representation of Rules in First Order Logic Sub rule for low-level features Centroid(F), maxdistance(G)  class(LowlevelCluster_A), with F cluster center and G distance value. Complete rules Tag(Tag1)  class(LowlevelCluster_A) (tag  low-level features: if image is tagged with Tag1, image is probably in cluster LowlevelCluster_A). class(LowlevelCluster_A)  Tag(Tag1) (low-level features  tag: if image is in cluster LowlevelCluster_A liegt, image is probably tagged with Tag1).

Sub rule for low-level features

Centroid(F), maxdistance(G)  class(LowlevelCluster_A), with F cluster center and G distance value.

Complete rules

Tag(Tag1)  class(LowlevelCluster_A) (tag  low-level features: if image is tagged with Tag1, image is probably in cluster LowlevelCluster_A).

class(LowlevelCluster_A)  Tag(Tag1) (low-level features  tag: if image is in cluster LowlevelCluster_A liegt, image is probably tagged with Tag1).

Conclusions & Outlook Search & Retrieval of Images by integration of Web 2.0 Tagging (High-level Semantics) CBIR (Low-level Semantics) Rule-based representation of Image Mining Results OWL Syntax Semantic Web Reasoner Outlook: Integration in Multimedia Information System „Virtual Campfire“ Integration in Context-aware Mining Suite

Search & Retrieval of Images by integration of

Web 2.0 Tagging (High-level Semantics)

CBIR (Low-level Semantics)

Rule-based representation of Image Mining Results

OWL Syntax

Semantic Web Reasoner

Outlook:

Integration in Multimedia Information System „Virtual Campfire“

Integration in Context-aware Mining Suite