ImageSemantics : User-Generated Metadata, Content-Based Retrieval & Beyond

2,198 views

Published on

ImageSemantics : User-Generated Metadata, Content-Based Retrieval & Beyond
Marc Spaniol, Ralf Klamma, Mathias Lux
TRIPLE-I, Graz, Austria, September 7, 2007

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,198
On SlideShare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
1
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • ,
  • ImageSemantics : User-Generated Metadata, Content-Based Retrieval & Beyond

    1. 1. Ralf Klamma , Marc Spaniol Mathias Lux RWTH Aachen University University of Klagenfurt http://www.multimedia-metadata.info I-Media 2007 Graz, September 7, 2007 ImageSemantics : User-Generated Metadata, Content-Based Retrieval & Beyond
    2. 2. Agenda <ul><li>Web 2.0 Image Mining </li></ul><ul><ul><li>Flickr.com </li></ul></ul><ul><ul><li>Caliph & Emir </li></ul></ul><ul><li>ImageSemantics </li></ul><ul><ul><li>Challenges & Requirements </li></ul></ul><ul><ul><li>Architecture & Algorithms </li></ul></ul><ul><ul><li>Rule Representation </li></ul></ul><ul><li>Conclusions and Outlooks </li></ul>
    3. 3. Flickr.com <ul><li>Digital Image </li></ul><ul><li>Retrieval Results for Tag „gardenia“ </li></ul>Tags: gardenia, green Very dífferent results
    4. 4. Caliph & Emir <ul><li>Digital Image: </li></ul><ul><li>Retrieved images for low-level features </li></ul>Visualization of low-level features Not very similar
    5. 5. Web 2.0 Image Mining: ImageSemantics <ul><li>Retrieved Images for Low-level Features and Tags </li></ul>
    6. 6. Problem Definition (Image Mining) <ul><li>Challenges: </li></ul><ul><ul><li>Image Search based on Human Semantics </li></ul></ul><ul><ul><li>Classification of Images </li></ul></ul><ul><li>Strategies: </li></ul><ul><ul><li>CBIR </li></ul></ul><ul><ul><li>Web 2.0 Tagging </li></ul></ul>Multimedia semantics as rule-based system }
    7. 7. Requirements <ul><li>Download & analyse training sets from Flickr.com </li></ul><ul><li>Rule extraction for building semantics </li></ul><ul><li>Semantic Web representation of rules </li></ul><ul><li>Rule execution in multimedia information system </li></ul>
    8. 8. State-of-the-Art <ul><li>MPEG-7 as a multimedia language </li></ul><ul><ul><li>Metadata </li></ul></ul><ul><ul><li>V isual descriptors </li></ul></ul><ul><li>Image Data Mining / Machine Learning </li></ul><ul><ul><li>K-Means Clustering </li></ul></ul><ul><ul><li>Hierarchical Clustering </li></ul></ul><ul><li>Ontologies / Predicate logic </li></ul><ul><ul><li>RDF </li></ul></ul><ul><ul><li>OWL </li></ul></ul><ul><li>Database Support </li></ul><ul><ul><li>XML database (eXist, IBM DB2 Viper, Oracle 10g/11g) </li></ul></ul>
    9. 9. Rule Component Data Management Maschine Learning OWL Representation XML Database Reasoner Testing
    10. 10. MPEG-7 Feature Descriptors <ul><li>Metadata / Image URI </li></ul><ul><li>Tags </li></ul><ul><li>Visual Descriptors </li></ul><ul><ul><li>Edge Histogram </li></ul></ul><ul><ul><li>Scalable Color </li></ul></ul><ul><ul><li>Color Layout </li></ul></ul>
    11. 11. Rule Extraction Algorithm Clustering for low-level features Training data Sub rule extraction for low-level features Clustering for tags in the low-level cluster Sub rule extraction for tags Complete rule extraction
    12. 12. OWL Rule Representation <ul><li>Interval_A_B: </li></ul><ul><li><owl:Class rdf:about=&quot;Interval_0_100&quot;> </li></ul><ul><li><rdfs:subClassOf rdf:resource=&quot;Interval_A_B&quot;/> </li></ul><ul><li><Hasminvalue>0<Hasminvalue> </li></ul><ul><li><Hasmaxvalue>100</ Hasmaxvalue> </li></ul><ul><li></owl:Class> </li></ul><ul><li><owl:DatatypeProperty rdf:about=&quot;Hasminvalue&quot;> </li></ul><ul><li><rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/> </li></ul><ul><li><rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/> </owl:DatatypeProperty> </li></ul><ul><li>  <owl:DatatypeProperty rdf:about=&quot;Hasmaxvalue&quot;> </li></ul><ul><li><rdfs:domain rdf:resource=&quot;Interval_A_B&quot;/> </li></ul><ul><li><rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#double&quot;/> </li></ul><ul><li></owl:DatatypeProperty> </li></ul>0 100
    13. 13. OWL Representation of Rules <ul><li>Sub Rule for low-level features: </li></ul><ul><li><owl:Class rdf:about=&quot;LowlevelCluster_A&quot;> </li></ul><ul><li><Centriod> Values</Centriod> </li></ul><ul><li><rdfs:subClassOf> </li></ul><ul><li><owl:Restriction> </li></ul><ul><li><owl:onProperty> </li></ul><ul><li><owl:ObjectProperty rdf:about=&quot;Distance&quot;/> </li></ul><ul><li></owl:onProperty> </li></ul><ul><li><owl:allValuesFrom> </li></ul><ul><li><owl:Class rdf:about=&quot;Interval_A_B&quot;/> </li></ul><ul><li></owl:allValuesFrom> </li></ul><ul><li></owl:Restriction> </li></ul><ul><li></rdfs:subClassOf> </li></ul>centroid Low-level Cluster max. Distance min. Distance
    14. 14. OWL Representation of Rules <ul><li>Complete rule </li></ul><ul><li><owl:Class rdf:about=&quot;Lowlevel_A_TagContent_Cluster&quot;> </li></ul><ul><li><Hastag>Tagcontent</Hastag> </li></ul><ul><li><rdfs:subClassOf rdf:resource=&quot;LowlevelCluster_A&quot;/> </li></ul><ul><li></owl:Class>  </li></ul><ul><li><owl:DatatypeProperty rdf:about=&quot;Hastag&quot;> </li></ul><ul><li><rdfs:range rdf:resource=&quot;http://www.w3.org/2001/XMLSchema#string&quot;/> </li></ul><ul><li><rdfs:domain rdf:resource=&quot; Lowlevel_A_Tag_Cluster&quot;/> </owl:DatatypeProperty> </li></ul><ul><li>OWL dataset representation </li></ul><ul><li><LowlevelCluster_A rdf:about=&quot;260407965_5c177d3703.mp7.xml&quot;> </li></ul><ul><li><rdf:type rdf:resource=&quot;Lowlevel_A_Tag1_Cluster &quot;/> </li></ul><ul><li><rdf:type rdf:resource=&quot;Lowlevel_A_Tag2_Cluster&quot;/> </LowlevelCluster_A> </li></ul>Lowlevel_A_TagContent_Cluster
    15. 15. Representation of Rules in First Order Logic <ul><li>Sub rule for low-level features </li></ul><ul><li>Centroid(F), maxdistance(G)  class(LowlevelCluster_A), with F cluster center and G distance value. </li></ul><ul><li>Complete rules </li></ul><ul><li>Tag(Tag1)  class(LowlevelCluster_A) (tag  low-level features: if image is tagged with Tag1, image is probably in cluster LowlevelCluster_A).  </li></ul><ul><li>class(LowlevelCluster_A)  Tag(Tag1) (low-level features  tag: if image is in cluster LowlevelCluster_A liegt, image is probably tagged with Tag1). </li></ul>
    16. 16. Conclusions & Outlook <ul><li>Search & Retrieval of Images by integration of </li></ul><ul><ul><li>Web 2.0 Tagging (High-level Semantics) </li></ul></ul><ul><ul><li>CBIR (Low-level Semantics) </li></ul></ul><ul><li>Rule-based representation of Image Mining Results </li></ul><ul><ul><li>OWL Syntax </li></ul></ul><ul><ul><li>Semantic Web Reasoner </li></ul></ul><ul><li>Outlook: </li></ul><ul><ul><li>Integration in Multimedia Information System „Virtual Campfire“ </li></ul></ul><ul><ul><li>Integration in Context-aware Mining Suite </li></ul></ul>

    ×