Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Semantics for visual resources: use cases from e-culture


Published on

Keynote Semantic Web Summer School, Cercedilla, Spain, July 2006

  • If u need a hand in making your writing assignments - visit ⇒ ⇐ for more detailed information.
    Are you sure you want to  Yes  No
    Your message goes here
  • Überprüfen Sie die Quelle ⇒ ⇐ . Diese Seite hat mir geholfen, eine Diplomarbeit zu schreiben.
    Are you sure you want to  Yes  No
    Your message goes here
  • You might get some help from ⇒ ⇐ Success and best regards!
    Are you sure you want to  Yes  No
    Your message goes here
  • This program and community of women has been the single most influential piece of my recovery after 11 years of bulimia. I recommend it to anyone who wants to learn to love themselves and claim a life of joy and freedom. ●●●
    Are you sure you want to  Yes  No
    Your message goes here
  • I know that while making the choice to recover was crucial, and having the will comes in handy, but it is the girls in this program and their support that has helped me make it this far. I have been in recovery without purging for two months. I know I still have a long way to go, but I could not imagine doing it on my own. Even in this short amount of time, I can see myself emerging and the world is opening up before me. I have started making plans to do the things I’ve wanted to do for years, but that got lost in my bulimia cycle. Just taking that first step in recovery, has made me see what is really important. ▲▲▲
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Semantics for visual resources: use cases from e-culture

  1. 1. Semantics for visual resources Use Cases from E-Culture Guus Schreiber Free University Amsterdam
  2. 2. 2 Purpose  Analyze a number of use cases from e-culture domain – Multimedia plays key role  Required technology – Typically combination of technologies  Relation to state of the art Acknowledgements: This presentations contains slides and images provided by Laura Hollink, Giang Nguyen and Cees Snoek. Also thanks to the MultimediaN E-Culture team
  3. 3. 3 Use case: Asian chairs User has found an image of an Asian chair Annotation: ex:image vra:stylePeriod aat:Guangxu . How can we find images of Asian chairs from the same historical period?
  4. 4. 4 AAT info on Guangxu
  5. 5. 5 Importance of time and space information  Many queries require time/space knowledge, either absolute or abstracted  For the chair image we can establish – Country = China (link Chinese => China) – Period = 1644-1911 (from Qing description)  Technology requirements: – Thesuari relating time/space concepts – NLP for unstructured descriptions – Time/space reasoning techniques
  6. 6. 6
  7. 7. 7
  8. 8. 8 Sample place information in TGN <tgn:AdministrativePlace rdf:about="&tgn;1000111" tgn:standardLatitude="35" tgn:standardLongitude="105“> <vp:parentPreferred rdf:resource="&tgn;1000004"/> …….. </tgn:AdministrativePlace>
  9. 9. 9 Issues when searching for “nearby” Asian chairs  Close in space: – Other country in (East) Asia – Latitude/longitude  Close in time: – Links between style periods – Match time periods (and handle incomplete information)
  10. 10. 10
  11. 11. 11 Use case: painting style Find paintings of a similar style MATISSE, Henri Le bonheur de vivre (The Joy of Life) 1905-1906 Oil on canvas, 69 1/8 x 94 7/8 in. (175 x 241 cm) Barnes Foundation, Merion, PA
  12. 12. 12 How can we find this other Fauve painting? DERAIN, Andre The Turning Road, L'Estaque, 1906 Oil on canvas, 51 x 76 3/4 in. (129.5 x 195 cm) Museum of Fine Arts, Houston, Texas
  13. 13. 13 Issues  Parse annotation to find matches with thesauri terms – E.g. match artists to ULAN individuals  Artists-style links – AAT contains styles; ULAN contains artists, but there is no link • Learn link from corpora • Derive it from other annotations – Domain-specific rules/reasoning needed • see example in SWRL doc • Painters may have painted in multiple styles
  14. 14. 14
  15. 15. 15
  16. 16. 16 Search: WordNet patterns that increase recall without sacrificing precision (Hollink)
  17. 17. 17 Issues w.r.t. thesauri  Public availability!  RDF/OWL representation  Learning/specifying term/concept mapping – owl:equivalentClass, owl:sameAs, rdf:type, rdfs:subClassOf – Domain-specific links  Managing the evolution of the thesauri and the mappings
  18. 18. 18 Use case: find images with the same subject Find another painting which portrays dancing
  19. 19. 19 Issues  Same subjects can be visually very different  Subject is often missing from the annotation  Mismatch: users often search for subjects of images
  20. 20. 20 Conceptual subject descriptions 85% of the user queries: General Descriptions of generally known items. Only general, everyday knowledge is necessary. Descriptions are at the level of the Natural categories of E. Rosch (1973), or more general. E.g An ape eating a banana. Specific Descriptions of objects or scenes that can be identified and named. Specific domain knowledge is necessary to recognize the objects or scenes. E.g. The old male gorilla Kumba, born in Cameroon and now living in Artis, Amsterdam Abstract Descriptions for which interpretative knowledge is used. This category is subjective. E.g An animal threatened with extinction.
  21. 21. 21 Example concepts in image  Specific – Fall of the Berlin Wall  General – People walking at night  Abstract – Fall of the Iron Curtain
  22. 22. 22 Use of conceptual categories by people searching for images Conceptual level: 83% 0% 20% 40% 60% 80% 100% event time place relation scene object Characteristics Nuberofelementsin%of conceptualelements Abstract Specific General
  23. 23. 23 Thesauri for scenes: Iconclass
  24. 24. 24
  25. 25. 25
  26. 26. 26 Annotation of image content  Template for subject description Agent Action Object Recipient  Guidelines for manual annotation – Annotate as specific as possible  Default reasoning  CBIR support: – Object identification – Spatial relations
  27. 27. 27
  28. 28. 28
  29. 29. 29 Some forms of image content are well suited to image analysis Collection of clothes Abstract painting
  30. 30. 30 The semantic gap  The distance between Content-Based Image Retrieval and semantics: – Smeulders, Worring, Santini, Gupta, Jain. Content- based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), December 2000.  Direct links between visual features and semantic concepts become more difficult when the domain is broader / more general
  31. 31. 31 Example semantic bridge: microscopic cell images mpeg7 : StillRegion(region) ^ mpeg7x : Dense(region) ^ mpeg7 : DominantColor(region, col) ^ swrlb : lessThan(col, 100) => mpeg7 : Depicts(region, mesh : MatureGranule)
  32. 32. 32 Segmentation often requires user interaction
  33. 33. 33 Automatic detection of concepts can be difficult even in “easy” cases What is the color of this ape?
  34. 34. 34 Image analysis useful for collection navigation
  35. 35. 35 Bridging the semantic gap: CBIR and ontologies Visual WordNet (GE paper) – Adding knowledge about visual characteristics to WordNet: mobility, color, … – Build detectors for the visual features – Use visual data to prune the tree of categories when analyzing a visual object
  36. 36. 36 Sample visual features and their mapping to WordNet
  37. 37. 37 Experiment: pruning the search for “conveyance” concepts 6 concepts found Including taxi cab 12 concepts found Including passenger train and commuter train Three visual features: material, motion, environment Assumption is that these work perfectly
  38. 38. 38 Bridging the semantic gap: concept detectors  Snoek et al., TRECVID2004 – 185 hours of news video  32 detectors for concepts in news video – Through machine learning  Similarity detectors based on keywords and visual analysis  Query interface in which these functions can be combined
  39. 39. 39 “Concepts” for which visual detectors were built
  40. 40. 40 LSCOM lexicon: 229 - Weather  Context-specific (i.e. news broadcast) interpretation: “Weather forecast”
  41. 41. 41 LSCOM lexicon: 110 – Female Anchor  Composite concept  Alignment needed for semantic search, e.g. with WordNet
  42. 42. 42 Natural-lang proc. automatic annotation text stings → concepts Distributed collections OAI-based access Reasoning support time/space reasoning Web interface support for web collections Presentation facilities semantic presentation device-specific Interoperability XML/RDF/OWL Scalability > 10,000,000 triples Ontologies WordNet, AAT, TGN ULAN, Dutch labels Search strategies sibling search semantic distance Dublin Core specializations dumb-down semantic annotation DIGITAL HERITAGE COLLECTIONS semantic search BASELINEENHANCEDENHANCED FEATURESFEATURES NEWNEW FEATURESFEATURES
  43. 43. 43
  44. 44. 44 Main observation A combination of many different techniques is needed to be able to cope with the complexity of multimedia semantics – NLP, segmentation, CBIR, visual feature detectors, visual ontologies, publicly available thesauri, thesauri mappings, dedicated reasoning techniques (time, space, default), personalization, presentation generation  Key role for user studies