ISSLOD2011 - Semantic Multimedia

3,565 views
3,436 views

Published on

My lecture at the Indian-summer School on Linked Open Data 2011 at the University Leipzig (Germany) on 15. Sep 2011

Published in: Technology, Education
1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
3,565
On SlideShare
0
From Embeds
0
Number of Embeds
194
Actions
Shares
0
Downloads
220
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

ISSLOD2011 - Semantic Multimedia

  1. 1. SemanticMultimediaIndian Summer School on Linked Data Leipzig, 15 Sep. 2011 Dr. Harald SackHasso-Plattner-Institut for IT-Systems Engineering University of Potsdam
  2. 2. ■ HPI was founded in October 1998 as a Public- Private-Partnership ■ HPI Research and Teaching is focussed on IT Systems Engineering ■ 10 Professors and 100 Scientific Coworkers ■ 450 Bachelor / Master Students ■ HPI is winner of CHE-Ranking 2010Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011 http://hpi.uni-potsdam.de/
  3. 3. Semantic Technologies & Multimedia Retrieval ■ Research Topics □ Semantic Web Technologies □ Ontological Engineering □ Information Retrieval □ Multimedia Analysis & Retrieval □ Social Networking □ Data/Information Visualization ■ Research ProjectsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  4. 4. Semantic Multimedia Indian Summer School on Linked Data, Leipzig, 15 Sep. 2011 Overview (1) Multimedia and Semantics (2) Multimedia Metadata and Ontologies (3) Semantic Multimedia Analysis (4) Semantic Multimedia RetrievalHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  5. 5. 1. Multimedia and Semantics Communication is the activity of conveying meaningful informationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  6. 6. 1. Multimedia and Semantics Shannon‘s Model of Communication Sender Receiver Message Message Channel Encoding Decoding Claude E. Shannon (1916-2001) Information Information Claude E. Shannon: ,A mathematical theory of communication‘, Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, July, October, 1948Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  7. 7. 1. Multimedia and Semantics Shannon‘s Model of Communication Sender Receiver Message Message Channel Encoding Decoding Media Claude E. Shannon (1916-2001) Information Information Claude E. Shannon: ,A mathematical theory of communication‘, Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, July, October, 1948Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  8. 8. 1. Multimedia and Semantics Sender Receiver Message Message Channel Encoding Decoding Media Information InformationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  9. 9. 1. Multimedia and Semantics Message Message Channel Media MEDIA: In communications, media (singular medium) are the storage and transmission channels or tools used to store and deliver information or data.Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  10. 10. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  11. 11. TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message.Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  12. 12. TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message.Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  13. 13. Text TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message.Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  14. 14. Text TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message. ImagesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  15. 15. Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  16. 16. TextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  17. 17. Image TextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  18. 18. Image Video / Audio TextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  19. 19. Image Interactive Elements Video / Audio TextHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  20. 20. 1. Multimedia and Semantics MediaHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  21. 21. 1. Multimedia and Semantics Media time-independent text imageHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  22. 22. 1. Multimedia and Semantics Media time-dependent time-independent audio text video / animation imageHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  23. 23. 1. Multimedia and Semantics One Small Step ... This video shows Neil Armstrong climbing down the lunar module ladder to the lunar surface. The video compares existing footage with the partially restored video. The thumbnail image shows the new footage on the left and the old on the right. • Information is encoded in media content • Media content contains implicite semanticsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  24. 24. 1. Multimedia and Semantics One Small Step ... This video shows Neil Armstrong climbing down the lunar module ladder to the lunar surface. The video compares existing footage with the partially restored video. The thumbnail image shows the new footage on the left and the old on the right. • Information is encoded in media content • Media content contains implicite semanticsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  25. 25. 1. Multimedia and Semantics One Small Step ... This video shows Neil Armstrong climbing down the lunar module ladder to the lunar surface. The video compares existing footage with the partially restored video. The thumbnail image shows the new footage on the left and the old on the right. • Information is encoded in media content • Media content contains implicite semanticsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  26. 26. SEMANTIC MULTIMEDIA facilitates • explicite semantic annotation • of multimedia content • on different levels of abstraction w.r.t. • time, • space, and • provenance.Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  27. 27. 1. Multimedia and Semantics One Small Step ... This video shows Neil Armstrong climbing down the lunar module ladder to the lunar surface. The video compares existing footage with the partially restored video. The thumbnail image shows the new footage on the left and the old on the right. Text (1)Identify media fragment (2)Annotate with explicite semantics dbpedia:Neil_ArmstrongHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  28. 28. 1. Multimedia and Semantics One Small Step ... This video shows Neil Armstrong climb ladder to the lunar surface. The video with the partially restored video. The t new footage on the left and the old on Video dbpedia:Flag (1)Identify media fragment dbpedia:Astronaut (2)Annotate with explicite semanticsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  29. 29. Semantic Multimedia Indian Summer School on Linked Data, Leipzig, 15 Sep. 2011 Overview (1) Multimedia and Semantics (2) Multimedia Metadata and Ontologies (3) Semantic Multimedia Analysis (4) Semantic Multimedia RetrievalHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  30. 30. 2. Multimedia Metadata and Ontologies One Small Step ... This video shows Neil Armstrong climb ladder to the lunar surface. The video with the partially restored video. The th new footage on the left and the old on Video dbpedia:Flag dbpedia:AstronautHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  31. 31. 2. Multimedia Metadata and Ontologies One Small Step ... This video shows Neil Armstrong climb ladder to the lunar surface. The video with the partially restored video. The th new footage on the left and the old on Video dbpedia:Flag dbpedia:Astronaut How can we put (semantic) metadata at the appropriate place within the media?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  32. 32. 2. Multimedia Metadata and Ontologies What is ,Metadata‘?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  33. 33. 2. Multimedia Metadata and Ontologies What is ,Metadata‘? „Metadata is defined as data providing information about one or more aspects of the data“ (informal Definition, Wikipedia)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  34. 34. 2. Multimedia Metadata and Ontologies What is ,Metadata‘? „Metadata is defined as data providing information about one or more aspects of the data“ (informal Definition, Wikipedia) „Metadata is structured, encoded data that describe characteristics of information-bearing entities to aid in the identification, discovery, assessment, and management of the described entities.“ (W.R. Durell, 1985)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  35. 35. 2. Multimedia Metadata and Ontologies What is ,Metadata‘? „Metadata is defined as data providing information about one or more aspects of the data“ (informal Definition, Wikipedia) „Metadata is structured, encoded data that describe characteristics of information-bearing entities to aid in the identification, discovery, assessment, and management of the described entities.“ (W.R. Durell, 1985) „Metadata is machine understandable information about web resources or other things.“ (T.Berners-Lee, Axioms of Web Architecture: Metadata, W3C, 1997)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  36. 36. 2. Multimedia Metadata and Ontologies Metadata • Simple example: bibliographic metadata Identification via ISBN / ISSN author(s) titel ... Classification via categories keywords abstract ...Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  37. 37. 2. Multimedia Metadata and Ontologies Structured Metadata • name-value pairs (e.g. author=‘Ernest Hemingway‘) • typed (e.g. author is of type string) • Meaning (semantics) of structured data is only implicite, i.e. it relies on mutual agreement about the proper usage of the data (e.g. Standardization for Dublin Core) • Title: A name given to the resource. • Creator: An entity primarily responsible for making the resource. • Subject: The topic of the resource. • Description: An account of the resource. • Publisher: An entity responsible for making the resource available. • Contributor: An entity responsible for making contributions to the resource. .... http://dublincore.org/documents/dces/Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  38. 38. 2. Multimedia Metadata and Ontologies Structured Metadata • can also be structured hierarchically Carl von Linné (1707-1787) Systema Naturae (1735)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  39. 39. 2. Multimedia Metadata and Ontologies Structured Metadata • Classification Systems, as e.g. Dewey Decimal Classification DDC 1 (1876) DDC 23 (2011) • 44 pages • 4 volumes • >4.000 pages Melvil Dewey • >45.000 classes (1851-1931) • >96.000 registered terms 10 Main DDC Classes 000 Computer science, information & general works 100 Philosophy & psychology 200 Religion 300 Social sciences 400 Language 500 Science 600 Technology 700 Arts & recreation 800 Literature 900 History & geographyHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011 http://www.oclc.org/dewey/
  40. 40. 2. Multimedia Metadata and Ontologies Unstructured Metadata • Text based metadata without a predefined structure, where the meaning (semantics) is determined implicitely by the (natural language) content. • e.g. abstract/summary Melville Louis Kossuth (Melvil) Dewey was an American librarian and educator, inventor of the Dewey Decimalsystem of library classification, and a founder of the Lake Placid Club.. Dewey was born in Adams Center, New York, the fifth and last child of Joel and Eliza Greene Dewey. He attended rural schools and determined early that his destiny was to be a reformer in educating the masses. At Amherst College he belonged to Delta Kappa Epsilon, earning a bachelors degree in 1874 and a masters in 1877....Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  41. 41. 2. Multimedia Metadata and Ontologies Authoritative vs. non-authoritative Metadata Authoritative Metadata are generated by a reliable (authoritative) source, as e.g. • the author of the original information • a certified expert Non-authoritative Metadata are created by an unreliable source, as e.g. • the user • Social Tagging SystemsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  42. 42. 2. Multimedia Metadata and Ontologies Collaborative Annotation -- Social Tagging tasty Ressource Author apple apple fruit breakfast fruit Users to buy © E.C. Publications, Inc. non- authoritative authoritative Metadata MetadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  43. 43. 2. Multimedia Metadata and Ontologies Collaborative Annotation -- Folksonomies http://www.wordle.net/Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  44. 44. 2. Multimedia Metadata and Ontologies Semantic Metadata • can be structured or semi-structured metadata • the semantics of metadata is defined explicitely in a formal way (Ontologies) and therefore machine readable (as well as machine understandable) "An ontology is an explicit, formal specification of a shared conceptualization. The term is borrowed from philosophy, where an Ontology is a systematic account of Existence. For AI systems, what ‘exists’ is that which can be represented.“ (Thomas R. Gruber, 1993) conceptualization: abstract model (domain, relevant terms, relations) explicit: semantics of all terms must be defined formal: machine understandable shared: consensus about ontologyHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  45. 45. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata publicationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  46. 46. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties • titel • keywords • ... publicationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  47. 47. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties book • titel • keywords is a • ... publicationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  48. 48. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publicationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  49. 49. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publication publishes publisherHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  50. 50. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publication is written by publishes publisher writes AutorHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  51. 51. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publication 1..n is written by publishes 1..n publisher writes AutorHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  52. 52. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publication 1..n is written by publishes 1..n publisher writes Autor Person is aHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  53. 53. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties journal book • titel • keywords is a is a • ... publication address 1..n is written by has a publishes 1..n publisher writes Autor Person is aHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  54. 54. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a publishes 1..n publisher writes Autor Person is aHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  55. 55. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a publishes 1..n publisher writes Autor Person is a is a Springer VerlagHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  56. 56. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a publishes 1..n publisher writes Autor Person is a is a is a Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  57. 57. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a publishes 1..n publisher writes Autor Person is a is a is a Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  58. 58. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a female is a publishes 1..n publisher writes Autor Person is a is a is a is a male Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  59. 59. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a female is a publishes 1..n publisher writes Autor is a Person ≠ is a is a is a male Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  60. 60. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking entity is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication address 1..n is written by has a female is a publishes 1..n publisher writes Autor is a Person ≠ is a is a is a male Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  61. 61. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking entity is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication class address 1..n is written by has a female is a publishes 1..n publisher writes Autor is a Person ≠ is a is a is a male Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  62. 62. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking entity is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication class address 1..n is written by has a female is a publishes 1..n publisher writes Autor is a Person ≠ is a is a is a male relation Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  63. 63. 2. Multimedia Metadata and Ontologies Example for Semantic Metadata Internetworking entity is a properties properties journal book • titel • surname • keywords • first name is a is a • ... • street... publication class address axiom 1..n is written by has a female is a publishes 1..n publisher writes Autor is a Person ≠ is a is a is a male relation Springer Verlag Harald SackHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  64. 64. 2. Multimedia Metadata and Ontologies Semantic Metadata • enable the definition of formal Axioms • e.g. „It is not possible that the publishing date is earlier than the birth date of the author of the publication.“ • enable deduction of new facts • e.g. „All men are mortal.“ „Socrates is a man.“ „Therefore Socrates is mortal.“ • semantic Metadata enable to make implicitely given information explicite with the help of deduction and inference Raffael: The School of Athens, 1510Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  65. 65. 2. Multimedia Metadata and Ontologies Multimedia Metadata Description Languages • for time-based media time • annotatation of temporal media fragmentsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  66. 66. 2. Multimedia Metadata and Ontologies Multimedia Metadata Description Languages • for media with spatial extend metadata metadata • annotatation of spatial media fragmentsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  67. 67. 2. Multimedia Metadata and Ontologies Multimedia Metadata • MPEG-7 The MPEG-7 standard, formerly named „Multimedia Content Description Interface“, provides a rich set of standard tools to describe multimedia content. Both human users and automatic systems that process audiovisual information are within the scope of MPEG-7. • Components of the MPEG-7 Standard • MPEG-7 Systems • MPEG-7 Description Definition Language • MPEG-7 Visual • MPEG-7 Audio • MPEG-7 Multimedia Description Schemes MDS • MPEG-7 Reference Software • MPEG-7 Conformance • MPEG-7 Extraction and Use of DescriptionsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  68. 68. 2. Multimedia Metadata and Ontologies MPEG-7 Description of a Video Segment <Mpeg7 xmlns="..."> <Description xsi:type="ContentEntityType"> ... <Video> <TemporalDecomposition> <VideoSegment> <CreationInformation>...</CreationInformation> <TextAnnotation> <KeywordAnnotation> <Keyword>mouse</Keyword> </KeywordAnnotation> </TextAnnotation> <MediaTime> <MediaTimePoint>T00:05:05:0F25</MediaTimePoint> <MediaDuration>PT00H00M31S0N25F</MediaDuration> </MediaTime> </VideoSegment> </TemporalDecomposition> </Video> ... </Description> </Mpeg7>Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  69. 69. 2. Multimedia Metadata and Ontologies MPEG-7 Description of a Still ImageHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  70. 70. 2. Multimedia Metadata and Ontologies MPEG-7 and the Semantic Web • MDS Upper Layer represented in RDF(S) (2001: Hunter, later with link to ABC upper ontology) • MDS fully represented in OWL-DL (2004: Tsinaraki et al., DS-MIRF model) • MPEG-7 fully represented in OWL-DL (2005: Garcia & Celma, Rhizomik model) • MDS and Visual Parts represented in OWL-DL (2007: Arndt et al., COMM model, re-engineering of MPEG-7 with DOLCE design patterns)Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  71. 71. 2. Multimedia Metadata and Ontologies Example: Tagging with an MPEG-7 Ontology Reg1 • Localize a region → Draw a bounding box • Annotate the content → Interpret the content → Tag ,Astronaut‘ :Reg1 foaf:depicts dbpedia:AstronautHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  72. 72. 2. Multimedia Metadata and Ontologies Example: Tagging with an MPEG-7 Ontology Reg1 mpeg7:StillRegion rdf:type decom position Reg1 mpeg7 :spatial_ mpeg7:image mpeg7:SpatialMask mpeg7:depicts mpeg7:depicts mpeg7:polygon dbpedia:Astronaut mpeg7:Coords Man on the MoonHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  73. 73. 2. Multimedia Metadata and Ontologies Media Fragment Identification • Multimedia data has temporal and spatial dimension • pinpoint access on media fragments (on the web) with media fragment identifiers • (W3C Media Fragments URI 1.0, Juli 2009, Working Draft) • simple examples http://www.example.com/example.ogg#track=‘audio‘ http://www.example.com/example.ogg#track=‘audio‘&t=10s,20s http://www.example.com/example.ogg#track=‘video‘&xywh=160,120,320,240 • requires different handling of media data by http client-server transactionsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  74. 74. Semantic Multimedia Indian Summer School on Linked Data, Leipzig, 15 Sep. 2011 Overview (1) Multimedia and Semantics (2) Multimedia Metadata and Ontologies (3) Semantic Multimedia Analysis (4) Semantic Multimedia RetrievalHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  75. 75. How do we find somethi ng in a Multime dia Arch ive?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  76. 76. How does Google find a video?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  77. 77. How do you find something in an audiovisual archive?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  78. 78. How do you find something in an audiovisual archive? Step 1: Digitalization of analogue dataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  79. 79. How do you find something in an audiovisual archive? Step 1: Digitalization of analogue data Step 2: Annotation with (textbased) metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  80. 80. How do you find something in an audiovisual archive? • Manual annotation of AV-content with descriptive metadataHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  81. 81. ...can th is also b achieved e in an automat ed way?Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  82. 82. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complexHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  83. 83. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complexHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  84. 84. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complexHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  85. 85. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  86. 86. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre Analysis Face DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  87. 87. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre overlay Face Analysis text DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  88. 88. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre Logo overlay Face Analysis Detection text DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  89. 89. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre Logo overlay Face Analysis Detection text Detection scene textHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  90. 90. Automated AV-Media Analysis automated content-based analysis is • difficult (error prone) and • complex Genre Logo overlay Face Analysis Detection text Detection scene text { Audio-Mining structural speaker transcription analysis identificationHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  91. 91. Automated AV-Media Analysis • Result: Video segments with time-based metadata annotations timeMetadata Extraction • Metadata consist of combined low level / high level feature descriptors • Metadata serve as a basis for traditional and semantic retrievalHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  92. 92. Semantic Multimedia Analysis Video Analysis / time Metadata Extraction Entity Recognition/ Mapping e.g., person xy location yz event abc e.g., bibliographical data, geographical data, encyclopedic data, ..Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  93. 93. Some Ex amples o Automat f ed Video AnalysisHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  94. 94. • Structural Analysis • Intelligent Character Recognition (ICR) • Character/Logo Detection • Character Filtering • Character Recognition • Audio Analysis • Speaker Detection • Automated Speech Recognition (ASR) • Genre Analysis / Categorization •graphic / real •indoor / outdoor •day / night •... • Face/Body Detection, Tracking & ClusteringHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  95. 95. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments videoHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  96. 96. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments video scenesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  97. 97. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments video scenes shotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  98. 98. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments video scenes shots subhotsHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  99. 99. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments video scenes shots subhots framesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  100. 100. Structural Analysis • Automated subdivision of AV media data by structural segmentation • Subdivision of data streams in contentual coherent segments video scenes shots subhots framesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  101. 101. Structural Analysis • Shot Boundary Detection Histogram Difference Analysis shots • Identification of • Hard Cuts • Drop Outs • Soft Cuts, as e.g., Dissolve, Wipe, Cross-Fade, etc. Analytical Shot Boundary Detection Motion Vector Analysis • Analysis of Luminance/Chrominance Histograms • Analysis of Edge Distribution • Analysis of Motion Vectors Machine Learning • Classification of Hard/Soft Cuts based on Image Features • K-Nearest Neighbor • Random Forrest • Support Vector MachinesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  102. 102. Structural Analysis • Shot Boundary Detection shots • Identification of • Hard Cuts Feature Analysis • Luminance Histogram Difference • Chrominance Histogram Difference • Edge Distribution 91927 91928 91929 91930 91931 91932Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  103. 103. Structural Analysis • Shot Boundary Detection shots • Identification of • Hard Cuts • Drop Outs Histogram/Chrominance Difference Analysis Drop OutHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  104. 104. Structural Analysis • Shot Boundary Detection shots • Identification of • Hard Cuts • Drop Outs • Soft Cuts, as e.g., Dissolve, Wipe, Cross-Fade, etc. Fade Out Fade InHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  105. 105. Structural Analysis • Shot Boundary Detection Histogram Difference Analysis shots • Identification of • Hard Cuts • Drop Outs • Soft Cuts, as e.g., Dissolve, Wipe, Cross-Fade, etc. Analytical Shot Boundary Detection • Analysis of Luminance/Chrominance Histograms • Analysis of Edge Distribution Motion Vector Analysis • Analysis of Motion Vectors Machine Learning • Classification of Hard/Soft Cuts based on Image Features • K-Nearest Neighbor • Random Forrest • Support Vector MachinesHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  106. 106. Automated AV-Media Analysis Character Detection Face-Detection Character Recognition Face Clustering Face Tracking Logo-Detection Genre DetectionHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011
  107. 107. Automated AV-Media Analysis Intelligent Character Recognition • Preprocessing • Character Identification • Text Preprocessing • Text Filtering • Adaption of script geometry (Deskew) • Image quality enhancement • Optical Character Recognition (OCR) • Standard OCR software (OCRopus) • Postprocessing • Lexical analysis • Statistical / context based filtering Ermittlungen nach BombenfundenHarald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011

×