Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Provenance for Multimedia

2,040 views

Published on

Provenance for Multimedia: COMM, D&S, M3O. Talk given for the W3C Provenance Incubator Group, May 28th 2010.

Published in: Technology, Education
  • Be the first to comment

Provenance for Multimedia

  1. 1. Provenance for Multimedia: COMM, D&S, M3O Raphaël Troncy <raphael.troncy@eurecom.fr> Multimedia Semantics, EURECOM (FR)
  2. 2. Common Scenario: Tagging Approach Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Localize a region  Draw a bounding box, a circle around a shape  Annotate the content  Interpret the content  Tag: Winston Churchill, UK Prime Minister, Allied Forces, WWII 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -2
  3. 3. Common Scenario: SW Approach Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Localize a region  Draw a bounding box, a circle around a shape  Annotate the content  Interpret the content  Link to knowledge on the Web :Reg1 foaf:depicts dbpedia:Winston_Churchill dbpedia:Winston_Churchill skos:altLabel "Sir Winston Leonard Spencer-Churchill" dbpedia:Winston_Churchill rdf:type foaf:Person 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -3
  4. 4. Research Problem Seq4 Reg1 Seq1 The "Big Three" at the Yalta Conference (Wikipedia) A history of G8 violence (video) (© Reuters)  Multimedia objects are complex  MPEG-7  Compound information objects, fragment identification  Semantic annotation  Subjective interpretation, context dependent  D&S | OIO  Linked data principle  Open to reuse existing knowledge  RDF 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -4
  5. 5. 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -5
  6. 6. COMM: Design Rationale  Approach:  NO 1-to-1 translation from MPEG-7 to OWL/RDF  Need for patterns: use DOLCE, a well designed foundational ontology as a modeling basis  Design patterns:  Ontology of Information Objects (OIO) Formalization of information exchange Multimedia = complex compound information objects  Descriptions and Situations (D&S) Formalization of context Multimedia = contextual interpretation (situation)  Define multimedia patterns that translate MPEG-7 in the DOLCE vocabulary 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -6
  7. 7. COMM: Core Functionalities  Most important MPEG-7 functionalities:  Decomposition of multimedia content into segments  Annotation of segments with metadata Administrative metadata: creation & production Content-based metadata: audio/visual descriptors Semantic metadata: interface with domain specific ontologies  Note that all are subjective and context dependent situations 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -7
  8. 8. COMM: D&S / OIO Patterns 3 2 1 Definition of design patterns for decomposition and annotation based on D&S and OIO 1. MPEG-7 describes digital data (multimedia information objects) with digital data (annotation) 2. Digital data entities are information objects 3. Decompositions and annotations are situations that satisfy the rules of a method or algorithm 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -8
  9. 9. COMM: Decomposition Pattern MPEG- MPEG-7 7 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -9
  10. 10. COMM: Semantic Pattern Domain Ontologies 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 10
  11. 11. COMM Example: Region Annotation http://en.wikipedia.org/wiki/ Image:Yalta_Conference.jpg dns:realized-by dns:setting core:semantic- core:image-data annotation dns:plays dns:defines foaf:Person loc:region- loc:spatial-mask- core:semantic-label- locator-descriptor role role dns:played-by rdf:type dns:defines dns:played-by http://en.wikipedia.org/wiki/ loc:bounding-box 5 25 10 20 15 15 10 10 5 15"^^xsd:string Churchill data:has-rectangle 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 11
  12. 12. Provenance of Annotation: Naïve Approach dbpedia:Winston_Churchill dc:creator foaf:depicts :me :Reg1 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 12
  13. 13. Provenance of Annotation: COMM / M3O ex:annotation-pattern defines ex:annotated-concept ex:annotation-concept classifies classifies satisfies ex:big-3-yalta dbpedia:Winston_Churchillin hasSetting ex:annotation-situation 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 13
  14. 14. Provenance of Annotation: COMM / M3O ex:annotation-pattern defines ex:annotated-concept ex:annotation-concept ex:applied-method-role dc:creator classifies classifies satisfies classifies classifies ex:big-3-yalta dbpedia:Winston_Churchillin ex:manual-annotation :me hasSetting ex:annotation-situation http://m3o.semantic-multimedia.org/ontology/2010/02/28/provenance.owl 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 14
  15. 15.  Low-level features extraction  Classifier: SVM  Input parameters  Threshold 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 15
  16. 16. Summary: Provenance in multimedia semantics  Direct approach:  Reification: what is the semantics of what is being reified?  Named Graph: how to represent the parameters used for detecting a face in an image?  Provenance vocabularies:  OPM in RDF [Moreau et al.], http://openprovenance.org/model/opm.owl  Provenir Vocab [Sahoo et al.], http://knoesis.wright.edu/provenir/provenir.owl  Provenance Vocab [Hartig and Zhao], http://purl.org/net/provenance/  prv:createdBy, prv:performedAt, prv:performedBy, etc.  D&S pattern:  Formalized description of a particular context (situation)  Specialized for multimedia in COMM and M3O 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 16

×