Provenance for Multimedia:
         COMM, D&S, M3O

Raphaël Troncy <raphael.troncy@eurecom.fr>
Multimedia Semantics, EUREC...
Common Scenario: Tagging Approach
Reg1

                                                                  The "Big Three" ...
Common Scenario: SW Approach
Reg1

                                                                 The "Big Three" at the...
Research Problem
                                                                                                Seq4
Reg1...
28/05/2010 -   Provenance for Multimedia - W3C Provenance XG   -5
COMM: Design Rationale
 Approach:
    NO 1-to-1 translation from MPEG-7 to OWL/RDF
    Need for patterns: use DOLCE, a ...
COMM: Core Functionalities

 Most important MPEG-7 functionalities:
   Decomposition of multimedia content into segments...
COMM: D&S / OIO Patterns

                                                                                   3

          ...
COMM: Decomposition Pattern




                 MPEG-
                 MPEG-7
                 7



  28/05/2010 -       ...
COMM: Semantic Pattern




                   Domain
                   Ontologies



  28/05/2010 -   Provenance for Mult...
COMM Example: Region Annotation

                                                                                         ...
Provenance of Annotation: Naïve Approach


                 dbpedia:Winston_Churchill


                                  ...
Provenance of Annotation: COMM / M3O
ex:annotation-pattern


                                           defines
          ...
Provenance of Annotation: COMM / M3O
ex:annotation-pattern


                                           defines
          ...
 Low-level features
                    extraction
                   Classifier: SVM
                            Input...
Summary: Provenance in multimedia semantics
 Direct approach:
   Reification: what is the semantics of what is being rei...
Upcoming SlideShare
Loading in...5
×

Provenance for Multimedia

1,605

Published on

Provenance for Multimedia: COMM, D&S, M3O. Talk given for the W3C Provenance Incubator Group, May 28th 2010.

Published in: Technology, Education

Provenance for Multimedia

  1. 1. Provenance for Multimedia: COMM, D&S, M3O Raphaël Troncy <raphael.troncy@eurecom.fr> Multimedia Semantics, EURECOM (FR)
  2. 2. Common Scenario: Tagging Approach Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Localize a region  Draw a bounding box, a circle around a shape  Annotate the content  Interpret the content  Tag: Winston Churchill, UK Prime Minister, Allied Forces, WWII 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -2
  3. 3. Common Scenario: SW Approach Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Localize a region  Draw a bounding box, a circle around a shape  Annotate the content  Interpret the content  Link to knowledge on the Web :Reg1 foaf:depicts dbpedia:Winston_Churchill dbpedia:Winston_Churchill skos:altLabel "Sir Winston Leonard Spencer-Churchill" dbpedia:Winston_Churchill rdf:type foaf:Person 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -3
  4. 4. Research Problem Seq4 Reg1 Seq1 The "Big Three" at the Yalta Conference (Wikipedia) A history of G8 violence (video) (© Reuters)  Multimedia objects are complex  MPEG-7  Compound information objects, fragment identification  Semantic annotation  Subjective interpretation, context dependent  D&S | OIO  Linked data principle  Open to reuse existing knowledge  RDF 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -4
  5. 5. 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -5
  6. 6. COMM: Design Rationale  Approach:  NO 1-to-1 translation from MPEG-7 to OWL/RDF  Need for patterns: use DOLCE, a well designed foundational ontology as a modeling basis  Design patterns:  Ontology of Information Objects (OIO) Formalization of information exchange Multimedia = complex compound information objects  Descriptions and Situations (D&S) Formalization of context Multimedia = contextual interpretation (situation)  Define multimedia patterns that translate MPEG-7 in the DOLCE vocabulary 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -6
  7. 7. COMM: Core Functionalities  Most important MPEG-7 functionalities:  Decomposition of multimedia content into segments  Annotation of segments with metadata Administrative metadata: creation & production Content-based metadata: audio/visual descriptors Semantic metadata: interface with domain specific ontologies  Note that all are subjective and context dependent situations 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -7
  8. 8. COMM: D&S / OIO Patterns 3 2 1 Definition of design patterns for decomposition and annotation based on D&S and OIO 1. MPEG-7 describes digital data (multimedia information objects) with digital data (annotation) 2. Digital data entities are information objects 3. Decompositions and annotations are situations that satisfy the rules of a method or algorithm 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -8
  9. 9. COMM: Decomposition Pattern MPEG- MPEG-7 7 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -9
  10. 10. COMM: Semantic Pattern Domain Ontologies 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 10
  11. 11. COMM Example: Region Annotation http://en.wikipedia.org/wiki/ Image:Yalta_Conference.jpg dns:realized-by dns:setting core:semantic- core:image-data annotation dns:plays dns:defines foaf:Person loc:region- loc:spatial-mask- core:semantic-label- locator-descriptor role role dns:played-by rdf:type dns:defines dns:played-by http://en.wikipedia.org/wiki/ loc:bounding-box 5 25 10 20 15 15 10 10 5 15"^^xsd:string Churchill data:has-rectangle 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 11
  12. 12. Provenance of Annotation: Naïve Approach dbpedia:Winston_Churchill dc:creator foaf:depicts :me :Reg1 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 12
  13. 13. Provenance of Annotation: COMM / M3O ex:annotation-pattern defines ex:annotated-concept ex:annotation-concept classifies classifies satisfies ex:big-3-yalta dbpedia:Winston_Churchillin hasSetting ex:annotation-situation 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 13
  14. 14. Provenance of Annotation: COMM / M3O ex:annotation-pattern defines ex:annotated-concept ex:annotation-concept ex:applied-method-role dc:creator classifies classifies satisfies classifies classifies ex:big-3-yalta dbpedia:Winston_Churchillin ex:manual-annotation :me hasSetting ex:annotation-situation http://m3o.semantic-multimedia.org/ontology/2010/02/28/provenance.owl 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 14
  15. 15.  Low-level features extraction  Classifier: SVM  Input parameters  Threshold 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 15
  16. 16. Summary: Provenance in multimedia semantics  Direct approach:  Reification: what is the semantics of what is being reified?  Named Graph: how to represent the parameters used for detecting a face in an image?  Provenance vocabularies:  OPM in RDF [Moreau et al.], http://openprovenance.org/model/opm.owl  Provenir Vocab [Sahoo et al.], http://knoesis.wright.edu/provenir/provenir.owl  Provenance Vocab [Hartig and Zhao], http://purl.org/net/provenance/  prv:createdBy, prv:performedAt, prv:performedBy, etc.  D&S pattern:  Formalized description of a particular context (situation)  Specialized for multimedia in COMM and M3O 28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 16

×