The document discusses provenance in multimedia semantics, focusing on tagging approaches and semantic annotation of multimedia objects using MPEG-7 standards. It highlights the complexity of multimedia content, the need for patterns and ontologies for effective information exchange, and several design patterns for decomposition and annotation. Additionally, it outlines various provenance vocabularies and emphasizes the importance of formalized contexts for multimedia interpretation.

1. Provenance for Multimedia:
COMM, D&S, M3O
Raphaël Troncy <raphael.troncy@eurecom.fr>
Multimedia Semantics, EURECOM (FR)

2. Common Scenario: Tagging Approach
Reg1
The "Big Three" at the Yalta
Conference (Wikipedia)
 Localize a region
 Draw a bounding box, a circle around a shape
 Annotate the content
 Interpret the content
 Tag: Winston Churchill, UK Prime Minister, Allied Forces, WWII
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -2

3. Common Scenario: SW Approach
Reg1
The "Big Three" at the Yalta
Conference (Wikipedia)
 Localize a region
 Draw a bounding box, a circle around a shape
 Annotate the content
 Interpret the content
 Link to knowledge on the Web
:Reg1 foaf:depicts dbpedia:Winston_Churchill
dbpedia:Winston_Churchill skos:altLabel
"Sir Winston Leonard Spencer-Churchill"
dbpedia:Winston_Churchill rdf:type foaf:Person
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -3

4. Research Problem
Seq4
Reg1
Seq1
The "Big Three" at the Yalta
Conference (Wikipedia) A history of G8 violence (video)
(© Reuters)
 Multimedia objects are complex
 MPEG-7
 Compound information objects, fragment identification
 Semantic annotation
 Subjective interpretation, context dependent  D&S | OIO
 Linked data principle
 Open to reuse existing knowledge  RDF
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -4

5. 28/05/2010 - Provenance for Multimedia - W3C Provenance XG -5

6. COMM: Design Rationale
 Approach:
 NO 1-to-1 translation from MPEG-7 to OWL/RDF
 Need for patterns: use DOLCE, a well designed foundational
ontology as a modeling basis
 Design patterns:
 Ontology of Information Objects (OIO)
Formalization of information exchange
Multimedia = complex compound information objects
 Descriptions and Situations (D&S)
Formalization of context
Multimedia = contextual interpretation (situation)
 Define multimedia patterns that translate MPEG-7 in the
DOLCE vocabulary
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -6

7. COMM: Core Functionalities
 Most important MPEG-7 functionalities:
 Decomposition of multimedia content into segments
 Annotation of segments with metadata
Administrative metadata: creation & production
Content-based metadata: audio/visual descriptors
Semantic metadata: interface with domain specific ontologies
 Note that all are subjective and context
dependent situations
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -7

8. COMM: D&S / OIO Patterns
3
2
1
Definition of design patterns for decomposition and
annotation based on D&S and OIO
1. MPEG-7 describes digital data (multimedia information objects)
with digital data (annotation)
2. Digital data entities are information objects
3. Decompositions and annotations are situations that satisfy the
rules of a method or algorithm
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -8

9. COMM: Decomposition Pattern
MPEG-
MPEG-7
7
28/05/2010 - Provenance for Multimedia - W3C Provenance XG -9

10. COMM: Semantic Pattern
Domain
Ontologies
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 10

11. COMM Example: Region Annotation
http://en.wikipedia.org/wiki/
Image:Yalta_Conference.jpg
dns:realized-by
dns:setting
core:semantic-
core:image-data
annotation
dns:plays dns:defines foaf:Person
loc:region- loc:spatial-mask- core:semantic-label-
locator-descriptor role role
dns:played-by
rdf:type
dns:defines dns:played-by
http://en.wikipedia.org/wiki/
loc:bounding-box 5 25 10 20 15 15 10 10 5 15"^^xsd:string
Churchill
data:has-rectangle
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 11

12. Provenance of Annotation: Naïve Approach
dbpedia:Winston_Churchill
dc:creator
foaf:depicts
:me
:Reg1
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 12

13. Provenance of Annotation: COMM / M3O
ex:annotation-pattern
defines
ex:annotated-concept ex:annotation-concept
classifies classifies
satisfies
ex:big-3-yalta dbpedia:Winston_Churchillin
hasSetting
ex:annotation-situation
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 13

14. Provenance of Annotation: COMM / M3O
ex:annotation-pattern
defines
ex:annotated-concept ex:annotation-concept ex:applied-method-role dc:creator
classifies classifies
satisfies
classifies classifies
ex:big-3-yalta dbpedia:Winston_Churchillin ex:manual-annotation
:me
hasSetting
ex:annotation-situation
http://m3o.semantic-multimedia.org/ontology/2010/02/28/provenance.owl
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 14

15.  Low-level features
extraction
 Classifier: SVM
 Input parameters
 Threshold
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 15

16. Summary: Provenance in multimedia semantics
 Direct approach:
 Reification: what is the semantics of what is being reified?
 Named Graph: how to represent the parameters used for detecting a
face in an image?
 Provenance vocabularies:
 OPM in RDF [Moreau et al.], http://openprovenance.org/model/opm.owl
 Provenir Vocab [Sahoo et al.],
http://knoesis.wright.edu/provenir/provenir.owl
 Provenance Vocab [Hartig and Zhao], http://purl.org/net/provenance/
 prv:createdBy, prv:performedAt, prv:performedBy, etc.
 D&S pattern:
 Formalized description of a particular context (situation)
 Specialized for multimedia in COMM and M3O
28/05/2010 - Provenance for Multimedia - W3C Provenance XG - 16