A Multimedia Service with MPEG-7 Metadata and Context Semantics Yiwei Cao, Ralf Klamma, and Maziar Khodaei Informatik 5 (Information Systems), RWTH Aachen University 20.03.2009 Toulouse, France The 9 th Workshop on Multimedia Metadata (WMM‘09)

Agenda Motivation and Scenario System Design of CA3M Service (Context-aware Mobile Multimedia Service) A MPEG-7ToRDF Converter Implementation and Evaluation Conclusions and Outlook CA3M Mobility Multimedia Uncertainty

Motivation and Scenario

System Design of CA3M Service (Context-aware Mobile Multimedia Service)

A MPEG-7ToRDF Converter

Implementation and Evaluation

Conclusions and Outlook

Motivation Context [Dey 1999] Interaction between users und applications Context awareness: location, time, device, community, activities Context-aware Mobile Multimedia Systems Presentation of relevant information: right information in the right format to the right community at right time and place Multimedia adaptation of 3D models, videos, audios, images, texts Data Uncertainty in Location Based Services (LBS) [Henrickson 2004] Inconsistencies between models und real world Inconsistencies between local environment models Representation: unknown; ambiguous; imprecise; erroneous

Context [Dey 1999]

Interaction between users und applications

Context awareness: location, time, device, community, activities

Context-aware Mobile Multimedia Systems

Presentation of relevant information: right information in the right format to the right community at right time and place

Multimedia adaptation of 3D models, videos, audios, images, texts

Data Uncertainty in Location Based Services (LBS) [Henrickson 2004]

Inconsistencies between models und real world

Inconsistencies between local environment models

Representation: unknown; ambiguous; imprecise; erroneous

Car Race Scenario Context-Aware Multimedia Sharing Communities produce and consume media at different places, times, etc. Semantic mapping of context data on semantic base types needed GPS data -> Tribunes, Curves, etc. (Places) Time data -> Lap, Start, Finish, Box stop, etc. (Events) Racing car data –> Ferrari, McLaren-Mercedes, BMW-Sauber Community, etc. (Agents) Multimedia adaption needed because of different mobile devices

Communities produce and consume media at different places, times, etc.

Semantic mapping of context data on semantic base types needed

GPS data -> Tribunes, Curves, etc. (Places)

Time data -> Lap, Start, Finish, Box stop, etc. (Events)

Racing car data –> Ferrari, McLaren-Mercedes, BMW-Sauber Community, etc. (Agents)

Multimedia adaption needed because of different mobile devices

Concepts of CA3M Community-Aware Static Multimedia Adaptation based on Clustering Algorithms [KSCa06] Context-Aware Adaptation based on Semantic Einrichment of Multimedia Information [CKHJ08] How to support MPEG-7/MPEG-21 Semantic Enrichment for further Semantic Processing? How to systematically handle Uncertain Information in further Semantic Processing? Context-Aware Adaptation System (OWL) [CKHJ08, MDM‘08] Media Adaptation System (MPEG-7/21) [KSCa06, ECTEL‘06] MPEG-7ToRDF Context Reasoner [CKHJ08, MDM‘08] Uncertainty Management

Community-Aware Static Multimedia Adaptation based on Clustering Algorithms [KSCa06]

Context-Aware Adaptation based on Semantic Einrichment of Multimedia Information [CKHJ08]

How to support MPEG-7/MPEG-21 Semantic Enrichment for further Semantic Processing?

How to systematically handle Uncertain Information in further Semantic Processing?

Selection of MPEG7ToRDF Converter Technologies MPEG-7 Ontology [García et al., 2006] Developed by Roberto Garcia Gonzales within Rhizomik Based on MPEG-7 as of 2001 In OWL-Full-Format Alternatives TBD: MPEG-21 Conversion Multimedia Ontologies Format Mapping MPEG-7 Upper DMS [Hunter, 2001] OWL-Full - COMM [Bloehdorn et al., 2005] OWL-DL - aceMedia [ace, 2005] RDFS-DS - MPEG-7 Ontologie (Rhizomik) [ García et al., 20 06] OWL-Full XSD2OWL XML2RDF

MPEG-7 Ontology [García et al., 2006]

Developed by Roberto Garcia Gonzales within Rhizomik

Based on MPEG-7 as of 2001

In OWL-Full-Format

Alternatives

TBD: MPEG-21 Conversion

The MPEG7ToRDF Converter The MPEG-7 Ontology based on MPEG-7 Schema as of 2004 Mapping MPEG-7 XML Schema to MPEG-7 Ontology in OWL Mapping MPEG-7 XML Documents to RDF Uncertainty Management Context Reasoner [CKHJ08] <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:mpeg7=&quot;http://manet.informatik.rwth-aachen.de/ ~khodaei/mpeg7-v2.owl#&quot;> <mpeg7:SemanticPlaceType rdf:about=&quot;http://manet.informatik.rwth- aachen.de/~khodaei/place.xml#SemanticPlaceType_ admin20080227090806191&quot;> <mpeg7:Place rdf:parseType=&quot;Resource&quot;> <mpeg7:GeographicPosition rdf:parseType=&quot;Resource&quot;> <mpeg7:Point rdf:parseType=&quot;Resource&quot;> <mpeg7:longitude>7.438253814591</mpeg7:longitude> <mpeg7:latitude>43.774566243600006</mpeg7:latitude> <mpeg7:altitude>29</mpeg7:altitude> </mpeg7:Point> </mpeg7:GeographicPosition> </mpeg7:Place> <mpeg7:Label rdf:parseType=&quot;Resource&quot;> <mpeg7:Definition rdf:parseType=&quot;Resource&quot;>curve-1</mpeg7:Definition> <mpeg7:Name>43:46.47397N, 7:26.29523E, 29m</mpeg7:Name> </mpeg7:Label> </mpeg7:SemanticPlaceType> </rdf:RDF> MPEG-7ToRDF Media Adaptation System (MPEG-7/21) [KSCa06] Context-Aware Adaptation System (OWL) [CKHJ08]

The MPEG-7 Ontology based on MPEG-7 Schema as of 2004

Mapping MPEG-7 XML Schema to MPEG-7 Ontology in OWL

Mapping MPEG-7 XML Documents to RDF

Context Reasoner Reasoning on context information SPARQL RDF Query Language Ontology Reasoning Language Implementation using the Jena Framework Uncertainty Management MPEG-7ToRDF Media Adaptation System (MPEG-7/21) [KSCa06] Context-Aware Adaptation System (OWL) [CKHJ08] PREFIX mpeg7: < http://manet.informatik.rwth-aachen.de/~khodaei/mpeg7-v2.owl#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?agent ?place WHERE { ?agent mpeg7:name “FirstName LastName&quot; ?agent mpeg7:hasGeographicPosition ?loc1 ?loc1 mpeg7:longitude ?lon1 ?loc1 mpeg7:latitude ?lat1 ?place mpeg7:hasGeographicPosition ?loc2 ?place mpeg7:definition “curve-n” ?loc2 mpeg7:longitude ?lon2 ?loc2 mpeg7:latitude ?lat2 FILTER Distance(?loc1, ?loc2) <= distance } Context Reasoner [CKHJ08]

Reasoning on context information

SPARQL

RDF Query Language

Ontology Reasoning Language

Implementation using the Jena Framework

Uncertainty Management Clustering Algorithms based on User Preferences Calculation Euclidean Distance of Communities Clusters are recalculated when new users come Rating/Ranking as User Query Feedback Context Reasoner [CKHJ08] MPEG-7ToRDF Media Adaptation System (MPEG-7/21) [KSCa06] Context-Aware Adaptation System (OWL) [CKHJ08] Uncertainty Management

Clustering Algorithms based on User Preferences

Calculation Euclidean Distance of Communities

Clusters are recalculated when new users come

Rating/Ranking as User Query Feedback

Evaluation Evaluating within the DFG UMIC Excellence Cluster 2006-2011 Concepts and demonstrators for smart, mobile, broadband, low-cost systems will be developed which support the demanding applications of the next-decade mobile Internet Mobile Multimedia Information for Cultural Heritage Management (Virtual Campfire) Virtual City Guides Immersive Games Semantic Enrichment of Metadata (MPEG-7, MPEG-21, …) for Location-Based Services Systematic Evaluation of Services with MoBSOS [RKSp08] Process Converting MPEG-7 / MPEG-21 Files into OWL/RDF SPARQL Queries: Context Search on Semantic Base Types

Evaluating within the DFG UMIC Excellence Cluster 2006-2011

Concepts and demonstrators for smart, mobile, broadband, low-cost systems will be developed which support the demanding applications of the next-decade mobile Internet

Mobile Multimedia Information for

Cultural Heritage Management (Virtual Campfire)

Virtual City Guides

Immersive Games

Semantic Enrichment of Metadata (MPEG-7, MPEG-21, …) for Location-Based Services

Systematic Evaluation of Services with MoBSOS [RKSp08]

Process

Converting MPEG-7 / MPEG-21 Files into OWL/RDF

SPARQL Queries: Context Search on Semantic Base Types

Conclusions CA3M - A mobile context-aware multimedia service Integration into existing LAS architecture as a web service Enhancement of information interoperability Evaluation within UMIC Cluster of Excellence Technical Improvements Generating MPEG-7 ontology based on the 2004 MPEG-7 Schema Realizing a converter mapping XML document to OWL/RDF Integrated reasoning on multimedia and context information Semantic query processing for multimedia information Systematic handling of uncertainty for location based services (rating/ranking, clustering)

CA3M - A mobile context-aware multimedia service

Integration into existing LAS architecture as a web service

Enhancement of information interoperability

Evaluation within UMIC Cluster of Excellence

Technical Improvements

Generating MPEG-7 ontology based on the 2004 MPEG-7 Schema

Realizing a converter mapping XML document to OWL/RDF

Integrated reasoning on multimedia and context information

Semantic query processing for multimedia information

Systematic handling of uncertainty for location based services (rating/ranking, clustering)

Outlook Data Management Improvement of SPARQL queries for large amount of data Performing SPARQL queries on relational databases Field tests with mobile communities Uncertainty Management Improvement of context reasoning Handling uncertainty with probabilistic methods Fuzzy logic Bayesian network Applications & User-Generated Content Mobile Storytelling Cultural Heritage Management Services on various mobile devices e.g. iPhone, Google G1

Data Management

Improvement of SPARQL queries for large amount of data

Performing SPARQL queries on relational databases

Field tests with mobile communities

Uncertainty Management

Improvement of context reasoning

Handling uncertainty with probabilistic methods

Fuzzy logic

Bayesian network

Applications & User-Generated Content

Mobile Storytelling

Cultural Heritage Management

Services on various mobile devices e.g. iPhone, Google G1