MPEG-7 Services in Community Engines

MPEG-7 Services in Community Engines Ralf Klamma Leuven, Belgium October 26, 2007

Agenda

MPEG-7 Multimedia Community

MPEG-7

Overview

Visual & Audio Descriptors

Multimedia Community Hosting

Applications

Conclusions & Outlook

Multimedia Metadata Community Our Goals

The Multimedia Metadata Community ( www.multimedia-metadata.info ) aims at

establishing a community of professionals in research and industry

bridging the gap between an academic research and an industrial scale development of innovative products with a strong focus on MPEG-7 & MPEG-21

establishing better metadata interoperability among different research and development groups

collecting of applications and tools

Multimedia Metadata Community Our Goals

Multimedia Analysis and Retrieval

Content Adaptation and Personalization

Learning and Multimedia

Multimedia and Web 2.0

Semantic and Knowledge in Multimedia

Mobile Multimedia and Digital Television

Multimedia Interoperability

Multimedia standards and related issues

Multimedia Metadata Standards Cross-walking

Multimedia Metadata Community Partners

University of Passau, Distributed Information Systems

Lehrstuhl Informatik 5, RWTH Aachen University

T-Systems International GmbH

Siemens AG

Know Center, Graz

Klagenfurt University, Department of Information Technology (ITEC)

JOANNEUM RESEARCH, Graz

Vienna University of Technology, Interactive Media Systems Group

University of Vienna - Multimedia Information Systems Group

Graz University of Technology, Institute for Information Systems and New Media (IICM)

ENSEEIHT Toulouse

Franche-Comté University

Institut National de Science Appliqués de Lyon

Institut National des Télécommunications, Evry

Multimedia Metadata Community – Events

MMC Workshop Klagenfurt - March 10, 2005 MPEG-7 and Multimedia Metadata: Standardization and Application

MMC Workshop, Graz, I-Know - June 30 – July 1, 2005 What can MPEG-7 do for you?

MMC Workshop, Aachen - November 24-25, 2006 Test-Set for Multimedia Interoperability and Application Scenario

MMC Workshop, Berlin - March 2-3, 2006 Multimedia Interoperability & MPEG-21

MMC Workshop Graz, I-Know -September 7-8, 2006 Extending the Community

MMC Workshop Aachen, BTW - March 5-6, 2007 Multimedia Semantics – The Emergent Role of Metadata

MMC Workshop Graz, I-Media – September 6-7, 2007 Multimedia Metadata Applications

MMC Workshop Klagenfurt, WIAMIS, March 7-8, 2008 Multimedia Metadata Services

Multimedia management features of MPEG-7 Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

XML Schema Language - Descriptors, DS, and DDL DDL DS DS DS D D D D DS D D Structural aspects of domains Domain spec. MPEG-7 extension MPEG-7 descriptor or DS non MPEG-7 descriptor or DS

Schema Tools Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

MPEG-7 Root Element

MPEG-7 Top Elements

Graph data type represents a graph a a list of nodes and relations between nodes in an arbitrary sorting

Basis Data Type – Graphs D B E A C r3 r3 r2 r1 r1 r4 <Graph> <Node id="A" /><Node id="B" /> <Node id="C" /><Node id="D" /> <Node id="E" /> <Relation type= ="#r1" source= "#A" target= ="#B" /> <Relation type= ="#r2" source= "#A" target= ="#C" /> <Relation type= ="#r3" source= "#B" target= ="#D" /> <Relation type= ="#r3" source= "#C" target= ="#D" /> <Relation type= ="#r4" source= "#B" target= ="#E" /> <Relation type= ="#r1" source= "#E" target= ="#A" /> </Graph>

Annotations <TextAnnotation> <StructuredAnnotation> <Who><Name>Spain</Name</Who> <Where><Name>AC</Name></Where> <When><Name>Mar15</Name></When> </StructuredAnnotation> </TextAnnotation> Structured Annotation <TextAnnotation> <KeywordAnnotation> <Keyword>score</Keyword> <Keyword>Sweden</Keyword> <Keyword>Spain</Keyword> </KeywordAnnotation> </TextAnnotation> Keyword Annotation <TextAnnotation xml:lang="en-us"> This is a nice apartment. </ TextAnnotation > <TextAnnotation xml:lang="en-uk"> This is a nice flat. </TextAnnotation > Free Text Annotation

Textual Annotations Score Spain A goal Sweden Against Governer Dependent Governer Dependent Dependent Dependent Dependency structure for "Spain scored a goal against Sweden“

Content Management and Description Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

Content Structures – Segment Entities Räumliche und zeitliche Segmente

Content Structures – Segment Entities Raum-zeitliches Segment

Content Structures – Segment Decomposition Segment decomposition

Content Structures – Structural Relations

The tools for structural relations can be used for the description of general segment structures

Tools are

SpatialRelation Classification Scheme

TemporalRelation Classification Scheme

Typed normative structural relations in MPEG-7 Precedes, follows, meets, metBy, overlaps, overlappedBy, contains, during, strictContains, stricrtDuring, starts, startedBy, finishes, finishedBy, coOccurs, contiguous, sequential, coBegin, coned, parallel, overlapping. Temporal South, North, West, East, Northwest, Northeast, Southwest, Southeast, Left, Right, Below, Above, Over, Under. Spatial Normative Relations Type

Example for Segmentation Tree Foreground Background SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation

Example for Content Semantics Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis

Putting it all together

Navigation and Access Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

Summaries – Keyframes Original Conten (represented by Keyframes) and highlighted summary content MovieClip1 Summary MovieClip2 MovieClip3

Hierarchical Summary Hierarchical Summary Summary Theme List Summary SegmentGroup Summary Segment 0, 1 1, * 1, * 0, *

Sequential Summary Sequential Summary Visual SummaryComponent Audio SummaryComponent Textual SummaryComponent 0, * 0, * 0, *

Views Multimedia program Multimedia program View Filtering Partition Region View Signal Source Signal View graphs View trees View sets Source Target View decompositions

Views – Space and Frequency Graph F F F S S S

Variations Variation Set Multimedia program Variation Fidelity Relationship Priority Multimedia program Variation Fidelity Relationship Priority Multimedia program Image Video Video Source Variation Variation

Content Organization Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

Collections Collection (abstract) Segment collection Content collection Descriptor collection Concept collection Mixed collection Collection structure

Content Collections and Collection Structure Different collection can be combined according to their collection structure Collection A Collection B Collection C R ac R bc R ab

Models Model (abstract) Probability Model Analytic Model Cluster Model Classification Model

Probability Model

Discrete distribution

Continuous distribution

Finite State Model

Collection Model

Probability Model

Class

Cluster Model

ClusterClassification

Model

Probabilty

Classification

Model

Cluster Model – Relevance Feedback After relevant content is marked by the user, the retrieval system can deliver more precise content clusters

State Transition Models State Transition models can be used for summarizing or classifying transitions of events on the timeline of a video sequence Video sequence with events, e.g. staging of an actor A C B p ab p ac p bc p ca Label Scene A C B Event A Event B Event C

User Interaction Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

Multimedia System with User Interaction Content filter & search engine Content browsing engine User profiling engine Local multimedia system Multimedia content description Multimedia content Multimedia content description Multimedia content description User preferences User action history Other devices Content/service provider User

Usage History Usage history 1, * 0, * UserIdentifier User action history Observation period User action list ActionType Action data item User action ProgramIdentifier ActionTime 1, * 1, * 0, *

User Preferences User preferences UserIdentifier Browsing preferences Summary preferences Preference condition Filtering and search preferences Preference condition Classification preferences Creation preferences Source preferences 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, *

Mapping Usage Hisotries on User Preferences Usage history description UserAction Program ID 1 UserAction Program ID 2 Content description Program ID 1 Title 1 Genre A Content description Program ID 2 Title 2 Genre B User preference description Classification Preferences Genre A preference Value a Genre B preference Value b

Only a short overview

MPEG-7 provides dozens of descriptors

Implementation is sometimes tricky

New descriptors can be developed by everybody

Color Space Descriptor I

Color Space Descriptor II

Texture Browsing Descriptor I

Texture Browsing Descriptor II Directions of textures:

Edge Histogram Descriptor

Overview Shape Descriptors

MPEG-7 defines three basis types of shape descriptors (SD):

Region-based SD (2-D objects)

Contour-based SD (2-D objects)

3D SD

Special type: Multiview Type Combination of Multiview and SD for 3D Object description (3D Form derived from multiple perspective 2-D views) Region-based notion of similarity Contour-based notion of similarity

Region Based SD

Expresses the distribution of pixels of a region or a 2D object

Compact and efficient description of complex objects (multiple not connected regions and simple object resp. holes)

Contour Based SD

Emulates the form based simularity perception of humans

Application on single regions with complex or simple outer lines (also open)

Motion Descriptors

Multimedia Community Hosting

Current Trends

Mobile multimedia sharing

Mobile community hosting

Mobile story-telling

Aim: MPEG-7 services

combination with other services

replaceable/extendable during runtime

available across different communities

Individual access rights communities

integration into other systems

LAS – Architecture Overview Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository Invoking services Data flows

Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository MPEG-7 Services Context Services Media Creation GPS-augmented multimedia creation Multimedia GIS Map Services Media Search Semantic Browsing MPEG-7 Services Context aware multimedia GIS Map Services Media Search Semantic Browsing Context Services MPEG-7 Services MPEG-7 multimedia tagging and commsonomy SNA Tools Media Search Media Tagging MPEG-7 Services MPEG-7 Services Context Services Media Search Semantic Browsing MPEG-7 enabled context aware multimedia search MPEG-7 Services Storytelling Service Semantic Browsing Non-linear digital storytelling Storytelling Service Storytelling on the ipod Context aware multimedia search Media Search MPEG-7 Services Context Services

MPEG-7 LAS Service Framework