• Save
MPEG-7 Services in Community Engines
Upcoming SlideShare
Loading in...5
×
 

MPEG-7 Services in Community Engines

on

  • 2,174 views

 

Statistics

Views

Total Views
2,174
Views on SlideShare
2,165
Embed Views
9

Actions

Likes
1
Downloads
0
Comments
0

3 Embeds 9

http://beamtenherrschaft.blogspot.com 7
http://s3.amazonaws.com 1
http://feeds.feedburner.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MPEG-7 Services in Community Engines MPEG-7 Services in Community Engines Presentation Transcript

  • MPEG-7 Services in Community Engines Ralf Klamma Leuven, Belgium October 26, 2007
  • Agenda
    • MPEG-7 Multimedia Community
    • MPEG-7
      • Overview
      • Visual & Audio Descriptors
    • Multimedia Community Hosting
    • Applications
    • Conclusions & Outlook
  • Multimedia Metadata Community Our Goals
      • The Multimedia Metadata Community ( www.multimedia-metadata.info ) aims at
      • establishing a community of professionals in research and industry
      • bridging the gap between an academic research and an industrial scale development of innovative products with a strong focus on MPEG-7 & MPEG-21
      • establishing better metadata interoperability among different research and development groups
      • collecting of applications and tools
  • Multimedia Metadata Community Our Goals
      • Multimedia Analysis and Retrieval
      • Content Adaptation and Personalization
      • Learning and Multimedia
      • Multimedia and Web 2.0
      • Semantic and Knowledge in Multimedia
      • Mobile Multimedia and Digital Television
      • Multimedia Interoperability
      • Multimedia standards and related issues
      • Multimedia Metadata Standards Cross-walking
  • Multimedia Metadata Community Partners
      • University of Passau, Distributed Information Systems
      • Lehrstuhl Informatik 5, RWTH Aachen University
      • T-Systems International GmbH
      • Siemens AG 
      • Know Center, Graz
      • Klagenfurt University, Department of Information Technology (ITEC)    
      • JOANNEUM RESEARCH, Graz
      • Vienna University of Technology, Interactive Media Systems Group
      • University of Vienna - Multimedia Information Systems Group
      • Graz University of Technology, Institute for Information Systems and New Media (IICM)
      • ENSEEIHT Toulouse
      • Franche-Comté University  
      • Institut National de Science Appliqués de Lyon  
      • Institut National des Télécommunications, Evry  
  • Multimedia Metadata Community – Events
    • MMC Workshop Klagenfurt - March 10, 2005 MPEG-7 and Multimedia Metadata: Standardization and Application
    • MMC Workshop, Graz, I-Know - June 30 – July 1, 2005 What can MPEG-7 do for you?
    • MMC Workshop, Aachen - November 24-25, 2006 Test-Set for Multimedia Interoperability and Application Scenario
    • MMC Workshop, Berlin - March 2-3, 2006 Multimedia Interoperability & MPEG-21
    • MMC Workshop Graz, I-Know -September 7-8, 2006 Extending the Community
    • MMC Workshop Aachen, BTW - March 5-6, 2007 Multimedia Semantics – The Emergent Role of Metadata
    • MMC Workshop Graz, I-Media – September 6-7, 2007 Multimedia Metadata Applications
    • MMC Workshop Klagenfurt, WIAMIS, March 7-8, 2008 Multimedia Metadata Services
  • Multimedia management features of MPEG-7 Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • XML Schema Language - Descriptors, DS, and DDL DDL DS DS DS D D D D DS D D Structural aspects of domains Domain spec. MPEG-7 extension MPEG-7 descriptor or DS non MPEG-7 descriptor or DS
  • Schema Tools Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • MPEG-7 Root Element
  • MPEG-7 Top Elements
    • Graph data type represents a graph a a list of nodes and relations between nodes in an arbitrary sorting
    Basis Data Type – Graphs D B E A C r3 r3 r2 r1 r1 r4 <Graph> <Node id=&quot;A&quot; /><Node id=&quot;B&quot; /> <Node id=&quot;C&quot; /><Node id=&quot;D&quot; /> <Node id=&quot;E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#A&quot; target= =&quot;#B&quot; /> <Relation type= =&quot;#r2&quot; source= &quot;#A&quot; target= =&quot;#C&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#B&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#C&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r4&quot; source= &quot;#B&quot; target= =&quot;#E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#E&quot; target= =&quot;#A&quot; /> </Graph>
  • Annotations <TextAnnotation> <StructuredAnnotation> <Who><Name>Spain</Name</Who> <Where><Name>AC</Name></Where> <When><Name>Mar15</Name></When> </StructuredAnnotation> </TextAnnotation> Structured Annotation <TextAnnotation> <KeywordAnnotation> <Keyword>score</Keyword> <Keyword>Sweden</Keyword> <Keyword>Spain</Keyword> </KeywordAnnotation> </TextAnnotation> Keyword Annotation <TextAnnotation xml:lang=&quot;en-us&quot;> This is a nice apartment. </ TextAnnotation >   <TextAnnotation xml:lang=&quot;en-uk&quot;> This is a nice flat. </TextAnnotation > Free Text Annotation
  • Textual Annotations Score Spain A goal Sweden Against Governer Dependent Governer Dependent Dependent Dependent Dependency structure for &quot;Spain scored a goal against Sweden“
  • Content Management and Description Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • Content Structures – Segment Entities Räumliche und zeitliche Segmente
  • Content Structures – Segment Entities Raum-zeitliches Segment
  • Content Structures – Segment Decomposition Segment decomposition
  • Content Structures – Structural Relations
    • The tools for structural relations can be used for the description of general segment structures
    • Tools are
      • SpatialRelation Classification Scheme
      • TemporalRelation Classification Scheme
      Typed normative structural relations in MPEG-7 Precedes, follows, meets, metBy, overlaps, overlappedBy, contains, during, strictContains, stricrtDuring, starts, startedBy, finishes, finishedBy, coOccurs, contiguous, sequential, coBegin, coned, parallel, overlapping. Temporal South, North, West, East, Northwest, Northeast, Southwest, Southeast, Left, Right, Below, Above, Over, Under. Spatial Normative Relations Type
  • Example for Segmentation Tree Foreground Background SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation
  • Example for Content Semantics Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis
  • Putting it all together
  • Navigation and Access Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • Summaries – Keyframes Original Conten (represented by Keyframes) and highlighted summary content MovieClip1 Summary MovieClip2 MovieClip3
  • Hierarchical Summary Hierarchical Summary Summary Theme List Summary SegmentGroup Summary Segment 0, 1 1, * 1, * 0, *
  • Sequential Summary Sequential Summary Visual SummaryComponent Audio SummaryComponent Textual SummaryComponent 0, * 0, * 0, *
  • Views Multimedia program Multimedia program View Filtering Partition Region View Signal Source Signal View graphs View trees View sets Source Target View decompositions
  • Views – Space and Frequency Graph F F F S S S
  • Variations Variation Set Multimedia program Variation Fidelity Relationship Priority Multimedia program Variation Fidelity Relationship Priority Multimedia program Image Video Video Source Variation Variation
  • Content Organization Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • Collections Collection (abstract) Segment collection Content collection Descriptor collection Concept collection Mixed collection Collection structure
  • Content Collections and Collection Structure Different collection can be combined according to their collection structure Collection A Collection B Collection C R ac R bc R ab
  • Models Model (abstract) Probability Model Analytic Model Cluster Model Classification Model
    • Probability Model
    • Discrete distribution
    • Continuous distribution
    • Finite State Model
    • Collection Model
    • Probability Model
    • Class
    • Cluster Model
    • ClusterClassification
    • Model
    • Probabilty
    • Classification
    • Model
  • Cluster Model – Relevance Feedback After relevant content is marked by the user, the retrieval system can deliver more precise content clusters
  • State Transition Models State Transition models can be used for summarizing or classifying transitions of events on the timeline of a video sequence Video sequence with events, e.g. staging of an actor A C B p ab p ac p bc p ca Label Scene A C B Event A Event B Event C
  • User Interaction Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
  • Multimedia System with User Interaction Content filter & search engine Content browsing engine User profiling engine Local multimedia system Multimedia content description Multimedia content Multimedia content description Multimedia content description User preferences User action history Other devices Content/service provider User
  • Usage History Usage history 1, * 0, * UserIdentifier User action history Observation period User action list ActionType Action data item User action ProgramIdentifier ActionTime 1, * 1, * 0, *
  • User Preferences User preferences UserIdentifier Browsing preferences Summary preferences Preference condition Filtering and search preferences Preference condition Classification preferences Creation preferences Source preferences 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, *
  • Mapping Usage Hisotries on User Preferences Usage history description UserAction Program ID 1 UserAction Program ID 2 Content description Program ID 1 Title 1 Genre A Content description Program ID 2 Title 2 Genre B User preference description Classification Preferences Genre A preference Value a Genre B preference Value b
  • Visual & Audio Descriptors
    • Only a short overview
    • MPEG-7 provides dozens of descriptors
    • Implementation is sometimes tricky
    • New descriptors can be developed by everybody
  • Color Space Descriptor I
  • Color Space Descriptor II
  • Texture Browsing Descriptor I
  • Texture Browsing Descriptor II Directions of textures:
  • Edge Histogram Descriptor
  • Overview Shape Descriptors
    • MPEG-7 defines three basis types of shape descriptors (SD):
    • Region-based SD (2-D objects)
    • Contour-based SD (2-D objects)
    • 3D SD
    Special type: Multiview Type Combination of Multiview and SD for 3D Object description (3D Form derived from multiple perspective 2-D views) Region-based notion of similarity Contour-based notion of similarity
  • Region Based SD
    • Expresses the distribution of pixels of a region or a 2D object
    • Compact and efficient description of complex objects (multiple not connected regions and simple object resp. holes)
  • Contour Based SD
    • Emulates the form based simularity perception of humans
    • Application on single regions with complex or simple outer lines (also open)
  • Motion Descriptors
  • Multimedia Community Hosting
    • Current Trends
      • Mobile multimedia sharing
      • Mobile community hosting
      • Mobile story-telling
    • Aim: MPEG-7 services
      • combination with other services
      • replaceable/extendable during runtime
      • available across different communities
      • Individual access rights communities
      • integration into other systems
  • LAS – Architecture Overview Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository Invoking services Data flows
  • Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository MPEG-7 Services Context Services Media Creation GPS-augmented multimedia creation Multimedia GIS Map Services Media Search Semantic Browsing MPEG-7 Services Context aware multimedia GIS Map Services Media Search Semantic Browsing Context Services MPEG-7 Services MPEG-7 multimedia tagging and commsonomy SNA Tools Media Search Media Tagging MPEG-7 Services MPEG-7 Services Context Services Media Search Semantic Browsing MPEG-7 enabled context aware multimedia search MPEG-7 Services Storytelling Service Semantic Browsing Non-linear digital storytelling Storytelling Service Storytelling on the ipod Context aware multimedia search Media Search MPEG-7 Services Context Services
  • MPEG-7 LAS Service Framework
    • LAS Extension for MPEG-7 Service Development
      • convenience methods for developers
      • create, retrieve, update, delete, validate MPEG-7
      • persistency with MPEG-7 XML database
      • automatic binding with Apache XMLBeans
  • Image/Video Tagging with NMV
    • powered by MPEG-7 LAS services
    • MPEG-7 keyword/semantic tagging
    • image retrieval by expression over tags
    • visualization of tag relationships
  • Virtual Campfire
  • Conclusions & Outlook
    • Conclusions
      • MPEG-7 Multimedia Content Management
      • Visual & Audio Descriptors
      • MPEG-7 Community Hosting
      • MPEG-7 Services
    • Outlook
      • Virtual Campfire