The document discusses MPEG-7, a standard for multimedia metadata. It provides an overview of MPEG-7 descriptors for describing visual and audio content, as well as multimedia communities that promote MPEG-7 adoption and interoperability. The document also outlines various applications of MPEG-7 for multimedia content management, description, navigation and retrieval.

1. MPEG-7 Services in Community Engines Ralf Klamma Leuven, Belgium October 26, 2007

2. Agenda MPEG-7 Multimedia Community MPEG-7 Overview Visual & Audio Descriptors Multimedia Community Hosting Applications Conclusions & Outlook

3. Multimedia Metadata Community Our Goals The Multimedia Metadata Community ( www.multimedia-metadata.info ) aims at establishing a community of professionals in research and industry bridging the gap between an academic research and an industrial scale development of innovative products with a strong focus on MPEG-7 & MPEG-21 establishing better metadata interoperability among different research and development groups collecting of applications and tools

4. Multimedia Metadata Community Our Goals Multimedia Analysis and Retrieval Content Adaptation and Personalization Learning and Multimedia Multimedia and Web 2.0 Semantic and Knowledge in Multimedia Mobile Multimedia and Digital Television Multimedia Interoperability Multimedia standards and related issues Multimedia Metadata Standards Cross-walking

5. Multimedia Metadata Community Partners University of Passau, Distributed Information Systems Lehrstuhl Informatik 5, RWTH Aachen University T-Systems International GmbH Siemens AG Know Center, Graz Klagenfurt University, Department of Information Technology (ITEC) JOANNEUM RESEARCH, Graz Vienna University of Technology, Interactive Media Systems Group University of Vienna - Multimedia Information Systems Group Graz University of Technology, Institute for Information Systems and New Media (IICM) ENSEEIHT Toulouse Franche-Comté University Institut National de Science Appliqués de Lyon Institut National des Télécommunications, Evry

6. Multimedia Metadata Community – Events MMC Workshop Klagenfurt - March 10, 2005 MPEG-7 and Multimedia Metadata: Standardization and Application MMC Workshop, Graz, I-Know - June 30 – July 1, 2005 What can MPEG-7 do for you? MMC Workshop, Aachen - November 24-25, 2006 Test-Set for Multimedia Interoperability and Application Scenario MMC Workshop, Berlin - March 2-3, 2006 Multimedia Interoperability & MPEG-21 MMC Workshop Graz, I-Know -September 7-8, 2006 Extending the Community MMC Workshop Aachen, BTW - March 5-6, 2007 Multimedia Semantics – The Emergent Role of Metadata MMC Workshop Graz, I-Media – September 6-7, 2007 Multimedia Metadata Applications MMC Workshop Klagenfurt, WIAMIS, March 7-8, 2008 Multimedia Metadata Services

7. Multimedia management features of MPEG-7 Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

8. XML Schema Language - Descriptors, DS, and DDL DDL DS DS DS D D D D DS D D Structural aspects of domains Domain spec. MPEG-7 extension MPEG-7 descriptor or DS non MPEG-7 descriptor or DS

9. Schema Tools Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

10. MPEG-7 Root Element

11. MPEG-7 Top Elements

12. Graph data type represents a graph a a list of nodes and relations between nodes in an arbitrary sorting Basis Data Type – Graphs D B E A C r3 r3 r2 r1 r1 r4 <Graph> <Node id=&quot;A&quot; /><Node id=&quot;B&quot; /> <Node id=&quot;C&quot; /><Node id=&quot;D&quot; /> <Node id=&quot;E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#A&quot; target= =&quot;#B&quot; /> <Relation type= =&quot;#r2&quot; source= &quot;#A&quot; target= =&quot;#C&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#B&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#C&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r4&quot; source= &quot;#B&quot; target= =&quot;#E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#E&quot; target= =&quot;#A&quot; /> </Graph>

13. Annotations <TextAnnotation> <StructuredAnnotation> <Who><Name>Spain</Name</Who> <Where><Name>AC</Name></Where> <When><Name>Mar15</Name></When> </StructuredAnnotation> </TextAnnotation> Structured Annotation <TextAnnotation> <KeywordAnnotation> <Keyword>score</Keyword> <Keyword>Sweden</Keyword> <Keyword>Spain</Keyword> </KeywordAnnotation> </TextAnnotation> Keyword Annotation <TextAnnotation xml:lang=&quot;en-us&quot;> This is a nice apartment. </ TextAnnotation > <TextAnnotation xml:lang=&quot;en-uk&quot;> This is a nice flat. </TextAnnotation > Free Text Annotation

14. Textual Annotations Score Spain A goal Sweden Against Governer Dependent Governer Dependent Dependent Dependent Dependency structure for &quot;Spain scored a goal against Sweden“

15. Content Management and Description Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

16. Content Structures – Segment Entities Räumliche und zeitliche Segmente

17. Content Structures – Segment Entities Raum-zeitliches Segment

18. Content Structures – Segment Decomposition Segment decomposition

19. Content Structures – Structural Relations The tools for structural relations can be used for the description of general segment structures Tools are SpatialRelation Classification Scheme TemporalRelation Classification Scheme Typed normative structural relations in MPEG-7 Precedes, follows, meets, metBy, overlaps, overlappedBy, contains, during, strictContains, stricrtDuring, starts, startedBy, finishes, finishedBy, coOccurs, contiguous, sequential, coBegin, coned, parallel, overlapping. Temporal South, North, West, East, Northwest, Northeast, Southwest, Southeast, Left, Right, Below, Above, Over, Under. Spatial Normative Relations Type

20. Example for Segmentation Tree Foreground Background SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation

21. Example for Content Semantics Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis

22. Putting it all together

23. Navigation and Access Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

24. Summaries – Keyframes Original Conten (represented by Keyframes) and highlighted summary content MovieClip1 Summary MovieClip2 MovieClip3

25. Hierarchical Summary Hierarchical Summary Summary Theme List Summary SegmentGroup Summary Segment 0, 1 1, * 1, * 0, *

26. Sequential Summary Sequential Summary Visual SummaryComponent Audio SummaryComponent Textual SummaryComponent 0, * 0, * 0, *

27. Views Multimedia program Multimedia program View Filtering Partition Region View Signal Source Signal View graphs View trees View sets Source Target View decompositions

28. Views – Space and Frequency Graph F F F S S S

29. Variations Variation Set Multimedia program Variation Fidelity Relationship Priority Multimedia program Variation Fidelity Relationship Priority Multimedia program Image Video Video Source Variation Variation

30. Content Organization Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

31. Collections Collection (abstract) Segment collection Content collection Descriptor collection Concept collection Mixed collection Collection structure

32. Content Collections and Collection Structure Different collection can be combined according to their collection structure Collection A Collection B Collection C R ac R bc R ab

33. Models Model (abstract) Probability Model Analytic Model Cluster Model Classification Model Probability Model Discrete distribution Continuous distribution Finite State Model Collection Model Probability Model Class Cluster Model ClusterClassification Model Probabilty Classification Model

34. Cluster Model – Relevance Feedback After relevant content is marked by the user, the retrieval system can deliver more precise content clusters

35. State Transition Models State Transition models can be used for summarizing or classifying transitions of events on the timeline of a video sequence Video sequence with events, e.g. staging of an actor A C B p ab p ac p bc p ca Label Scene A C B Event A Event B Event C

36. User Interaction Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

37. Multimedia System with User Interaction Content filter & search engine Content browsing engine User profiling engine Local multimedia system Multimedia content description Multimedia content Multimedia content description Multimedia content description User preferences User action history Other devices Content/service provider User

38. Usage History Usage history 1, * 0, * UserIdentifier User action history Observation period User action list ActionType Action data item User action ProgramIdentifier ActionTime 1, * 1, * 0, *

39. User Preferences User preferences UserIdentifier Browsing preferences Summary preferences Preference condition Filtering and search preferences Preference condition Classification preferences Creation preferences Source preferences 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, *

40. Mapping Usage Hisotries on User Preferences Usage history description UserAction Program ID 1 UserAction Program ID 2 Content description Program ID 1 Title 1 Genre A Content description Program ID 2 Title 2 Genre B User preference description Classification Preferences Genre A preference Value a Genre B preference Value b

41. Visual & Audio Descriptors Only a short overview MPEG-7 provides dozens of descriptors Implementation is sometimes tricky New descriptors can be developed by everybody

42. Color Space Descriptor I

43. Color Space Descriptor II

44. Texture Browsing Descriptor I

45. Texture Browsing Descriptor II Directions of textures:

46. Edge Histogram Descriptor

47. Overview Shape Descriptors MPEG-7 defines three basis types of shape descriptors (SD): Region-based SD (2-D objects) Contour-based SD (2-D objects) 3D SD Special type: Multiview Type Combination of Multiview and SD for 3D Object description (3D Form derived from multiple perspective 2-D views) Region-based notion of similarity Contour-based notion of similarity

48. Region Based SD Expresses the distribution of pixels of a region or a 2D object Compact and efficient description of complex objects (multiple not connected regions and simple object resp. holes)

49. Contour Based SD Emulates the form based simularity perception of humans Application on single regions with complex or simple outer lines (also open)

50. Motion Descriptors

51. Multimedia Community Hosting Current Trends Mobile multimedia sharing Mobile community hosting Mobile story-telling Aim: MPEG-7 services combination with other services replaceable/extendable during runtime available across different communities Individual access rights communities integration into other systems

52. LAS – Architecture Overview Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository Invoking services Data flows

53. Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository MPEG-7 Services Context Services Media Creation GPS-augmented multimedia creation Multimedia GIS Map Services Media Search Semantic Browsing MPEG-7 Services Context aware multimedia GIS Map Services Media Search Semantic Browsing Context Services MPEG-7 Services MPEG-7 multimedia tagging and commsonomy SNA Tools Media Search Media Tagging MPEG-7 Services MPEG-7 Services Context Services Media Search Semantic Browsing MPEG-7 enabled context aware multimedia search MPEG-7 Services Storytelling Service Semantic Browsing Non-linear digital storytelling Storytelling Service Storytelling on the ipod Context aware multimedia search Media Search MPEG-7 Services Context Services

54. MPEG-7 LAS Service Framework LAS Extension for MPEG-7 Service Development convenience methods for developers create, retrieve, update, delete, validate MPEG-7 persistency with MPEG-7 XML database automatic binding with Apache XMLBeans

55. Image/Video Tagging with NMV powered by MPEG-7 LAS services MPEG-7 keyword/semantic tagging image retrieval by expression over tags visualization of tag relationships

56. Virtual Campfire

57. Conclusions & Outlook Conclusions MPEG-7 Multimedia Content Management Visual & Audio Descriptors MPEG-7 Community Hosting MPEG-7 Services Outlook Virtual Campfire