The document discusses MPEG-7, a standard for multimedia metadata. It provides an overview of MPEG-7 descriptors for describing visual and audio content, as well as multimedia communities that promote MPEG-7 adoption and interoperability. The document also outlines various applications of MPEG-7 for multimedia content management, description, navigation and retrieval.

1. MPEG-7 Services in Community Engines Ralf Klamma Leuven, Belgium October 26, 2007

7. Multimedia management features of MPEG-7 Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

8. XML Schema Language - Descriptors, DS, and DDL DDL DS DS DS D D D D DS D D Structural aspects of domains Domain spec. MPEG-7 extension MPEG-7 descriptor or DS non MPEG-7 descriptor or DS

9. Schema Tools Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

10. MPEG-7 Root Element

11. MPEG-7 Top Elements

13. Annotations <TextAnnotation> <StructuredAnnotation> <Who><Name>Spain</Name</Who> <Where><Name>AC</Name></Where> <When><Name>Mar15</Name></When> </StructuredAnnotation> </TextAnnotation> Structured Annotation <TextAnnotation> <KeywordAnnotation> <Keyword>score</Keyword> <Keyword>Sweden</Keyword> <Keyword>Spain</Keyword> </KeywordAnnotation> </TextAnnotation> Keyword Annotation <TextAnnotation xml:lang=&quot;en-us&quot;> This is a nice apartment. </ TextAnnotation > <TextAnnotation xml:lang=&quot;en-uk&quot;> This is a nice flat. </TextAnnotation > Free Text Annotation

14. Textual Annotations Score Spain A goal Sweden Against Governer Dependent Governer Dependent Dependent Dependent Dependency structure for &quot;Spain scored a goal against Sweden“

15. Content Management and Description Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

16. Content Structures – Segment Entities Räumliche und zeitliche Segmente

17. Content Structures – Segment Entities Raum-zeitliches Segment

18. Content Structures – Segment Decomposition Segment decomposition

20. Example for Segmentation Tree Foreground Background SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation

21. Example for Content Semantics Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis

22. Putting it all together

23. Navigation and Access Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

24. Summaries – Keyframes Original Conten (represented by Keyframes) and highlighted summary content MovieClip1 Summary MovieClip2 MovieClip3

25. Hierarchical Summary Hierarchical Summary Summary Theme List Summary SegmentGroup Summary Segment 0, 1 1, * 1, * 0, *

26. Sequential Summary Sequential Summary Visual SummaryComponent Audio SummaryComponent Textual SummaryComponent 0, * 0, * 0, *

27. Views Multimedia program Multimedia program View Filtering Partition Region View Signal Source Signal View graphs View trees View sets Source Target View decompositions

28. Views – Space and Frequency Graph F F F S S S

29. Variations Variation Set Multimedia program Variation Fidelity Relationship Priority Multimedia program Variation Fidelity Relationship Priority Multimedia program Image Video Video Source Variation Variation

30. Content Organization Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

31. Collections Collection (abstract) Segment collection Content collection Descriptor collection Concept collection Mixed collection Collection structure

32. Content Collections and Collection Structure Different collection can be combined according to their collection structure Collection A Collection B Collection C R ac R bc R ab

34. Cluster Model – Relevance Feedback After relevant content is marked by the user, the retrieval system can deliver more precise content clusters

35. State Transition Models State Transition models can be used for summarizing or classifying transitions of events on the timeline of a video sequence Video sequence with events, e.g. staging of an actor A C B p ab p ac p bc p ca Label Scene A C B Event A Event B Event C

36. User Interaction Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects

37. Multimedia System with User Interaction Content filter & search engine Content browsing engine User profiling engine Local multimedia system Multimedia content description Multimedia content Multimedia content description Multimedia content description User preferences User action history Other devices Content/service provider User

38. Usage History Usage history 1, * 0, * UserIdentifier User action history Observation period User action list ActionType Action data item User action ProgramIdentifier ActionTime 1, * 1, * 0, *

39. User Preferences User preferences UserIdentifier Browsing preferences Summary preferences Preference condition Filtering and search preferences Preference condition Classification preferences Creation preferences Source preferences 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, *

40. Mapping Usage Hisotries on User Preferences Usage history description UserAction Program ID 1 UserAction Program ID 2 Content description Program ID 1 Title 1 Genre A Content description Program ID 2 Title 2 Genre B User preference description Classification Preferences Genre A preference Value a Genre B preference Value b

42. Color Space Descriptor I

43. Color Space Descriptor II

44. Texture Browsing Descriptor I

45. Texture Browsing Descriptor II Directions of textures:

46. Edge Histogram Descriptor

50. Motion Descriptors

52. LAS – Architecture Overview Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository Invoking services Data flows

53. Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository MPEG-7 Services Context Services Media Creation GPS-augmented multimedia creation Multimedia GIS Map Services Media Search Semantic Browsing MPEG-7 Services Context aware multimedia GIS Map Services Media Search Semantic Browsing Context Services MPEG-7 Services MPEG-7 multimedia tagging and commsonomy SNA Tools Media Search Media Tagging MPEG-7 Services MPEG-7 Services Context Services Media Search Semantic Browsing MPEG-7 enabled context aware multimedia search MPEG-7 Services Storytelling Service Semantic Browsing Non-linear digital storytelling Storytelling Service Storytelling on the ipod Context aware multimedia search Media Search MPEG-7 Services Context Services

56. Virtual Campfire