MPEG-7 Services in Community Engines

1,786 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,786
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • MPEG-7 Services in Community Engines

    1. 1. MPEG-7 Services in Community Engines Ralf Klamma Leuven, Belgium October 26, 2007
    2. 2. Agenda <ul><li>MPEG-7 Multimedia Community </li></ul><ul><li>MPEG-7 </li></ul><ul><ul><li>Overview </li></ul></ul><ul><ul><li>Visual & Audio Descriptors </li></ul></ul><ul><li>Multimedia Community Hosting </li></ul><ul><li>Applications </li></ul><ul><li>Conclusions & Outlook </li></ul>
    3. 3. Multimedia Metadata Community Our Goals <ul><ul><li>The Multimedia Metadata Community ( www.multimedia-metadata.info ) aims at </li></ul></ul><ul><ul><li>establishing a community of professionals in research and industry </li></ul></ul><ul><ul><li>bridging the gap between an academic research and an industrial scale development of innovative products with a strong focus on MPEG-7 & MPEG-21 </li></ul></ul><ul><ul><li>establishing better metadata interoperability among different research and development groups </li></ul></ul><ul><ul><li>collecting of applications and tools </li></ul></ul>
    4. 4. Multimedia Metadata Community Our Goals <ul><ul><li>Multimedia Analysis and Retrieval </li></ul></ul><ul><ul><li>Content Adaptation and Personalization </li></ul></ul><ul><ul><li>Learning and Multimedia </li></ul></ul><ul><ul><li>Multimedia and Web 2.0 </li></ul></ul><ul><ul><li>Semantic and Knowledge in Multimedia </li></ul></ul><ul><ul><li>Mobile Multimedia and Digital Television </li></ul></ul><ul><ul><li>Multimedia Interoperability </li></ul></ul><ul><ul><li>Multimedia standards and related issues </li></ul></ul><ul><ul><li>Multimedia Metadata Standards Cross-walking </li></ul></ul>
    5. 5. Multimedia Metadata Community Partners <ul><ul><li>University of Passau, Distributed Information Systems </li></ul></ul><ul><ul><li>Lehrstuhl Informatik 5, RWTH Aachen University </li></ul></ul><ul><ul><li>T-Systems International GmbH </li></ul></ul><ul><ul><li>Siemens AG  </li></ul></ul><ul><ul><li>Know Center, Graz </li></ul></ul><ul><ul><li>Klagenfurt University, Department of Information Technology (ITEC)     </li></ul></ul><ul><ul><li>JOANNEUM RESEARCH, Graz </li></ul></ul><ul><ul><li>Vienna University of Technology, Interactive Media Systems Group </li></ul></ul><ul><ul><li>University of Vienna - Multimedia Information Systems Group </li></ul></ul><ul><ul><li>Graz University of Technology, Institute for Information Systems and New Media (IICM) </li></ul></ul><ul><ul><li>ENSEEIHT Toulouse </li></ul></ul><ul><ul><li>Franche-Comté University   </li></ul></ul><ul><ul><li>Institut National de Science Appliqués de Lyon   </li></ul></ul><ul><ul><li>Institut National des Télécommunications, Evry   </li></ul></ul>
    6. 6. Multimedia Metadata Community – Events <ul><li>MMC Workshop Klagenfurt - March 10, 2005 MPEG-7 and Multimedia Metadata: Standardization and Application </li></ul><ul><li>MMC Workshop, Graz, I-Know - June 30 – July 1, 2005 What can MPEG-7 do for you? </li></ul><ul><li>MMC Workshop, Aachen - November 24-25, 2006 Test-Set for Multimedia Interoperability and Application Scenario </li></ul><ul><li>MMC Workshop, Berlin - March 2-3, 2006 Multimedia Interoperability & MPEG-21 </li></ul><ul><li>MMC Workshop Graz, I-Know -September 7-8, 2006 Extending the Community </li></ul><ul><li>MMC Workshop Aachen, BTW - March 5-6, 2007 Multimedia Semantics – The Emergent Role of Metadata </li></ul><ul><li>MMC Workshop Graz, I-Media – September 6-7, 2007 Multimedia Metadata Applications </li></ul><ul><li>MMC Workshop Klagenfurt, WIAMIS, March 7-8, 2008 Multimedia Metadata Services </li></ul>
    7. 7. Multimedia management features of MPEG-7 Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    8. 8. XML Schema Language - Descriptors, DS, and DDL DDL DS DS DS D D D D DS D D Structural aspects of domains Domain spec. MPEG-7 extension MPEG-7 descriptor or DS non MPEG-7 descriptor or DS
    9. 9. Schema Tools Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    10. 10. MPEG-7 Root Element
    11. 11. MPEG-7 Top Elements
    12. 12. <ul><li>Graph data type represents a graph a a list of nodes and relations between nodes in an arbitrary sorting </li></ul>Basis Data Type – Graphs D B E A C r3 r3 r2 r1 r1 r4 <Graph> <Node id=&quot;A&quot; /><Node id=&quot;B&quot; /> <Node id=&quot;C&quot; /><Node id=&quot;D&quot; /> <Node id=&quot;E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#A&quot; target= =&quot;#B&quot; /> <Relation type= =&quot;#r2&quot; source= &quot;#A&quot; target= =&quot;#C&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#B&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r3&quot; source= &quot;#C&quot; target= =&quot;#D&quot; /> <Relation type= =&quot;#r4&quot; source= &quot;#B&quot; target= =&quot;#E&quot; /> <Relation type= =&quot;#r1&quot; source= &quot;#E&quot; target= =&quot;#A&quot; /> </Graph>
    13. 13. Annotations <TextAnnotation> <StructuredAnnotation> <Who><Name>Spain</Name</Who> <Where><Name>AC</Name></Where> <When><Name>Mar15</Name></When> </StructuredAnnotation> </TextAnnotation> Structured Annotation <TextAnnotation> <KeywordAnnotation> <Keyword>score</Keyword> <Keyword>Sweden</Keyword> <Keyword>Spain</Keyword> </KeywordAnnotation> </TextAnnotation> Keyword Annotation <TextAnnotation xml:lang=&quot;en-us&quot;> This is a nice apartment. </ TextAnnotation >   <TextAnnotation xml:lang=&quot;en-uk&quot;> This is a nice flat. </TextAnnotation > Free Text Annotation
    14. 14. Textual Annotations Score Spain A goal Sweden Against Governer Dependent Governer Dependent Dependent Dependent Dependency structure for &quot;Spain scored a goal against Sweden“
    15. 15. Content Management and Description Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    16. 16. Content Structures – Segment Entities Räumliche und zeitliche Segmente
    17. 17. Content Structures – Segment Entities Raum-zeitliches Segment
    18. 18. Content Structures – Segment Decomposition Segment decomposition
    19. 19. Content Structures – Structural Relations <ul><li>The tools for structural relations can be used for the description of general segment structures </li></ul><ul><li>Tools are </li></ul><ul><ul><li>SpatialRelation Classification Scheme </li></ul></ul><ul><ul><li>TemporalRelation Classification Scheme </li></ul></ul>  Typed normative structural relations in MPEG-7 Precedes, follows, meets, metBy, overlaps, overlappedBy, contains, during, strictContains, stricrtDuring, starts, startedBy, finishes, finishedBy, coOccurs, contiguous, sequential, coBegin, coned, parallel, overlapping. Temporal South, North, West, East, Northwest, Northeast, Southwest, Southeast, Left, Right, Below, Above, Over, Under. Spatial Normative Relations Type
    20. 20. Example for Segmentation Tree Foreground Background SR1: · Creation, Usage meta information · Media description · Textual annotation · Color histogram, Texture SR2: · Shape · Color Histogram · Textual annotation SR6: · Color Histogram · Textual annotation SR5: · Shape · Textual annotation SR4: · Shape · Color Histogram · Textual annotation SR3: · Shape · Color Histogram · Textual annotation
    21. 21. Example for Content Semantics Segment Tree Shot1 Shot2 Shot3 Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7 Semantic DS (Events) • Introduction • Summary • Program logo • Studio • Overview • News Presenter • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Sports • Closing Time Axis
    22. 22. Putting it all together
    23. 23. Navigation and Access Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    24. 24. Summaries – Keyframes Original Conten (represented by Keyframes) and highlighted summary content MovieClip1 Summary MovieClip2 MovieClip3
    25. 25. Hierarchical Summary Hierarchical Summary Summary Theme List Summary SegmentGroup Summary Segment 0, 1 1, * 1, * 0, *
    26. 26. Sequential Summary Sequential Summary Visual SummaryComponent Audio SummaryComponent Textual SummaryComponent 0, * 0, * 0, *
    27. 27. Views Multimedia program Multimedia program View Filtering Partition Region View Signal Source Signal View graphs View trees View sets Source Target View decompositions
    28. 28. Views – Space and Frequency Graph F F F S S S
    29. 29. Variations Variation Set Multimedia program Variation Fidelity Relationship Priority Multimedia program Variation Fidelity Relationship Priority Multimedia program Image Video Video Source Variation Variation
    30. 30. Content Organization Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    31. 31. Collections Collection (abstract) Segment collection Content collection Descriptor collection Concept collection Mixed collection Collection structure
    32. 32. Content Collections and Collection Structure Different collection can be combined according to their collection structure Collection A Collection B Collection C R ac R bc R ab
    33. 33. Models Model (abstract) Probability Model Analytic Model Cluster Model Classification Model <ul><li>Probability Model </li></ul><ul><li>Discrete distribution </li></ul><ul><li>Continuous distribution </li></ul><ul><li>Finite State Model </li></ul><ul><li>Collection Model </li></ul><ul><li>Probability Model </li></ul><ul><li>Class </li></ul><ul><li>Cluster Model </li></ul><ul><li>ClusterClassification </li></ul><ul><li>Model </li></ul><ul><li>Probabilty </li></ul><ul><li>Classification </li></ul><ul><li>Model </li></ul>
    34. 34. Cluster Model – Relevance Feedback After relevant content is marked by the user, the retrieval system can deliver more precise content clusters
    35. 35. State Transition Models State Transition models can be used for summarizing or classifying transitions of events on the timeline of a video sequence Video sequence with events, e.g. staging of an actor A C B p ab p ac p bc p ca Label Scene A C B Event A Event B Event C
    36. 36. User Interaction Basis Schema Links&Media Basic Basic tools localization tools data types [IEEE 02] User interaction User preferences Usage History Navigation & Access Index Views Variations Content organization Collections Models Creation & Production Content Management Media Usage Content description Structural aspects Semantic aspects
    37. 37. Multimedia System with User Interaction Content filter & search engine Content browsing engine User profiling engine Local multimedia system Multimedia content description Multimedia content Multimedia content description Multimedia content description User preferences User action history Other devices Content/service provider User
    38. 38. Usage History Usage history 1, * 0, * UserIdentifier User action history Observation period User action list ActionType Action data item User action ProgramIdentifier ActionTime 1, * 1, * 0, *
    39. 39. User Preferences User preferences UserIdentifier Browsing preferences Summary preferences Preference condition Filtering and search preferences Preference condition Classification preferences Creation preferences Source preferences 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, * 0, *
    40. 40. Mapping Usage Hisotries on User Preferences Usage history description UserAction Program ID 1 UserAction Program ID 2 Content description Program ID 1 Title 1 Genre A Content description Program ID 2 Title 2 Genre B User preference description Classification Preferences Genre A preference Value a Genre B preference Value b
    41. 41. Visual & Audio Descriptors <ul><li>Only a short overview </li></ul><ul><li>MPEG-7 provides dozens of descriptors </li></ul><ul><li>Implementation is sometimes tricky </li></ul><ul><li>New descriptors can be developed by everybody </li></ul>
    42. 42. Color Space Descriptor I
    43. 43. Color Space Descriptor II
    44. 44. Texture Browsing Descriptor I
    45. 45. Texture Browsing Descriptor II Directions of textures:
    46. 46. Edge Histogram Descriptor
    47. 47. Overview Shape Descriptors <ul><li>MPEG-7 defines three basis types of shape descriptors (SD): </li></ul><ul><li>Region-based SD (2-D objects) </li></ul><ul><li>Contour-based SD (2-D objects) </li></ul><ul><li>3D SD </li></ul>Special type: Multiview Type Combination of Multiview and SD for 3D Object description (3D Form derived from multiple perspective 2-D views) Region-based notion of similarity Contour-based notion of similarity
    48. 48. Region Based SD <ul><li>Expresses the distribution of pixels of a region or a 2D object </li></ul><ul><li>Compact and efficient description of complex objects (multiple not connected regions and simple object resp. holes) </li></ul>
    49. 49. Contour Based SD <ul><li>Emulates the form based simularity perception of humans </li></ul><ul><li>Application on single regions with complex or simple outer lines (also open) </li></ul>
    50. 50. Motion Descriptors
    51. 51. Multimedia Community Hosting <ul><li>Current Trends </li></ul><ul><ul><li>Mobile multimedia sharing </li></ul></ul><ul><ul><li>Mobile community hosting </li></ul></ul><ul><ul><li>Mobile story-telling </li></ul></ul><ul><li>Aim: MPEG-7 services </li></ul><ul><ul><li>combination with other services </li></ul></ul><ul><ul><li>replaceable/extendable during runtime </li></ul></ul><ul><ul><li>available across different communities </li></ul></ul><ul><ul><li>Individual access rights communities </li></ul></ul><ul><ul><li>integration into other systems </li></ul></ul>
    52. 52. LAS – Architecture Overview Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository Invoking services Data flows
    53. 53. Data Sources Context-Aware Services Community Engine WWW MPEG-7 Services Context Services User Manager Map Services Storytelling Service Object Manager (Mobile) Interfaces Session Manager SNA Tools Multimedia Processor Multimedia Annotation Multimedia Extractor Multimedia Adaptation Multimedia Player Metadata Databases Connectors: HTTP, SOAP Multimedia Repository Media Creation Media Search Media Tagging Semantic Browsing Mashups Automatic Discovery & Configuration Multimedia Input Data Access Multimedia Repository Multimedia Repository MPEG-7 Services Context Services Media Creation GPS-augmented multimedia creation Multimedia GIS Map Services Media Search Semantic Browsing MPEG-7 Services Context aware multimedia GIS Map Services Media Search Semantic Browsing Context Services MPEG-7 Services MPEG-7 multimedia tagging and commsonomy SNA Tools Media Search Media Tagging MPEG-7 Services MPEG-7 Services Context Services Media Search Semantic Browsing MPEG-7 enabled context aware multimedia search MPEG-7 Services Storytelling Service Semantic Browsing Non-linear digital storytelling Storytelling Service Storytelling on the ipod Context aware multimedia search Media Search MPEG-7 Services Context Services
    54. 54. MPEG-7 LAS Service Framework <ul><li>LAS Extension for MPEG-7 Service Development </li></ul><ul><ul><li>convenience methods for developers </li></ul></ul><ul><ul><li>create, retrieve, update, delete, validate MPEG-7 </li></ul></ul><ul><ul><li>persistency with MPEG-7 XML database </li></ul></ul><ul><ul><li>automatic binding with Apache XMLBeans </li></ul></ul>
    55. 55. Image/Video Tagging with NMV <ul><li>powered by MPEG-7 LAS services </li></ul><ul><li>MPEG-7 keyword/semantic tagging </li></ul><ul><li>image retrieval by expression over tags </li></ul><ul><li>visualization of tag relationships </li></ul>
    56. 56. Virtual Campfire
    57. 57. Conclusions & Outlook <ul><li>Conclusions </li></ul><ul><ul><li>MPEG-7 Multimedia Content Management </li></ul></ul><ul><ul><li>Visual & Audio Descriptors </li></ul></ul><ul><ul><li>MPEG-7 Community Hosting </li></ul></ul><ul><ul><li>MPEG-7 Services </li></ul></ul><ul><li>Outlook </li></ul><ul><ul><li>Virtual Campfire </li></ul></ul>

    ×