C14   fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing results of automatic annotation services
Upcoming SlideShare
Loading in...5
×
 

C14 fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing results of automatic annotation services

on

  • 622 views

 

Statistics

Views

Total Views
622
Views on SlideShare
502
Embed Views
120

Actions

Likes
0
Downloads
13
Comments
0

3 Embeds 120

http://innovatie.vrt.be 74
http://fiatifta.org 39
http://innovation.vrt.be 7

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

C14   fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing results of automatic annotation services C14 fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing results of automatic annotation services Presentation Transcript

  • “THE MPEG-7 AUDIOVISUAL DESCRIPTION PROFILE STANDARD FOR DESCRIBING RESULTS OF AUTOMATIC ANNOTATION SERVICES” Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina, Jean-Pierre Évain, Peter Schallauer Vlaamse Radio- en Televisieomroeporganisatie, Belgium JOANNEUM RESEARCH, Austria Get in contact: mike.matton@vrt.be; peter.schallauer@joanneum.at  of  this   the  property  of  t uthor(s).   granted  permission  to   Copyright  ©opies  of  ptresenta1on  is  urposes  relevant  he  ahe  above  FcIAT/IFTA  is  and  future  communica1on   reproduce  c his  work  for  p to  t onference   by  FIAT/IFTA  without  limita1on,  provided  that  the  author(s),  source  and  copyright  no1ce  are  included  in   each  copy.  For  other  uses,  including  extended  quota1on,  please  contact  the  author(s).   #FIATIFTADubai2013 AVDP” Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Contents •  •  •  •  Introduction & motivation MPEG-7 AVDP Applications of AVDP Conclusion #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • MPEG-7 Audiovisual Description Profile •  Motivated by activities of EBU MIM/SCAIE group –  Study and evaluation of Content analysis-based Automatic Information Extraction tools in media production –  facilitate the introduction of new production tools based on automatic extraction of information •  Relevant tools include •  •  •  •  •  •  Speech Recognition Audiovisual Segmentation Personality Identification Semantic Detection Text Recognition Subject Classification #FIATIFTADubai2013 •  •  •  •  •  •  Format Detection" Named Entities Detection" Copy / Repetition Detection" Content Summarisation" Keyword Extraction" Automatic text Translation" Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • MPEG-7 Audiovisual Description Profile •  Automatic analysis tools generate –  detailed time-based metadata –  low-level audio/video features (e.g., color descriptors) •  Common metadata format is needed to –  represent results –  exchange between organisations –  perform automated evaluation #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • MPEG-7 AVDP Motivation (TOSCA-MP project) •  Put automatic annotation in automated workflow •  Services need to communicate with one another (e.g. Scene segmentation needs input from shot boundary detection) –  Even across vendors –  MPEG-7 AVDP specifically created for this purpose –  Different such services later in this presentation #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • MPEG-7 •  Multimedia Content Description Interface –  ISO/IEC 15938, version 1 (2001) and 2 (2004) •  Metadata of multimedia content –  descriptors, description schemes –  definition based on an extension of XML Schema –  serialised in XML or binary format •  Multimedia description schemes (part 5) –  content structure (decompositions, segments) –  descriptive and technical metadata •  Visual and Audio (parts 3 and 4) –  descriptors for visual and audio low-level features •  use of parts of MPEG-7 for TV Anytime, EBU Classification Schemes #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • MPEG-7 Pros and Cons •  Support for describing arbitrary content structures –  time segments, regions, moving regions •  •  Support for low-level feature descriptors Benefits of MPEG-7 –  –  –  –  •  designed as a data model, not just as an exchange format broad range of applications flexible, fine-grained description content structuring capabilities Drawbacks –  complexity: generic types, deep hierarchical structures –  interoperability problems: several ways to describe the same, conformance to standard can only be checked on syntactic level but not on a semantic level #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Why profiles for MPEG-7? •  Address issues of complexity and interoperability •  Profiles are a proven concept –  subsets of the standard for certain functionalities and/or applications –  cf. MPEG-2, MPEG-4 #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Why profiles for MPEG-7? •  Definition of an MPEG-7 profile –  Selection of description tools → Profile XML Schema •  which description schemes and descriptors may be used –  Constraints on description tools → Profile XML Schema •  restrictions on cardinalities, child elements, attributes –  Semantic constraints → Textual description •  semantics of the elements in the description •  specify use of the selected tools in the context of the profile •  ensure interoperability #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Source used for defining the new profile •  Detailed Audiovisual Profile (DAVP) –  –  –  –  single multimedia content entities comprehensive structural description of the content audio and visual low-level feature descriptors (parts 3 and 4) defines semantic constraints on structure and elements to address interoperability issues •  NHK Metadata Production Framework (MPF) –  –  –  –  based on the Core Description Profile (CDP) addresses the complexity and ambiguity problems of MPEG-7 audio and visual low-level feature descriptors (parts 3 and 4) semantic constraints for the structure of the description as well as several syntactic and semantic constraints on different elements #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Design goals for a new profile •  Fulfill the requirements of EBU MIM/SCAIE •  Harmonise DAVP and MPF –  both share many design principles –  some implementation choices are different •  Simplify –  based on experience from using DAVP and MPF –  some included elements are never used –  some constructs can be simplified in terms of cardinality/flexibility •  First profile based on MPEG-7 v2 (2004) #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AudioVisual Description " Profile – AVDP •  An MPEG-7 AVDP file contains –  0 or more descriptions of " audiovisual contents –  0 or 1 summaries of the content(s) •  Describe audio, video or " audiovisual content Mpeg7 Description  type=“ContentEntityType” (a)  A  ContentEntityType for  one  audiovisual  content. Mpeg7 Description  type=“ContentEntityType” Description  type=“ContentEntityType” Description  type=“ContentEntityType” –  no other types of multimedia " content –  no collections of content –  same top-level structure for audio, " video or audiovisual content (b)  Several  ContentEntityTypes for  copy  detection   result. Mpeg7 Description  type=“SummaryDescriptionType (c)  A  SummaryDescription. Mpeg7 Description  type=“SummaryDescriptionType Description  type=“ContentEntityType” Description  type=“ContentEntityType” (d)  A  SummaryDescription with  several  ContentEntityTypes. Figure  1    Top  level  structure #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AVDP content description structure #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Constraint definition •  Profile specification document defines –  –  –  –  list of included types constraints on elements and cardinalities semantic constraints on use of these types semantics of description structure •  Profile XML schema –  included types –  excluded elements, restrictions on cardinality –  few of the semantic and structure constraints #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Examples •  Examples provided with AVDP specification –  –  –  –  –  General description structure Content segmentation Speech-to-text Copy detection Summarization •  Annex –  example outputs from RAI and JRS content analysis tools –  MIM/MDN Knowledge base •  http://workspace.ebu.ch/display/ecmmdn/Knowledge+base+-+contributions •  EBU MIM/SCAIE is currently working on a guidelines document #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Validation •  Validation against MPEG-7 XML Schema or Profile XML Schema –  e.g., using validator of an XML editor, Xerces, MS XML, etc. •  many constraints of a profile cannot be represented in XML schema –  e.g., the AudioVisual element, representing the entire content (root a/v segment) is of type AudiovisualSegmentType –  the same type is used to represent many audiovisual segments at other places in the description –  the specific constraints on the elements that need to be present on the root a/v segment but not on others cannot be checked #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AVDP Applications #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • VAMP: Validation of " MPEG-7 Profile Descriptions (including AVDP) •  validate metadata documents –  conforming to complex metadata standards such as MPEG-7 –  semantics of temporal annotations –  beyond XML schema validation •  prevent interoperability problems •  http://vamp.joanneum.at •  Further MPEG-7 ressources •  http://mpeg7.joanneum.at/ #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • VidiCert: AVDP for Video Essence QC Metadata •  Video Essence QC •  Automatic Analysis + Interactice Verification AVDP   AVDP   Request Quality Report Interactive Verification Automatic Analysis Verified Quality Report Storage Analysis Profile Setup (once) #FIATIFTADubai2013 Video / Movie Movie, Video Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AVDP for " Automatic Video Essence Quality Analysis •  Metadata in VidiCert Analyser –  Technical MD –  Structural Video MD –  Essence Defects MD •  •  •  •  •  •  Video Breakup (major video disruptions) Tape Dropouts (e.g. Digital BETACAM™) Monochrome Frames Test Pattern Silence Dolby®E™ –  Essence Quality Measures •  Noise/Grain •  Blurriness –  All metadata are AVDP compliant –  More at www.vidicert.com #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AVDP for " Interactive Essence Quality Verification •  Metadata in VidiCert Summary –  Timeline based summarisation metadata views of automatic analysis results –  Synchronised navigation in all metadata views –  Efficient human quality judgement by defect severity based inspection –  Overall rating metadata support for multi-stage QC –  Optional metadata filtering (verified vs. non-verified) before final export –  All metadata are AVDP compliant –  More at www.vidicert.com #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • AVDP for " Interactive Essence Quality Verification #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP” 22  
  • AVDP for Web based metadata visualisation #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • NHK Metadata Production Framework ! hJp://www.nhk.or.jp/strl/mpf/english/index.htm   #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Conclusion •  AVDP new metadata standard for automated information extraction •  AVDP enables metadata extraction service interoperability •  Several tools working with AVDP are already available #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • Acknowledgements hJp://www.tosca-­‐mp.eu   hJp://david-­‐preserva1on.eu   #FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
  • “THE MPEG-7 AUDIOVISUAL DESCRIPTION PROFILE STANDARD FOR DESCRIBING RESULTS OF AUTOMATIC ANNOTATION SERVICES” Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina, Jean-Pierre Évain, Peter Schallauer Vlaamse Radio- en Televisieomroeporganisatie, Belgium JOANNEUM RESEARCH, Austria Get in contact: mike.matton@vrt.be; peter.schallauer@joanneum.at  of  this   the  property  of  t uthor(s).   granted  permission  to   Copyright  ©opies  of  ptresenta1on  is  urposes  relevant  he  ahe  above  FcIAT/IFTA  is  and  future  communica1on   reproduce  c his  work  for  p to  t onference   by  FIAT/IFTA  without  limita1on,  provided  that  the  author(s),  source  and  copyright  no1ce  are  included  in   each  copy.  For  other  uses,  including  extended  quota1on,  please  contact  the  author(s).   #FIATIFTADubai2013 AVDP” Mike Matton, Peter Schallauer: “MPEG-7 AVDP”