3. MPEG-7 Audiovisual Description Profile
• Motivated by activities of EBU MIM/SCAIE group
– Study and evaluation of Content analysis-based Automatic Information
Extraction tools in media production
– facilitate the introduction of new production tools based on automatic
extraction of information
• Relevant tools include
•
•
•
•
•
•
Speech Recognition
Audiovisual Segmentation
Personality Identification
Semantic Detection
Text Recognition
Subject Classification
#FIATIFTADubai2013
•
•
•
•
•
•
Format Detection"
Named Entities Detection"
Copy / Repetition Detection"
Content Summarisation"
Keyword Extraction"
Automatic text Translation"
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
4. MPEG-7 Audiovisual Description Profile
• Automatic analysis tools generate
– detailed time-based metadata
– low-level audio/video features (e.g., color descriptors)
• Common metadata format is needed to
– represent results
– exchange between organisations
– perform automated evaluation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
5. MPEG-7 AVDP Motivation (TOSCA-MP
project)
• Put automatic annotation in automated workflow
• Services need to communicate with one another
(e.g. Scene segmentation needs input from shot
boundary detection)
– Even across vendors
– MPEG-7 AVDP specifically created for this purpose
– Different such services later in this presentation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
6. MPEG-7
•
Multimedia Content Description Interface
– ISO/IEC 15938, version 1 (2001) and 2 (2004)
•
Metadata of multimedia content
– descriptors, description schemes
– definition based on an extension of XML Schema
– serialised in XML or binary format
•
Multimedia description schemes (part 5)
– content structure (decompositions, segments)
– descriptive and technical metadata
•
Visual and Audio (parts 3 and 4)
– descriptors for visual and audio low-level features
•
use of parts of MPEG-7 for TV Anytime, EBU Classification Schemes
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
7. MPEG-7 Pros and Cons
•
Support for describing arbitrary content structures
– time segments, regions, moving regions
•
•
Support for low-level feature descriptors
Benefits of MPEG-7
–
–
–
–
•
designed as a data model, not just as an exchange format
broad range of applications
flexible, fine-grained description
content structuring capabilities
Drawbacks
– complexity: generic types, deep hierarchical structures
– interoperability problems: several ways to describe the same, conformance to
standard can only be checked on syntactic level but not on a semantic level
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
8. Why profiles for MPEG-7?
• Address issues of complexity and
interoperability
• Profiles are a proven concept
– subsets of the standard for certain
functionalities and/or applications
– cf. MPEG-2, MPEG-4
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
9. Why profiles for MPEG-7?
• Definition of an MPEG-7 profile
– Selection of description tools
→ Profile XML Schema
• which description schemes and descriptors may be used
– Constraints on description tools
→ Profile XML Schema
• restrictions on cardinalities, child elements, attributes
– Semantic constraints
→ Textual description
• semantics of the elements in the description
• specify use of the selected tools in the context of the
profile
• ensure interoperability
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
10. Source used for defining the new profile
• Detailed Audiovisual Profile (DAVP)
–
–
–
–
single multimedia content entities
comprehensive structural description of the content
audio and visual low-level feature descriptors (parts 3 and 4)
defines semantic constraints on structure and elements to address
interoperability issues
• NHK Metadata Production Framework (MPF)
–
–
–
–
based on the Core Description Profile (CDP)
addresses the complexity and ambiguity problems of MPEG-7
audio and visual low-level feature descriptors (parts 3 and 4)
semantic constraints for the structure of the description as well as several
syntactic and semantic constraints on different elements
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
11. Design goals for a new profile
• Fulfill the requirements of EBU MIM/SCAIE
• Harmonise DAVP and MPF
– both share many design principles
– some implementation choices are different
• Simplify
– based on experience from using DAVP and MPF
– some included elements are never used
– some constructs can be simplified in terms of cardinality/flexibility
• First profile based on MPEG-7 v2 (2004)
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
12. AudioVisual Description "
Profile – AVDP
• An MPEG-7 AVDP file contains
– 0 or more descriptions of "
audiovisual contents
– 0 or 1 summaries of the content(s)
• Describe audio, video or "
audiovisual content
Mpeg7
Description
type=“ContentEntityType”
(a)
A
ContentEntityType for
one
audiovisual
content.
Mpeg7
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
– no other types of multimedia "
content
– no collections of content
– same top-level structure for audio, "
video or audiovisual content
(b)
Several
ContentEntityTypes for
copy
detection
result.
Mpeg7
Description
type=“SummaryDescriptionType
(c)
A
SummaryDescription.
Mpeg7
Description
type=“SummaryDescriptionType
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
(d)
A
SummaryDescription with
several
ContentEntityTypes.
Figure
1
Top
level
structure
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
13. AVDP content description structure
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
14. Constraint definition
• Profile specification document defines
–
–
–
–
list of included types
constraints on elements and cardinalities
semantic constraints on use of these types
semantics of description structure
• Profile XML schema
– included types
– excluded elements, restrictions on cardinality
– few of the semantic and structure constraints
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
15. Examples
• Examples provided with AVDP specification
–
–
–
–
–
General description structure
Content segmentation
Speech-to-text
Copy detection
Summarization
• Annex
– example outputs from RAI and JRS content analysis tools
– MIM/MDN Knowledge base
• http://workspace.ebu.ch/display/ecmmdn/Knowledge+base+-+contributions
• EBU MIM/SCAIE is currently working on a guidelines document
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
16. Validation
• Validation against MPEG-7 XML Schema or Profile XML
Schema
– e.g., using validator of an XML editor, Xerces, MS XML, etc.
• many constraints of a profile cannot be represented in
XML schema
– e.g., the AudioVisual element, representing the entire content (root
a/v segment) is of type AudiovisualSegmentType
– the same type is used to represent many audiovisual segments at
other places in the description
– the specific constraints on the elements that need to be present on
the root a/v segment but not on others cannot be checked
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
18. VAMP: Validation of "
MPEG-7 Profile Descriptions (including AVDP)
• validate metadata documents
– conforming to complex metadata standards such as MPEG-7
– semantics of temporal annotations
– beyond XML schema validation
• prevent interoperability problems
• http://vamp.joanneum.at
• Further MPEG-7 ressources
• http://mpeg7.joanneum.at/
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
19. VidiCert: AVDP for Video Essence QC
Metadata
• Video Essence QC
• Automatic Analysis + Interactice Verification
AVDP
AVDP
Request
Quality
Report
Interactive Verification
Automatic Analysis
Verified
Quality
Report
Storage
Analysis Profile
Setup (once)
#FIATIFTADubai2013
Video / Movie
Movie, Video
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
20. AVDP for "
Automatic Video Essence Quality Analysis
•
Metadata in VidiCert Analyser
– Technical MD
– Structural Video MD
– Essence Defects MD
•
•
•
•
•
•
Video Breakup (major video disruptions)
Tape Dropouts (e.g. Digital BETACAM™)
Monochrome Frames
Test Pattern
Silence
Dolby®E™
– Essence Quality Measures
• Noise/Grain
• Blurriness
– All metadata are AVDP compliant
– More at www.vidicert.com
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
21. AVDP for "
Interactive Essence Quality Verification
• Metadata in VidiCert Summary
– Timeline based summarisation
metadata views of automatic analysis
results
– Synchronised navigation in all metadata
views
– Efficient human quality judgement by
defect severity based inspection
– Overall rating metadata support for
multi-stage QC
– Optional metadata filtering (verified vs.
non-verified) before final export
– All metadata are AVDP compliant
– More at www.vidicert.com
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
22. AVDP for "
Interactive Essence Quality Verification
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
22
23. AVDP for Web based metadata visualisation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
24. NHK Metadata Production Framework
!
hJp://www.nhk.or.jp/strl/mpf/english/index.htm
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
25. Conclusion
• AVDP new metadata standard for
automated information extraction
• AVDP enables metadata extraction service
interoperability
• Several tools working with AVDP are already
available
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”