1. “THE MPEG-7 AUDIOVISUAL
DESCRIPTION PROFILE STANDARD
FOR DESCRIBING RESULTS OF
AUTOMATIC ANNOTATION SERVICES”
Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina,
Jean-Pierre Évain, Peter Schallauer
Vlaamse Radio- en Televisieomroeporganisatie, Belgium
JOANNEUM RESEARCH, Austria
Get in contact: mike.matton@vrt.be; peter.schallauer@joanneum.at
of
this
the
property
of
t
uthor(s).
granted
permission
to
Copyright
©opies
of
ptresenta1on
is
urposes
relevant
he
ahe
above
FcIAT/IFTA
is
and
future
communica1on
reproduce
c
his
work
for
p
to
t
onference
by
FIAT/IFTA
without
limita1on,
provided
that
the
author(s),
source
and
copyright
no1ce
are
included
in
each
copy.
For
other
uses,
including
extended
quota1on,
please
contact
the
author(s).
#FIATIFTADubai2013
AVDP”
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

2. Contents
•
•
•
•
Introduction & motivation
MPEG-7 AVDP
Applications of AVDP
Conclusion
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

3. MPEG-7 Audiovisual Description Profile
• Motivated by activities of EBU MIM/SCAIE group
– Study and evaluation of Content analysis-based Automatic Information
Extraction tools in media production
– facilitate the introduction of new production tools based on automatic
extraction of information
• Relevant tools include
•
•
•
•
•
•
Speech Recognition
Audiovisual Segmentation
Personality Identification
Semantic Detection
Text Recognition
Subject Classification
#FIATIFTADubai2013
•
•
•
•
•
•
Format Detection"
Named Entities Detection"
Copy / Repetition Detection"
Content Summarisation"
Keyword Extraction"
Automatic text Translation"
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

4. MPEG-7 Audiovisual Description Profile
• Automatic analysis tools generate
– detailed time-based metadata
– low-level audio/video features (e.g., color descriptors)
• Common metadata format is needed to
– represent results
– exchange between organisations
– perform automated evaluation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

5. MPEG-7 AVDP Motivation (TOSCA-MP
project)
• Put automatic annotation in automated workflow
• Services need to communicate with one another
(e.g. Scene segmentation needs input from shot
boundary detection)
– Even across vendors
– MPEG-7 AVDP specifically created for this purpose
– Different such services later in this presentation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

6. MPEG-7
•
Multimedia Content Description Interface
– ISO/IEC 15938, version 1 (2001) and 2 (2004)
•
Metadata of multimedia content
– descriptors, description schemes
– definition based on an extension of XML Schema
– serialised in XML or binary format
•
Multimedia description schemes (part 5)
– content structure (decompositions, segments)
– descriptive and technical metadata
•
Visual and Audio (parts 3 and 4)
– descriptors for visual and audio low-level features
•
use of parts of MPEG-7 for TV Anytime, EBU Classification Schemes
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

7. MPEG-7 Pros and Cons
•
Support for describing arbitrary content structures
– time segments, regions, moving regions
•
•
Support for low-level feature descriptors
Benefits of MPEG-7
–
–
–
–
•
designed as a data model, not just as an exchange format
broad range of applications
flexible, fine-grained description
content structuring capabilities
Drawbacks
– complexity: generic types, deep hierarchical structures
– interoperability problems: several ways to describe the same, conformance to
standard can only be checked on syntactic level but not on a semantic level
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

8. Why profiles for MPEG-7?
• Address issues of complexity and
interoperability
• Profiles are a proven concept
– subsets of the standard for certain
functionalities and/or applications
– cf. MPEG-2, MPEG-4
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

9. Why profiles for MPEG-7?
• Definition of an MPEG-7 profile
– Selection of description tools
→ Profile XML Schema
• which description schemes and descriptors may be used
– Constraints on description tools
→ Profile XML Schema
• restrictions on cardinalities, child elements, attributes
– Semantic constraints
→ Textual description
• semantics of the elements in the description
• specify use of the selected tools in the context of the
profile
• ensure interoperability
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

10. Source used for defining the new profile
• Detailed Audiovisual Profile (DAVP)
–
–
–
–
single multimedia content entities
comprehensive structural description of the content
audio and visual low-level feature descriptors (parts 3 and 4)
defines semantic constraints on structure and elements to address
interoperability issues
• NHK Metadata Production Framework (MPF)
–
–
–
–
based on the Core Description Profile (CDP)
addresses the complexity and ambiguity problems of MPEG-7
audio and visual low-level feature descriptors (parts 3 and 4)
semantic constraints for the structure of the description as well as several
syntactic and semantic constraints on different elements
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

11. Design goals for a new profile
• Fulfill the requirements of EBU MIM/SCAIE
• Harmonise DAVP and MPF
– both share many design principles
– some implementation choices are different
• Simplify
– based on experience from using DAVP and MPF
– some included elements are never used
– some constructs can be simplified in terms of cardinality/flexibility
• First profile based on MPEG-7 v2 (2004)
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

12. AudioVisual Description "
Profile – AVDP
• An MPEG-7 AVDP file contains
– 0 or more descriptions of "
audiovisual contents
– 0 or 1 summaries of the content(s)
• Describe audio, video or "
audiovisual content
Mpeg7
Description
type=“ContentEntityType”
(a)
A
ContentEntityType for
one
audiovisual
content.
Mpeg7
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
– no other types of multimedia "
content
– no collections of content
– same top-level structure for audio, "
video or audiovisual content
(b)
Several
ContentEntityTypes for
copy
detection
result.
Mpeg7
Description
type=“SummaryDescriptionType
(c)
A
SummaryDescription.
Mpeg7
Description
type=“SummaryDescriptionType
Description
type=“ContentEntityType”
Description
type=“ContentEntityType”
(d)
A
SummaryDescription with
several
ContentEntityTypes.
Figure
1
Top
level
structure
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

13. AVDP content description structure
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

14. Constraint definition
• Profile specification document defines
–
–
–
–
list of included types
constraints on elements and cardinalities
semantic constraints on use of these types
semantics of description structure
• Profile XML schema
– included types
– excluded elements, restrictions on cardinality
– few of the semantic and structure constraints
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

15. Examples
• Examples provided with AVDP specification
–
–
–
–
–
General description structure
Content segmentation
Speech-to-text
Copy detection
Summarization
• Annex
– example outputs from RAI and JRS content analysis tools
– MIM/MDN Knowledge base
• http://workspace.ebu.ch/display/ecmmdn/Knowledge+base+-+contributions
• EBU MIM/SCAIE is currently working on a guidelines document
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

16. Validation
• Validation against MPEG-7 XML Schema or Profile XML
Schema
– e.g., using validator of an XML editor, Xerces, MS XML, etc.
• many constraints of a profile cannot be represented in
XML schema
– e.g., the AudioVisual element, representing the entire content (root
a/v segment) is of type AudiovisualSegmentType
– the same type is used to represent many audiovisual segments at
other places in the description
– the specific constraints on the elements that need to be present on
the root a/v segment but not on others cannot be checked
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

17. AVDP Applications
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

18. VAMP: Validation of "
MPEG-7 Profile Descriptions (including AVDP)
• validate metadata documents
– conforming to complex metadata standards such as MPEG-7
– semantics of temporal annotations
– beyond XML schema validation
• prevent interoperability problems
• http://vamp.joanneum.at
• Further MPEG-7 ressources
• http://mpeg7.joanneum.at/
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

19. VidiCert: AVDP for Video Essence QC
Metadata
• Video Essence QC
• Automatic Analysis + Interactice Verification
AVDP
AVDP
Request
Quality
Report
Interactive Verification
Automatic Analysis
Verified
Quality
Report
Storage
Analysis Profile
Setup (once)
#FIATIFTADubai2013
Video / Movie
Movie, Video
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

20. AVDP for "
Automatic Video Essence Quality Analysis
•
Metadata in VidiCert Analyser
– Technical MD
– Structural Video MD
– Essence Defects MD
•
•
•
•
•
•
Video Breakup (major video disruptions)
Tape Dropouts (e.g. Digital BETACAM™)
Monochrome Frames
Test Pattern
Silence
Dolby®E™
– Essence Quality Measures
• Noise/Grain
• Blurriness
– All metadata are AVDP compliant
– More at www.vidicert.com
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

21. AVDP for "
Interactive Essence Quality Verification
• Metadata in VidiCert Summary
– Timeline based summarisation
metadata views of automatic analysis
results
– Synchronised navigation in all metadata
views
– Efficient human quality judgement by
defect severity based inspection
– Overall rating metadata support for
multi-stage QC
– Optional metadata filtering (verified vs.
non-verified) before final export
– All metadata are AVDP compliant
– More at www.vidicert.com
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

22. AVDP for "
Interactive Essence Quality Verification
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”
22

23. AVDP for Web based metadata visualisation
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

24. NHK Metadata Production Framework
!
hJp://www.nhk.or.jp/strl/mpf/english/index.htm
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

25. Conclusion
• AVDP new metadata standard for
automated information extraction
• AVDP enables metadata extraction service
interoperability
• Several tools working with AVDP are already
available
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

26. Acknowledgements
hJp://www.tosca-­‐mp.eu
hJp://david-­‐preserva1on.eu
#FIATIFTADubai2013
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

27. “THE MPEG-7 AUDIOVISUAL
DESCRIPTION PROFILE STANDARD
FOR DESCRIBING RESULTS OF
AUTOMATIC ANNOTATION SERVICES”
Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina,
Jean-Pierre Évain, Peter Schallauer
Vlaamse Radio- en Televisieomroeporganisatie, Belgium
JOANNEUM RESEARCH, Austria
Get in contact: mike.matton@vrt.be; peter.schallauer@joanneum.at
of
this
the
property
of
t
uthor(s).
granted
permission
to
Copyright
©opies
of
ptresenta1on
is
urposes
relevant
he
ahe
above
FcIAT/IFTA
is
and
future
communica1on
reproduce
c
his
work
for
p
to
t
onference
by
FIAT/IFTA
without
limita1on,
provided
that
the
author(s),
source
and
copyright
no1ce
are
included
in
each
copy.
For
other
uses,
including
extended
quota1on,
please
contact
the
author(s).
#FIATIFTADubai2013
AVDP”
Mike Matton, Peter Schallauer: “MPEG-7 AVDP”