The document discusses carrying timed text and graphics in ISOBMFF. It proposes making the carriage of these media more generic and future-proof. Specifically, it suggests allowing 'meta' boxes in movie fragments to carry timed data samples and using either timed metadata samples or 'meta' boxes as samples to store text/graphics documents and their resources. It also comments on issues in the current working draft.

1. Comments on carriage of
timed text (and graphics)
Cyril Concolato, Jean Le Feuvre
M25978
July 2012, Stockholm, Sweden
INSTITUT MINES-TÉLÉCOM

2. Timed text has a long history …
■ So many formats
• See http://wiki.videolan.org/Subtitles
■ MPEG has already looked at it, some time ago…
• Analysis of the streaming text
requirements, MPEG, Shanghai, China, October 2002, M8931
• Existing MPEG technologies:
− Scene description streams (BIFS, LASeR)
− MPEG-4 Part 17
■ Now new formats (TTML, WebVTT, …)
• Each format so far requires MPEG to standardize a new
mechanism for its carriage
• What will we do for the next formats (now or in 10 years) ?
■ MPEG should design future-proof technology for the
carriage in ISOBMF of timed text !!
1 7/19/2012 INSTITUT MINES-TÉLÉCOM

3. Not only of timed text but also of timed
graphics
■ Proposed new use case
• Frame-based synchronized graphics overlay on top of a
video
− Ex: Graphics and video derived from Kinect devices
− Ex: Recordings of Augmented Reality applications
− Ex: SVG-based cartoons à la Flash
■ Same requirements as “timed text”
• Selecting a graphics track
• Playing while keeping synchronization
• Accessing randomly in the graphics stream
• Enabling progressive download and streaming or
adaptive streaming,
• Positioning the track on top of the video …
■ Demo
2 7/19/2012 INSTITUT MINES-TÉLÉCOM

4. Example of mis-synchronization
3 7/19/2012 INSTITUT MINES-TÉLÉCOM

5. We need more generic requirements (1/2)
■ The ISOBMFF should be able to carry timed data,
in a generic manner, for which the exact type or
format can be identified.
• Ex: to carry timed TTML, SVG, HTML, WebVTT...
■ The ISOBMFF should be able to carry samples of
timed data composed of a main sample data
referencing several individual pieces of data
(sample resource), each of them carried
efficiently, without requiring modifications to the
main sample data.
• Ex: Efficient carriage of JPEG images used by the
timed text or graphics document
4 7/19/2012 INSTITUT MINES-TÉLÉCOM

6. We need more generic requirements (2/2)
■ The ISOBMFF should be able store sample
resources together with or separately of the main
sample data, possibly using movie fragments.
• Ex: Share a JPEG across samples
■ The ISOBMFF should enable the storage of timed
data in a fragmented manner across samples, for
progressive loading by the application
consuming sample data.
• Ex: if an XML progressive loader can be used, use it!
5 7/19/2012 INSTITUT MINES-TÉLÉCOM

7. Technical elements towards a solution
■ Situation
• MPEG has already almost all tools for timed text and
graphics
− Metadata tracks (part 12)
− Scene description tracks (part 14)
• Reuse as much existing tools
• Adapting them if needed
■ 2 proposals
• Generic Tool: Usage of ‘meta’ in movie fragments
• Specific adaptations to carry timed text and/or graphics
− Option 1: Usage of timed metadata samples
− Option 2: Usage of ‘meta’ box as samples
6 7/19/2012 INSTITUT MINES-TÉLÉCOM

8. „meta‟ in movie fragments
■ The „meta‟ box provides
• Carriage of un-timed metadata
• a useful mapping between a URL and the location of
the metadata in the file (ItemInfoBox and
ItemLocationBox)
• Gives way to protect the metadata
■ Current situation
• Fragmenting movie with ‘meta’, what happens?
− Media data not allowed in initialization segments!
• Why ‘meta’ not allowed in movie fragments?
■ Proposal
• Allow at most one ‘meta’ box (and possibly one ‘meco’
box to be consistent with the rest of the specification)
• At the ‘traf’ level (not at the ‘moof’ level)
7 7/19/2012 INSTITUT MINES-TÉLÉCOM

9. Option 1: Usage of “timed metadata”
samples to carry timed text and graphics
■ Track handler: „meta‟
■ Sample entry:
• XMLMetaDataSampleEntry if XML (TTML, SVG, …)
• TextMetaDataSampleEntry if textual (WebVTT, HTML, …)
• URIMetaSampleEntry if needed.
■ Sample
• Use of given mime type or namespace to identify the content of the
sample
• Complete XML document or text chunks or binary content
• Storage of secondary resources (eg. JPEG …) as items in a ‘meta’
box:
− At ‘traf’ level, if fragmented
− At the ‘trak’ level, at the ‘moov’ level, at the file level
■ Flatenning
• Merge ‘meta’
• Or store ‘traf’-level meta boxes at the ‘trak’ level with ‘meco’ boxes
− Use a new sampleGroup type to associate ‘meta’ to sample
8 7/19/2012 INSTITUT MINES-TÉLÉCOM

10. Option 2: Usage of „meta‟ box as samples to
carry timed text and graphics
■ Track handler: „metb‟
■ Sample entry:
• MetaBoxSampleEntry (merge of Text- and
XMLMetadataSampleEntry)
■ Sample format
• A ‘meta’ box with text, XML or graphics document stored
as primary item
• Storage of secondary resources (eg. JPEG …) as items
in a ‘meta’ box:
− This one, at the sample level (if any)
− At ‘traf’ level, if fragmented
− At the ‘trak’ level, at the ‘moof’ level, at the file level
■ Flatenning
• As usual with audio, video, …
9 7/19/2012 INSTITUT MINES-TÉLÉCOM

11. Important points
■ Sample mapping
• Empty samples
− Option 1: an « empty » sample is a zero-length string
− Option 2: an « empty » sample is a meta box with an
empty primary document
• Overlapping samples
− Use sample start time as CTS
− Use delta CTS as duration (as usual)
− Let the sample define « real » presentation duration
and overlap
− No « artificial » duplicate content
■ Simple processing
• Import/export
• Fragmentation/Un-fragmentation
10 7/19/2012 INSTITUT MINES-TÉLÉCOM

12. Misc points: Current WD problems
■ « Time mapping »
• What is needed?
• What is the timeBase for TTML?
• « adjacent time ranges » not defined in SMPTE?
■ Spatial registration
• We agree with m25859: we want to be able to store a
text/graphics track in a separate file from the video
track
• We agree to reuse 3GPP-style positioning
■ Unnecessary Restrictions
• We agree with m25859: do not add restrictions to the
ISOBMFF on timescale, …
■ Be careful not to restrict to complete XML
documents only
11 7/19/2012 INSTITUT MINES-TÉLÉCOM

13. Summary
■ Proposed one new use case: timed graphics
■ Reformulated requirements (more generic)
■ Proposed clarifications for „meta‟ box in movie
fragments
■ Proposed 2 options based on existing tools to
carry timed text or graphics
■ Comments about current WD
Questions?
12 7/19/2012 INSTITUT MINES-TÉLÉCOM