This document discusses emerging technologies and optimization techniques for media workflows and content management. It covers topics like virtual, augmented and mixed reality, 3D spatial audio, high dynamic range video, media over IP, object-based media, video compression techniques, and streaming. Specific technologies and standards discussed include 360-degree video, MPEG-H for 3D audio, HDR10, Dolby Vision, HLG, and SMPTE ST 2110 for media over IP. Applications and use cases are also presented for mixed reality, spatial computing, and next-generation audiovisual experiences.
Introduction to the presentation, emphasizing the journey from dreams and ideas to plans and implementations in media workflows.
Focusing on media workflows, this section outlines acquisition, production, post-production, and distribution, along with smart technologies and optimization techniques.
Discussion on new technologies in media including mixed reality, spatial audio, and video compression standards.
Explains 360-degree video, virtual reality, augmented reality, and mixed reality, detailing their characteristics and impacts on viewing experiences.
Highlighting mixed reality devices and human-machine interfaces, alongside various applications and technologies enhancing user interaction.
Details on 3D spatial audio, its coding standards, and capabilities for enhancing audio experiences in media.
Breakdown of various television formats and standards, emphasizing high-definition and ultra-high-definition technologies.
Discussion on the motivation for 8K technology and NHK's plans for 4K/8K broadcasting rollout, including satellite services.
Overview of TICO compression standards and advancements in video transport formats for broadcasting.
Forecasting UHD TV penetration and discussing various bitrates and quality enhancements related to HDR and UHD content. Insights into quality assessments and the evolving standards for next-gen audio visual technologies.
Exploration of media delivery systems including IPTV, CDN, HbbTV, and edge computing impacts on the content distribution landscape.
Concluding thoughts on the transformation in media workflows, advancements, and open floor for Q&A.
What we aregoing to look at?
New techniques & optimization for Media Workflows and Content Management
– Smart technologies
– Intelligent systems and processes
– New applications and human factors
4.
Smart Technologies, IntelligentSystems
& New Applications
Creation Management Distribution
Mixed Reality Experience
- VR
- AR
- MR
3D Spatial Audio
- MPEG-H
- Coding
TV Picture
- High Dynamic Range
- Pixel Resolution
Media over IP
- SMPTE 2110
- Routeing
- Production
Object-based Media
- Metadata
- The Cloud
Video Compression
- HEVC (MPEG5)
- VP9 & AV1
- ML techniques
Streaming
- DASH
- CMAF
Delivery Platform
- Skylink satellite
- 5G
Consumer
- Multimedia IoT
- Content at the edge
- AI & Smart HMI
5.
360-degree Video, VR,AR and MR
360-degree video
Video recordings where a view in every direction is
recorded at the same time, shot using an omnidirectional
camera or a collection of cameras.
Virtual reality
Replicates an environment, real or imagined, and
simulates a user’s physical presence and environment to
allow for user interaction and sensory experience.
Real world blocked out, user can only see the virtual world
and virtual objects.
5
6.
360-degree Video, VR,AR and MR
Augmented Reality
View of a physical, real world environment whose elements
are augmented by computergenerated items (data,
graphics, video, sound).
• Real world is not blocked out: users can see the real
world and virtual objects
• Interaction with virtual objects is limited
Mixed Reality or Hybrid
The merging of real and virtual worlds to produce new
environments and visualizations where physical and digital
objects
coexist and interact in real time.
6
7.
Mixed Reality Experiences
OculusQuest
Microsoft Hololens
Magic Leap
Virtual Reality
Kit Prices $
$2.3k
$1.4k
$0.4k
Augmented Reality Mixed Reality
Human Machine Interface
= “spatial computing”:
- physical/motion control …
- gaze tracking …
- brain EEG signals …
- natural language …
- special machine vocab …
- LINGUISTICS
8.
The Senses ofthe Human Body
Vision:
360 deg
HD / UHD
HDR
Audition/Sound
3D Spatial audio
Binaural
Olfaction / Smell:
1960 Smellovision
‘4D’ Cinema effects
- smoke …
Gustation/Taste:
University of Singapore
- Digital lollipop
University of Tokyo – Virtual food
Somatosensation/
Haptics/Touch:
Ultra-haptics (Bristol)
Teslasuit …
Perception Understanding Behavior
3D Spatial Sound
–MPEG-H 3D Audio is an audio coding standard developed by the ISO/IEC Moving Picture Experts
Group (MPEG) to support coding audio as audio channels, audio objects, or Higher
Order Ambisonics (HOA).
– MPEG-H 3D Audio can support up to 64 loudspeaker channels and 128 codec core channels.
13.
8 -
13
12bit, HFR,BT.2020, HDR, 4:2:2, 4:4:4,
Abject based audio
Resolution, 14bit, Scalability
DVB UHD Phases and HDR Delivery
14.
UHDTV 1
3840 x2160
8.3 MPs
Digital Cinema 2K
2048 x 1080 2.21 MPs
4K
4096 x 2160 8.84 MPs
SD (PAL)
720 x 576
0.414MPs
HDTV 720P
1280 x 720
0.922 MPs
HDTV 1920 x 1080
2.027 MPs
UHDTV 2
7680 x 4320
33.18 MPs
8K
8192×4320
35.39 MPs
Wider viewing angle
More immersive
Spatial Resolution
16
NAB 2015, IkegamiSHK-810: 8K Super Hi-Vision Camera System (Single 33 MP Super 35 CMOS sensor)
In Japan, 4K and 8K broadcasting is called “super high-vision”, or SHV.
8K/UHD2
Price: $300/000~$400/000
– The oddchannel 17 is being used for the 4K/8K test broadcasting since 2016.
– Channel 7 and 17, will be used for 4K broadcasting by NHK and commercial satellite broadcasting stations in future.
– The even channels will also be used for 4K/8K broadcasting when full service broadcasting starts in 2018.
– Full operation of 8K broadcasting will start in 2018 using channel 14.
20
NHK UHD satellite broadcasting
21.
– NHK isdue to start regular satellite broadcasting service of 4K/8K Super Hi-Vision on December 1st, 2018.
– Transmission System of Mezzanine-compressed 8K over IP (a single 10Gb/s Ethernet cable)
21
The First 8K service by NHK in 2018
48Gb/s Quad-Link 12G-SDI 48Gb/s Quad-Link 12G-SDI
22.
22
– At IBC2013, intoPIX launched TICO for utilization as a mezzanine compression format.
– Extremely tiny footprint in FPGA/ASIC fabric.
– Visually lossless using a 2:1 to 4:1 compression ratio, but keeps power and bandwidth at a reasonable budget.
– Bitrates capable for IP transport
– Low-delay suitable for switches, effects, etc. (Latency down to 1 pixel line ensures absolute real-time and instant operation).
TICO Low Light Visually Lossless Compression
Video Services Forum
Studio Video over IP
More that halfof broadcaster will deliver UHD content till 2022 25
USA population:
326,506,644 (2018)
26.
Delivered
via
Transfer
Function
Frame
Rate
Bit Depth
(bit)
Color SpaceAudio Codec(s)
Bit rate
(Mbps)
Note
Satellite SDR 30 27
Satellite
HLG or SDR 59.94 10 BT.2020 Audio adds 5Mbps 35
Bitrate works for
sports and HDR
content
Satellite or
IPTV
SDR 50 8 or 10 AC-3 24-30
Satellite or
IPTV
PQ 59.94 10 AC-3, AAC 32
IPTV PQ 50 10 BT.2020 MPEG2, AC-3, DD 25 Sport
Satellite SDR 50 10 BT.709 30-38 Drama, Movie
All the examples in Table are 2160p spatial resolution and use HEVC encoding with 4:2:0 color subsampling.
Resolution Frame Rate Approximately Bit Rate
3840×2160 P60/50 15-20 Mbps
1920×1080 P60/50 5-10 Mbps
720×404 P30/25 < 1 Mbps
Example bitrates that could be used for OTT services
26
Example “real world” Bitrates in 2016/2017 for 4K Services
27.
Images : DolbyLaboratories
Standard Dynamic Range
High Dynamic Range
(More Vivid, More Detail)
High Dynamic Range
PQ (Perceptual Quantization)HDR EOTF
Code words are equally spaced in perceived brightness over this range nits.
BrightnessCodeWords
33
Minimum Detectable Contrast (%) =
𝐌𝐢𝐧𝐢𝐦𝐮𝐦 𝐃𝐞𝐭𝐞𝐜𝐭𝐚𝐛𝐥𝐞 𝐃𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐢𝐧 𝐋𝐮𝐦𝐢𝐧𝐚𝐧𝐜𝐞
𝐋𝐮𝐦𝐢𝐧𝐮𝐧𝐜𝐞
× 𝟏𝟎𝟎 =
∆𝑳
𝑳
×100
2
L
34.
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.40.6 0.8 1
SignalValue
Linear light
SDR gamma curve
SDR with Knee
HDR HLG
Hybrid Log-Gamma (HLG) HDR-TV
E : The signal for each color component {RS, GS, BS} proportional to scene linear light and scaled by camera exposure,
normalized to the range [0:12].
E′ :The resulting non-linear HLG coded signal {R',G',B'} in the range [0:1].
a = 0.17883277, b = 0.28466892, c = 0.55991073
More CodeWords for DarkArea
112ln
03
OETF
12
1
12
1
EcbEa
EE
EE
34
Less CodeWords for BrightArea
ITU-R Application 2 ,ARIB B67 (Association of Radio Industries and Businesses)
35.
– Opto-Electronic TransferFunction (OETF): Scene light to electrical signal
– Electro-Optical Transfer Function (EOTF): Electrical signal to scene light
Gamma, EOTF, OETF
35
36.
– Opto-Electronic TransferFunction (OETF): Scene light to electrical signal
– Electro-Optical Transfer Function (EOTF): Electrical signal to scene light
Gamma, EOTF, OETF
The CRT EOTF is commonly
known as gamma
36
37.
Scene-Referred and Display-Referred
Scene-Referred:
–The HLG signal describes the relative light in the scene
– Every pixel in the image represents the light intensity in the captured scene
– The signal produced by the camera is independent of the display
– The signal is specified by the camera OETF characteristic
Display-Referred:
– The PQ signal describes the absolute output light from the mastering display
– The signal is specified by the display EOTF
37
PQ10 & HLG10HDR Systems in DVB Phase 2
– PQ10: PQ OETF, BT.2020,10-bit, non-constant luminance YCbCr, narrow range
– HLG10: HLG OETF, BT.2020, 10-bit, non-constant luminance YCbCr, narrow range
39
40.
– Deeper Colors
–More realistic pictures
– More Colorful
– Rec. 2020 color space covers 75.8%, of CIE 1931 while
Rec. 709 covers 35.9%.
Wide Color Space (ITU-R Rec. BT.2020)
Color Space (ITU-R Rec. BT.709)
Wide Color Gamut
WCG
41.
Motion Blur
Motion Judder
ConventionalFrame Rate High Frame Rate
Wider viewing angle
Increased perceived
motion artifacts
Higher frame rates needed
50fps minimum (100fps being vetted)
41
High Frame Rate (HFR)
What is Quality? Many definitions but in general, it’s like an
elephant
42.
What is Quality?
•Many definitions but in general, it’s like an elephant (The blind man and the elephant)
42
43.
Next Gen Audio
WCG
HDR
NewEOTF
HFR (> 50 fps)
Screen Size
4K Resolution
0 1 2 3 4 5 6 7 8 9 10
43
Added Value Score/ Importance
Source: Ericsson Co. 2015 and Ultra HD Froum
What’s Important in UHD
44.
4H UHDTV
High FrameRate 120FPS
High Frame Rate 60FPS
HDR
Color Gamut
10-Bit Bit Depth
44
Percentage of bit rate incensement
(Reference Bit Rate: HD SDR BT.709 8-Bit,30FPS)
www.atsc.org
Relative Bandwidth Demands of 4K,HDR, WCG, HFR
45.
Source Destination AddedValue
SD HD Medium
HD UHD Medium
HD UHD+HDR+WCG High
HD HD+HDR+WCG High
HD+HDR+WCG UHD+HDR+WCG
Medium or High (Frequently High)
(According to Program and Display is
variable, QDs Display)
SFR HFR
Medium or High
(According to Program is variable)
45
Added Value for 4K/UHD, HDR, WCG, HFR
46.
46
IP and SDI
(ToSynchronize Video Equipment over an IP Network)
(Seamless Protection Switching of SMPTE ST 2022 IP Datagram)
(Transport of High Bit Rate Media Signals over IP Networks (HBRTM))
Single or Quad 12G-SDI
Baseband or Lightly
Compressed UHD/4K
IP (10/40/100G)
Compressed TS, Mezzanine
and baseband UHD/4K
47.
– SDI challengeswill emerge from 40Gb/s and 100Gb/s Ethernet variants with the capacity to handle baseband production signals.
– Broadcasting no longer an independent silo
–Take advantage of evolution speed of Internet
– Broadcast & Broadband as peer delivery mechanisms
–Enables new types of hybrid services
–Ability to seamlessly incorporate niche content
– Enable new business models
–Localized Insertion
– Ads or other content
– Allows revenue model for broadcasters that has been available to cable or IPTV operators
47www.atsc.org
Benefits of IP transport
The mild compression schemes for editing and long GOP compression may breath life into SDI, especially in12-G variants.
48.
IP in theStudio and the Broadcast Chain
Internet / Cloud
Consumer
Post Processing
House
Intranet
IP Studio
PlayOut
Server
In the IP Studio every piece of video, audio
and data is given a unique identifier &
timestamp as soon as it is captured or
created. This allows us to find and
synchronise the most appropriate content
whenever it is needed.
Ref: BBC R&D
SMPTE Standard ST2110 is the main
specification enabling fully
IP-based operations
Object-Based Media Group
–Media + Metadata = Objects
– The idea behind ‘object-based’ media is that you take all the assets for a given TV or radio
show – the video clips, the accompanying audio, any music soundtrack, and extras like
subtitles and sign language translations.
– Wrap it up in useful metadata and then use software to ‘remix’ it when needed
51
52.
Object-Based Media Group
–Media + Metadata = Objects
– The Object-Based Media Group at the MIT Media Lab explores the creative and
technological applications and implications of the intersection of context-aware consumer
electronics and self-aware digital content.
– Projects include immersive, interactive, and personalized television, 3-D display
technologies (in particular computer generated holography), novel user interfaces, and
applications that run on ecosystems of consumer electronics devices.
– Graduate students in the group typically have backgrounds in signal processing and
understanding, image capture and display (especially in 3-D), user interfaces, technical
and creative aspects of new media forms, and/or consumer electronics product design.
52
Video Compression
– Streamingand recording in 4K video resolution yields significant benefits in terms of definition
and video quality, and post-production processes.
– H.265, also known as HEVC, is the successor to the current generation of MPEG4 H.264 codec
(also known as AVC) and is fully capable of encoding /decoding 4K resolutions. HEVC comes
with most 4K TVs and computer monitors.
– Google’s VP9 has gained popularity in web applications with the gradual shift from Flash to
HTML5 technology, and is implemented in many web browsers and used in Google’s
YouTube.
– Unhappy with the licensing and patent issues associated with HEVC, tech
giants Microsoft, Google, Mozilla, Cisco, Intel, Netflix and Amazon launched a new
consortium: the Alliance for Open Media (AOMedia). Their open source AV1 codec was
released earlier this year; it aims to improve bitrate efficiency over HEVC and VP9 by 30 - 50%
and is optimised for 4K image resolution and above.
– New coders include: “Noniterative Content-Adaptive Distributed Encoding Through Machine
Learning Techniques”; SMPTE Motion Imaging V127 No9; Sriram Sethuraman ; V. S. Nithya ; D.
Venkata Narayanababu Laveti. Oct 2018.
Dynamic Adaptive Streamingover HTTP (DASH)
– An MPEG-DASH client can seamlessly adapt to changing network conditions and provide
high quality playback with fewer stalls or re-buffering events.
65.
– Dynamic AdaptiveStreaming over HTTP (DASH), also known as MPEG-DASH, is an adaptive
bitrate streaming technique that enables high quality streaming of media content over
the Internet delivered from conventional HTTP web servers.
– Similar to Apple's HTTP Live Streaming (HLS) solution, MPEG-DASH works by breaking the
content into a sequence of small HTTP-based file segments, each segment containing a
short interval of playback time of content that is potentially many hours in duration, such
as a movie or the live broadcast of a sports event.
65
Dynamic Adaptive Streaming over HTTP (DASH)
66.
– The contentis made available at a variety of different bit rates, i.e., alternative segments
encoded at different bit rates covering aligned short intervals of playback time.
– While the content is being played back by an MPEG-DASH client, the client automatically
selects from the alternatives the next segment to download and play based on current
network conditions.
– The client automatically selects the segment with the highest bit rate possible that can be
downloaded in time for playback without causing stalls or re-buffering events in the
playback.
– Thus, an MPEG-DASH client can seamlessly adapt to changing network conditions and
provide high quality playback with fewer stalls or re-buffering events.
66
Dynamic Adaptive Streaming over HTTP (DASH)
67.
AV Streaming –Common Media Application Format (CMAF)
– Container: Streamed video is compressed using
standardised encoding techniques – H.264 or
H.265/HEVC for the video and mainly AAC for the
audio – the resulting files have to be wrapped up in an
additional layer of information known as a ‘container’,
which adds timing information as well as extra
metadata about what the files contain.
– CMAF is a standardised container format, whose
standardisation is being led within MPEG by Microsoft
and Apple, to making it both cheaper and easier to
deliver OTT video streams to multiple devices.
68.
New Delivery Platforms- Satellite
• 4,425 mini satellites in low-Earth orbit, with a further 7,518 in
very low- Earth orbit – these latter satellites positioned over
major cities in order to provide greater redundancy and
lower latency, in a similar way to how there are more
mobile phone masts in cities, to ensure that everyone can
get good service and that connections aren’t slow or
overloaded.
• The ground stations will use solar generation and battery
storage technology from Musk’s electric car technology
developments
Elon Musk’s Starlink Project:
69.
Development of MobilePhone Standards
- 10Gbps d’load
- Latency <1ms
- Spectrum:
600MHz-6GHz
24GHz-86GHz
70.
Internet of Things(IoT)
• The Internet of things (IoT) is the network of devices, vehicles, and home
appliances that contain electronics, software, actuators, and connectivity which
allows these things to connect, interact and exchange data.
• IoT involves extending Internet connectivity beyond standard devices, such as
desktops, laptops, smartphones and tablets, to any range of traditionally dumb or
non-internet-enabled physical devices and everyday objects.
• Embedded with technology, these devices can communicate and interact over
the Internet, and they can be remotely monitored and controlled.
70
– Edge computingis a distributed computing paradigm in which computation is largely or completely
performed on distributed device nodes known as smart devices or edge devices as opposed to
primarily taking place in a centralized cloud environment.
– The eponymous "edge" refers to the geographic distribution of computing nodes in the network as
Internet of Things devices, which are at the "edge" of an enterprise, metropolitan or other network.
– The motivation is to provide server resources, data analysis and artificial intelligence ("ambient
intelligence") closer to data collection sources and cyber-physical systems such as smart sensors and
actuators.
– Edge computing is seen as important in the realization of physical computing, smart cities, ubiquitous
computing and the Internet of Things.
74
Edge Computing
Content Distribution Network
–CDN: A content distribution network—also known as a content delivery network — is a large,
geographically distributed network of specialized servers that accelerate the delivery of web
content and rich media to internet-connected devices.
– 1. Your Server Load will decrease
– 2. Content Delivery will become faster
– 3. Segmenting your audience becomes easy
– 4. Lower Network Latency and packet loss
– 5. Higher Availability and better usage analytics
– 6. Storage and Security
Internet Protocol television(IPTV)
– Internet Protocol television (IPTV) is the delivery of television content over Internet Protocol
(IP) networks.
– Unlike downloaded media, IPTV offers the ability to stream the source media continuously.
As a result, a client media player can begin playing the content (such as a TV channel)
almost immediately. This is known as streaming media.
– Although IPTV uses the Internet protocol it is not limited to television streamed from the
Internet, (Internet television).
79
80.
– IPTV iswidely deployed in subscriber-based telecommunications networks with high-speed
access channels into end-user premises via set-top boxes or other customer-premises
equipment. IPTV is also used for media delivery around corporate and private networks.
– IPTV services may be classified into three main groups:
• Live television and live media, with or without related interactivity;
• Time-shifted media: e.g. catch-up TV (replays a TV show that was broadcast hours or days ago),
start-over TV (replays the current TV show from its beginning);
• Video on demand (VOD): browse and view items in a stored media catalogue.
80
Internet Protocol television (IPTV)
81.
Over the top(OTT)
– Over the top (OTT) is a term used to refer to content providers that distribute streaming
media as a standalone product directly to viewers over the Internet, bypassing
telecommunications, multichannel television, and broadcast television platforms that
traditionally act as a controller or distributor of such content.
– The term is most synonymous with subscription-based video on demand services that offer
access to film and television content.
– Over the top services are typically accessed via websites on personal computers, as well
as via apps on mobile devices (such as smartphones and tablets), digital media players
(including video game consoles), or televisions with integrated smart TV platforms.
81
82.
82
The primary differencebetween IPTV and OTT streaming is that OTT streamed content
comes over the same open, unmanaged network as your email and web browsing
while IPTV uses a private, dedicated network to deliver more consistent service.
83.
• Hybrid BroadcastBroadband TV (HbbTV) is both an industry standard and promotional initiative for hybrid digital TV to harmonise the
broadcast, IPTV, and broadband delivery of entertainment to the end consumer through connected TVs (smart TVs) and set-top boxes.
• The HbbTV Association, comprising digital broadcasting and Internet industry companies, has established a standard for the delivery of
broadcast TV and broadband TV to the home, through a single user interface, creating an open platform as an alternative to
proprietary technologies.
• Products and services using the HbbTV standard can operate over different broadcasting technologies, such as satellite, cable, or
terrestrial networks.
• HbbTV can show digital television content from a number of different sources including traditional broadcast TV, Internet, and
connected devices in the home.
• To watch hybrid digital TV, consumers will need a hybrid IPTV set-top box with a range of input connectors, including Ethernet as well as
at least one tuner for receiving broadcast TV signals.
• The tuner in a hybrid set-top box can be digital terrestrial television (DVB-T, DVB-T2), digital cable (DVB-C, DVB-C2) and digital satellite
(DVB-S, DVB-S2).
83
Hybrid Broadcast Broadband TV (HbbTV)
Main Benefits ofthe HbbTV Operator App
More control of the TV user experience
- Add remote controller keys
- Channel Up/Down
- Volume control
- TV-Guide
- Channel lists
- Co-existence of broadcasted Apps
- Maintain access to ”Red Button” service
- Possibility to launch without broadcast signal
- Reach households who does not have access to Antenna /Satellite RF signal
- Operator-specific HbbTV OpApps
- Especially for Set-top Box environment
- Replace whole UI of the device
85
86.
HbbTV Trends in2018
HbbTV 2.0.x is coming
- HEVC / UHD / NGA / ETC
- Media syncronization
- Better HTML+CSS support
- Seven new companies joining in HbbTV
- i.e Google, Sky, RAI, Skyworth
- Content substitution
- Audio and/or video replacement
- Localized or personal experience
- Ads replacement
- HbbTV 1.5 going strong while 2.0.x is still coming
- HbbTV Operator Apps gain a lot of interest!
- DVB-I, how to Bring TV services over Internet
86
87.
Conclusions
Networks Next GenNetworks, Edge Computing & Delivery
for new AR/VR/360 Content
Always connected to the Cloud?
Creation to Consumption – end to end IP
8K UHD Workflow, better Stitching of 360 degree Video, 3D image scanning &
transmission
HDR compatibility/HFR/3D Sound …
Cloud Services, AI, Understanding Human Behaviour and Human Machine Interfaces
New Engines for 8K, VR/AR/360 Video