Virtual reality (VR) endows any user with a sense of full immersion within a virtual environment. Omnidirectional or spherical video content is the new multimedia format that provides this immersive sensation. The viewer is placed at the centre of the sphere and dynamically changes the displayed portion of the spherical content, viewport. This new spherical multimedia format and this new interactive way of consuming the content have created novel challenges: • large volume of data to store, deliver and display, • ultra-low-delay constraints over bandwidth-limited resources • uncertainty on the portion of content that will be displayed by the user.

1. Adaptive Streaming for Immersive Communication
Silvia Rossi and Laura Toni
{s.rossi, l.toni}@ucl.ac.uk
Communications and Information System Group
Department of Electronic and Electrical Engineering
INSTITUTE OF COMMUNICATIONS & CONNECTED SYSTEMS
Our project
Description
Virtual reality (VR) endows any user with a sense of full immersion within a virtual environment. Omnidirectional or spherical video content is the new multimedia format that
provides this immersive sensation. The viewer is placed at the centre of the sphere and dynamically changes the displayed portion of the spherical content, viewport.
This new spherical multimedia format and this new interactive way of consuming the content have created novel challenges:
• large volume of data to store, deliver and display,
• ultra-low-delay constraints over bandwidth-limited resources
• uncertainty on the portion of content that will be displayed by the user.
How can adaptive streaming systems be efficiently improved to achieve the suitable flexibility and interactivity for omnidirectional content ?
User navigation prediction
VR and the User Centric Era
A novel QoE for a new content geometry?
Which content to send?
Which content to store?
Overview
• Omnidirectional video (ODV) is projected on a planar frame to be encoded, stored at the server and delivered
• Projection introduces artifacts that affect the user quality of experience
• Equirectangular (ERP) projection is the most popular and simplest projection
Need for a geometry-aware distortion evaluation
• Prefetching requested viewport prevents freezing of the video
• Delivering non-homogenous quality content saves bandwidth
• Resolution of content improves user QoE
Need for navigation-aware adaptive logic
• High cost of storage and coding
• Limited storage space
• Non-homogenous user requests and content characteristics
Need for a popular-representation set to store
Spherical domain
Planar domain
From the sphere to planar with ERP projection. User VP is in red.
STEP 1:
Users profiling
STEP 2:
Viewport prediction
• Analysis on the sphere taking into
account both physical structure
and users’ features
• Users clustered based on their
position on the sphere during the
navigation
Need for users analysis on the
spherical domain
• Users in the same class have
similar characteristics
• The user attention is strongly
related to content features
• Popular navigation paths
can give a global behaviour
Need for head movement
prediction algorithm
Quality
Spatial
Resolution
High quality
Low quality
Client
Content Provider
HTTP request
HTTP segment
Understanding where the user is most likely going to look at is the key to improve the delivery of VR applications
Our goal is to optimise the entire pipeline of immersive communication leading to the best Quality of Experience (QoE) to the user. To achieve this, we propose:
• novel sphere-based machine learning techniques to predict users’ behaviour
• improved coding and streaming strategies able to take into account: users’ prediction and the new geometry of the content.
x x
Content Provider CDN Interactive Users
Features +
Saliency
Extraction
Content
Aware
Coding
planar
Features
Compressed
Video
Ingest
Server Origin
Server
Edge
Server
Navigation
Prediction
Navigation Based
Adaptation Logic
Viewport
Head
Movement
MPD file
HTTP get