1. Tile-based Streaming of 8K
Omnidirectional Video: Subjective and
Objective QoE Evaluation
Raimund Schatz
AIT Austrian Institute
of Technology
11th International Conference on Quality of Multimedia Experience (QoMEX 2019)
June 5-7, 2019, Berlin
Christian Timmerer
Alpen-Adria-Universität
Klagenfurt / Bitmovin Inc.
Anatoliy Zabrovskiy
Alpen-Adria-Universität
Klagenfurt

2. § QoE of Omnidirectional Video (ODV, aka 360º Movie) Streaming
§ Immersive panorama video surrounding the user
§ User can turn head to change gaze direction
§ ODV streaming = content class with
dramatically increasing popularity
§ Cisco VNI: globally, virtual–reality traffic will
increase 61-fold between 2015 and 2020
Focus of this Talk & Motivation
208.06.19 © AIT
Immersive
Media
VR
Responsive
VR
ODV
MR AR

3. Motivation
§ Current deployments of ODV: technically straightforward …
Drawback: viewport-agnostic!
à High storage & bandwidth requirements
à Impaired QoE (under constrained conditions)

4. Tile-based Streaming to the Rescue!
Mario Graf, Christian Timmerer, and Christopher Mueller. 2017. Towards Bandwidth Efficient
Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation.
In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York,
NY, USA, 261-271. DOI: https://doi.org/10.1145/3083187.3084016
https://multimediacommunication.blogspot.com/2017/04/acm-mmsys17-special-session-paper.html

5. Tile-based Streaming, The Challenge:
Lots of Knobs & Levers …
§ Tiling Patterns?
§ Encoding Quality Levels?
§ Delivery Strategy?
§ …
àWANTED: Valid Ground Truths & Guidelines
based on subjective ODV Quality Studies!
à This Talk: Results (subjective & objective) from
a Joint ODV QoE Study by AIT & AAU
508.06.19

6. The Study
608.06.19

7. Research Questions
RQ1: How does full vs. partial delivery impact ODV
streaming QoE? Is there a clear user preference?
RQ2: What is the QoE impact of different ODV tile quality
encoding levels? (RQ2)
RQ3: Do camera movement (static vs. moving) or head
turning speed exert an influence on quality perception?
708.06.19

8. Subjective Lab Test: SRCs & HRCs
§ SRCs: Two 8k high-quality source clips (duration: 30s)
§ HRCs, Factors:
§ Delivery: Full vs. Partial
§ Head turn: Slow vs. Fast
§ Tile encoding quality:
QP 46, 32, 22
808.06.19

9. SRCs: Source Clips (8k)
908.06.19
URLs:
http://medialab.sjtu.edu.cn/vr8K/index.html
https://www.itu.int/en/ITU-T/studygroups/2017-2020/16/Pages/video/jctvc.aspx

10. HRCs: Full vs. Partial Delivery
1008.06.19
Gray Tiles

11. HRCs: Head-Turn
§ Head turning movement in the middle of the clip
§ 90 deg to the right
§ Fast (1s) vs. Slow (4s)
§ Challenge: slight changes in head movement à huge differences in tile-
based streaming behavior
à Solution: render FoV into PVS!
1108.06.19

12. Processing Pipeline: from 360 SRCs to 2D PVSes
1308.06.19

13. Subjective Lab Test: Setup
§ Tiled ODV QoE Lab Study Sessions in Summer 2018
§ Setup: Large 4k screen (65“, SONY) viewed at 1,5m distance, ratings via
tablet
1408.06.19
Similar setup used in
standardization,
e,g, Versatile Video
Coding (VVC)

14. Subjective Lab Test: Protocol
1508.06.19
Ratings provided on table using the open source software http://www.thefragebogen.de

15. Test Subjects
§ N=35
§ 21 male, 14 female
§ 50% had (some) VR experience
§ Mixed education / backgrounds
§ Age: mean 32, median 33, range: 22-50 yrs
1608.06.19 nafi 4.0

16. The Results
1708.06.19

17. Subjective Ratings: MOS [0-100]
18
§ Delivery: Partial Delivery à visibly low QoE
§ Tile Encoding Quality: QP46 = significant QoE drop, but only very little
difference between QP22 & 32
QP: 46 32 22 | 46 ….

18. Subjective Ratings: Acceptability
19
§ Acceptability Ratings: confirm MOS results
§ In particular: Partial Delivery = really unacceptable
Full Delivery Partial Delivery
QP: 46 32 22 | 46 ….

19. Mixed Model ANOVA Results (Full Delivery only)
2008.06.19
Head
movement
Content /
Cam movement

20. Objective Metrics: wPSNR, SSIM and VMAF for FoV PVS
§ Low sensitivity of SSIM w.r.t. full vs. partial delivery
§ Little impact of content / cam movement speed
§ Consistent impact of head movement speed (but not pronounced, too)
2108.06.19
Results for segments 3-4-5 (i.e. head turn part only):
More objective analysis results à see the paper!

21. Findings & Conclusions
§ RQ1 (Full vs partial delivery):
§ Avoid partial delivery (or find better visual workarounds), since
unacceptable for end users
§ RQ2 (Impact of tile encoding quality):
§ Sufficient to use QP 32 for tile encoding for practical deployments as
saturation kicks in here
§ RQ3 (Impact of camera movement and head turn speed):
§ Camera movement masking low encoding quality (known)
§ Head turn speed: only weak impact, interaction with content/cam
movement (+ for static, - for dynamic scene)
§ Objective Evaluation:
§ Largely in line with subjective results, with some exceptions depending on
metric and impairment
2208.06.19

22. Issues, Challenges & Outlook
§ Several limitations: 2D Display used (not HMD), limited number of SRCs,
head-motion patterns used, etc.
§ Further research needed!
§ Future studies featuring 2D Display vs. HMD viewing impact, larger variety
of content, head-motions, etc.
§ Also: tile-based rendering pipeline = technically challenging, lots of tweaking
& error-checking required à joint efforts required
2308.06.19

23. Thank you for your attention!
Raimund Schatz
AIT Austrian Institute of Technology GmbH
Donau-City-Str. 1, 1220 Vienna
raimund.schatz@ait.ac.at
This work was supported in part by the Austrian Research Promotion Agency
(FFG) under the Next Generation Video Streaming project “PROMETHEUS.
Christian Timmerer
Alpen-Adria-Universität
Klagenfurt / Bitmovin Inc.

24. Objective Metrics: Metrics Calculated for Whole 8k Content

25. Questionnaire
2608.06.19