"Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos" in proceeding of ACM Multimedia System (MMSys'18). Full immersion inside a Virtual Reality (VR) scene requires six Degrees of Freedom (6DoF) applications, where the user is allowed to perform translation and rotation movements within the video content. The implementation of 6DoF applications is however still an open question. We study in this paper a multi-viewpoint (MVP) 360-degree video streaming system, where a scene is simultaneously captured by multiple synchronized omnidirectional cameras. The user can only switch positions to predefined viewpoints (VPs). We focus on the new challenges that are introduced by adaptive MVP 360-degree video streaming. We introduce several options for the video encoding with existing technologies such as High Efficiency Video Coding (HEVC) tiles and for the implementation of VP switching. We model three video-segment download strategies for an adaptive streaming client into Mixed Integer Linear Programming (MILP) problems: an omniscient download scheduler, and two essentially different strategies; one where the client proactively downloads all VPs to guarantee fast VP switch, and one where the client reacts to client commands. We recorded a first MVP 360-degree video with three VPs, implemented a mobile MVP 360-degree video player, and recorded different users navigating inside the content. We solved the adaptive streaming optimization problems on this video and those navigation traces. The results emphasize the gains of using tiles on the quality metrics. They also emphasize the importance of performing further study on VP switching prediction to reduce the bandwidth consumption, and to measure the impact of VP switching delay on the subjective Quality of Experience (QoE).

1. IMT Atlantique
Bretagne-Pays de la Loire
École Mines-Télécom
Dynamic Adaptive Streaming
for Multi-Viewpoint
Omnidirectional Videos
Xavier Corbillon Francesca De Simone
Gwendal Simon Pascal Frossard

2. Omnidirectionnal Videos 2/17
Three degrees of freedom (3DoF)
v0 = O
k

ı u
v
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

3. Six Degrees of Freedom 3/17
Full immersion requires six degrees of freedom (6DoF)
Industry (e.g. MPEG) roadmap:
• 3DoF+
• Windowed 6DoF
• Omnidirectional 6DoF
In this paper: discrete 6DoF
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

4. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

5. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

6. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

7. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
Click
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

8. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

9. Open Research Topics 5/17
Service implementation?
• resource explosion: storage, processing, delivery
Viewpoint 1 Viewpoint 2
Origin Server
? ?
Viewpoint 1 Viewpoint 2
Edge Server Client
Predictions:
• Viewpoint switching in addition to the bandwidth and head orientation
Downloading strategy
QoE assessment for viewpoint switching
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

10. Contributions 6/17
Introduce some implementation proposals
Optimal downloading strategy
Evaluation of 2 practical strategies
A dataset:
• A 4-minute long multi-viewpoint video with 3 viewpoints
• Recorded navigation of 4 users
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

11. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

12. Single Viewpoint Video Characteristics 7/17
Most pixels are not displayed
Immersion requires hight resolution: > 16K
No simulation sickness requires low latency: < 10 ms
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

13. Single Viewpoint Video Streaming 8/17
Tile encoding Composition
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

14. Single Viewpoint Video Streaming 8/17
Viewport adaptive streaming
high
s1
low
s2 s3
t
server
bw
t
low high low
client
connect
mpd
s1:QER2 lo
s2:QER3 hi
s3:QER1 lo
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

15. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

16. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

17. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

18. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

19. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

20. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

21. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

22. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

23. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

24. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

25. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

26. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

27. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step (in the following)
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

28. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

29. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

30. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

31. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
Switch
wanted
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

32. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
Switch
wanted
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

33. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

34. Multi-Viewpoint: Quality of Experience 11/17
Distortion: encoding + projection
Stalls: pause for re-buffering
Switching lag: delay before actual viewpoint change
t
switch
requested
Immediate switch
direct switch
1 segment switch
2 segment switch
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

35. Multi-Viewpoint: Client Download Strategies 12/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

36. Multi-Viewpoint: Client Download Strategies 12/17
Optimal
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

37. Multi-Viewpoint: Client Download Strategies 12/17
Proactive: aims at direct switch
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

38. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

39. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

40. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

41. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

42. Test-Bed 13/17
A video: 4 min long, 3 viewpoints
• Encoded with no tile, 3x2 or 6x4 tiles
• 1 second long chunks
No Tile 3x2 6x4
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

43. Test-Bed 13/17
A video: 4 min long, 3 viewpoints
• Encoded with no tile, 3x2 or 6x4 tiles
• 1 second long chunks
Recorded trajectories of 4 users
A software to solve the three download strategy problems
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

44. Users behaviors 14/17
Users’ statistics:
User A User B User C User D
Sex F F F F
Age 34 31 27 27
Nb. switch event 18 10 4 13
Ang. Vel. (Deg./s):
global 28.81 20.88 28.44 20.00
2 s before a switch 8.48 3.42 3.24 7.53
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

45. Users behaviors 14/17
Users’ statistics:
User A User B User C User D
Sex F F F F
Age 34 31 27 27
Nb. switch event 18 10 4 13
Ang. Vel. (Deg./s):
global 28.81 20.88 28.44 20.00
2 s before a switch 8.48 3.42 3.24 7.53
2 s after a switch 59.57 46.88 54.25 51.21
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

46. Distortion 15/17
Reactive Proactive
0 10 20 30
−0.4
−0.2
0
Bandwidth (Mbps)
MedianPSNRGap(dB)
(g) No tiles
0 10 20 30
Bandwidth (Mbps)
(h) 3x2 tiles
0 10 20 30
Bandwidth (Mbps)
(i) 6x4 tiles
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

47. Switching Lag 16/17
Omni React
No tile
Omni React
3x2 tiles
Omni React
6x4 tiles
0
0.2
0.4
0.6
0.8
1
Ratiooflagdurations
Direct 1 segment 2 segments more than 2 segments
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

48. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos

49. Takeaway 17/17
Context
High immersion in omnidirectional media requires 6DoF
Discrete 6DoF is a realistic first step with new challenges
Contributions
A first dataset: users + video
Model the client optimal download strategies
Evaluate two radical strategies: proactive vs. reactive
Next?
Evaluate the switching lag impacts on the QoE
More MPV 360 videos + More user navigation traces
Predict viewpoint switching decisions
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos