aims at 1 segment switchViewtLegend:high quality medium quality low quality not requestedDownloaded:Requested:Selected ViewpointNowIMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional VideosMulti-Viewpoint: Client Download Strategies 12/17Proactive: aims at direct switchView
"Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos" in proceeding of ACM Multimedia System (MMSys'18).
Full immersion inside a Virtual Reality (VR) scene requires six
Degrees of Freedom (6DoF) applications, where the user is allowed
to perform translation and rotation movements within the video
content. The implementation of 6DoF applications is however still
an open question. We study in this paper a multi-viewpoint (MVP)
360-degree video streaming system, where a scene is
simultaneously
captured
by
multiple
synchronized
omnidirectional cameras. The user can only switch positions to
predefined viewpoints (VPs). We focus on the new challenges that
are introduced by adaptive MVP 360-degree video streaming. We
introduce several options for the video encoding with existing
technologies such as High Efficiency Video Coding (HEVC) tiles
and for the implementation of VP switching. We model three
video-segment download strategies for an adaptive streaming
client into Mixed Integer Linear Programming (MILP) problems:
an omniscient download scheduler, and two essentially different
strategies; one where the client proactively downloads all VPs to
guarantee fast VP switch, and one where the client reacts to client
commands. We recorded a first MVP 360-degree video with three
VPs, implemented a mobile MVP 360-degree video player, and
recorded different users navigating inside the content. We solved
the adaptive streaming optimization problems on this video and
those navigation traces. The results emphasize the gains of using
tiles on the quality metrics. They also emphasize the importance of
performing further study on VP switching prediction to reduce the
bandwidth consumption, and to measure the impact of VP
switching delay on the subjective Quality of Experience (QoE).
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
More Related Content
Similar to aims at 1 segment switchViewtLegend:high quality medium quality low quality not requestedDownloaded:Requested:Selected ViewpointNowIMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional VideosMulti-Viewpoint: Client Download Strategies 12/17Proactive: aims at direct switchView
Similar to aims at 1 segment switchViewtLegend:high quality medium quality low quality not requestedDownloaded:Requested:Selected ViewpointNowIMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional VideosMulti-Viewpoint: Client Download Strategies 12/17Proactive: aims at direct switchView (20)
WordPress Websites for Engineers: Elevate Your Brand
aims at 1 segment switchViewtLegend:high quality medium quality low quality not requestedDownloaded:Requested:Selected ViewpointNowIMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional VideosMulti-Viewpoint: Client Download Strategies 12/17Proactive: aims at direct switchView
1. IMT Atlantique
Bretagne-Pays de la Loire
École Mines-Télécom
Dynamic Adaptive Streaming
for Multi-Viewpoint
Omnidirectional Videos
Xavier Corbillon Francesca De Simone
Gwendal Simon Pascal Frossard
2. Omnidirectionnal Videos 2/17
Three degrees of freedom (3DoF)
v0 = O
k
ı u
v
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
3. Six Degrees of Freedom 3/17
Full immersion requires six degrees of freedom (6DoF)
Industry (e.g. MPEG) roadmap:
• 3DoF+
• Windowed 6DoF
• Omnidirectional 6DoF
In this paper: discrete 6DoF
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
4. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
5. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
6. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
7. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
Click
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
8. Discrete 6DoF: Use Case 4/17
cam1
cam2
cam3
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
9. Open Research Topics 5/17
Service implementation?
• resource explosion: storage, processing, delivery
Viewpoint 1 Viewpoint 2
Origin Server
? ?
Viewpoint 1 Viewpoint 2
Edge Server Client
Predictions:
• Viewpoint switching in addition to the bandwidth and head orientation
Downloading strategy
QoE assessment for viewpoint switching
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
10. Contributions 6/17
Introduce some implementation proposals
Optimal downloading strategy
Evaluation of 2 practical strategies
A dataset:
• A 4-minute long multi-viewpoint video with 3 viewpoints
• Recorded navigation of 4 users
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
11. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
12. Single Viewpoint Video Characteristics 7/17
Most pixels are not displayed
Immersion requires hight resolution: > 16K
No simulation sickness requires low latency: < 10 ms
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
13. Single Viewpoint Video Streaming 8/17
Tile encoding Composition
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
14. Single Viewpoint Video Streaming 8/17
Viewport adaptive streaming
high
s1
low
s2 s3
t
server
bw
t
low high low
client
connect
mpd
s1:QER2 lo
s2:QER3 hi
s3:QER1 lo
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
15. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
16. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
17. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
18. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
19. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
20. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
21. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
22. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
23. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
24. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
25. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
26. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
27. Multi-Viewpoint: Viewpoint Switching 9/17
cam1
cam2
cam3
Movement restriction:
• Full teleportation
• Viewport teleportation
• Step-by-step
• Viewport step-by-step (in the following)
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
28. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
29. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
30. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
31. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
Switch
wanted
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
32. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
Switch
wanted
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
33. Multi-Viewpoint: Client Side 10/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
34. Multi-Viewpoint: Quality of Experience 11/17
Distortion: encoding + projection
Stalls: pause for re-buffering
Switching lag: delay before actual viewpoint change
t
switch
requested
Immediate switch
direct switch
1 segment switch
2 segment switch
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
35. Multi-Viewpoint: Client Download Strategies 12/17
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
36. Multi-Viewpoint: Client Download Strategies 12/17
Optimal
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
37. Multi-Viewpoint: Client Download Strategies 12/17
Proactive: aims at direct switch
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
38. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
39. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
40. Multi-Viewpoint: Client Download Strategies 12/17
Reactive: aims at maximum quality
View0 View1 View2
Tile 0 Tile 1 Tile 0 Tile 1 Tile 0 Tile 1
Seg
i
Seg
i+1
Seg
i+2
Seg
i+3
t
Legend:
high quality medium quality low quality not requested
Downloaded:
Requested:
Selected Viewpoint Wanted Viewpoint
Now
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
41. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
42. Test-Bed 13/17
A video: 4 min long, 3 viewpoints
• Encoded with no tile, 3x2 or 6x4 tiles
• 1 second long chunks
No Tile 3x2 6x4
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
43. Test-Bed 13/17
A video: 4 min long, 3 viewpoints
• Encoded with no tile, 3x2 or 6x4 tiles
• 1 second long chunks
Recorded trajectories of 4 users
A software to solve the three download strategy problems
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
44. Users behaviors 14/17
Users’ statistics:
User A User B User C User D
Sex F F F F
Age 34 31 27 27
Nb. switch event 18 10 4 13
Ang. Vel. (Deg./s):
global 28.81 20.88 28.44 20.00
2 s before a switch 8.48 3.42 3.24 7.53
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
45. Users behaviors 14/17
Users’ statistics:
User A User B User C User D
Sex F F F F
Age 34 31 27 27
Nb. switch event 18 10 4 13
Ang. Vel. (Deg./s):
global 28.81 20.88 28.44 20.00
2 s before a switch 8.48 3.42 3.24 7.53
2 s after a switch 59.57 46.88 54.25 51.21
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
47. Switching Lag 16/17
Omni React
No tile
Omni React
3x2 tiles
Omni React
6x4 tiles
0
0.2
0.4
0.6
0.8
1
Ratiooflagdurations
Direct 1 segment 2 segments more than 2 segments
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
48. Table of contents
1 Background
2 Multi-Viewpoint
3 Evaluation
4 Conclusion
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos
49. Takeaway 17/17
Context
High immersion in omnidirectional media requires 6DoF
Discrete 6DoF is a realistic first step with new challenges
Contributions
A first dataset: users + video
Model the client optimal download strategies
Evaluate two radical strategies: proactive vs. reactive
Next?
Evaluate the switching lag impacts on the QoE
More MPV 360 videos + More user navigation traces
Predict viewpoint switching decisions
IMT Atlantique 12 June 2018 Xavier Corbillon et al. Dynamic Adaptive Streaming for Multi-Viewpoint Omnidirectional Videos