My talk at Streaming Media East 2017, covers 360 streaming workflow and dives into challenges; starting from delivering 6 times data compared to traditional 4K quality in 360, to processing this data on the client playback browsers and mobile devices. Learn about different projection formats, compression, tiles based streams, and field of view adaptive streaming on the player side. You will also learn how the WebVR standards enables 360 streaming across VR devices, and the challenges and workarounds Yahoo has implemented to support 360 viewing across browsers like Edge and Safari on mobile.

1. 360 VR at Yahoo
1

2. Hello!
Satender Saroha
VR Enthusiast at Yahoo
satender@yahoo-inc.com
2

3. “
Virtual reality is a technology
that could actually allow you to
connect on a real human
level, soul-to-soul, regardless
of where you are in the world.
Chris Milk
3

4. Outline
▪ 360 Streaming Workflow
▫ Stitching
▫ Player Rendering
▪ Mobile VR
▪ Challenges
4

5. 360 Streaming Workflow
5

6. Stitching
6
3 Step Process
▪ Raw frame image
processing
▪ Calibration
▪ Optical Flow
Softwares
▪ Vahana VR
▪ Surround 360
▪ Radeon Loom

7. Player Rendering - Steps
▪ Equirectangular video as source
▪ Create sphere and apply equirectangular video as
its texture
▪ Create camera and place it in the center of sphere
▪ Build user interactions and device motion to control
the camera
7

8. Player Rendering - WebGL/Metal
Concepts
 GL renderer
 Scene
 Camera
● PerspectiveCamera
 3D object
● Geometry
● Material
● Geometry + Material = Mesh
 Vector3 position 8

9. Player Rendering
9

10. Demo
http://codepen.io/ssaroha/pen/oWWbLx
10

11. Mobile 3D VR Experience
▪ Enable 3D VR room scale experience for all videos (not just 360)
▪ Uses DayDream VR + Unity SDK
11

12. Challenges
12

13. Problem 1: Motion to Photon Latency on mobile
13

14. Solution: Motion to Photon Latency on mobile
Double Buffered Rendering Single Rendering
14

15. Challenge 2: 360 Support in Safari
mobile web
CORS Bug
▪ Safari does not
respect CORS
headers with webgl
▪ Unable to draw
content to Canvas
Fix - Server Side
▪ Embed player in
iframe on same
domain as CDN
▪ Reverse proxy
mobile-web traffic to
CDN through own
server
15

16. Challenge 3: Enormous Data
▪ True immersive experience needs 8K per eye
▪ Need > 100 Mbps
▪ Avg Wifi in US 20 Mbps
16

17. Solution: Compression
▪ Cubemap vs Equirectangular - 30% BW reduction
▪ HEVC, VP9 vs H.264 - 40% reduction
17

18. Solution: FOV Adaptive Streaming
▪ FOV Adaptive Streaming
▪ Spatially divide videos
into multiple tiles
▪ MPEG-DASH SRD
18

19. Solution: FOV Adaptive Streaming
(Cont’d)
https://arxiv.org/pdf/1609.08042.pdf
19

20. Challenge 4: Achieving 6 DoF
http://stanford.edu/~jingweih/papers/6dof.pdf20

21. Solution: 6 DoF by Adobe Research
▪ Enhances existing monoscopic 360 videos with 6 DoF
▪ Infers camera path and 3D scene geometry using structure-
from-motion techniques
▪ A novel warping algorithm synthesizes views on the fly by
warping original content
21

22. Summary
▪ Successful 360 streaming of the Presidential inauguration
▪ Production ready 360 iOS player using Metal
22

23. Thanks
!!
More questions?
Twitter: https://twitter.com/ssaroha
Email: satender@yahoo-inc.com
23