Recommended
More Related Content
Similar to Light Field Video for 6 Degrees-of-Freedom VR Playback
Similar to Light Field Video for 6 Degrees-of-Freedom VR Playback (8)
Recently uploaded
Recently uploaded (20)
Light Field Video for 6 Degrees-of-Freedom VR Playback
- 1. Light Field Video for 6-DOF VR Playback William Jiang Lytro, Inc. EI 2017 DPMI
- 2. ● True presence ● Light Field video approach ● Challenges ● Conclusion Outline
- 3. Presence
- 5. 360 2D Panorama LG 360 G5 GoPro Omni Bublcam Giroptic 360 cam ALLie 360 Camera
- 6. ● 360 immersion ● Stereoscopic vision Presence
- 7. Directional Stereo Nokia Ozo Hubblo VR camera Vuze camera
- 8. Horizontal 360 Stereo Google Jump Facebook 360 Surround Samsung Project Beyond Jaunt VR camera
- 9. Omni-Directional Stereo An anaglyph of rendered ODS images (source)
- 10. ● 360 Immersion ● Stereoscopic Vision ● Motion Parallax Presence
- 13. Omni-Directional Stereo Light Field Volume
- 14. Omni-Directional Stereo Light Field Volume
- 15. Light Field Volume Omni-Directional Stereo
- 16. Light Field Volume Omni-Directional Stereo
- 17. Light Field Volume Omni-Directional Stereo
- 19. ● 360 Immersion ● Stereoscopic Vision ● Motion Parallax ● Convergence & Accommodation Presence
- 21. ● 360 Immersion ● Stereoscopic Vision ● Motion Parallax ● Convergence & Accommodation ● Photorealism Presence
- 22. ● 360 Immersion ● Stereoscopic Vision ● Motion Parallax ● Convergence & Accommodation ● Photorealism Presence 964 Carrera RS by Djordje Ilic (source)
- 24. Light Field Video Pipeline
- 25. Capture Stanford plenoptic camera array used for Light Field research in 2004.
- 27. Light Field Volume Merging
- 31. Light Field Video Post-Production
- 37. Data, Data, Data
- 38. Light Field Video Viewing Volume
- 40. Light Field Video Mastering ● Viewing volume and data size ● Compression
- 41. Light Field Video Playback ● Position tracking ● View synthesis ● Performance
- 43. ● True Presence is more than 360 stereo. ● Light Field volumetric video is a huge step forward that combines motion parallax and photo-realism. ● The Holodeck is near us. Conclusions
Editor's Notes
- I will first attempt to define the true presence, and then the Light Field video approach spearheaded by Lytro, along with the technical challenges.
- The true promise of Virtual Reality is Presence, taking you to a completely different time and/or different space. It does that by recreating all sensory inputs that matches that time and space.
- The first thing in presence is 360-degree immersive visual perception.
- There are numerous camera rigs that have been released to capture 360-degree video, but what they capture is simply 2D panorama, which means there is no sense of depth.
- The second requirement for Presence is stereoscopic vision, giving you slightly different view for each eye, thus generate the sense of depth.
- The first group of cameras rigs captures stereoscopic visual data only for certain directions, and not all 360 directions.
- Then the second group of camera rigs capture video for all directions, and use panoramic stitching techniques to generate stereoscopic vision for all 360 degrees. However, as soon as you look up or down, you gradually lose the stereo vision. That’s why I call this Horizontal 360 stereo.
- Then Omni-directional stereo would give you stereoscopic vision for all 360-180 directions. For example, this is an anaglyph of a rendered omni-directional stereo image. No cameras in the market today can capture omni-directional stereo video, yet.
- Because we humans likes to move, motion parallax is another important depth-cue, which means objects at different distances move at different rate as you move your head.
- The way Lytro solves this problem is to capture a Light Field Volume.
- This diagram shows Omni-Directional stereo, with blue light-rays coming into your left eye, and red light-rays coming into your right-eye. As both eyes rotate around the center of perspective, the light-rays generate stereoscopic vision for all 360-degree directions. However, as soon as you move away from the center of perspective, you don’t have any captured video information to render for the new perspective.
- A Light Field volume is a volumetric video that has all the light-rays in order to recreate the perspective at any point inside the spherical volume. This is done by capturing a sampling of all the light-rays entering the spherical surface of the volume.
- In comparison, you can think of the Light Field volume as enabling Omni-Directional Stereo for any point inside the spherical volume.
- This is what we refer to as 6-degrees of freedom inside the Light Field Volume.
- This is an example of the Light Field volumetric video produced by Lytro, called Moon. The white sphere you see here is the volume, about 0.6 meters in diameter. As long as you are looking from inside the sphere, you can experience complete motion parallax and view dependent lighting.
- Then there is Convergence and Accommodation.
- It means when we look into a VR headset, both of our eyes always focus on the imaging plane, rather than focusing on the object at the right distance. This causes sensory conflicts and thus fatigue after short period of time. I won’t get into much details into this.
- Then finally, what you see needs to be photo real. To me, this means view-dependent lighting, reflection, and shadows. It not only needs to be real, but it needs to be photo-real.
- Like this rendered video of 964 Carrera created by Director Djordje Ilic. It looks super real, because it has all the imperfections just at the right place.
- In these two axis of immersion and photo-realism, Games Engines does real-time graphics rendering based on head-position tracking, it gives you motion parallax, but with only milliseconds of rendering time, it cannot give you the high-quality photo-realism. On the other hand, the 360-degree videos captures high-quality panorama, but fails to provide motion parallax. The Lytro Light Field volumetric video approach provides both motion parallax and high-quality photoreal imaging experiences.
- I will now run through our Light Field volumetric video pipeline and some of the technical challenges.
- This is the Stanford plenoptic camera array that was used in their Light Field research in 2004. It is not what we are using today, but the Lytro technology originates from the Light Field research done at Stanford during that time. The challenges though are still similar, for example, synchronizing cameras, capturing video at high frame-rate, and handling huge amount of data.
- Light Field processing is the core the pipeline, where the captured camera data is calibrated, color-matched, and processed to reconstruct the 3D scene for the viewing volume. This shows an example of the processed depth-map from one of our captured scenes.
- When your camera rig does not capture 360-degree FOV all at once, you need to merge Light Field volumes. It’s a relatively painless process.
- Let’s say your camera rig captures a limited field-of-view and generates a spherical volume for that field-of-view.
- Using the same camera rig, you can then capture another area of the scene, with only slight overlap between this and the previous captured field-of-view.
- You can then merge the two captured data into the same scene, resulting the same viewing volume. A couple things to be careful with here, first, you need to ensure identical lighting between the two captures; second, you need to calibrate for the two camera positions, using some calibration workflow. Once you do these, you can even capture action in different directions at different time, and have them show up in the same 360-degree frame.
- Post production for volumetric video is very challenging, in terms of adding visual-effects for the 3D geometry, and ensuring consistent view-dependent lighting and color, across the viewing volume.
- For example, the Moon video was captured in the historic Mack Sennett soundstage in LA. As you can see, there is only a small patch of the Moon surface and the ladder of the Moon lander, in the scene.
- The floor is then blended with computer-generated moon surface.
- A computer generated moon lander got inserted.
- Then the Earth was inserted into the background.
- Then changed the ceiling. These all look easy and simple right? But, the challenge is to do this for the entire viewing volume, which requires the development of a whole new suite of post-production tools.
- It is no surprise that we are dealing with a large volume of data. The volumetric video data size is cubic relative to the diameter of the viewing volume. Hence we need tools for propagating visual-effect change spatially as well as temporally.
- One of the things we realized in the process was that the captured volume may not be the same as the Playback volume.
- For example, this shows a heat-map of the locations inside the viewing volume that are frequently visited for this particular piece of content. Red dots mean highly-visited positions, and green-dots mean less-frequently visited. We can take this information, and optimize the size and shape of the volume for distribution.
- This process is called Mastering. Here, the grey sphere represents the captured Light Field volume, and the golden area inside the sphere means the actual volume you package and ship for playback. For instance, you may choose to release a smaller volume to reduce the video size; or in the second case, you may choose to release a volume with maximal horizontal motion parallax.
- During playback, the Lytro application tracks the precise positions of both eyes, and uses the volumetric video data, to render a 360-degree view of the scene for the perspective of each eye positions. This is done in real-time, and hence performance is supercritical.
- In summary, the true presence promised by Virtual Reality is a lot more than just 360-degree panoramic video. Lytro’s Light Field volumetric video approach is a huge step forward that combines motion parallax and high-quality photo-real imaging. There are still a number of challenges ahead of us, but the vision of the Holodeck is already very near. Thank you!