Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

246 views

Published on

Project Site: http://montage4d.com

The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution towards real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to their approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

  1. 1. Montage4D: Interactive Seamless Fusion of Multiview Video Textures Ruofei Du†‡, Ming Chuang‡¶ , Wayne Chang‡, Hugues Hoppe‡§ , and Amitabh Varshney† †Augmentarium and GVIL | UMIACS | University of Maryland, College Park ‡Microsoft Research, Redmond ¶ Apple Inc. § Google Inc. THE AUGMENTARIUM VIRTUAL AND AUGMENTED REALITY LABORATORY AT THE UNIVERSITY OF MARYLAND COMPUTER SCIENCE UNIVERSITY OF MARYLAND
  2. 2. Motivation Popularity of VR and AR devices 2
  3. 3. Motivation Popularity of VR and AR devices 3
  4. 4. Motivation Popularity of VR and AR devices 4
  5. 5. Motivation Popularity of VR and AR devices 5
  6. 6. Motivation Popularity of VR and AR devices 6
  7. 7. Motivation Assorted VR and AR applications 7
  8. 8. Motivation Assorted VR and AR applications 8
  9. 9. Motivation Assorted VR and AR applications 9
  10. 10. These VR/AR applications have Huge Datarequirements
  11. 11. These VR/AR applications have Huge Datarequirements Where is the 3D data going to come from?
  12. 12. Introduction Fusion4D and Holoportation Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016) RGB Depth RGB Depth Mask RGB Depth Mask RGB Depth Mask Depth estimation & segmentation 8 Pods Capture SITE A SITE B Volumetric fusion Color Rendering Remote rendering Mesh, color, audio streams Network 13
  13. 13. Fusing multiview video textures onto dynamic meshes with real-time constraint remains a challenging task 14
  14. 14. 15Screen recorded with OBS. Computed in real-time from 8 videos.
  15. 15. 16 of the participants does not believe the 3D reconstructed person looks real Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
  16. 16. 17 of the participants does not believe the 3D reconstructed person looks real Screen recorded with OBS. Computed in real-time from 8 videos.
  17. 17. Related Work 3D Texture Montage 18
  18. 18. Related Work 3D Texture Montage 19
  19. 19. Related Work 3D Texture Montage 20
  20. 20. Related Work 3D Texture Montage 21
  21. 21. Related Work 3D Texture Montage 22
  22. 22. Related Work 3D Texture Montage 23
  23. 23. Related Work 3D Texture Montage 24 Screen-space optical flow could fix many misregistration issues, but heavily relies on RGB features and screen resolution, and fails when rapidly changing viewpoints.
  24. 24. Related Work 3D Texture Montage 25 Up to now, few systems but Holoportation could fuse dynamic meshes with multiple cameras in real time. This recent SIGGRAPH 2017 paper produces excellent dynamic reconstruction results, but uses a single RGBD camera, which may result in lots of occlusion.
  25. 25. Related Work 3D Texture Montage 26 𝑤𝑖 = 𝑉 ⋅ max 0, 𝑛 ⋅ 𝑣𝑖 𝛼 Normal Weighted Blending (200 FPS) Majority Voting for color correction For each vertex, and for each texture, test if the projected color agrees with more than half of the other textures, if not, set the texture weight field to 0. Visibility test Normal vector Texture camera view direction
  26. 26. Motivation Visual Quality Matters 27 Holoportation (Escolano et al. 2016) Montage4D (Du et al. 2018)
  27. 27. Motivation Visual Quality Matters Montage4D (Du et al. 2018) 28 Holoportation (Escolano et al. 2016)
  28. 28. What is our approach for real-time seamless texture fusion?
  29. 29. Workflow Identify and diffuse the seams 30
  30. 30. Workflow Identify and diffuse the seams 31
  31. 31. Workflow Identify and diffuse the seams 32
  32. 32. Workflow Identify and diffuse the seams 33
  33. 33. What are the causes for the seams?
  34. 34. Motivation Causes for Seams 35 Self-occlusion (depth seams) Field-of-View (background seams) Majority-voting (color seams)
  35. 35. Self-occlusion (Depth Seams) One or two vertices of the triangle are occluded in the depth map while the others are not. Seams Causes 36
  36. 36. Seams Causes 37 Depth: 1.3 Depth: 1.4 Depth: 30
  37. 37. Seams Causes 38 Raw projection mapping results Seams after occlusion test
  38. 38. Majority Voting (Color Seams) Each vertex is assigned with 8 colors coming from the 8 cameras. These colors are classified into different clusters in LAB color space with a 0.1 threshold. The mean color of the largest cluster is named majority voting color. Seams Causes 39
  39. 39. Majority Voting (Color Seams) The triangle vertices have different results of the majority voting colors, which may be caused by either mis- registration or self-occlusion. Seams Causes 40
  40. 40. Seams Causes 41 Inside Cluster of Mean L*A*B Color Inside Cluster of Mean L*A*B Color Outside Cluster of Mean L*A*B Color
  41. 41. Seams Causes 42 Raw projection mapping results Seams after occlusion test Seams after majority voting test
  42. 42. Field of View (Background Seams) One or two triangle vertices lie outside the camera’s field of view or in the subtracted background region while the rest are not. Seams Causes 43
  43. 43. Seams Causes 44 Foreground Foreground Background
  44. 44. Seams Causes 45 Raw projection mapping results Seams after field-of-view test
  45. 45. Seams Causes 46
  46. 46. 47 of the million-level triangles are seams (for each view) Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
  47. 47. For a static frame, how can we get rid of the annoying seams at interactive frame rate?
  48. 48. How can we spatially smooth the texture (weight) field near the seams so that we cannot see visible seams in the results?
  49. 49. Workflow Identify and diffuse the seams 50
  50. 50. Geodesics For diffusing the seams 51 Geodesic is the shortest route between two points on the surface.
  51. 51. Geodesics For diffusing the seams 52 On triangle meshes, this is challenging because of the computation of tangent directions. And shortest paths are defined on edges instead of the vertices.
  52. 52. Geodesics For diffusing the seams 53 We use the algorithm by Surazhsky and Hoppe for computing the approximate geodesics. The idea is to maintain only 2~3 shortest paths along each edge to reduce the computational cost.
  53. 53. 54
  54. 54. What are the causes for the blurring?
  55. 55. Motivation Causes for blurring 56 Texture projection errors Careful calibration + Bundle adjustment Normal-weighted blending Imprecise geometries / Noisy point clouds / Different specular highlights 𝑤𝑖 = 𝑉 ⋅ max 0, 𝑛 ⋅ 𝑣𝑖 𝛼
  56. 56. Motivation Causes for blurring 57 Texture projection errors Normal-weighted blending View-dependent rendering 𝑤𝑖 = 𝑉 ⋅ max 0, 𝑣 ⋅ 𝑣𝑖 𝛼 𝑤𝑖 = 𝑉 ⋅ max 0, 𝑛 ⋅ 𝑣𝑖 𝛼
  57. 57. 58 Visibility test Visibility% of view i User camera’s view direction Texture camera view direction Geodesics 𝑫 𝑣 𝑖 (𝑡) For frame at time 𝑡, for each camera 𝑖, for each vertex 𝑣, We define the Desired Texture Field:
  58. 58. 59
  59. 59. 60
  60. 60. Workflow Identify and diffuse the seams 61
  61. 61. Temporal Texture Field Temporally smooth the texture fields 62 For frame at time 𝑡, for each camera 𝑖, for each vertex 𝑣, We define the Temporal Texture Field (exponential smoothing) 𝑻 𝑣 𝑖 𝑡 = 1 − 𝜆 𝑻 𝑣 𝑖 𝑡 − 1 + 𝜆𝑫 𝑣 𝑖 (𝑡) Texture field of the previous frame Temporal smoothing factor 1 𝐹𝑃𝑆
  62. 62. Temporal Texture Fields Transition between views 63
  63. 63. 64 Holoportaion Montage4D
  64. 64. 65 Holoportaion Montage4D
  65. 65. 66
  66. 66. 67
  67. 67. 68 Note: Results use default parameters of Floating Textures, which may not be optimal for our datasets. Still, for optical flow based approach, it would be better if seam vertices are assigned with less texture weights.
  68. 68. 69
  69. 69. 70
  70. 70. With additional computation for seams, geodesics, and temporal texture fields, is our approach still in real time?
  71. 71. Exmperiment Cross-validation 72 Montage4D achieves better quality with over 90 FPS on NVIDIA GTX 1080 • Root mean square error (RMSE) ↓ • Structural similarity (SSIM) ↑ • Signal-to-noise ratio (PSNR) ↑
  72. 72. 73 of the participants does not believe the 3D reconstructed person looks real
  73. 73. Experiment Break-down of a typical frame 74 Most of the time is used in communication between CPU and GPU
  74. 74. In conclusion, Montage4D uses seam identification, geodesic fields, and temporal texture field to provides a practical texturing solution for real-time 3D reconstructions. In the future, we envision that Montage4D is useful for fusing the massive multi-view video data into VR applications like remote business meeting, remote training, and live broadcasting.
  75. 75. Thank you With a Starry Night Stylization Ruofei Du ruofei@cs.umd.edu Amitabh Varshney varshney@cs.umd.edu Wayne Chang wechang@microsoft.com Ming Chuang mingchuang82@gmail.com Hugues Hoppe hhoppe@gmail.com • www.montage4d.com • www.duruofei.com • shadertoy.com/user/starea • github.com/ruofeidu • I am graduating December 2018.
  76. 76. Introduction Fusion4D and Holoportation image courtesy: Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
  77. 77. Limitations Holoportation image courtesy: afterpsychotherapy.com 1.6 Gbps per second
  78. 78. Introduction Mobile Holoportation image courtesy: Wayne Chang and Spencer Fowers
  79. 79. Introduction Mobile Holoportation image courtesy: Jeff Kramer 30-50 Mbps
  80. 80. 83

×