Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Montage4D: Interactive Seamless Fusion of
Multiview Video Textures
Ruofei Du†‡, Ming Chuang‡¶
, Wayne Chang‡, Hugues Hoppe...

Motivation
Popularity of VR and AR devices
2

Motivation
Popularity of VR and AR devices
3

Motivation
Popularity of VR and AR devices
4

Motivation
Popularity of VR and AR devices
5

Motivation
Popularity of VR and AR devices
6

Motivation
Assorted VR and AR applications
7

Motivation
Assorted VR and AR applications
8

Motivation
Assorted VR and AR applications
9

These VR/AR applications have
Huge
Datarequirements

These VR/AR applications have
Huge
Datarequirements
Where is the
3D data going
to come from?

Introduction
Fusion4D and Holoportation
Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
R...

Fusing multiview video textures onto dynamic meshes
with real-time constraint remains a challenging task
14

15Screen recorded with OBS. Computed in real-time from 8 videos.

16
of the participants does not believe the 3D reconstructed person looks real
Escolano et al. Holoportation: Virtual 3D T...

17
of the participants does not believe the 3D reconstructed person looks real
Screen recorded with OBS. Computed in real-...

Related Work
3D Texture Montage
18

Related Work
3D Texture Montage
19

Related Work
3D Texture Montage
20

Related Work
3D Texture Montage
21

Related Work
3D Texture Montage
22

Related Work
3D Texture Montage
23

Related Work
3D Texture Montage
24
Screen-space optical flow could fix
many misregistration issues, but
heavily relies on ...

Related Work
3D Texture Montage
25
Up to now, few systems but Holoportation
could fuse dynamic meshes with
multiple camera...

Related Work
3D Texture Montage
26
𝑤𝑖 = 𝑉 ⋅ max 0, 𝑛 ⋅ 𝑣𝑖
𝛼
Normal Weighted Blending
(200 FPS)
Majority Voting for color c...

Motivation
Visual Quality Matters
27
Holoportation (Escolano et al. 2016) Montage4D (Du et al. 2018)

Motivation
Visual Quality Matters
Montage4D (Du et al. 2018)
28
Holoportation (Escolano et al. 2016)

What is our approach for real-time seamless
texture fusion?

Workflow
Identify and diffuse the seams
30

Workflow
Identify and diffuse the seams
31

Workflow
Identify and diffuse the seams
32

Workflow
Identify and diffuse the seams
33

What are the causes for the seams?

Motivation
Causes for Seams
35
Self-occlusion (depth seams)
Field-of-View (background seams)
Majority-voting (color seams)

Self-occlusion (Depth Seams)
One or two vertices of the triangle are occluded in the
depth map while the others are not.
S...

Seams
Causes
37
Depth: 1.3
Depth: 1.4
Depth: 30

Seams
Causes
38
Raw projection mapping results Seams after occlusion test

Majority Voting (Color Seams)
Each vertex is assigned with 8 colors coming from the 8
cameras. These colors are classified...

Majority Voting (Color Seams)
The triangle vertices have different results of the majority
voting colors, which may be cau...

Seams
Causes
41
Inside Cluster of
Mean L*A*B Color
Inside Cluster of
Mean L*A*B Color
Outside Cluster of
Mean L*A*B Color

Seams
Causes
42
Raw projection mapping results Seams after occlusion test Seams after majority voting test

Field of View (Background Seams)
One or two triangle vertices lie outside the
camera’s field of view or in the subtracted ...

Seams
Causes
44
Foreground
Foreground
Background

Seams
Causes
45
Raw projection mapping results Seams after field-of-view test

Seams
Causes
46

47
of the million-level triangles are seams (for each view)
Escolano et al. Holoportation: Virtual 3D Teleportation in Rea...

For a static frame, how can we get rid of the
annoying seams at interactive frame rate?

How can we spatially smooth the texture
(weight) field near the seams so that we
cannot see visible seams in the results?

Workflow
Identify and diffuse the seams
50

Geodesics
For diffusing the seams
51
Geodesic is the shortest route between two points on the surface.

Geodesics
For diffusing the seams
52
On triangle meshes, this is challenging because of the computation of tangent directi...

Geodesics
For diffusing the seams
53
We use the algorithm by Surazhsky and Hoppe for computing the approximate geodesics.
...

What are the causes for the blurring?

Motivation
Causes for blurring
56
Texture projection errors
Careful calibration + Bundle adjustment
Normal-weighted blendi...

Motivation
Causes for blurring
57
Texture projection errors
Normal-weighted blending
View-dependent rendering
𝑤𝑖 = 𝑉 ⋅ max...

58
Visibility test
Visibility% of view i
User camera’s view direction
Texture camera view direction
Geodesics
𝑫 𝑣
𝑖 (𝑡)
Fo...

Workflow
Identify and diffuse the seams
61

Temporal Texture Field
Temporally smooth the texture fields
62
For frame at time 𝑡, for each camera 𝑖, for each vertex 𝑣,
...

Temporal Texture
Fields
Transition between views
63

64
Holoportaion
Montage4D

65
Holoportaion
Montage4D

68
Note: Results use default parameters of Floating Textures, which may not be optimal for our datasets.
Still, for optica...

With additional computation for seams,
geodesics, and temporal texture fields, is
our approach still in real time?

Exmperiment
Cross-validation
72
Montage4D achieves better quality with over 90 FPS on NVIDIA GTX 1080
• Root mean square e...

73
of the participants does not believe the 3D reconstructed person looks real

Experiment
Break-down of a typical frame
74
Most of the time is used in communication between CPU and GPU

In conclusion, Montage4D uses seam identification, geodesic fields,
and temporal texture field to provides a practical tex...

Thank you
With a Starry Night Stylization
Ruofei Du
ruofei@cs.umd.edu
Amitabh Varshney
varshney@cs.umd.edu
Wayne Chang
wec...

Introduction
Fusion4D and Holoportation
image courtesy: Escolano et al. Holoportation: Virtual 3D Teleportation in Real-ti...

Limitations
Holoportation
image courtesy: afterpsychotherapy.com
1.6 Gbps per second

Introduction
Mobile Holoportation
image courtesy: Wayne Chang and Spencer Fowers

Introduction
Mobile Holoportation
image courtesy: Jeff Kramer
30-50 Mbps

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

You are reading a preview.

Activate your 30 day free trial to continue reading.

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 83 Ad

1 of 83 Ad