Carved Visual Hulls for Image-Based Modeling

Carved Visual Hulls for Image-Based Modeling
Yasutaka Furukawa, Jean Ponce

presented by
Phongsathorn Eakamongul

Department of Computer Science
Asian Institute of Technology

2009, July 27

Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 1 / 49

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims

5 Global Optimization

6 Local Reﬁnement

7 Result

8 Comparison

9 Limitation & Future work


Reference

silhouette image from http://www.ﬂickr.com/photos/alexfrance/3693058077/
space carving, shadow curving, voxel coloring from Silvio Savarese’s slide.
http://www.cs.cornell.edu/courses/cs664/2005fa/Lectures/lecture15.pdf
graph cut from ECCV 2006 tutorial
Multi-view shape reconstruction, lecture from Dana Cobzas, PIMS Postdoc


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Introduction

image-based Modeling
optimization



accuracy 1/200 ( 1 mm for 20 cm wide object ) from a set of low resolution
(640x480px 2 ) images∗

∗ From a comparison and evaluation of multi-view stereo reconstruction algorithms by ( Seitz et al. 2006 ), comparision of different method also available at

http://vision.middlebury.edu/mview/



choose a surface representation
deﬁne a photo-consistency function ( discrepancy between different projections
of their surface points )
solve the following minimization


Representation


Volumetric representation

shape from silhouette ( Visual Hull )
space carving
voxel coloring
others i.e. Voxel-based, Image ray based, Axis-aligned, etc.


shape from silhouette ( Visual Hull )
image object’s contour

Why contour ?
No texture
No shading


shape from silhouette
visual cones intersection

Carve all voxels outside the cone
Not photo-consistent (only to binary images)


Space carving algorithm

Initialize to a volume V containing the true scene
Choose a voxel on the current surface
Project to visible input images
Carve away voxels if not photo-consistent with images
Repeat until convergence


Voxel Coloring

Assign colors (RGBA : color + opacity) to voxels consistent with the input images.


Optimization


Graph cut

ﬁnd the minimum cut , by compute the maximum ﬂow , and look for the cut(s) that
separate origin and destination by cutting through bottlenecks of the network.


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Steps

shape from silhouette
initialize deformation of a surface mesh under photoconsistency constriants
output : rims that are used in graph cuts


Steps

global optimization process : use graph cuts with photoconsistency constraints +
geometric contraints ( rims )

local reﬁnement : enforce geometric contraints + photoconsistency constraints


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Identify Rims


Identify Rims
cone strips

( Lazebnik et al 2007 ) represent visual hull in terms of polyhedral cone strips
Γ : rim


Identify Rims
cone strips


Identify Rims
triangulated mesh model

( Lazebnik et al 2007 ) algorithm to obtain triangulated mesh model

In practices, measurement errors result to multiple horizontal neighbour


Identify Rims
Image Discrepancy

Since rim segments are only part that touch surface of object, they can be found
as strip curves that minimize some measure of image discrepancy

Image Discrepancy Score ( Faugeras and Keriven 1998 )
2
Pτ Pτ (1−NCC(hi ,hj ))2
f (p) = τ (τ −1) i=1 j=i+1 1 − exp(− 2
2σ1
)

NCC(hi , hj ) : normalizated cross correlation between hi and hj


Identify Rims

image discrepancy is smallest values, so ﬁnd shortest path ; path length = image
discrepancy function
ﬁnd shortest path by dynamic programming


Identify Rims
ﬁnd shortest path by dynamic programming

rim can be discontinous due to T-junction
First, Assume rim discontinuities occur only at right or left end points of each
connected strip component
change from undirected graph to direct graph and apply dynamic programming
result :


Identify Rims
remove false rim segmentation

assumption break at complicated structure i.e. the fold of human cloth
remove false rim segmentation
Among all vertics identiﬁed as rim points, ﬁlter out false-positives
vertex v is detected as false positive if
either 4Rl < g(v ) or Rl < g(v ) [vertical size is too large] and
f ∗ (v ) < η [vertical size is not small enough] and [average NCC wrose than η]

average NCC score ( Faugeras and Keriven 1998 )
Pτ Pτ
f ∗ (p) = 2
τ (τ −1) i=1 j=i+1 NCC(hi , hj )


Identify Rims


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Global Optimization


Global Optimization

¯
use rim Γ to split surface Ω into Gi (i = 1, ..., k )
¯
iteratively deform Gi inwards to generate multiple layers of 3D graph Ji and
associate photoconsistency weights to the edge of this graph
use graph cuts to carve surface


Global Optimization
Move Vertices down

At each iteration, move every vertex v along its surface normal N(v ) and apply
smoothing

v ← v − λ (ζ1 f (v ) + ζ2 )N(v ) + s(v )

notice that surface shrinks faster where image discrepancy function is larger
Constant they use in all their experiment
¯
ζ1 = 100, ζ2 = 0.1, β1 = 0.4, β2 = 0.3, λ = 20, ρ = 40, = 0.5 ∗ AverageLengthInGi


Global Optimization
Building a graph and Apply Graph Cuts

build vertical edge
build horizontal edge
assign vertical and horizontal edge weight

weight of edge (vi , vj )
α(f (vi )+f (vj )(δi +δj ))
wij = d(vi ,vj )

f (vi ) : photoconsistency function; d(vi , vj ) : length of edge; δi : sparity of vertices
around vi , α = 1.0 for horizontal, 6.0 for vertical
∞ edge weight for edge connected to source and sink
apply graph cuts

Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Local Reﬁnement


Local Reﬁnement
local minimum

increase resolution of mesh ( Hoppe et al 1993 ) until image projections of edges
become approximiately 2 px in length

rim consistency ( Hernandex Esteban and Schmitt 2004 )
exp(−vk rj −vk¯ rj )2 /2σ2
¯ ∗ 2
r (vk ) = vj¯rj P
∗
¯
exp(−(vk¯ rj −v ∗ rj )2 /2σ22 )
v ∈Vj j
k


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Result
7 data sets


Result


Result
Rim Identification Result

Filtering ratio : how many % of identified rim points has been filtered out as outliers ( for each
contour )
Sizes of components : show 3 largest connected components inside identified rim-segments

From table, visual hull boundary is mostly covered by a single large connected component except
for Twin data set, which has many input images, and hence, many rim curves.


Result
Running Time (with 3.4 GHz Pentium 4)

bottleneck of computation is global optimization and local reﬁnement step ( takes about
2 hr )


Result
Testing algorithm w/o graph cuts phase ( use only local method )

local minimum problem


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Comparisons

( This paper ) rim constriants
( Voiazis et al 2005 ) add inﬂationary ballooning term to enery function in graph cuts to
prevent over-carving


Comparisons

( Hernandex Exteban and Schmitt 2004 ) In local refinement, use gradient flow instead
of direct derivatives f (v )
Some other differences, i.e. local iterative deformation, that have problem avoiding
local minima.

outperforms in reconstructing
mistake in rim-identification steps concave structure


Comparisons


Comparisons
Multi-view stereo evaluation ( http://vision.middlebury. edu/mview/ )

Temple data set

Accurracy : distance d that bring 90% of result within ground-truth surface
Completeness : % ground-truth surface that lies within 1.25 mm of the result


Outline

1 Reference

2 Introduction

3 Steps

4 Identify Rims


6 Local Reﬁnement

7 Result

8 Comparison



Limitation & Future work

Since, cannot handle concavities too deep to be carved by the graph cuts. i.e. eye
sockets of skulls
To overcomes this, combine their works with sparse wide-baseline stereo from
interest point (e.g. Schaffalitzky and Zisserman 2001) in order to incorporate
stronger geometric constraints in the carving and local reﬁnement stages
handle non-Lambertain surfaces ( Soatto et al 2003 )
simutaneous camera calibration where both camera parameters and surface
shape are reﬁned simultaneously using bundle adjustment ( Uffenkamp 1993 )


Carved Visual Hulls for Image-Based Modeling

More Related Content

What's hot

Similar to Carved Visual Hulls for Image-Based Modeling

Recently uploaded

Carved Visual Hulls for Image-Based Modeling