CARVED VISUAL HULLS FOR IMAGE-BASED MODELING

1. Furukawa, Yasutaka, and Jean Ponce
CARVED VISUAL HULLS FOR IMAGE-BASED MODELING
European Conference on Computer Vision. Springer Berlin Heidelberg, 2006.
Aftab Alam
Department of Computer Engineering, Kyung Hee University

2. Carved Visual Hulls for Image-Based Modeling
Contents
Preliminaries
Conclusion
Results & Comparison
Introduction
Local Refinement
7
6
5
2
1
4
3 Identify Rims
Global Optimization

3. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Preliminaries
• Silhouettes
– The dark shape and outline
o of an object/something
o Machining the outlines of the subject
o visible against a lighter background.
Silhouettes, Visual Hull
• Visual Hull
– Shape from silhouette
– A geometric entity
o created by shape-from-silhouette
o 3D reconstruction technique

4. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Preliminaries
Geometric camera calibration
• Geometric camera calibration
– Parameters of a lens and image sensor
o of an image/video camera.
– Prerequisite for making accurate geometric measurements from image data
– You can use these parameters to correct for lens distortion
o Measure the size of an object (in units)
o Applications:
 object measurement, navigation systems, and 3-D scene reconstruction.
Ref: Kannala, Juho, Janne Heikkilä, and Sami S. Brandt. "Geometric camera calibration." Wiley Encyclopedia of Computer Science & Engg (2008).

5. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Preliminaries
Silhouette-based 3D Reconstruction

6. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Preliminaries
Photo-consistency
• Photo-consistency
– Global method for estimating the depth variation in a scene
– Determines whether a given voxel is occupied.
– A voxel is considered to be photo-consistent
o when its color appears to be similar to all the cameras that can see it.
• RIM
– Visual rays from a camera which grazes the true surface tangentially give rise to a
smooth continuous curve on the true surface called the rim
• Carved: to cut so as to form something

7. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Introduction
What?
• Carved visual hulls for image-based modeling (optimization)

8. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Introduction
How?
• 3 Steps Process
• 3D shape from silhouette
• initialize deformation of a surface mesh
o under photo-consistency constraints
o output : rims that are used in graph cuts
1
• the visual hull is carved using graph cuts
• Global optimization process:
– use graph cuts with
– photoconsistency constraints +
– geometric constraints (rims)
2
• Local refinement Step:
– Recover surface details
– enforce geometric constraints
+
– photoconsistency constraints
3

9. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
1 - Identifying Rims on Visual Hull Surfaces
• 3D shape from silhouette
1. Corn Strips
2. Measuring Image Discrepancy
3. Identifying a Rim in a Cone Strip

10. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
1 - Identifying Rims on Visual Hull Surfaces
• consider an object observed by
– n calibrated cameras
– with optical centers O1, . . . ,On, &
– denote by γi its apparent contour in the image Ii
• The corresponding visual cone is the solid
– bounded by the surface Φi
– swept by the rays joining Oi to γi
• Φi grazes the object along a surface curve,
– the rim Γi.
• The visual hull is the solid formed
– by the intersection of the visual cones
– and its boundary can be decomposed into a set
of cone strips φi formed by patches from the
cone boundaries that connect to each other
at frontier points where two rims intersect
(Fig. 2(b)).
1- Cone strips (Lazebnik et al 2007 )

11. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
1 - Identifying Rims on Visual Hull Surfaces
• Fig. 2(c), each strip can be mapped onto a plane by parameterizing
– its boundary by the arc length of the corresponding image contour.
• Once the visual hull and the corresponding cone strips have been constructed
– using the algorithm propose by [Lazebnik et al 2007 ]
• the next step is to identify the rim that runs “horizontally” inside each strip
• Rim segments touch the surface of an object,
– the strip curves are used to minimize some measure of image discrepancy.
1- Cone strips (Lazebnik et al 2007 )

12. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
No of pictures = 5
1 - Identifying Rims on Visual Hull Surfaces
• Since rim segments are only part that touch surface of object,
– they can be found as strip curves that minimize some measure of image discrepancy.
– Used to determine the path length.
• Image Discrepancy Score/measure ( Faugeras and Keriven 1998 )
2. Image Discrepancy Score (Faugeras and Keriven 1998)
• Normalized cross
correlation B/W hi and hj
• hi … hj : Winows of the the
corresponding input image.
Grid μ ×μ = 11

13. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
1 - Identifying Rims on Visual Hull Surfaces
• the image discrepancy function should have small values along rims
– these curves can be found as shortest paths within the strips
– where path length is determined by the image discrepancy function
• A cone strip φi is represented by the undirected graph G
– with its polyhedral vertices V and edges E
– find shortest path by dynamic programming
3. Identifying a Rim in a Cone Strip

14. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
2 - Global Optimization
1. Deforming the surface
2. Building a graph and applying graph cuts

15. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
2- Global Optimization
1- Deforming the surface (Creating multiple layers)
– A
– independently & iteratively deform the surface of each component Gi inwards
– to generate multiple layers forming a 3D graph
2- Building a graph and applying graph cuts
– associate photoconsistency weights to the edges of this graph,
and use graph cuts to carve the surface

16. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
3 - Local Refinement
Local Minimum

17. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
3 - Local Refinement
• Iteratively refine the surface while enforcing all available
– photometric and geometric information.
• At every iteration, move each vertex v along its surface normal
– by a linear combination of three terms:
o an image discrepancy term,
o A smoothness term, and
o a rim consistency term.
(Hernandex Esteban and Schmitt 2004)
V = set of vertices
v = single vertex
S = Sink of vertex V
r(v) = rays
k = scalar coefficient (depends on obj. res.)

18. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Results
7 datasets

19. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Results

20. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Result
• Filtering ratio:
– how many % of identified rim points has been filtered out as outliers ( for each
contour )
• Sizes of components:
– show 3 largest connected components inside identified rim-segments
• From table, visual hull boundary is mostly covered by a single large connected
component except for Twin data set, which has many input images, and hence,
many rim curves.
Rim Identification Result

21. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Result
• bottleneck of computation is
– global optimization and local refinement step
– takes about 2 hr
Running Time (with 3.4 GHz Pentium 4)

22. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Comparisons
• Temple dataset

23. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
Conclusion and Limitations
• proposed a method for acquiring high-quality geometric models
– of complex 3D shapes
– by enforcing the photometric and geometric consistencies associated
– with multiple calibrated images
• Promising results and evaluation
• Since, cannot handle concavities too deep to be carved by the graph cuts.
– i.e. eye sockets of skulls

24. Data & Knowledge Engineering Lab, Department of Computer Engineering, Kyung Hee University, Korea.
References
1. https://pdfs.semanticscholar.org/2bea/e911649eb461bd430cf56afdc3340fcb9137.pdf
2. https://kr.mathworks.com/help/vision/ug/camera-calibration.html
3. https://www.youtube.com/watch?v=1hh9c4FOa2U
4. Furukawa, Yasutaka, and Jean Ponce. "Carved visual hulls for image-based modeling." European
Conference on Computer Vision. Springer Berlin Heidelberg, 2006.

25. Your Logo
THANK YOU!
?