Carved Visual Hulls for Image-Based Modeling
                  Yasutaka Furukawa, Jean Ponce


                           ...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Reference




   silhouette image from http://www.flickr.com/photos/alexfrance/3693058077/
   space carving, shadow curving...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Introduction




    image-based Modeling
    optimization




      Phongsathorn (AIT)   Carved Visual Hulls for Image-Ba...
image-based Modeling


        accuracy 1/200 ( 1 mm for 20 cm wide object ) from a set of low resolution
        (640x480...
image-based Modeling




   choose a surface representation
   define a photo-consistency function ( discrepancy between di...
Representation




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Present...
Volumetric representation




   shape from silhouette ( Visual Hull )
   space carving
   voxel coloring
   others i.e. V...
shape from silhouette ( Visual Hull )
image object’s contour




Why contour ?
      No texture
      No shading




     ...
shape from silhouette
visual cones intersection




      Carve all voxels outside the cone
      Not photo-consistent (on...
Space carving algorithm




   Initialize to a volume V containing the true scene
   Choose a voxel on the current surface...
Voxel Coloring




Assign colors (RGBA : color + opacity) to voxels consistent with the input images.




       Phongsath...
Optimization




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Presentat...
Graph cut




   find the minimum cut , by compute the maximum flow , and look for the cut(s) that
   separate origin and de...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Steps




  shape from silhouette
  initialize deformation of a surface mesh under photoconsistency constriants
  output :...
Steps

  global optimization process : use graph cuts with photoconsistency constraints +
  geometric contraints ( rims )
...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Identify Rims




      Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Present...
Identify Rims
cone strips

      ( Lazebnik et al 2007 ) represent visual hull in terms of polyhedral cone strips
      Γ ...
Identify Rims
cone strips




         Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision ...
Identify Rims
triangulated mesh model




        ( Lazebnik et al 2007 ) algorithm to obtain triangulated mesh model




...
Identify Rims
Image Discrepancy




     Since rim segments are only part that touch surface of object, they can be found
...
Identify Rims




   image discrepancy is smallest values, so find shortest path ; path length = image
   discrepancy funct...
Identify Rims
find shortest path by dynamic programming




     rim can be discontinous due to T-junction
     First, Assu...
Identify Rims
remove false rim segmentation




     assumption break at complicated structure i.e. the fold of human clot...
Identify Rims




      Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Present...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Global Optimization




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Pr...
Global Optimization




                                     ¯
   use rim Γ to split surface Ω into Gi (i = 1, ..., k )
  ...
Global Optimization
Move Vertices down




At each iteration, move every vertex v along its surface normal N(v ) and apply...
Global Optimization
Building a graph and Apply Graph Cuts




     build vertical edge
     build horizontal edge
     ass...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Local Refinement




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Presen...
Local Refinement
local minimum




increase resolution of mesh ( Hoppe et al 1993 ) until image projections of edges
become...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Result
7 data sets




         Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student...
Result




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Presentation   ...
Result
Rim Identification Result




Filtering ratio : how many % of identified rim points has been filtered out as outliers ...
Result
Running Time (with 3.4 GHz Pentium 4)




bottleneck of computation is global optimization and local refinement step...
Result
Testing algorithm w/o graph cuts phase ( use only local method )




local minimum problem




         Phongsathor...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Comparisons




( This paper ) rim constriants
( Voiazis et al 2005 ) add inflationary ballooning term to enery function in...
Comparisons

( Hernandex Exteban and Schmitt 2004 ) In local refinement, use gradient flow instead
of direct derivatives f (...
Comparisons




     Phongsathorn (AIT)   Carved Visual Hulls for Image-Based Modeling   Machine Vision Student Presentati...
Comparisons
Multi-view stereo evaluation ( http://vision.middlebury. edu/mview/ )




Temple data set




Accurracy : dist...
Outline


1   Reference

2   Introduction

3   Steps

4   Identify Rims

5   Global Optimization

6   Local Refinement

7  ...
Limitation & Future work




   Since, cannot handle concavities too deep to be carved by the graph cuts. i.e. eye
   sock...
Upcoming SlideShare
Loading in …5
×

Carved Visual Hulls for Image-Based Modeling

1,538 views

Published on

paper by Yasutaka Furukawa, Jean Ponce. Presented in class by me. Apologized if any mistaken.

Published in: Education, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,538
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Carved Visual Hulls for Image-Based Modeling

  1. 1. Carved Visual Hulls for Image-Based Modeling Yasutaka Furukawa, Jean Ponce presented by Phongsathorn Eakamongul Department of Computer Science Asian Institute of Technology 2009, July 27 Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 1 / 49
  2. 2. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 2 / 49
  3. 3. Reference silhouette image from http://www.flickr.com/photos/alexfrance/3693058077/ space carving, shadow curving, voxel coloring from Silvio Savarese’s slide. http://www.cs.cornell.edu/courses/cs664/2005fa/Lectures/lecture15.pdf graph cut from ECCV 2006 tutorial Multi-view shape reconstruction, lecture from Dana Cobzas, PIMS Postdoc Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 3 / 49
  4. 4. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 4 / 49
  5. 5. Introduction image-based Modeling optimization Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 5 / 49
  6. 6. image-based Modeling accuracy 1/200 ( 1 mm for 20 cm wide object ) from a set of low resolution (640x480px 2 ) images∗ ∗ From a comparison and evaluation of multi-view stereo reconstruction algorithms by ( Seitz et al. 2006 ), comparision of different method also available at http://vision.middlebury.edu/mview/ Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 6 / 49
  7. 7. image-based Modeling choose a surface representation define a photo-consistency function ( discrepancy between different projections of their surface points ) solve the following minimization Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 7 / 49
  8. 8. Representation Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 8 / 49
  9. 9. Volumetric representation shape from silhouette ( Visual Hull ) space carving voxel coloring others i.e. Voxel-based, Image ray based, Axis-aligned, etc. Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 9 / 49
  10. 10. shape from silhouette ( Visual Hull ) image object’s contour Why contour ? No texture No shading Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 10 / 49
  11. 11. shape from silhouette visual cones intersection Carve all voxels outside the cone Not photo-consistent (only to binary images) Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 11 / 49
  12. 12. Space carving algorithm Initialize to a volume V containing the true scene Choose a voxel on the current surface Project to visible input images Carve away voxels if not photo-consistent with images Repeat until convergence Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 12 / 49
  13. 13. Voxel Coloring Assign colors (RGBA : color + opacity) to voxels consistent with the input images. Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 13 / 49
  14. 14. Optimization Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 14 / 49
  15. 15. Graph cut find the minimum cut , by compute the maximum flow , and look for the cut(s) that separate origin and destination by cutting through bottlenecks of the network. Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 15 / 49
  16. 16. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 16 / 49
  17. 17. Steps shape from silhouette initialize deformation of a surface mesh under photoconsistency constriants output : rims that are used in graph cuts Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 17 / 49
  18. 18. Steps global optimization process : use graph cuts with photoconsistency constraints + geometric contraints ( rims ) local refinement : enforce geometric contraints + photoconsistency constraints Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 18 / 49
  19. 19. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 19 / 49
  20. 20. Identify Rims Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 20 / 49
  21. 21. Identify Rims cone strips ( Lazebnik et al 2007 ) represent visual hull in terms of polyhedral cone strips Γ : rim Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 21 / 49
  22. 22. Identify Rims cone strips Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 22 / 49
  23. 23. Identify Rims triangulated mesh model ( Lazebnik et al 2007 ) algorithm to obtain triangulated mesh model In practices, measurement errors result to multiple horizontal neighbour Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 23 / 49
  24. 24. Identify Rims Image Discrepancy Since rim segments are only part that touch surface of object, they can be found as strip curves that minimize some measure of image discrepancy Image Discrepancy Score ( Faugeras and Keriven 1998 ) 2 Pτ Pτ (1−NCC(hi ,hj ))2 f (p) = τ (τ −1) i=1 j=i+1 1 − exp(− 2 2σ1 ) NCC(hi , hj ) : normalizated cross correlation between hi and hj Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 24 / 49
  25. 25. Identify Rims image discrepancy is smallest values, so find shortest path ; path length = image discrepancy function find shortest path by dynamic programming Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 25 / 49
  26. 26. Identify Rims find shortest path by dynamic programming rim can be discontinous due to T-junction First, Assume rim discontinuities occur only at right or left end points of each connected strip component change from undirected graph to direct graph and apply dynamic programming result : Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 26 / 49
  27. 27. Identify Rims remove false rim segmentation assumption break at complicated structure i.e. the fold of human cloth remove false rim segmentation Among all vertics identified as rim points, filter out false-positives vertex v is detected as false positive if either 4Rl < g(v ) or Rl < g(v ) [vertical size is too large] and f ∗ (v ) < η [vertical size is not small enough] and [average NCC wrose than η] average NCC score ( Faugeras and Keriven 1998 ) Pτ Pτ f ∗ (p) = 2 τ (τ −1) i=1 j=i+1 NCC(hi , hj ) Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 27 / 49
  28. 28. Identify Rims Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 28 / 49
  29. 29. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 29 / 49
  30. 30. Global Optimization Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 30 / 49
  31. 31. Global Optimization ¯ use rim Γ to split surface Ω into Gi (i = 1, ..., k ) ¯ iteratively deform Gi inwards to generate multiple layers of 3D graph Ji and associate photoconsistency weights to the edge of this graph use graph cuts to carve surface Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 31 / 49
  32. 32. Global Optimization Move Vertices down At each iteration, move every vertex v along its surface normal N(v ) and apply smoothing v ← v − λ (ζ1 f (v ) + ζ2 )N(v ) + s(v ) notice that surface shrinks faster where image discrepancy function is larger Constant they use in all their experiment ¯ ζ1 = 100, ζ2 = 0.1, β1 = 0.4, β2 = 0.3, λ = 20, ρ = 40, = 0.5 ∗ AverageLengthInGi Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 32 / 49
  33. 33. Global Optimization Building a graph and Apply Graph Cuts build vertical edge build horizontal edge assign vertical and horizontal edge weight weight of edge (vi , vj ) α(f (vi )+f (vj )(δi +δj )) wij = d(vi ,vj ) f (vi ) : photoconsistency function; d(vi , vj ) : length of edge; δi : sparity of vertices around vi , α = 1.0 for horizontal, 6.0 for vertical ∞ edge weight for edge connected to source and sink apply graph cuts Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 33 / 49
  34. 34. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 34 / 49
  35. 35. Local Refinement Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 35 / 49
  36. 36. Local Refinement local minimum increase resolution of mesh ( Hoppe et al 1993 ) until image projections of edges become approximiately 2 px in length rim consistency ( Hernandex Esteban and Schmitt 2004 ) exp(−vk rj −vk¯ rj )2 /2σ2 ¯ ∗ 2 r (vk ) = vj¯rj P ∗ ¯ exp(−(vk¯ rj −v ∗ rj )2 /2σ22 ) v ∈Vj j k Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 36 / 49
  37. 37. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 37 / 49
  38. 38. Result 7 data sets Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 38 / 49
  39. 39. Result Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 39 / 49
  40. 40. Result Rim Identification Result Filtering ratio : how many % of identified rim points has been filtered out as outliers ( for each contour ) Sizes of components : show 3 largest connected components inside identified rim-segments From table, visual hull boundary is mostly covered by a single large connected component except for Twin data set, which has many input images, and hence, many rim curves. Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 40 / 49
  41. 41. Result Running Time (with 3.4 GHz Pentium 4) bottleneck of computation is global optimization and local refinement step ( takes about 2 hr ) Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 41 / 49
  42. 42. Result Testing algorithm w/o graph cuts phase ( use only local method ) local minimum problem Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 42 / 49
  43. 43. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 43 / 49
  44. 44. Comparisons ( This paper ) rim constriants ( Voiazis et al 2005 ) add inflationary ballooning term to enery function in graph cuts to prevent over-carving Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 44 / 49
  45. 45. Comparisons ( Hernandex Exteban and Schmitt 2004 ) In local refinement, use gradient flow instead of direct derivatives f (v ) Some other differences, i.e. local iterative deformation, that have problem avoiding local minima. outperforms in reconstructing mistake in rim-identification steps concave structure Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 45 / 49
  46. 46. Comparisons Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 46 / 49
  47. 47. Comparisons Multi-view stereo evaluation ( http://vision.middlebury. edu/mview/ ) Temple data set Accurracy : distance d that bring 90% of result within ground-truth surface Completeness : % ground-truth surface that lies within 1.25 mm of the result Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 47 / 49
  48. 48. Outline 1 Reference 2 Introduction 3 Steps 4 Identify Rims 5 Global Optimization 6 Local Refinement 7 Result 8 Comparison 9 Limitation & Future work Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 48 / 49
  49. 49. Limitation & Future work Since, cannot handle concavities too deep to be carved by the graph cuts. i.e. eye sockets of skulls To overcomes this, combine their works with sparse wide-baseline stereo from interest point (e.g. Schaffalitzky and Zisserman 2001) in order to incorporate stronger geometric constraints in the carving and local refinement stages handle non-Lambertain surfaces ( Soatto et al 2003 ) simutaneous camera calibration where both camera parameters and surface shape are refined simultaneously using bundle adjustment ( Uffenkamp 1993 ) Phongsathorn (AIT) Carved Visual Hulls for Image-Based Modeling Machine Vision Student Presentation 49 / 49

×