Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this presentation? Why not share!

3,465 views

Published on

No Downloads

Total views

3,465

On SlideShare

0

From Embeds

0

Number of Embeds

2,729

Shares

0

Downloads

0

Comments

0

Likes

1

No embeds

No notes for slide

- 1. Smooth Signed Distance Surface Reconstruction Gabriel Taubin Brown University IMPA/CMA2011 October 20 2011
- 2. Typical Surface Reconstruction PipelineOriented Reconstruction Surface Points Method RepresentationPositions & Normals Implicit Surface Polygon Mesh
- 3. SSD: Smooth Signed Distance Surface Reconstruction F. Calakli, G. Taubin, Computer Graphics Forum, 2011.• A new mathematical formulation• And a particular algorithm• To reconstruct a watertight surface From a static oriented point cloud
- 4. Particularly Good at Extrapolating Missing Data
- 5. http://mesh.brown.edu/ssd
- 6. Motivation• Industry – Reverse engineering – Fast metrology – Physical simulations
- 7. Motivation• Industry – Reverse engineering – Fast metrology – Physical simulations• Entertainment – Animating digital clays for movies or games
- 8. Motivation• Industry – Reverse engineering – Fast metrology – Physical simulations• Entertainment – Animating digital clays for movies or games• Archeology and Art – Digitization of cultural heritage and artistic works
- 9. Motivation• Industry – Reverse engineering – Fast metrology – Physical simulations• Entertainment – Animating digital clays for movies or games• Archeology and Art – Digitization of cultural heritage and artistic works• Medical Imaging – Visualization – Segmentation• …
- 10. Data AcquisitionLaser range scanning devices Multi-camera systems Structured lighting systems
- 11. http://mesh.brown.edu/byo3d/
- 12. http://mesh.brown.edu/byo3d
- 13. Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive WindowsY. Zhao, and G. Taubin, Image and Vision Computing 2011. Screen shots of our real-time stereo system working on the field
- 14. Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive Windows Y. Zhao, and G. Taubin, Image and Vision Computing 2011. Stereo Frame Stereo Multi-Resolution Grabbing Rectification Pyramid Generating Resolution Scan from Low to Background Modeling With Shadow Removal Foreground Detection with Dilation & Erosion High Stereo Matching Using Adaptive Window with Cross-Checking Disparity Refinement¼ Resolution ½ Resolution Full Resolution Coarse-to-fine matching on multiple resolutions Processing Pipeline
- 15. 3D Reconstruction & Analysis of Bat Flight Maneuvers• 3D Reconstruction of Bat Flight Kinematics from Sparse Multiple Views, by A. Bergou, S. Swartz, and G. Taubin, and K. Breuer, 4DMOD, 2011.• 3D Reconstruction and Analysis of Bat Flight Maneuvers from Sparse Multiple View Video, by A. Bergou, S. Swartz, S. K. Breuer, G. Taubin, BioVis, 2011.• Falling with Style - The Role of Wing Inertia in Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Annual Meeting, Society for Integrative and Comparative Biology, 2011.• Falling with Style-Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Bulletin of the American Physical Society, Vol. 55, 2010.
- 16. How do we measure bats ?• Multiple synchronized 1000fps+ cameras• Controlled environment (backdrop & illumination)• Bats trained to land on landing pad• Experiments with several species
- 17. • Bats have highly articulated wings• Very complex wing motion• Current goal: Detailed reconstruction of wing and body kinematics and derivatives from visual data• Skeleton model with 52 degrees of freedom• Geometry parameterized by 37 constants• Future Goal: Model-less Dynamic Shape Reconstruction
- 18. Some Methods to Capture 3D Point CloudsMulti-Flash Camera Shadow Multi-Flash Backdrop Attachment Turntable8 Megapixel Camera
- 19. Beyond Silhouettes: Surface Reconstruction using Multi-Flash PhotographyD. Crispell, D. Lanman, P. Sibley, Y. Zhao and G. Taubin [3DPVT 2006]
- 20. Multi-Flash RecoveredMulti-Flash 3D Photography: Turntable Sequence: Estimated Shape: 3D Point Cloud Appearance: Input Image Phong BRDF ModelCapturing the Shape and Appearanceof 3D ObjectsA new approach for reconstructing 3D objectsusing shadows cast by depth discontinuities, asdetected by a multi-flash camera. Unlike existingstereo vision algorithms, this method works evenwith plain surfaces, including unpainted ceramicsand architecture.Data Capture: A turntable and a digital cameraare used to acquire data from 670 viewpoints. Foreach viewpoint, we capture a set of images usingillumination from four different flashes. Futureembodiments will include a small, inexpensivehandheld multi-flash camera. Multi-Flash Camera Shadow Turntable Rotation Recovering a Smooth Surface Backdrop Multi-Flash Attachment The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the Turntable 3D points. This method can be applied to any 3D point cloud, including those 8 Megapixel Camera generated by laser scanners. 21
- 21. Raskar et al. [Siggraph 2004]• Depth discontinuity estimation for Non-Photorealistic Rendering – Camera static with respect to object – 4 Images are captured with object – Each illuminated by a different flash – Flashes located close to camera lens – Image processing extracts and combines shadows
- 22. What do we gain?Using only silhouettes Using all depth discontinuities
- 23. Silhouettes vs Depth Discontinuities [2D]
- 24. Silhouettes vs Depth Discontinuities [2D] Silhouette
- 25. Silhouettes vs Depth Discontinuities [2D] Depth Discontinuity Silhouette
- 26. Differential Shape From Silhouette [Cipolla & Blake 92]
- 27. Differential Shape From Silhouette [Cipolla & Blake 92]Known fromcamera motion
- 28. Differential Shape From Silhouette [Cipolla & Blake 92]Known fromcamera motion Measured in epipolar slice
- 29. Differential Shape From Silhouette [Cipolla & Blake 92] Measured in depthKnown from discontinuity imagecamera motion Measured in epipolar slice
- 30. Differential Shape From Silhouette [Cipolla & Blake 92] Measured in depthKnown from discontinuity imagecamera motion Measured in epipolar slice
- 31. Computing r’(t) : Depth Discontinuity Edge Tracking12 1 234 3 4
- 32. Vectorfield Isosurface-Based Reconstruction From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]• Surface reconstruction from cloud of oriented points• Implicit representation can deal with missing data• Rather than fitting analytic function (RBFs, etc), and then extract isosurface for visualization, fit isosurface directly to data
- 33. Vectorfield Isosurface-Based Reconstruction From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]• Surface reconstruction from cloud of oriented points• Implicit representation can deal with missing data• Rather than fitting analytic function (RBFs, etc), and then extract isosurface for visualization, fit isosurface directly to data• Initial approach: 3 steps
- 34. Vectorfield Isosurface-Based Reconstruction From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]• Surface reconstruction from cloud of oriented points• Implicit representation can deal with missing data• Rather than fitting analytic function (RBFs, etc), and then extract isosurface for visualization, fit isosurface directly to data• Initial approach: 3 steps 1. Fit continuous vector field to data normal vectors
- 35. Vectorfield Isosurface-Based Reconstruction From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]• Surface reconstruction from cloud of oriented points• Implicit representation can deal with missing data• Rather than fitting analytic function (RBFs, etc), and then extract isosurface for visualization, fit isosurface directly to data• Initial approach: 3 steps 1. Fit continuous vector field to data normal vectors 2. Integrate vector field on regular grid
- 36. Vectorfield Isosurface-Based Reconstruction From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]• Surface reconstruction from cloud of oriented points• Implicit representation can deal with missing data• Rather than fitting analytic function (RBFs, etc), and then extract isosurface for visualization, fit isosurface directly to data• Initial approach: 3 steps 1. Fit continuous vector field to data normal vectors 2. Integrate vector field on regular grid 3. Determine Isolevel
- 37. VFIso: Representation• Regular voxel grid
- 38. VFIso: Representation• Regular voxel grid• Implicit function defined by values at vertices f
- 39. VFIso: Representation• Regular voxel grid• Implicit function defined by values at vertices• And tri-linear interpolation f ( p) f ( p) f (p)
- 40. VFIso: Representation• Regular voxel grid• Implicit function defined by values at vertices• And tri-linear interpolation f ( p) f ( p) • The output is: { f } f (p)
- 41. VFIso: Variational Approach• Separate gradient field from scalar field 1. Extend data to vector field on the volume 2. Integrate vector field• Minimize: E (v ) v ( p) n v v 2 2 D , ) ( data regulariza tion where : v ( p) v ( p)
- 42. VFIso: Details• Problem: Vector field v(p) not integrable!
- 43. VFIso: Details• Problem: Vector field v(p) not integrable!• Fit scalar field { f } directly by minimizing E ({ f }) f ( p) n f f 2 2 D ( , )
- 44. VFIso: Details• Problem: Vector field v(p) not integrable!• Fit scalar field { f } directly by minimizing E ({ f }) f ( p) n f f 2 2 D ( , )• Where f ( p) f ( p)
- 45. VFIso: Details• Problem: Vector field v(p) not integrable!• Fit scalar field { f } directly by minimizing E ({ f }) f ( p) n f f 2 2 D ( , )• Where f ( p) f ( p)• And f is discretized using central differences
- 46. VFIso: Details• Problem: Vector field v(p) not integrable!• Fit scalar field { f } directly by minimizing E ({ f }) f ( p) n f f 2 2 D ( , )• Where f ( p) f ( p)• And f is discretized using central differences• This is a standard (Laplacian) Least Squares problem
- 47. VFIso: Details• Problem: Vector field v(p) not integrable!• Fit scalar field { f } directly by minimizing E ({ f }) f ( p) n f f 2 2 D ( , )• Where f ( p) f ( p)• And f is discretized using central differences• This is a standard (Laplacian) Least Squares problem• Minimize to find isolevel E ( f 0 ) 2 f ( p) f 0 D
- 48. VFIso: Results [2005]Squirrel(9K) Angel(24K) Bunny(35K) Ram(678K)
- 49. VFIso: Uniform Downsampling [2005]#Pt 133K 50K 20K 5K
- 50. VFIso: Curvature Sampling [2005]#Pt 35K 21K 9K 4K
- 51. VFIso Results [2006 110x110x110 grid]
- 52. Multi-Flash 3D Photography: Photometric ReconstructionUsing the implicit surface, we can determine which points are visiblefrom each viewpoint. To model the material properties of thesurface, we fit a per-point Phong BRDF model to the set of visiblereflectance observations (using a total of 67 viewpoints). Ambient Diffuse Specular … Multi-Flash Turntable Sequence Phong (Specular) Images 56 3D Point Implicit Surface Phong (Diffuse) Estimated Phong Appearance Model
- 53. Gray Code Structured Lighting Point Grey Flea2 (15 Hz @ 1024 x 768) Mitsubishi XD300U (50-85 Hz @ 1024 x 768)3D Reconstruction using Structured Light [Inokuchi 1984]• Recover 3D depth for each pixel using ray-plane intersection• Determine correspondence between camera pixels and projector planes by projecting a temporally-multiplexed binary image sequence• Each image is a bit-plane of the Gray code for each projector row/column
- 54. Gray Code Structured Lighting Point Grey Flea2 (15 Hz @ 1024 x 768) Mitsubishi XD300U (50-85 Hz @ 1024 x 768)3D Reconstruction using Structured Light [Inokuchi 1984]• Recover 3D depth for each pixel using ray-plane intersection• Determine correspondence between camera pixels and projector planes by projecting a temporally-multiplexed binary image sequence• Each image is a bit-plane of the Gray code for each projector row/column• Encoding algorithm: integer row/column index binary code Gray code
- 55. Gray Code Structured Lighting Results
- 56. Surround Structured Lighting:3-D Scanning with Orthographic IlluminationD. Lanman, D. Crispell, G. Taubin [CVIU 2009]
- 57. 3D Slit Scanning with Planar Constraints M. Leotta, A. Vandergon, and G. Taubin [CGF 2008] Laser pointer Camera 2 +cylindrical lens Camera 1
- 58. Catadioptric Stereo Implementation
- 59. Can Estimate Points Visible From One Camera
- 60. A Variable Resolution Probabilistic 3-D Model for Change Detection D. Crispell, J. Mundy, and G. Taubin IEEE Transactions on Geoscience and Remote Sensing 2011.
- 61. Surface Reconstruction from Multi-View Data
- 62. Digital Archaeology [NSF IIS]• The main goal is to automate the tedious processes of data collection and documentation at the excavation site, as well as to provide visualization tools to explore the data collected in the database• Also to solve specific problems in Archaeology using computer vision techniques.• We use a network of cameras to capture the activity at the excavation site, to reconstruct the shape of the environment as it is being excavated, to reconstruct layers, and to locate finds in 3D
- 63. REVEAL Archaeological Data AcquisitionAssisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-formatanalysis Data Acquisition Advanced Algorithms Improve Automatically Convert Photos to 3D Models speed and accuracy with computer assisted data Import photos, entry videos, and Semi-automatically laser scans and Assemble connect them to Fragments database Into Artifacts objects Import External Data Laser Scanned Site Plans Models REVEAL Database Objects: Data: Rule-based Artifacts Text Reconstructions Excavations Photos Areas Video Sites 3D Models
- 64. REVEAL Archaeological Analysis Data integrated and synchronized in tabular, plan drawing, 3D spatial, image, and video formatsTypical Activity Sequence Examine Relationship of Artifacts in- situ in auto-generated 3D Excavation Display Photos of Model Selected Artifacts Artifacts Excavations ExportSelect Artifacts on Areas FormattedSite Plan Sites Artifact Data for inclusion in Site Publication
- 65. [Snavely et. al. 2006]http://phototour.cs.washington.edu/bundler/ MVS softwarePatch-based Multi-View Stereo (PMVS)http://grail.cs.washington.edu/software/pmvs/ [Furukawa and Ponce 2008]
- 66. Accurate 3D Footwear Impression Recovery From Photographs, F. A. Andalo, F. Calakli, G. Taubin, and S. Goldenstein, International Conference on Imaging for Crime Detection and Prevention (ICDP-2011). Comparable to 3D Laser Scanner
- 67. Challenges Uniform samplingNon-uniform sampling Noisy data Misaligned scans
- 68. General Approaches• Interpolating polygon meshes Boissonnat [1984], Edelsbrunner [1984] Amenta et al. [1998], Bernardini et al. [1999] Dey et al. [2003][2007], …• Implicit function fitting Taubin [1991], Hoppe et al. [1992], Curless et al. [1996] Whitaker [1998], Carr et al.[2001], Davis et al. [2002], Ohtake et al. [2004], Turk et al. [2004], Shen et al. [2004] Sibley-Taubin [2005]
- 69. Poisson Surface ReconstructionKazhdan et al. [2006] Manson et al. [2008]
- 70. Implicit function framework Oriented Points, D(samples from unknown surface S) Find a scalar valued function f : D , whose zero level set Z(f) = S’ is the estimate for true surface S
- 71. Implicit function framework <0 0 f >0 Oriented Points, D(samples from unknown surface S) Find a scalar valued function f : D , whose zero level set Z(f) = S’ is the estimate for true surface S
- 72. Implicit function framework Z( f ) Oriented Points, D Computed Implicit Surface, S’(samples from unknown surface S) Find a scalar valued function f : D , whose zero level set Z(f) = S’ is the estimate for true surface S
- 73. What kind of implicit function?0 1 0 Indicator Function Smooth Signed Distance Function
- 74. Continuous formulation• Oriented point set: D = { ( pi, ni ) } sampled from a surface S• Implicit surface: S = { x | f (x) = 0 } such that f (pi) = 0 and ∇f (pi) = ni ∀(pi,ni) ∈ D• Least squares energy: N N ED ( f ) f (p i ) 1 f (p i ) n i 2 2 i 1 i 1 ER ( f ) 2 Hf (x) dx 2 V N NE ( f ) f (p i ) 1 f (p i ) n i 2 || Hf (x) || 2 dx 2 2 V i 1 i 1
- 75. What does the regularization term do ?
- 76. What does the regularization term do ?• Near data points: since the data terms dominate, the function approximates the signed distance
- 77. What does the regularization term do ?• Near data points: since the data terms dominate, the function approximates the signed distance• Away from data points: the regularization term dominates and forces the gradient to be smooth and close to constant
- 78. Role of each energy term N NE ( f , v, M ) f (p i ) 1 v(p i ) n i 2 || M (x) || 2 dx 2 2 V i 1 i 1 Quadratic energy in f, v, and M
- 79. Role of each energy term N NE ( f , v, M ) f (p i ) 1 v(p i ) n i 2 || M (x) || 2 dx 2 2 V i 1 i 1 Quadratic energy in f, v, and M Relation to Poisson Reconstruction
- 80. Role of each energy term N NE ( f , v, M ) f (p i ) 1 v(p i ) n i 2 || M (x) || 2 dx 2 2 V i 1 i 1 Quadratic energy in f, v, and M Relation to Poisson Reconstruction NQ(v) E (0, v, Dv ) 1 v(p i ) n i 2 || Dv(x) || 2 dx, 2 V i 1 where Dv denotes the Jacobian of the vector field v. If ∇f = v, then Hf = Dv = M.
- 81. Role of each energy term N NE ( f , v, M ) f (p i ) 1 v(p i ) n i 2 || M (x) || 2 dx 2 2 V i 1 i 1 Quadratic energy in f, v, and M Relation to Poisson Reconstruction NQ(v) E (0, v, Dv ) 1 v(p i ) n i 2 || Dv(x) || 2 dx, 2 V i 1 where Dv denotes the Jacobian of the vector field v. If ∇f = v, then Hf = Dv = M. If f, v, and M are linear functions of the same parameters, then the minimization reduces to a least
- 82. Linear families of functions
- 83. Linear families of functions• Popular Smooth Basis Functions – Monomials [Taubin’91] – Radial basis functions [Carr et al., ‘01], – Compactly supported basis functions [Othake et al. ‘04], – Trigonometric polynomials [Kazhdan et al. ‘05], – B-splines [Kazhdan et al., 06], – Wavelets [Manson et al. ‘08],
- 84. Linear families of functions• Popular Smooth Basis Functions – Monomials [Taubin’91] – Radial basis functions [Carr et al., ‘01], – Compactly supported basis functions [Othake et al. ‘04], – Trigonometric polynomials [Kazhdan et al. ‘05], – B-splines [Kazhdan et al., 06], – Wavelets [Manson et al. ‘08], Non-homogenous, Quadratic energy E ( F ) F AF 2b F c t t
- 85. Linear families of functions• Popular Smooth Basis Functions – Monomials [Taubin’91] – Radial basis functions [Carr et al., ‘01], – Compactly supported basis functions [Othake et al. ‘04], – Trigonometric polynomials [Kazhdan et al. ‘05], – B-splines [Kazhdan et al., 06], – Wavelets [Manson et al. ‘08], Non-homogenous, Quadratic energy E ( F ) F AF 2b F c t t Global minimum AF b
- 86. We can use Independent Discretizations• Hybrid FE/FD discretization – Trilinear interplant for the function f(x) – Finite differences for the gradient ∇f(x) – Finite differences for the Hessian Hf(x)
- 87. We can use Independent Discretizations• Hybrid FE/FD discretization – Trilinear interplant for the function f(x) – Finite differences for the gradient ∇f(x) – Finite differences for the Hessian Hf(x)• As long as f(x), ∇f(x), and Hf(x) are written as a linear combinations of the same parameter vector F E ( F ) F AF 2b F c Non-homogenous, t t Quadratic energy Global minimum AF b
- 88. Implementation• Primal-Dual octree data structure• Iterative solver (conjugate gradient) with cascading multi-grid approach – Solve the problem on a much coarser level – Use the solution at that level to initialize the solution at the next level – Refine with the iterative solver
- 89. Implementation• Primal-Dual octree data structure• Iterative solver (conjugate gradient) with cascading multi-grid approach – Solve the problem on a much coarser level – Use the solution at that level to initialize the solution at the next level – Refine with the iterative solver• Iso-surface extraction (crack-free) – Dual marching cubes [Schaefer 2005]
- 90. Input point MPU Poisson D4 Wavelets SSDcloud [Othake ‘03 ] [Kazhdan ‘06 ] [Manson ‘08 ] [Calakli ’11]
- 91. Input point MPU Poisson D4 Wavelets SSDcloud [Othake ‘03 ] [Kazhdan ‘06 ] [Manson ‘08 ] [Calakli ’11]
- 92. Input point MPU Poisson D4 Wavelets SSDcloud [Othake ‘03 ] [Kazhdan ‘06 ] [Manson ‘08 ] [Calakli ’11]
- 93. Input point MPU Poisson D4 Wavelets SSDcloud [Othake ‘03 ] [Kazhdan ‘06 ] [Manson ‘08 ] [Calakli ’11]
- 94. Accuracy• Hausdorff distance between real surfaces and reconstructed surfaces using Metro [Cignoni ’96]
- 95. Performance characteristicsMPU Poisson D4 Wavelets Our SSD[Othake ‘03 ] [Kazhdan ‘06 ] [Manson ‘08 ]
- 96. Conclusion• Theoretical contributions: – Oriented point samples samples of signed Euclidean distance function – Reconstruction as global minimization problem – Yet sparse system• Empirical advantages: – Robustness to noise – Adaptive to sampling density• Future work: – Streaming out-of-core implementations – Parallel GPU implementations
- 97. Reconstruction of colored meshes
- 98. From Multi-View Video Cameras
- 99. View Interpolation From Multi-View Video Cameras
- 100. View Interpolation From Multi-View Video Cameras
- 101. View Interpolation From Multi-View Video Cameras
- 102. View Interpolation From Multi-View Video Cameras
- 103. View Interpolation From Multi-View Video Cameras
- 104. With Background Segmentation
- 105. Ongoing work
- 106. An Evaluation and Comparison of Surface Reconstruction M. Berger, J. Levine, L. Nonato, C. Silva, and G.Taubin ACM Transactions on Graphics (to appear)http://www.cs.utah.edu/~bergerm/recon_bench/
- 107. Questions? This material is based upon work supported bythe National Science Foundation under Grant No. IIS-0808718, CCF-0729126, and CCF-0915661.

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment