Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
•
4 likes•1,796 views
This document summarizes a talk on 3D reconstruction with priors. It discusses using priors to improve 3D reconstruction in cases where standard multi-view stereo fails, such as in the presence of noise, for compression purposes, or to avoid the uncanny valley effect. Methods for enforcing priors discussed include using Markov random fields with advanced smoothness terms or piecewise planar maps. Integrating top-down primitive detection approaches with bottom-up multi-view stereo can also help enforce piecewise planar or curved surface priors for scenes.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
Similar to Lecture 02 yasutaka furukawa - 3 d reconstruction with priors (20)
More from mustafa sarac
More from mustafa sarac (20)
Recently uploaded
Recently uploaded (20)
Lecture 02 yasutaka furukawa - 3 d reconstruction with priors
- 1. ICVSS 2014 3D Reconstruction with Priors Yasutaka Furukawa Washington University in St. Louis
- 3. Outline • Why priors? • Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 4. Brief Introduction of Multi-View Stereo (MVS) Camera pose 3d reconstruction Input: Calibrated photographs Output: 3D geometry
- 6. 1995 1999 1998 1998 2004 2005 2005 2007 Computer Vision: Algorithms and Applications [Richard Szeliski]
- 7. A decade ago... A Theory of Space by Space Carving [ Kutulakos and Seitz, 1999 ] Reconstructions w/ and w/o color
- 8. Volumetric graph cuts by Vogiatzis, Torr, and Cipolla CVPR 2005 1. This is called Haniwa. 2. This is Roberto Cipolla’s Haniwa. This will be a final quiz!
- 9. Results [ CVPR 2007 ]
- 10. Visual Effects
- 12. ComputerVision in Industry • Amazon • Apple • Facebook • Google • Microsoft • Nokia
- 13. ComputerVision in Industry • Amazon • Apple • Facebook • Google • Microsoft • Nokia Mountain View Seattle Zurich NYC LA …
- 14. ComputerVision in Industry • Amazon • Apple • Facebook • Google • Microsoft • Nokia Mountain View Seattle Zurich NYC LA …
- 15. Google Seattle - 3DVision Team • Bundling streetview data (by Sameer Agarwal) • Photo Tours (by the team) • Picasa face movie (by Ira Kemelmacher- Shlizerman and Rahul Garg) • Lens Blur (by Carlos Hernandez) Steve Seitz, Sameer Agarwal, Carlos Hernandez, David Gallup, Changchang Wu, Li Zhang
- 16. Face Movie [Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg, Steve Seitz]
- 19. Why am I here?
- 20. What if no texture?
- 23. Why priors? • Fails in some important cases • Noise suppression • Compression
- 24. Why priors? • Fails in some important cases • Noise suppression • Compression • Uncanny valley for 3D reconstruction
- 26. Uncanny valley for 3D reconstruction
- 27. More accurate, but looks worse Less accurate, but looks better
- 28. People like everything about this picture
- 29. 3D photography hates everything about this picture
- 38. Uncanny Valley for 3D Reconstruction Ten Things You Should Know About Large Scale 3D Reconstruction (3DV 2013) Martin Byröd Apple Maps
- 39. Why priors? • Fails in some important cases • Noise suppression • Compression • Uncanny valley for 3D reconstruction
- 40. Outline • Why priors? • Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 41. Prior enforcement through MRF
- 43. Motivations [Furukawa and Ponce, 2007]
- 44. Enforce Prior in MVS • Manhattan-world assumption • Planarity • Orthogonality
- 46. 8.0 8.2 8.5 8.9 9.3 camera center image screen Depthmap Problem Discretized depth values { 0.1, 0.2, …, 8.0, 8.1, … 9.8, 9.9}
- 47. Standard Depthmap Markov Random Field (MRF) A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]
- 48. A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008] Markov Random Field (MRF)
- 49. Markov Random Field (MRF)
- 50. Markov Random Field (MRF) 7 7 6 0 6 7
- 51. p q Markov Random Field (MRF)
- 52. p q Markov Random Field (MRF)
- 53. 0 p q Markov Random Field (MRF)
- 54. 1 p q Markov Random Field (MRF)
- 55. 2 p q Markov Random Field (MRF)
- 56. 4 p q Markov Random Field (MRF)
- 57. 0 p q Markov Random Field (MRF)
- 58. Markov Random Field (MRF) Graph-cuts (alpha-expansion) gives you very good solutions
- 59. Prefers front-parallel surfaces Markov Random Field (MRF)
- 60. We want piecewise planar
- 62. Piecewise planarity from MRF 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 63. Advanced MRF for Depthmap Estimation Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award Reference image Ground truth
- 64. Standard MRF
- 65. Standard MRF p q
- 66. Standard MRF p q
- 67. Standard MRF p q
- 68. Standard MRF p q MRF with a triple clique p q r
- 69. Standard MRF p q MRF with a triple clique p q r
- 70. Standard MRF p q MRF with a triple clique p q r 0 = |3 + 5 - 2 x 4|
- 71. Standard MRF p q MRF with a triple clique p q r 0 = |3 + 5 - 2 x 4|
- 72. Standard MRF p q MRF with a triple clique p q r
- 73. Standard MRF p q MRF with a triple clique p q r
- 74. MRF with a triple clique
- 75. Optimization becomes a challenge Standard MRF (submodular) MRF with a triple clique (non-submodular) • Graph-cuts (alpha-expansion) • Belief propagation (TRW) • QPBO • Belief propagation (TRW?)
- 76. Optimization becomes a challenge Standard MRF (submodular) MRF with a triple clique (non-submodular) • Graph-cuts (alpha-expansion) • Belief propagation (TRW) • QPBO • Belief propagation (TRW?) Fusion moves…
- 77. Experimental results Reference image Neighboring image Output depthmap [Woodford et al.]
- 78. Comparative experiment Reference image Ground truth Standard MRF MRF with a triple clique [Woodford et al.]
- 79. How to enforce piecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 81. Planemap [Furukawa and Ponce, 2007] 1. Extract Manhattan directions 2. Extract planes
- 82. Planemap Possible planes a b c d e f g h 1. Extract Manhattan directions 2. Extract planes
- 83. Planemap Possible plane ids a b c d e f g h e e e e e 1. Extract Manhattan directions 2. Extract planes
- 84. Depthmap to Planemap Depthmap ( ) Planemap ( )
- 85. Planemap MRF
- 86. Planemap MRF p
- 87. Planemap MRF p 7 6 0 6 7
- 88. Planemap MRF p 10 3 10
- 93. Comparison Standard method Manha9an Planemap
- 94. Kitchen -‐ 22 images house -‐ 148 images gallery -‐ 492 images Reconstruction Results
- 96. How to enforce piecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 97. 20 seconds break
- 98. How to enforce piecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 99. Relaxing Mahnattan Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Use sparse lines + sparse points to detect planes • MRF + Graph-cuts
- 100. Relaxing Manhattan [Sinha et al.]
- 101. Relaxing Manhattan [Sinha et al.]
- 102. Enable Curved Surfaces • Building reconstruction from a top down view Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
- 103. Enable Curved Surfaces[Zebedin et al.]
- 104. How to enforce piecewise planar 1. Advanced MRF and optimization • Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award 2. Integrate with top-down (primitive) approach • Manhattan World Stereo [Furukawa et al., CVPR 2009] • Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009] • Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008] • Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
- 105. Piecewise Planar and Non-‐Planar Stereo for Urban Scene Reconstruction David Gallup Jan-‐Michael Frahm Marc Pollefeys University of North Carolina ETH Zurich
- 106. 3D Reconstruction from Video Street-‐Side Video Real-‐Time Stereo 106
- 107. Idea 3D ReconstructionVideo Frames 107 Enforce planarity where it looks like a building
- 108. First step: Planar and Non-‐Planar Stereo 108 Video Real-‐Time Stereo Plane Detection Planemap (w/ non-‐planar)) Labels = planes non-‐plane discard
- 109. Second step: Appearance Prior 109 Video Frames
- 110. Classification • divide image into regular grid • RGB, HSV, hue histogram, edge orientation histogram • 5000 human-‐labeled examples (5 images) • k-‐nearest-‐neighbor classifier non-‐planar planar 110 Hoiem et al. 2005, Xiao et al. 2009
- 111. Algorithm Video Real-‐Time Stereo Planar/Non-‐Planar Classification Plane Detection Planar and Non-‐Planar MRF 111 non-‐planar planar
- 112. Effect of Classifier 112 Without image prior With image prior non-‐planar planar
- 113. Results 113 Video Frame Plane Detection Planar Classification Reconstruction non-‐planar planar
- 114. Results 114
- 115. Summary: Shape priors in Depthmap MVS MRF with a triple term Manhattan planemap Planemap Planemap w/ curved surfaces Mix of planemap and depthmap
- 116. Outline • Why priors? • Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 117. Priors through primitives for large-scale indoor modeling “Reconstructing the World’s Museums” [Xiao and Furukawa, 2012] (Best Student Paper Award)
- 119. Technical Contributions • Architectural shape priors through primitives • Single consistent 3D model
- 120. Technical Contributions • Architectural shape priors through primitives • Single consistent 3D model Inverse Constructive Solid Geometry (CSG)
- 121. Technical Contributions Inverse Constructive Solid Geometry (CSG) • Architectural shape priors through primitives! • Single consistent 3D model (real merging)
- 122. + Technical Contributions Inverse Constructive Solid Geometry (CSG) • Architectural shape priors through primitives • Single consistent 3D model (real merging) + + - + + -
- 123. CGAL • From CSG to a mesh
- 124. Algorithm
- 125. Cut into Slices gravity side view
- 126. Cut into Slices point count
- 127. Cut into Slices
- 129. Bottom-up approach point 2D line 2D rectangle 2D CSG
- 130. 2D CSG Reconstruction point à line
- 131. 2D CSG Reconstruction point à line line à rectangle From 4 line segments
- 132. 2D CSG Reconstruction point à line line à rectangle rectangle à 2D CSG
- 133. 2D CSG Reconstruction Explain the data • Laser points • Free space !
- 134. 2D CSG Reconstruction Repeat in each slice
- 136. Bottom-up approach point 2D line 2D rectangle 2D CSG
- 137. Bottom-up approach point 2D line 2D rectangle 2D CSG
- 138. Bottom-up approach point 2D line 2D rectangle
- 139. point 2D line 2D rectangle 3D rectangle 3D CSG
- 140. 2D CSG to 3D CSG
- 141. 2D CSG Rectangle primitive 2D CSG to 3D CSG
- 142. 2D CSG Rectangle primitive 2D CSG to 3D CSG
- 143. 1. Generate primitives (cuboids) 2D CSG Rectangle primitive 2D CSG to 3D CSG
- 144. 3D CSG Reconstruction 1. Generate primitives (cuboids) 2. Build a 3D CSG
- 145. Algorithm on Run
- 146. Last Step 1. Remove Ceiling 2. Texture Mapping
- 148. Challenging Navigation Frick Collection Gallery (NewYork City) Goal Start
- 151. Statistics # of sub problems
- 152. Ground Ground+Aerial Outdoor Indoor Ground vs. Aerial à Ground+Aerial Google Streetview Google MapsGL Furukawa et al. Aerial Google/Bing/NASA … This paper This paper
- 153. So, museum recons. from very high-end devices
- 154. “museums”
- 155. “restaurants”
- 156. “grocery stores”
- 157. {# of museums} << {# of restaurants} + {# of grocery stores} + {# of clinic} + ...
- 158. Millions of small/medium- scale businesses • Image-based for scalable deployment (especially for emerging markets) • Compact model (for browser on low-end PCs)
- 159. Millions of small/medium- scale businesses • Image-based for scalable deployment (especially for emerging markets) • Compact mesh model (for browser on low-end PCs)
- 160. [ Cabral and Furukawa, 2014 ]
- 161. Outline • Why priors? • Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 164. Why priors? • Only reconstruct planar outline • Texture mapping looks better • Challenge is how to IGNORE clutter
- 165. Pipeline (standard)
- 166. 1. Panorama images 2. MVS points 3. Point evidence 4. Free-space evidence 5. 2D room outline 6. 3D model
- 167. 2D room outline reconstruction 3. Point evidence 4. Free-space evidence Outline should pass through this point Threshold
- 168. 2D room outline reconstruction 4. Free-space evidence3. Point evidence 5. Shortest path formulation
- 169. Standard way
- 170. Standard way
- 171. Standard way
- 172. 2D room outline reconstruction 5. Shortest path formulation
- 173. 2D room outline reconstruction 5. Shortest path formulation
- 174. 2D room outline reconstruction 5. Shortest path formulation Model complexity penalty added to each edge
- 175. 2D room outline reconstruction 5. Shortest path formulation
- 176. 2D room outline reconstruction Standard Ours
- 177. Features • Regularize on {# of line segments} • Piecewise planarity enforcement • With Dynamic Programming, exactly control the number of vertices
- 184. Exact complexity control 13 verts. 15 verts. 17 verts. 19 verts. 23 verts. . . . . . .
- 189. Summary • Why priors? • Prior enforcement through MRF • Prior enforcement through primitives • Prior enforcement through shortest-path
- 190. Questions
- 191. A. Who owns the Haniwa in the volumetric graph-cuts paper? 1. Yasutaka Furukawa. 2. Roberto Cipolla. 3. British Museum. B. What is Uncanny valley for 3D reconstruction? 1. As the number of authors in a paper goes up, the model quality goes down. 2. As the face realism goes up, the model looks creepier. 3. As the model accuracy goes up, the rendering quality goes down. C. How can one enforce piecewise planarity with a standard MRF formulation easily? 1. Use super-pixel segmentation. 2. Use primitive detection. 3. Use object recognition.
- 192. Lunch Time!