Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators using Eigenvectors

6,585 views

Published on

My lecture notes for the course Geo1004 3D modelling of the built environment at TU Delft, Faculty of Architecture and the Built Environment

Published in: Engineering
  • Be the first to comment

Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators using Eigenvectors

  1. 1. 1Challenge the future Basic Point Cloud Processing Estimating Normal Vectors and Curvature Indicators Ir. Pirouz Nourian PhD candidate & Instructor, chair of Design Informatics, since 2010 MSc in Architecture 2009 BSc in Control Engineering 2005 Geo1004, Geomatics Master Track Directed by Dr. ir. Sisi Zlatannova
  2. 2. 2Challenge the future How do we make a computer ‘see’ what we understand from this? What do you make of this? What is this all about?
  3. 3. 3Challenge the future Data to Information: LIDAR to LOD2 How do we make sense out of such big data? Image courtesy of GEOCONNEXION ?
  4. 4. 4Challenge the future SOFTWARE APPLICATIONS • FME, Safe Software • LASTools, rapidlasso • Point Cloud Library (PCL) • Cloud Compare • MeshLab • LP360 QCoherent A NEW VERSION OF TOIDAR! YES! http://opentopo.sdsc.eduInformation from:
  5. 5. 5Challenge the future FME, Safe Software transform, convert, translate, extract, integrate, automated, repeatable
  6. 6. 6Challenge the future LASTools, rapidlasso filtering, clipping, reprojecting, compression, classification, DSM, DTM, TIN, contours bare-earth de facto standards for point cloud data: .las .laz
  7. 7. 7Challenge the future LP360, QCoherenet LIDAR, Classification, Breaklines, Visualization, Extraction, Automatic
  8. 8. 8Challenge the future Point Cloud Library PCL point clouds, visualization, processing, segmentation, filtering, feature estimation, registration Using this library in Rhino?
  9. 9. 9Challenge the future Cloud Compare Implements PCL and more methods, handy to use for point cloud processing Image from software.informer.com
  10. 10. 10Challenge the future MeshLab Has some surface reconstruction methods, handy to have when working with point clouds. Image from https://danieleferdani.wordpress.com/
  11. 11. 11Challenge the future Brainstorm A sample building extracted from the AHN2 dataset. ? ?
  12. 12. 12Challenge the future Part 1 finished We will continue with one specific way of dealing with point clouds and estimating normal vectors and curvatures…
  13. 13. 13Challenge the future An idea! A sample building extracted from the AHN2 dataset.
  14. 14. 14Challenge the future A Curvature-Based Approach to Point Cloud Segmentation & Feature Detection Removing Outliers Elevation Classification Slope Classification Aspect Classification Region-Growing Segmentation Point Cloud of a Building Noise reduction Forming Neighbourhoods: e.g. KNN or Range Forming Covariance Matrices for Neighbourhoods Using PCA, Estimating: Normal Vectors & Curvature Indicators Normal Vectors Curvature Indicators Eigen System Triangulation Mesh Surface
  15. 15. 15Challenge the future Different Approaches for Finding Fitting Planes/Edges 1. Using curvature values computed (estimated) by eigenvalues of covariance matrices, to run a segmentation algorithm based on region growing. 2. Using Octree Voxels, to detect edge voxels, remove them and create segments. 3. Using Hough transform on a 2.5D point cloud, converting to the parameter space and using DBScan clustering method to find clusters of parameters, each of which correspond to a segment in a point cloud. 4. And a few more in the literature…
  16. 16. 16Challenge the future Different Approaches for Finding Fitting Planes/Edges Region growing based on normals and curvature Segmentation usingOctree voxels Hough transformation
  17. 17. 17Challenge the future Different Approaches for Finding Fitting Planes/Edges: our experience! Curvature based region growing Voxel based region growing Hough transform Noise management Fair Good Good Density variation management Fair Fair Good Efficiency Good Good Poor Ambiguity of vantage point management Fair Good Good Avoiding priori knowledge Good Good Poor Ease of further processing Good Poor Poor
  18. 18. 18Challenge the future Neighbourhoods of Points Fixed Distance Neighbors (FDN) and K-Nearest Neighbors (KNN) • Rabbani, T., van den Heuvel, F., & Vosselmann, G. (2006). Segmentation of point clouds using smoothness constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 36(5), 248-253. • Pauling, Frederick, Michael Bosse, and Robert Zlot. "Automatic segmentation of 3d laser point clouds by ellipsoidal region growing." Australasian Conference on Robotics and Automation. 2009. The idea is to form neighborhoods based on [squared] distance These two options are usually provided. Apart from normal vector estimation KNN can be useful also for removal of outliers.
  19. 19. 19Challenge the future Fitting Planes to Neighbourhoods The idea is that the underlying surface is a 2-manifold; therefore it resembles a 2D plane locally Least Square Plane Fitting Estimation Problem  Principal Component Analysis Problem Intuition: Defining plane as the locus of lines that have direction vectors perpendicular to a normal vector; considering an origin for the plane in questions, we can consider the plane as the locus of points A such that A- O is perpendicular to N.
  20. 20. 20Challenge the future Fitting Planes to Neighbourhoods The idea is that the underlying surface is a 2-manifold; therefore it resembles a 2D plane locally, we look at ellipsoids showing local dispersions Images courtesy of Olga Sorkine
  21. 21. 21Challenge the future How to estimate normals using PCA The idea is that the underlying surface is a 2-manifold; therefore it resembles a 2D plane locally • Pauly, Mark, Markus Gross, and Leif P. Kobbelt. "Efficient simplification of point-sampled surfaces." Proceedings of the conference on Visualization'02. IEEE Computer Society, 2002. • Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W. Surface reconstruction from unorganized points. SIGGRAPH 92, 1992 • Shaffer, E., Garland, M. Efficient Adaptive Simplification of Massive Meshes. IEEE Visualization 01, 2001
  22. 22. 22Challenge the future How to estimate normals using PCA We form a covariance matrix for each neighborhood, showing how neighbors are dispersed around their average (centroid). This will be a 3 by 3 symmetric matrix!
  23. 23. 23Challenge the future How to estimate normals using PCA We form a covariance matrix for each neighborhood, showing how neighbors are dispersed around their average (centroid). This will be a 3 by 3 symmetric matrix! Form an Eigen system for this matrix using a linear algebra library, and the first eigenvector corresponding to least eigenvalue will be the normal vector at each neighbourhood. Explanation follows… PCL implementation: http://pointclouds.org/documentation/tutorials/normal_estimation.php
  24. 24. 24Challenge the future How to estimate curvature using PCA The idea is to use an indication of change along the normal vector Jolliffe, I. Principle Component Analysis. Springer-Verlag, 1986
  25. 25. 25Challenge the future How to do all this in code? • We try not to reinvent the wheel; the idea is to use free open source libraries like Math.NET and Accord.NET • We are not the first people dealing with such issues, these are generally matters of data mining and machine learning • We can find KNN neighbourhoods using Accord.NET http://accord-framework.net/ • We can find eigenvalues and eigenvectors using MetaNumerics.dll, MathNet.dll or Accord.NET • You will receive example code implementing MetaNumerics
  26. 26. 26Challenge the future Fitting Planes to Neighbourhoods How do we minimize the error in fitting a plane to the mentioned neighbourhood? First we define it… Images courtesy of Olga Sorkine
  27. 27. 27Challenge the future An explanation after Olga Sorkine: • Input points: • Centroid: • Vectors from the centroid: m
  28. 28. 28Challenge the future Continued… mm minimizes SSD and it can be shown that m is the centroid C We can rewrite the problem as:
  29. 29. 29Challenge the future Continued… m m minimizes SSD, formally it is the “arg min” of the following function (to be minimized), i.e. the argument that makes it reach its minimum. and it can be shown that m is the centroid C Solving this turns out to be equal to solving an Eigen system for the covariance matrix; you can see how…
  30. 30. 30Challenge the future Long story short… We find eigenvalues and eigenvectors of the covariance matrices…
  31. 31. 31Challenge the future Long story short… We find eigenvalues and eigenvectors of the covariance matrices… What was this all about? Segmenting the point cloud taking into account [underlying] surface variations, in search of smooth patches, made disjoint by edges (where we find high curvature/variation). Therefore we can use the above estimated vectors and values for such a segmentation.
  32. 32. 32Challenge the future Explanation from lecture notes of Olga Sorkine…
  33. 33. 33Challenge the future Continued from lecture notes of Olga Sorkine…
  34. 34. 34Challenge the future Continued from lecture notes of Olga Sorkine… This S is our covariance matrix, remember?
  35. 35. 35Challenge the future •Constrained minimization – Lagrange multipliers Continued from lecture notes of Olga Sorkine… Given this equality in differentiation of vector valued functions
  36. 36. 36Challenge the future •Constrained minimization – Lagrange multipliers Continued from lecture notes of Olga Sorkine… maximize f(x, y) subject to g(x, y) = c. Lagrangian: The prblem is transformed to finding a ‘’staionary point for the Langrangian at which the partial derivatives of Lagrangian function are zero.
  37. 37. 37Challenge the future •Constrained minimization – Lagrange multipliers Continued from lecture notes of Olga Sorkine… This one means that the normal vector is normalized, meaning its length is equal to 1; why, because the dot product of a vector by itself gives the squared length
  38. 38. 38Challenge the future How to form covariance matrices in code? VB.NET (confirm and debug for yourself) If(Pts.Count > 3) Then Dim CovMatrices As New list(Of Meta.Numerics.Matrices.SymmetricMatrix) For j As Integer=0 To Pts.Count - 1 Dim CovM As New Meta.Numerics.Matrices.SymmetricMatrix(3) Dim Neighbors As List(Of Integer) = DirectCast(Neigh(j), List(Of Integer)) Dim Centroid As New Point3d Dim NPts As New List(Of Point3d) For Each neighbor As Integer In Neighbors Centroid = Centroid + Pts(neighbor) NPts.Add(Pts(neighbor)) Next Centroid = Centroid / Neighbors.Count Dim CiCBar As New RectangularMatrix(3, Neighbors.Count) For k As Integer=0 To Neighbors.count - 1 Dim Diff As point3d = Pts(Neighbors(k)) - Centroid CiCBar(0, k) = Diff.X CiCBar(1, k) = Diff.y CiCBar(2, k) = Diff.z Next CovM = CiCBar.MultiplySelfByTranspose() CovM = (1 / (Neighbors.count - 1)) * CovM CovMatrices.Add(CovM) Next A = CovMatrices(0).ToArray() C = CovMatrices End If
  39. 39. 39Challenge the future How to form covariance matrices in code? C#.NET (confirm and debug for yourself) if ((Pts.Count > 3)) { List<Meta.Numerics.Matrices.SymmetricMatrix> CovMatrices = new List<Meta.Numerics.Matrices.SymmetricMatrix>(); for (int j = 0; j <= Pts.Count - 1; j++) { Meta.Numerics.Matrices.SymmetricMatrix CovM = new Meta.Numerics.Matrices.SymmetricMatrix(3); List<int> Neighbors = (List<int>)Neigh(j); Point3d Centroid = new Point3d(); List<Point3d> NPts = new List<Point3d>(); foreach (int neighbor in Neighbors) { Centroid = Centroid + Pts(neighbor); NPts.Add(Pts(neighbor)); } Centroid = Centroid / Neighbors.Count; RectangularMatrix CiCBar = new RectangularMatrix(3, Neighbors.Count); for (int k = 0; k <= Neighbors.count - 1; k++) { point3d Diff = Pts(Neighbors(k)) - Centroid; CiCBar(0, k) = Diff.X; CiCBar(1, k) = Diff.y; CiCBar(2, k) = Diff.z; } CovM = CiCBar.MultiplySelfByTranspose(); CovM = (1 / (Neighbors.count - 1)) * CovM; CovMatrices.Add(CovM); } A = CovMatrices(0).ToArray(); C = CovMatrices; }
  40. 40. 40Challenge the future Part 2 finished Some hints on previous assignment follow…
  41. 41. 41Challenge the future Grid Surface Reconstruction Pseudo-Code Define a new Mesh with a list of Vertices and a list of Faces Mesh.Vertices=Points For j as Integer=0 to V-2//choose only bottom-left corners as pivot points For i As Integer=0 To U - 2 define n0,n1,n2,n3 As Integer n0 = j * U + I //bottom-left n1 = n0 + 1 //bottom-right n2 = n0 + U //top-right n3 = n1 + U //top-left Define face As new MeshFace(n0, n1, n3, n2) Mesh.Faces.AddFace(face) Next Next
  42. 42. 42Challenge the future Grid Surface Reconstruction VB.NET Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For j As Integer=0 To V - 2 For i As Integer=0 To U - 2 Dim n0,n1,n2,n3 As Integer n0 = j * U + i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) Next Next with two nested loops
  43. 43. 43Challenge the future Grid Surface Reconstruction VB.NET Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For i As Integer=0 To U * (V - 1) - 1 If (i Mod u) < u - 1 Then Dim n0,n1,n2,n3 As Integer n0 = i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) End If Next with one loop
  44. 44. 44Challenge the future Grid Surface Reconstruction C#.NET Mesh M = new Mesh(); M.Vertices.AddVertices(P); if ((C != null)) M.VertexColors.AppendColors(C.toArray); for (int i = 0; i <= U * (V - 1) - 1; i++) { if ((i % u) < u - 1) { int n0 = 0; int n1 = 0; int n2 = 0; int n3 = 0; n0 = i; n1 = n0 + 1; n2 = n0 + U; n3 = n1 + U; MeshFace face = new MeshFace(n0, n1, n3, n2); M.Faces.AddFace(face); } } with one loop
  45. 45. 45Challenge the future Simple Estimation of Normal Vectors Pseudo-Code Form an empty list of normal vectors Define deviation as a double For each point as Point3d in the point cloud find neighbors fit a plane to neighbors Get the normal of this plane and put it out as the normal of the point form a vector from the vantage point VP to point=VP-point and call it dir if this normal.dir>0 then Add the normal to the list of normals Else Add –normal to the list of normals End Next
  46. 46. 46Challenge the future Estimation Normal Vectors C#.NET List<Vector3d> Normals = new List<Vector3d>(); Point3dList PCList = new Point3dList(); PCList.AddRange(x); double Dev = MD; foreach (Point3d point in PCList) { dynamic Neighbors = PCList.FindAll(V => V.DistanceTo(point) < D); plane NP = default(Plane); Plane.FitPlaneToPoints(Neighbors, NP, Dev); if (NP.Normal * (VP - point) > 0) { Normals.Add(NP.Normal); } else { Normals.Add(-NP.Normal); } } A = Normals; B = PCList.FindAll(VT => VT.DistanceTo(x(654)) < D);
  47. 47. 47Challenge the future Estimation Normal Vectors VB.NET Dim Normals As New list(Of Vector3d) Dim PCList As New point3dlist PClist.AddRange(x) Dim Dev As Double = MD For Each point As point3d In PClist Dim Neighbors = PClist.FindAll(Function(V) V.distanceto(point) < D) Dim NP As plane Plane.FitPlaneToPoints(Neighbors, NP, Dev) If NP.Normal * (VP - point) > 0 Then Normals.Add(NP.Normal) Else Normals.Add(-NP.Normal) End if Next A = Normals B = PClist.FindAll(Function(VT) VT.DistanceTo(x(654)) < D)
  48. 48. 48Challenge the future Basics of Scripting in Rhino + GH • VB.NET (Sub[routine], Function, ByVal, ByRef) • C#.NET (Void, [Functions], [val],ref) • Python (defs) Basic concepts of systems and modules, Inputs & Outputs Private Sub RunScript(ByVal x As Object, ByVal y As Object, ByRef A As Object) End Sub '<Custom additional code> '</Custom additional code> Function plus(A As Integer, B As Integer) As Integer Return A + B End Function private void RunScript(object x, object y, ref object A) { } // <Custom additional code> // </Custom additional code> public int plus(int a, int b) { return a + b; } def plus(a,b): return a+b Nothing visible.
  49. 49. 49Challenge the future Basics of Scripting in Rhino + GH • Rhinocommon • Rhinoscript • Grasshopper Kernel Rhinocommon Library of Geometry Operations VB.NET. C#.NET & Python Grasshopper Kernel Library of Some Special Geometry Operations VB.NET. C#.NET & Python Rhionscript (with [Iron]Python) Operations as in Rhino Command Line Python http://4.rhino3d.com/5/rhinocommon/Rhinocommon SDK: Grasshopper SDK: [Rhino]Command: GrasshopperGetSDKDocumentation Rhinoscript Syntax SDK: [GH Python]Help: Rhinoscript syntax help
  50. 50. 50Challenge the future Questions? p.nourian@tudelft.nl • I will give you the tools we have developed so far as open source; • You will make them better using the following libraries; • Using either Math.NET or MetaNumerics is a must; • Using Accord.NET is a big plus! • If you are using Python, use similar libraries such as SciPy • If you want to do something else let us discuss it now!

×