Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
The document discusses point cloud processing and analysis using the Point Cloud Library (PCL). It covers topics such as acquisition, feature estimation including fast point feature histograms (FPFH), surface classification, object recognition, and registration. The goal is to classify surfaces and objects from 3D point cloud data using geometric reasoning and machine learning techniques.
3. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (1/2)
Point-based classification
Figure out ways to “classify the world”
Radu Bogdan RUSU PCL :: 3D Features
4. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)
Point-based classification
Two basic alternatives:
1. create a powerful, discriminative 3D feature descriptor;
learn different classes of surface or object types;
use a machine learning classifier
.
(the solution efficiency depends on the method used − let your “neighbor” (the ML guy) solve it)
→
Radu Bogdan RUSU PCL :: 3D Features
5. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)
Point-based classification
Two basic alternatives:
1. create a powerful, discriminative 3D feature descriptor;
learn different classes of surface or object types;
use a machine learning classifier
.
(the solution efficiency depends on the method used − let your “neighbor” (the ML guy) solve it)
→
2. use geometric reasoning techniques: segmentation, region
growing, robust estimators, octrees, ...
fit linear (planes, lines) and non-linear (cylinders, spheres,
etc) models to your data
Radu Bogdan RUSU PCL :: 3D Features
7. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Acquisition (1/1)
How are Point Clouds acquired? Where do they come from?
Recap: Point Cloud acquisition sources:
laser/lidar sensors (2D/3D)
stereo cameras
time-of-flight (TOF) cameras
etc...
*
Radu Bogdan RUSU PCL :: 3D Features
8. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)
Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi , pj ) > µ + dthresh · σ}
Distance Analysis:
Before and after:
Radu Bogdan RUSU PCL :: 3D Features
9. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)
Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi , pj ) > µ + dthresh · σ}
Before and after:
Radu Bogdan RUSU PCL :: 3D Features
11. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)
Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
Given a point cloud with x,y,z 3D point coordinates
Radu Bogdan RUSU PCL :: 3D Features
12. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)
Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
Given a point cloud with x,y,z 3D point coordinates
Select each point’s k -nearest neighbors, fit a local plane,
and compute the plane normal
Radu Bogdan RUSU PCL :: 3D Features
13. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (3/5)
Surface Normal and Curvature Estimation
k 1 k
Cj = i=1 ξi · (pi − pj )T · (pi − pj ), p = k · i=1 pi
di2 λ0
− σp =
e µ2 , pi outlier λ0 +λ1 +λ2
ξi =
1, pi inlier
Radu Bogdan RUSU PCL :: 3D Features
14. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)
Consistent Normal Orientation
Before
Extended Gaussian Image
Orientation consistent for:
1. registration
2. feature estimation
3. surface representation
normals on the
Gaussian sphere
should be in the
same half-space
Radu Bogdan RUSU PCL :: 3D Features
15. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)
Consistent Normal Orientation
Before After
(viewpoint − pi ) · npi ≥ 0
or:
propagate consistency
through an EMST
Radu Bogdan RUSU PCL :: 3D Features
16. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (1/5)
What scale to choose ?
bad scale (too small) good scale
Selecting the right scale (k -neighborhood) is problematic:
Radu Bogdan RUSU PCL :: 3D Features
17. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (2/5)
Solution: Compute the persistence of the feature over multiple scales
Scales are independent, can be parallelized
Where to threshold ?
Computer Graphics: look for jumps in the feature curve
Datasets in “robotics” have non-smooth surfaces. Solution:
Radu Bogdan RUSU PCL :: 3D Features
18. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (3/5)
Multiple radii analysis
L1-Manhattan L2-Euclidean Jeffries-Matusita
Radu Bogdan RUSU PCL :: 3D Features
19. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (4/5)
Multiple radii analysis
Bhattacharyya Chi-Square Kullback-Leibler
Radu Bogdan RUSU PCL :: 3D Features
20. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (5/5)
Example Result
RAW/intensity data Persistent Feature Histograms
*
n−1
Pf = [Pfi ∩ Pfi+1 ]
i=1
Radu Bogdan RUSU PCL :: 3D Features
22. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation
For every point pair
(ps , ns ); (pt , nt ) , let
u = ns , v = (pt −ps )×u, w = u×v
f0 = v , n j
f1 = u, pj − pi /||pj − pi ||
x≤3 fx ·d
i = · dx
f2 = ||pj − pi || hist
x=0 fxmax −fxmin
f3 = atan( w, nj , u, nj )
Radu Bogdan RUSU PCL :: 3D Features
23. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation
f0 = v , n j
f1 = u, pj − pi /||pj − pi ||
x≤3 fx ·d
i = · dx
f2 = ||pj − pi || hist
x=0 fxmax −fxmin
f3 = atan( w, nj , u, nj )
Radu Bogdan RUSU PCL :: 3D Features
24. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation
For every point pair
(ps , ns ); (pt , nt ) , let
u = ns , v = (pt −ps )×u, w = u×v
Radu Bogdan RUSU PCL :: 3D Features
25. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Points lying on different geometric primitives
Radu Bogdan RUSU PCL :: 3D Features
26. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (5/5)
Complexity Analysis
Complexity is high: O(k 2 ). Optimizations to the rescue!
Unordered Ordered
Radu Bogdan RUSU PCL :: 3D Features
27. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (1/5)
Basic Concepts
1 k 1
Re-formulate: FPFH(p) = SPF (p) + k i=1 ωk · SPF (pk )
Point Feature Histograms (PFH) Fast Point Feature Histograms (FPFH)
Radu Bogdan RUSU PCL :: 3D Features
28. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (2/5)
Theoretical formulation
Radu Bogdan RUSU PCL :: 3D Features
29. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (3/5)
Noise Analysis
Synthetically noiseless Synthetically noisified
Radu Bogdan RUSU PCL :: 3D Features
30. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (4/5)
Noise Analysis
Radu Bogdan RUSU PCL :: 3D Features
31. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (5/5)
Persistence Similarity
Radu Bogdan RUSU PCL :: 3D Features
33. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (1/8)
Point-based classification recap
Figure out ways to “classify the world”
Radu Bogdan RUSU PCL :: 3D Features
34. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (2/8)
Learning classes of surfaces
Concave vs Convex
13 classes total
Radu Bogdan RUSU PCL :: 3D Features
35. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (3/8)
Noise Analysis
Radu Bogdan RUSU PCL :: 3D Features
36. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (4/8)
Most Discriminating Features Selection
How many training samples to generate ? Need to
simulate real scans.
Solution: generate a lot, then select the most
discriminative ones
Radu Bogdan RUSU PCL :: 3D Features
37. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (5/8)
Classification results using different methods
SVM
KNN
KMeans
Table 1. Classification results
Method used Noiseless Noisy
SVM RBF Sublinear kernel 95.26% 89.55%
KNN Bhattacharyya (µ-dist) 87.11% 83.53%
K-Means (81D) Bhattacharyya 73.63% 70.74%
Radu Bogdan RUSU PCL :: 3D Features
38. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (6/8)
FPFH classification results
Radu Bogdan RUSU PCL :: 3D Features
39. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (7/8)
Classification results using PFH and SVM: 89.95%
*
*
Radu Bogdan RUSU PCL :: 3D Features
40. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (8/8)
Classification results using FPFH and CRF: 97.36%
Radu Bogdan RUSU PCL :: 3D Features
42. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH Concepts
From local annotations (FPFH) to global (cluster) annotations (GFPFH)
Radu Bogdan RUSU PCL :: 3D Features
43. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
FPFH Classification
Classification results using FPFH and CRF: 98.27%
*
Radu Bogdan RUSU PCL :: 3D Features
44. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH Classification
Classification results using GFPFH and SVM: 95.13%
*
Radu Bogdan RUSU PCL :: 3D Features
46. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (1/7)
Multiple Scans :: PCL/Registration/IterativeClosestPoint*
points on similar surfaces
*
Radu Bogdan RUSU PCL :: 3D Features
47. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (2/7)
The classics
*
Radu Bogdan RUSU PCL :: 3D Features
48. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (3/7)
Outdoor Example: Persistence, Initial Alignment
*
Radu Bogdan RUSU PCL :: 3D Features
49. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (4/7)
Outdoor Example: Non-Linear Optimization
*
Radu Bogdan RUSU PCL :: 3D Features
50. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (5/7)
Outdoor Example: Non-Linear Optimization
Radu Bogdan RUSU PCL :: 3D Features
51. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (6/7)
Back to Indoor Environments
Radu Bogdan RUSU PCL :: 3D Features
52. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (7/7)
Registered 360◦ scans - 15 million points
Radu Bogdan RUSU PCL :: 3D Features