1. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Point Cloud Library
3D Features. Registration. Surface Classification
Radu Bogdan RUSU
June 14, 2010

2. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

3. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (1/2)
Point-based classification
Figure out ways to “classify the world”
Radu Bogdan RUSU PCL :: 3D Features

4. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)
Point-based classification
Two basic alternatives:
1. create a powerful, discriminative 3D feature descriptor;
learn different classes of surface or object types;
use a machine learning classifier
.
(the solution efficiency depends on the method used − let your “neighbor” (the ML guy) solve it)
→
Radu Bogdan RUSU PCL :: 3D Features

5. [Introduction] Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Motivation (2/2)
Point-based classification
Two basic alternatives:
1. create a powerful, discriminative 3D feature descriptor;
learn different classes of surface or object types;
use a machine learning classifier
.
(the solution efficiency depends on the method used − let your “neighbor” (the ML guy) solve it)
→
2. use geometric reasoning techniques: segmentation, region
growing, robust estimators, octrees, ...
fit linear (planes, lines) and non-linear (cylinders, spheres,
etc) models to your data
Radu Bogdan RUSU PCL :: 3D Features

6. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

7. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Acquisition (1/1)
How are Point Clouds acquired? Where do they come from?
Recap: Point Cloud acquisition sources:
laser/lidar sensors (2D/3D)
stereo cameras
time-of-flight (TOF) cameras
etc...
*
Radu Bogdan RUSU PCL :: 3D Features

8. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)
Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi , pj ) > µ + dthresh · σ}
Distance Analysis:
Before and after:
Radu Bogdan RUSU PCL :: 3D Features

9. Introduction [Acquisition] Feature Estimation F/PFH Surface Classification Object Recognition Registration Conclusion
Filtering (1-2/2)
Statistical Gross Outlier Removal :: PCL/Filters/StatisticalOutlierRemoval
P = {pi ∈ Praw | dist(pi , pj ) > µ + dthresh · σ}
Before and after:
Radu Bogdan RUSU PCL :: 3D Features

10. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

11. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)
Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
Given a point cloud with x,y,z 3D point coordinates
Radu Bogdan RUSU PCL :: 3D Features

12. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (1-2/5)
Surface Normal Estimation :: PCL/Features/NormalEstimationOMP,TBB
Given a point cloud with x,y,z 3D point coordinates
Select each point’s k -nearest neighbors, fit a local plane,
and compute the plane normal
Radu Bogdan RUSU PCL :: 3D Features

13. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (3/5)
Surface Normal and Curvature Estimation
k 1 k
Cj = i=1 ξi · (pi − pj )T · (pi − pj ), p = k · i=1 pi

di2 λ0
 − σp =
e µ2 , pi outlier λ0 +λ1 +λ2
ξi =
1, pi inlier

Radu Bogdan RUSU PCL :: 3D Features

14. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)
Consistent Normal Orientation
Before
Extended Gaussian Image
Orientation consistent for:
1. registration
2. feature estimation
3. surface representation
normals on the
Gaussian sphere
should be in the
same half-space
Radu Bogdan RUSU PCL :: 3D Features

15. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Local Features Estimation (4-5/5)
Consistent Normal Orientation
Before After
(viewpoint − pi ) · npi ≥ 0
or:
propagate consistency
through an EMST
Radu Bogdan RUSU PCL :: 3D Features

16. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (1/5)
What scale to choose ?
bad scale (too small) good scale
Selecting the right scale (k -neighborhood) is problematic:
Radu Bogdan RUSU PCL :: 3D Features

17. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (2/5)
Solution: Compute the persistence of the feature over multiple scales
Scales are independent, can be parallelized
Where to threshold ?
Computer Graphics: look for jumps in the feature curve
Datasets in “robotics” have non-smooth surfaces. Solution:
Radu Bogdan RUSU PCL :: 3D Features

18. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (3/5)
Multiple radii analysis
L1-Manhattan L2-Euclidean Jeffries-Matusita
Radu Bogdan RUSU PCL :: 3D Features

19. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (4/5)
Multiple radii analysis
Bhattacharyya Chi-Square Kullback-Leibler
Radu Bogdan RUSU PCL :: 3D Features

20. Introduction Acquisition [Feature Estimation] F/PFH Surface Classification Object Recognition Registration Conclusion
Feature Persistence (5/5)
Example Result
RAW/intensity data Persistent Feature Histograms
*
n−1
Pf = [Pfi ∩ Pfi+1 ]
i=1
Radu Bogdan RUSU PCL :: 3D Features

21. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

22. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation
For every point pair
(ps , ns ); (pt , nt ) , let
u = ns , v = (pt −ps )×u, w = u×v

f0 = v , n j 

f1 = u, pj − pi /||pj − pi ||

x≤3 fx ·d
i = · dx
f2 = ||pj − pi ||  hist

x=0 fxmax −fxmin


f3 = atan( w, nj , u, nj )
Radu Bogdan RUSU PCL :: 3D Features

23. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation

f0 = v , n j 

f1 = u, pj − pi /||pj − pi ||

x≤3 fx ·d
i = · dx
f2 = ||pj − pi ||  hist

x=0 fxmax −fxmin


f3 = atan( w, nj , u, nj )
Radu Bogdan RUSU PCL :: 3D Features

24. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Basic Concepts :: PCL/Features/FPFHEstimation
For every point pair
(ps , ns ); (pt , nt ) , let
u = ns , v = (pt −ps )×u, w = u×v
Radu Bogdan RUSU PCL :: 3D Features

25. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (1-4/5)
Points lying on different geometric primitives
Radu Bogdan RUSU PCL :: 3D Features

26. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Point Feature Histograms (PFH) (5/5)
Complexity Analysis
Complexity is high: O(k 2 ). Optimizations to the rescue!
Unordered Ordered
Radu Bogdan RUSU PCL :: 3D Features

27. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (1/5)
Basic Concepts
1 k 1
Re-formulate: FPFH(p) = SPF (p) + k i=1 ωk · SPF (pk )
Point Feature Histograms (PFH) Fast Point Feature Histograms (FPFH)
Radu Bogdan RUSU PCL :: 3D Features

28. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (2/5)
Theoretical formulation
Radu Bogdan RUSU PCL :: 3D Features

29. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (3/5)
Noise Analysis
Synthetically noiseless Synthetically noisified
Radu Bogdan RUSU PCL :: 3D Features

30. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (4/5)
Noise Analysis
Radu Bogdan RUSU PCL :: 3D Features

31. Introduction Acquisition Feature Estimation [F/PFH] Surface Classification Object Recognition Registration Conclusion
Fast Point Feature Histograms (FPFH) (5/5)
Persistence Similarity
Radu Bogdan RUSU PCL :: 3D Features

32. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

33. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (1/8)
Point-based classification recap
Figure out ways to “classify the world”
Radu Bogdan RUSU PCL :: 3D Features

34. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (2/8)
Learning classes of surfaces
Concave vs Convex
13 classes total
Radu Bogdan RUSU PCL :: 3D Features

35. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (3/8)
Noise Analysis
Radu Bogdan RUSU PCL :: 3D Features

36. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (4/8)
Most Discriminating Features Selection
How many training samples to generate ? Need to
simulate real scans.
Solution: generate a lot, then select the most
discriminative ones
Radu Bogdan RUSU PCL :: 3D Features

37. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (5/8)
Classification results using different methods
SVM
KNN
KMeans
Table 1. Classification results
Method used Noiseless Noisy
SVM RBF Sublinear kernel 95.26% 89.55%
KNN Bhattacharyya (µ-dist) 87.11% 83.53%
K-Means (81D) Bhattacharyya 73.63% 70.74%
Radu Bogdan RUSU PCL :: 3D Features

38. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (6/8)
FPFH classification results
Radu Bogdan RUSU PCL :: 3D Features

39. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (7/8)
Classification results using PFH and SVM: 89.95%
*
*
Radu Bogdan RUSU PCL :: 3D Features

40. Introduction Acquisition Feature Estimation F/PFH [Surface Classification] Object Recognition Registration Conclusion
Surface Classification (8/8)
Classification results using FPFH and CRF: 97.36%
Radu Bogdan RUSU PCL :: 3D Features

41. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

42. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH Concepts
From local annotations (FPFH) to global (cluster) annotations (GFPFH)
Radu Bogdan RUSU PCL :: 3D Features

43. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
FPFH Classification
Classification results using FPFH and CRF: 98.27%
*
Radu Bogdan RUSU PCL :: 3D Features

44. Introduction Acquisition Feature Estimation F/PFH Surface Classification [Object Recognition] Registration Conclusion
GFPFH Classification
Classification results using GFPFH and SVM: 95.13%
*
Radu Bogdan RUSU PCL :: 3D Features

45. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

46. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (1/7)
Multiple Scans :: PCL/Registration/IterativeClosestPoint*
points on similar surfaces
*
Radu Bogdan RUSU PCL :: 3D Features

47. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (2/7)
The classics
*
Radu Bogdan RUSU PCL :: 3D Features

48. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (3/7)
Outdoor Example: Persistence, Initial Alignment
*
Radu Bogdan RUSU PCL :: 3D Features

49. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (4/7)
Outdoor Example: Non-Linear Optimization
*
Radu Bogdan RUSU PCL :: 3D Features

50. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (5/7)
Outdoor Example: Non-Linear Optimization
Radu Bogdan RUSU PCL :: 3D Features

51. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (6/7)
Back to Indoor Environments
Radu Bogdan RUSU PCL :: 3D Features

52. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition [Registration] Conclusion
Registration (7/7)
Registered 360◦ scans - 15 million points
Radu Bogdan RUSU PCL :: 3D Features

53. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration [Conclusion]
Outline
1. Introduction
2. Acquisition
3. Feature Estimation
4. (Fast) Point Feature Histograms
5. Surface Classification
6. Object Recognition
7. Registration
8. Conclusion
Radu Bogdan RUSU PCL :: 3D Features

54. Introduction Acquisition Feature Estimation F/PFH Surface Classification Object Recognition Registration [Conclusion]
Questions?
http://www.ros.org/wiki/pcl
Radu Bogdan RUSU PCL :: 3D Features