The document describes a pipeline for 3D object recognition and 6-DOF pose estimation from an RGB-D image. It involves generating synthetic views for training, extracting global features combining color and geometry, recognizing objects by matching features, and optimizing the estimated pose using ICP. The feature descriptor encodes relationships between point pairs and correlates color and geometry information across segmented regions. The approach achieves 94% recognition rate on online objects and 100% on synthetic views.

1. http://www.lsr.ei.tum.d
e
www.lsr.ei.tum.de
Lehrstuhl für Steuerungs- und Regelungstechnik
Technische Universität München
Forschungspraxis : Object
recognition and pose estimation
from an RGB-D image
Chiraz Nafouki Supervisor: Shile Li
chiraz.nafouki@tum.de li.shile@mytum.de

2. Motivation
2
Why 3D based object recognition and 6-DOF
pose estimation?
●
Many robotic applications rely on object recognition
and 6-DOF pose estimation.
●
Emergence of low-cost RGB-D devices.

3. 3
PointCloudObjectP={p1,p2, p3,...,pN},withpi=(x,y,z,r,g,b)T
P
Globalfeaturedescriptor ⃗f=(f1, f 2, ...,fm)
Problem : Develop a global feature descriptor that combines
geometry and color information for object recognition and pose estimation.
Problem Formulation
P
Keypointsextraction
Localfeaturedescriptors
⃗f1 ,⃗f2 ,...⃗f N

4. Related Work
4
Viewpoint Feature Histogram (VFH):
●
Global descriptor of size 308.
●
Surflet-pair relation between centroid and other points.
●
Drawbacks: Only geometric information is used.
[Radu et. al. 2010]
Two objects with similar shapes
and different colors
β=arccos(
n5 .c
∥c∥
)
α=v.n5
Φ=u.
(p5−c)
d
θ=arctan(w.n5 ,u.n5)
d=∥p5−c∥

5. 5
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud
Testing data
Training data

6. 6
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud

7. Object Modeling and synthetic
views generation
7
- Generate 1260 synthetic views for each 3D object model.
- Compute a feature descriptor for each synthetic view.
3D Object models used in the
training dataset
Example of synthetic views of
two object models

8. Synthetic views generation: Naïve
Sampling and uniform sampling
Naïve Sampling Uniform Sampling
Naïve Sampling:
● Uniformly sampling each Euler angle independetly.
● Results in oversampling near polar regions.
Uniform sampling:
● Uniformly sampling the pitch angle.
● Use the inverse cosine for the yaw angle to avoid oversampling.
[James J.Kuffner 2004]

9. 9
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud

10. Segmentation and clustering
10
●
Extract the horizontal plane on which the objects lie using RANSAC algorithm.
●
Cluster the remaining points into individual objects.
Segmentation &
clustering

11. 11
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud

12. 12
Geometry
Information
Correlation between
Color & geometry
⃗f
Feature extraction
The feature descriptor is composed of two parts:
Part one : three features between random point pairs
●
Distance :
●
Angle with normals : ,
●
Angle with the viewpoint direction :
Each of the three features is encoded
in a 30-bin histogram.
d(pi , pj)=∥pi−pj∥
(pi , pj).
a1=^(ni ,⃗pi pj) a2=^(nj ,⃗pj pi)
a3=^(⃗v ,⃗pi pj)

13. Feature extraction
13
Part two :
●
Points are classified into 10 subregions
based on their distance to the centroid.
●
Color information is encoded using CIELab
color space in a 30-bin histogram.
●
Color and geometry are correlated.
⃗f
Definition of regions for an object

14. 14
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud

15. Object Recognition and pose
estimation
15
●
Given an object with a global descriptor :
Find the k-nearest neighbours to in the database.
●
Chi-squared distance is used :
Object from
real scene
⃗fFeature
extraction
⃗f
Database
Find k-nearest
neighbours
⃗f
⃗f 1
⃗f 2
⃗f k
...
Object recognition
and pose
estimation

16. 16
Pipeline of object recognition and pose estimation
Object modeling Feature extraction Database
Segmentation
& clustering
Feature extraction
Recognition
(matching)
Pose
optimization
Synthetic views
generation
Object recognition pipeline
Real scene
Point cloud

17. Pose Optimization
17
●
Using Iterative Closest Point (ICP).
●
ICP aims at minmizing the distance between two point clouds.
●
Find optimal rotation and translation to apply on the source point
cloud to match the reference.
Object from
real scene
(reference)
Retrieved nearest
neighbour from database
(source)
ICP
Optimize the pose of the
source point cloud

18. Experimental Results
18
Experimental setup
● Online recognition (integration into Ros node).
● Recognition rate of 94% for online objects.
● Recognition rate of 100% for synthetic views.
Superposition of the real object with the
recognized object in its estimated pose

19. Experimental Results
19

20. Occlusion handling
20

21. Summary
Feature part:
● Combining color and geometry information in a feature descriptor.
Training part:
● Uniformly distributed synthetic views generation (uniform sampling).
Testing part:
● Segmentation and clustering.
● Recognition (matching).
● Pose refinement after recognition (ICP).
Evaluation:
● Recognition rate.
● Pose accuracy.
● Occlusion handling.
21

22. Future Work
●
Make the second part of the descriptor independent of the centroid.
●
Use Locality-Sensitive-Hashing (LSH) algorithm instead of kd-tree.
●
Test the descriptor with uniformly sampled training data.
●
Estimate quantitatively the pose accuracy.

23. References
23
Radu Bogdan Rusu, Gary Bradski, Romain Thibaux and John Hsu.
Fast 3D Recognition and Pose Using the Viewpoint Feature Histogram.
In Proc. of the Int. Conf. on Intelligent Robot Systems (IROS), 2010, pp. 3467–3474.
Wei Wang , Shile Li , Lili Chen , Dongming Chen and Kolja Kühnlenz.
Fast object recognition and 6d pose estimation using viewpoint oriented color-shape
histogram.
In Multimedia and Expo (ICME), 2013 IEEE International Conference on, pages 1–6. IEEE, 2013.
James J.Kuffner
Effective Sampling and Distance Metrics for 3D Rigid Body Path Planning.
In Proc. of the Int. Conf. on Robotics and Automation (ICRA), 2004.
Walter Wohlkinger and Markus Vincze
Ensemble of Shape Functions for 3D Object Classification.
In Proc. of the Int. Conf. on Robotics and Biomimetics, 2011.