7. 1
2
x
y
θ
(x1, y1)
(x2, y2)
Template A
Template A
Search image S
Template image P
T
Recognition of 2D rigid objects
2015/10/19 7
Template
Search image
8. Similarity measure
1 Simialrity coefficient 1
Perfect matching Similarity coefficient =1
minimum difference
2015/10/19 8
Sum of Absolute Differences (SAD)
Sum of Squared Differences (SSD)
Normalized cross correlation (NCC)
Shape-based matching (SBM)
9. Sum of Absolute Differences (SAD)
Sum of Squared Differences (SSD)
Matching result : Minimum difference
Similarity measure
2015/10/19 9
/2 /2
/2 /2
( , ) ( , ) ( , )
w h
i w j h
SAD x y I x i y j T i j
/2 /2
2
/2 /2
( , ) ( , ) ( , )
w h
i w j h
SAD x y I x i y j T i j
Image I
Image T
10. Normalized cross correlation (NCC)
The size of the template image
and are the gray-level averages of the template image and the
windowed compared image.
Similarity measure
/2 /2
/2 /2
/2 /2 /2 /2
2 2 2 2
/2 /2 /2 /2
[ ( , ) ( , )]
( , ) ,
( , ) ( , )
w h
I T
i w j h
g
w h w h
I T
i w j h i w j h
I x i y j T i j w h
x y
I x i y j w h T i j w h
w h
Tu Iu
/2 /2
/2 /2
1
( , )
w h
T
i w j h
T i j
w h
/2 /2
/2 /2
1
( , )
w h
I
i w j h
I x i y j
w h
2015/10/19 10
11. Normalized cross correlation (NCC)
The size of the template image
and are the color-level averages of the template image and the
windowed compared image.
Similarity measure
w h
Tu Iu
/2 /2
/2 /2
/2 /2 /2 /2
2 22 2
/2 /2 /2 /2
[ ( , ) ( , )] 3
( , ) ,
( , ) 3 ( , ) 3
w h
I T
i w j h
c
w h w h
I T
i w j h i w j h
I x i y j T i j w h
x y
I x i y j w h T i j w h
R G B( , ) ( , ), ( , ), ( , )T i j T i j T i j T i j
R G B( , ) ( , ), ( , ), ( , )I x i y j I x i y j I x i y j I x i y j
/2 /2
R G B
/2 /2
1
[ ( , ) ( , ) ( , )]
3
w h
T
i w j h
T i j T i j T i j
w h
/2 /2
R G B
/2 /2
1
[ ( , ) ( , ) ( , )]
3
w h
I
i w j h
I x i x j I x i x j I x i x j
w h
2015/10/19 11
12. Shape-based matching (SBM)
Similarity measure
, ,
2 2 2 2
1 1 , ,
1 1
, ,i i i i
i i i i
n n
i x r y c i x r y ci i
eg
i ii i i i x r y c x r y c
t v u w
x y
n n t u v w
d e
d e
( , )T
i i it ud
, ,( , )i i i i
T
i r c r cv we
( , )T
i i ir cq
gradients direction vector of template image
gradients direction vector of scene image
edge points, these points are relative to the center of gravity
and the edge points of the object.
2015/10/19 12
Picture Ref: Halcon
13. Category Features Approaches Disadvantages
Feature-based
Edge maps
Interest point
Invariant descriptors
Orientation code
Hausdorff distance
Feature correspondence
Zernike moment
Dissimilarity measurement
Ring-projection
Inaccurate
feature
extraction
Area-based Intensity Cross correlation
Excessive
computation
time
Recognition of 2D rigid objects
Table. Summary of image alignment methods.
2015/10/19 13
15. Optimal (Coarse-to-fine) :
Image pyramid technique
SIMD
Search strategy
Recognition of 2D rigid objects
Template Image Input Image
Matching Results
Threshold of
Similarity Score
User
Defined
Detected Range of
Rotation Angle
Offline Model Generation
Image Pyramid Technique
Model Gengeration
Online Matching Process
Image Pyramid Technique
Object Selection Process
, 0, 1,..., 1m l
lT l n
l
n
m
lT
Sub-pixel Accuracy Estimation
minTh
min maxand
Optimal Matching
Process
Detected Range of
Scaling
min maxandS S
, 0, 1,..., 1m l
lI l n
T I
Detected Number
of objects
max
Obj
and
training data
2015/10/19 15
pyramid level
16. Image pyramid technique
Pyramid levels: as high as possible.
Top level: recognizable.
Recognition of 2D rigid objects
Level 0 1 2 3 4
2015/10/19 16
Inspection
image
Level 0
Level 1
Level 2
Search region
17. Search strategy
Recognition of 2D rigid objects
Inspection
image
Level 0
Level 1
Level 2
Search region
Level 2
Level 1
Level 0
2,1obj
1,1obj 1,2obj 1,3obj 1,4obj
0,{1,...,4}obj
0,5obj 0,6obj 0,7obj 0,8obj
0,{13,...,16}obj0,{9,...,12}obj
2,1obj
1,{1, , ,42 3 }obj
0,{5, , ,86 7 }obj
Yellow: search candidates
Red: with maximum similarity score and it exceeds the
threshold of similarity minTh
2015/10/19 17
19. Recognition of 2D rigid objects
Template Image Input Image
Matching Results
Threshold of
Similarity Score
User
Defined
Detected Range of
Rotation Angle
Offline Model Generation
Image Pyramid Technique
Model Gengeration
Online Matching Process
Image Pyramid Technique
Object Selection Process
, 0, 1,..., 1m l
lT l n
l
n
m
lT
Sub-pixel Accuracy Estimation
minTh
min maxand
Optimal Matching
Process
Detected Range of
Scaling
min maxandS S
, 0, 1,..., 1m l
lI l n
T I
Detected Number
of objects
max
Obj
and
training data
2015/10/19 19
20. Recognition of 2D rigid objects
↓2
Level
Level
Level 0
Search Image
↓2 Similarity
Coefficient
Calculation
2D Rigid
Transformation
2D Rigid
Transformation
Exhaustive
Search
Candidate Extraction and Ranking
Detected Range of
Rotation Angle
and Scaling
Threshold of
Similarity Score
Track Track
2D Rigid
Transformation
2D Rigid
Transformation
First
Tracking
Track Track
2D Rigid
Transformation
2D Rigid
Transformation
Final
Tracking
Sub-pixel Pose Estimation
Candidate Ranking
Threshold of
Similarity Score
Matching
Results
Local Maximum Local Maximum
Results
Similarity
Coefficient
Calculation
Detected Range of
Rotation Angle
and Scaling
Detected Range of
Rotation Angle
and Scaling
max max max max
( , , , )ix y S
1l
n
2l
n
Inspection
image
Level 0
Level 1
Level 2
Search region
2015/10/19 20
21. Recognition of 2D rigid objects
Tracking
Model Pyramid
Similarity
Coefficient
Calculation
Similarity
Coefficient
Calculation
Candidate
Transformed
Image Pyramid
Maximum Extraction
Candidate
max max max max , , ,
( , , , ) arg max ( , , , ),
[ , ], [ , ],
[ ' 2 , ' 2 ]
[ , ]
x y s
x x y y
x y s x y s
x x t x t y y t y t
s s s s s
Tracking
2015/10/19
Inspection
image
Level 0
Level 1
Level 2
Search region
21
22. Recognition of 2D rigid objects
↓2
Level
Level
Level 0
Search Image
↓2 Similarity
Coefficient
Calculation
2D Rigid
Transformation
2D Rigid
Transformation
Exhaustive
Search
Candidate Extraction and Ranking
Detected Range of
Rotation Angle
and Scaling
Threshold of
Similarity Score
Track Track
2D Rigid
Transformation
2D Rigid
Transformation
First
Tracking
Track Track
2D Rigid
Transformation
2D Rigid
Transformation
Final
Tracking
Sub-pixel Pose Estimation
Candidate Ranking
Threshold of
Similarity Score
Matching
Results
Local Maximum Local Maximum
Results
Similarity
Coefficient
Calculation
Detected Range of
Rotation Angle
and Scaling
Detected Range of
Rotation Angle
and Scaling
max max max max
( , , , )ix y S
1l
n
2l
n
Inspection
image
Level 0
Level 1
Level 2
Search region
2015/10/19 22
23. Sub-pixel estimation for NCC and SBM
Recognition of 2D rigid objects
2 2 2
0 1 2 3 4 5 6 7 8 9( , , )x y k x k y k k xy k x k y k x k y k k
x
y
1
0
1
101
1
0
1 x
y
Maximum
of * * *
( , , )x y
3 3 3 neighborhood space
0 0
0 0
0 0 0 0
[ 1, 1],
[ 1, 1],
[ ' , ' ]
x x x
y y y
1*
0 3 4 6
*
3 1 5 7
*
4 5 2 8
2
2
2
x k k k k
y k k k k
k k k k
2015/10/19 23
25. Methods Strength weakness
Stereo vision
Simple and inexpensive
High accuracy on well-defined targets
Computational burden
Limited to well defined scenes
Low data acquisition rata
Laser triangulation
Relative simplicity
High data acquisition rate
High accuracy
Safety constrains, if laser based
Limited range
Missing data in correspondence
with occlusions and shadows
High cost
Structured light
High data acquisition rate
High accuracy
Safety constrains, if laser based
Missing data in correspondence
with occlusions and shadows
High cost
Time of Flight
Medium to large measurement range
High data acquisition rate
High cost
Recognition of 3D objects
2015/10/19 25
26. Methods Data Strength weakness
View based
CAD model
or
model image
Efficiency
Robust to occlusions, Clutter
and contrast changes
Low cost
Limited accuracy on
viewpoints
Point Cloud based Point clouds
High accuracy
Robust to occlusions and
clutter
3D sensor required
High Computation complexity
High cost
Recognition of 3D objects
Table. Summarized of 3D pose estimation techniques.
2015/10/19 26
27. Perspective shape-based matching
Recognition of perspective objects
Image coordinate
Camerea coordinate
Object plane
coordinate
Point on plane
P
Q
extM
projectionM
R
C
1
r
c
Pixel coordinate
Pixel point
affineM
0
1
p
q
X
Y
Z
x
y
U
V
W
,
1
1
affine projection ext
u
r
v
c
w
M M M
11 12 13
21 22 23
31 32 33
11 12
21 22
31 32
0 0 0
0 0 0
0
1 0 0 1 0
0 0 0 1 1
0 0
0 0 ,
0 0 1 1
x
y
z
x
y
z
r r r t p
x f
r r r t q
y f
r r r t
f r r t p
f r r t q
r r t
𝐑 𝐓
11
21
31
x
r
r
r
r
12
22
32
y
r
r
r
r
( , , ) ( ) ( ) ( )x y z x x y y z z R R R R
2015/10/19 27
28. Perspective shape-based matching
Recognition of perspective objects
, ,
2 2 2 2
1 , ,
1
( , , , , ) ,i i i i
i i i i
n
i x r y c i x r y c
perspective x y z
i i i x r y c x r y c
t v u w
x y
n t u v w
cos( ) sin( )
,
sin( ) cos( )
i iz z
i iz z
t t
u u
cos cos cos sin sin cos sin1 0
,
cos sin cos cos sin sin sin0 1
y z z x y x zi i
y z x z x y zi i
r r
c c
( , )T
i i it ud
, ,( , )i i i i
T
i r c r cv we
( , )T
i i ir cq
gradients direction vector of template image
gradients direction vector of scene image
edge points, these points are relative to the center of gravity
and the edge points of the object.
, ,
2 2 2 2
1 , ,
1
( , , , ) ,i i i i
i i i i
n
i x r y c i x r y c
eg
i i i x r y c x r y c
t v u w
x y s
n t u v w
cos( ) sin( )
sin( ) cos( )
i i
i i
t ts s
u us s
cos( ) sin( )
sin( ) cos( )
i i
i i
r rs s
c cs s
2015/10/19 28
orthogonal projection
weak-perspective projection
29. Optimal (Coarse-to-fine) :
Image pyramid technique
SIMD
Search strategy
Recognition of perspective objects
Template Image
Input Image
Matching Results
Threshold of
Similarity Score
User
Defined
Detected Range of
Rotation Angle
Offline Model Generation
Image Pyramid Technique
Model Gengeration
Online Matching Process
Image Pyramid Technique
Object Selection Process
, 0, 1,..., 1m l
lT l n
l
n
minTh
_ min _ max
_ min _ max
_ min _ max
and ,
and ,
and
x x
y y
z z
Optimal Matching
Process
, 0, 1,..., 1m l
lI l n
T
I
Detected Number
of objects max
Obj
, ,,l i l ip d
,l ie
2015/10/19 29
30. Rotation requirements
Recognition of perspective objects
(roll)x
(pitch)y
(yaw)z
Image Plane x
y
z
Camera
(pitch)y
(yaw)z
x
Original Image Plane
Rotated Image Plane
d
l
Camera
(roll)x
(yaw)z
yOriginal Image Plane
Rotated Image Plane
d
l
Camera
1 1
cosx y
l
l
1
2
1
cos 1
2
z
l
2015/10/19 30
31. Recognition of perspective objects
Inspection
image
Level 0
Level 1
Level 2
Search region
↓2
Level
Level
Level 0
Search Image
↓2
Similarity
Coefficient
Calculation
3D Rigid
Transformation
3D Rigid
Transformation
Exhaustive
Search
Candidate Extraction and Ranking
Detected Range of
Rotation Angle
Threshold of
Similarity Score
Track Track
3D Rigid
Transformation
3D Rigid
Transformation
First
Tracking
Track Track
3D Rigid
Transformation
3D Rigid
Transformation
Final
Tracking
Candidate Ranking
Threshold of
Similarity Score
Matching
Results
Results
Similarity
Coefficient
Calculation
Detected Range of
Rotation Angle
Detected Range of
Rotation Angle
max max max max max_ _ _( , , , , )x y z ix y
1l
n
2l
n
2015/10/19 31
32. Recognition of perspective objects
max max max max max_ _ _
, , ,
( , , , , ) arg max ( , , , , ),
[ , ], [ , ],
[ ' 2 , ' 2 ],
[ ' 2 , ' 2 ],
[ ' 2 , ' 2 ]
x y z x y z
x y s
x x y y
x x x x x
y y y y y
z z z z z
x y x y
x x t x t y y t y t
Tracking
Tracking
Model Pyramid
Similarity
Coefficient
Calculation
Similarity
Coefficient
Calculation
Candidate
3D Transformed
Image Pyramid
Maximum Extraction
Candidate
2015/10/19 32
34. Stereo vision
Recognition of 3D objects
2015/10/19 34
l r
fb
Z
x x
Ref: Stefano Mattoccia - http://vision.deis.unibo.it/~smatt/Seminars/StereoVision.pdf
35. Stereo vision
Recognition of 3D objects
2015/10/19 35
Camera Calibration
(offline)
Rectification
Stereo Correspondence
Triangulation
Calibration is carried out acquiring and
processing 10+ stereo pairs of a known
pattern (typically a checkerboard)
Intrinsic parameters of the two cameras
(focal length, image center, parameters
of lenses distortion, etc)
Extrinsic parameters
(R and T that aligns the two cameras)
Left
i
j
i
j
..(i,j) (i,j)
Right
36. Laser triangulation
Recognition of 3D objects
2015/10/19 36
Ref: http://www.vision-doctor.co.uk/laser-illumination/principle-of-triangulation.html
42. Point Cloud Library (PCL)
Recognition of 3D objects
2015/10/19 42
•Module common
•Module features
•Module filters
•Module geometry
•Module io
•Module kdtree
•Module keypoints
•Module octree
•Module outofcore
•Module recognition
•Module registration
•Module sample_consensus
•Module search
•Module segmentation
•Module surface
•Module visualization
The Point Cloud Library (or PCL) is a large scale,
open project for point cloud processing, started by
Willow Garage, with the purpose to accelerate 3D
algorithmic work in perception for use in robotic
applications.
Ref: http://www.pointclouds.org/
43. Point Cloud Library (PCL) – Point Feature Histogram (PFH)
Recognition of 3D objects
2015/10/19 43
1D
2D
3D
4D
5D
6D
9D
7D
8D
qD
[D D ] Ut sV
W U V
sD
tD
U
W
V
tN
sN U
t sD D
sU N
( )t s
t s
D D U
V
D D
W U V
acos( )tV N
t sd D D
acos( )t sD D
U
d
atan( , )t tW N U N
44. Point Cloud Library (PCL) – Point Feature Histogram (PFH)
Recognition of 3D objects
2015/10/19 44
1
2 4
1
13
4
=
(s , ) 2
atan( , )
t
t s i
i
i it s
i
t t
f V N
f D D
idx step fD D
f U
d
f W N U N
0, if >
( , )
1, if
f s
step s f
f s
1D
2D
3D
4D
5D
6D
9D
7D
8D
qD
0 1 2 3 … 15
count
45. Point Cloud Library (PCL) – Fast Point Feature Histogram (FPFH)
Recognition of 3D objects
2015/10/19 45
1D
4D
6D
7D
9D
qD
5D
3D
8D
2D
1
1 1
( ) ( ) ( )
k
q q k
i k
FPFH D SPFH D SPFH D
k w
1
2 4
1
13
4
=
(s , ) 2
atan( , )
t
t s i
i
i it s
i
t t
f V N
f D D
idx step fD D
f U
d
f W N U N
46. Point Cloud Library (PCL) – PFH- FPFH
Recognition of 3D objects
2015/10/19 46
1D
4D
6D
7D
9D
qD
5D
3D
8D
2D
1D
2D
3D
4D
5D
6D
9D
7D
8D
qD
PFH FPFH
47. Point Cloud Library (PCL) – registration
Recognition of 3D objects
2015/10/19 47
R
T
R
T
Phase1 SAC-IA >>>
Phase2 ICP >>>
Template
pose
Search
Sample Consensus Initial Alignment (SAC-IA)
Iteration Closest Point (ICP)
