1
Change Detection of 3D Scene
with 3D and 2D Information for
Environment Checking
PhD Candidate: Baowei Lin
August 12th, ...
1. Introduction
Research Motivation
Change Detection
2. 3D Keypoints Based 3D-2D Matching
Background
3D Keypoints Detectio...
1. Introduction
Research Motivation
Change Detection
2. 3D Keypoints Based 3D-2D Matching
Background
3D Keypoints Detectio...
4
Changes should be alerted at these areas.
5
Original
configuration
Damaged
configuration
wave washing
if changed
dangerous
6
• Impossible to check
manually
 Wide range
 Huge number of blocks
• Important to check
automatically
7
• Impractical to check by
fixed cameras
8
• possible to check by
hand-held devices
9
Finding potential
change area.
Sub-goal 1:
10
Estimating accurate
changes.
Sub-goal 2:
offline
Finding potential
change area. online
Sub-goal 1:
1. Introduction
Research Motivation
Change Detection
2. 3D Keypoints Based 3D-2D Matching
Background
3D Keypoints Detectio...
A change is a difference of objects in the scene
at time A and at time B. 12
Time A Time B
13
3D point cloud
Training images (2D images)
1. 2D-2D Method
14
Need Fixed camera
input
output
Original
image
Change
image
Changed area
2D-2D
input
output
2. 3D-2D Method
15
Detection is fast but not accurate
Original
point cloud
Change
image
Changed area
3D-2D
input
output
3. 3D-3D Method
16
Detection is accurate but slow
Original
point cloud
Change
point cloud
Changed area
3D-3D
17
Potential changed areas
3D information
Camera poses
Online system Offline system
3D-2D based
change detection
3D-2D bas...
1. Introduction
Research Motivation
Change Detection
2. 3D Keypoints Based 3D-2D Matching
Background
3D Keypoints Detectio...
19
3D interesting
points
2D interesting points
Camera pose=[R,t]
3D point cloud
2D training images
20
2D-2D
3D-3D
SIFT[Lowe 2004], SURF [Bay 2006], etc.
spin image[Johnson 1998], NARF[Steder 2010], etc.
detector descripto...
21
3D point cloud
2D training images
3D detector and 2D
detector can not be
corresponded.
22
Image patch
Point distribution
Can not match
2D image
3D point cloud
23
• Detect keypoints
correctly
• Describe keypoints
appropriately
2D image
3D point cloud
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
Detected 2D
interesting points
25
Feature matching
Obviously, the SIFT features could be used
in 3D keypoints detection an...
Point cloud
26
P1
P2
P3
P4
P5
P6Camera position
3D keypoint
Projected 3D
points
2D images number
threshold used for
3D key...
27
th_v = 1
#3D keypoints ≅10,000
27 training images
105,779 3D points
th_v = 7
#3D keypoints ≅ 1,000
Reconstructed 3D poi...
28
2D SIFT keypoints and descriptors
3D keypoint&
descriptor
-Keep all 2D descriptors
Accurate but slow
Description method...
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
30
P1 P2 P3 P4 P5 P6
3D point cloud
Ground truth
Camera
positions
……
Training images
31
P1 P2 P3 P4 P5 P6
P6
’
Camera pose
estimation
3D keypoints
generation
32
P1
P2
P3
P4
P5
P6
P6
’
P6
’ =[R ’ |t ’]
P6 =[R | t ]
TranslationerrorRotationerror[rad]
33
1. Our method is accurate
2. th_v does not affect the result
th_v is used for 3D
key...
34
3D point cloud Query image
project 3D points
35
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
37
Potential changed areas
3D information
Camera poses
Online system Offline system
3D-2D
based
change
detection
3D-2D bas...
38
Our method:
1. Use local feature instead of color
2. Detect any shape of object
Using laser range
finder [Goncalves 201...
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
40
1. Find the nearest image
Query image
Nearest image
2. Find changed area
Nearest image Query image
changed area
3. Visu...
1st Nearest Query image
41
P1 P2 P3 P4 P5
3D keypoints
generation
Need fixed camera
Smallest distance
Ground truth
……
Trai...
42
the 1st nearest
image
Query
image
Points: 2D keypoints
Blue: correspondence
Red: no correspondence
Blue: correspondence...
3D point cloud
Visualized 3D points
43
change area
projection
Detected result
Projected 3D points
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
3D point cloud Query image
45
Quantitative results visualization
Changed 3D points Changed area
Results for different thresholds 46
Set as:0, 5, 10, 20, 30,
50, 70 and 90 pixels
0 5 10 20
30 50 70 90
Image resolution:2...
TP rate= True Positive
Ground Positive
FP rate=
Ground Negative
False Positive
47
Receiver operating characteristic (ROC) ...
48It is the parameter left for users.
1st
2nd
Query image Detection results
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
50
Potential changed areas
3D information
Camera poses
Online system Offline system
3D-2D
based
change
detection
3D-2D bas...
Different size scale because of
the character of Structure-
from-Motion (SfM)
51
3D-3D registration is actually, the
scale...
52
Iterative closest point (ICP)
based alignment [Besl 1991].
-Need simple scenes
-Need initial pose and scale
-Not robust...
53
1. Scale estimation
2. Scale Ratio estimation
Keyscale1=0.5
Keyscale2=0.1
Scale ratio=5
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
Bunny point cloud
Width=0.001
55
Similar to each other
Different to each other
Width=0.1
Width=1.0
3D keypoints
Spin image...
56
Decide which set of
spin images are
different to each
other by using
Contribution rate.
PCA
Robust to order of extracte...
57minimum
1 5 10 15
similaritysimilarity
d
w
Similar to each other
Different
Similar to each other
minimum is not unique
Finding them is not stable
58
minimumminimum
similarity
w
59
Bunny point cloud
Finding minimum is not stable
similarity
w
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
Point clouds
61
Register two plots to get scale ratio
Scale ratio ICP
similarity plots
Overlapping parts
Original bunny curves 5 times larger bunny curves
62
Original bunny
curves
5 times larger
bunny curves
63
Scale ratio t
Displaced
Original bunny
curves
5 times larger
bunny curves
64
65
Similarity
estimation
3D
registration
Scaleratio
estimation
input
Similarity plots
Scale
ratio
alignment
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
67
Original
point
cloud
The number of points:207,583
Scene size: 35m x 5m
overlapping area
68
Created 1st
point cloud
Created 2nd
point cloud
estimate
scale ratio
Original
point
cloud
69
The method provides perfect result when
the overlap rate is larger than 70%.
Ground truth = 1
Small blocks point clouds
70
Changed block
 Introduction
◦ Research Motivation
◦ Change Detection
 3D Keypoints Based 3D-2D Matching
◦ Background
◦ 3D Keypoints De...
72
Potential changed areas
3D information
Camera poses
Online system Offline system
3D-2D
based
change
detection
3D-2D bas...
73
Future work:
1. Improve computation speed and
detection accuracy for online system.
-current computation time: 20 secon...
Upcoming SlideShare
Loading in …5
×

Change Detection of 3D Scene with 3D and 2D Information for Environment Checking

1,121 views

Published on

presentation slides for PhD degree of Baowei Lin @Hiroshima University.
20130812

Published in: Education, Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,121
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
38
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Change Detection of 3D Scene with 3D and 2D Information for Environment Checking

  1. 1. 1 Change Detection of 3D Scene with 3D and 2D Information for Environment Checking PhD Candidate: Baowei Lin August 12th, 2013
  2. 2. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 2
  3. 3. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 3
  4. 4. 4 Changes should be alerted at these areas.
  5. 5. 5 Original configuration Damaged configuration wave washing if changed dangerous
  6. 6. 6 • Impossible to check manually  Wide range  Huge number of blocks • Important to check automatically
  7. 7. 7 • Impractical to check by fixed cameras
  8. 8. 8 • possible to check by hand-held devices
  9. 9. 9 Finding potential change area. Sub-goal 1:
  10. 10. 10 Estimating accurate changes. Sub-goal 2: offline Finding potential change area. online Sub-goal 1:
  11. 11. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 11
  12. 12. A change is a difference of objects in the scene at time A and at time B. 12 Time A Time B
  13. 13. 13 3D point cloud Training images (2D images)
  14. 14. 1. 2D-2D Method 14 Need Fixed camera input output Original image Change image Changed area 2D-2D
  15. 15. input output 2. 3D-2D Method 15 Detection is fast but not accurate Original point cloud Change image Changed area 3D-2D
  16. 16. input output 3. 3D-3D Method 16 Detection is accurate but slow Original point cloud Change point cloud Changed area 3D-3D
  17. 17. 17 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection Chapter 2 Chapter 3 Chapter 4 3D-3D3D-2D
  18. 18. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 18
  19. 19. 19 3D interesting points 2D interesting points Camera pose=[R,t] 3D point cloud 2D training images
  20. 20. 20 2D-2D 3D-3D SIFT[Lowe 2004], SURF [Bay 2006], etc. spin image[Johnson 1998], NARF[Steder 2010], etc. detector descriptor detector descriptor
  21. 21. 21 3D point cloud 2D training images 3D detector and 2D detector can not be corresponded.
  22. 22. 22 Image patch Point distribution Can not match 2D image 3D point cloud
  23. 23. 23 • Detect keypoints correctly • Describe keypoints appropriately 2D image 3D point cloud
  24. 24.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 24
  25. 25. Detected 2D interesting points 25 Feature matching Obviously, the SIFT features could be used in 3D keypoints detection and description.
  26. 26. Point cloud 26 P1 P2 P3 P4 P5 P6Camera position 3D keypoint Projected 3D points 2D images number threshold used for 3D keypoints decision. the points which can appear on multiple training images Back face points are not used for computation 3D keypoints th_v
  27. 27. 27 th_v = 1 #3D keypoints ≅10,000 27 training images 105,779 3D points th_v = 7 #3D keypoints ≅ 1,000 Reconstructed 3D points #3D points ≅30,000 Too many for real time calculating Smaller number and good distribution oursoriginal
  28. 28. 28 2D SIFT keypoints and descriptors 3D keypoint& descriptor -Keep all 2D descriptors Accurate but slow Description methods: -Average and Median SIFT features are different when view directions are different.
  29. 29.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 29
  30. 30. 30 P1 P2 P3 P4 P5 P6 3D point cloud Ground truth Camera positions …… Training images
  31. 31. 31 P1 P2 P3 P4 P5 P6 P6 ’ Camera pose estimation 3D keypoints generation
  32. 32. 32 P1 P2 P3 P4 P5 P6 P6 ’ P6 ’ =[R ’ |t ’] P6 =[R | t ]
  33. 33. TranslationerrorRotationerror[rad] 33 1. Our method is accurate 2. th_v does not affect the result th_v is used for 3D keypoints selection 2 degrees Dataset: 27 training images Image resolution:2256x1504 3D points number:105,779 3D scene size:40x25x5cm Bounding box size:10.6x5.7x1.4 0.24cm
  34. 34. 34 3D point cloud Query image project 3D points
  35. 35. 35
  36. 36.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 36
  37. 37. 37 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
  38. 38. 38 Our method: 1. Use local feature instead of color 2. Detect any shape of object Using laser range finder [Goncalves 2010, Ryle 2011 and Neuman 2011]. Not for wide area targets. Not applicable for our round shape or natural scenes. Matching 3D line segments [Eden 2008]. Using color differences [Sato 2006, Pollard 2007 and Taneja 2011] Not stable for illumination changes.
  39. 39.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 39
  40. 40. 40 1. Find the nearest image Query image Nearest image 2. Find changed area Nearest image Query image changed area 3. Visualization Project 3D points onto changed area
  41. 41. 1st Nearest Query image 41 P1 P2 P3 P4 P5 3D keypoints generation Need fixed camera Smallest distance Ground truth …… Training images 2nd 3rd P
  42. 42. 42 the 1st nearest image Query image Points: 2D keypoints Blue: correspondence Red: no correspondence Blue: correspondence Red: no correspondence Non-change area Uncovered area is the changed area Estimated changed area
  43. 43. 3D point cloud Visualized 3D points 43 change area projection Detected result Projected 3D points
  44. 44.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 44
  45. 45. 3D point cloud Query image 45 Quantitative results visualization Changed 3D points Changed area
  46. 46. Results for different thresholds 46 Set as:0, 5, 10, 20, 30, 50, 70 and 90 pixels 0 5 10 20 30 50 70 90 Image resolution:2256x1504 The number of Image: 54 The number of 3D points: 190,845
  47. 47. TP rate= True Positive Ground Positive FP rate= Ground Negative False Positive 47 Receiver operating characteristic (ROC) plot threshold = 30 threshold = 30 Ground truth is set manually We expect the 1st nearest image perform better than others, but the best result is the 2nd nearest image. Good performance Bad performance
  48. 48. 48It is the parameter left for users. 1st 2nd Query image Detection results
  49. 49.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 49
  50. 50. 50 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
  51. 51. Different size scale because of the character of Structure- from-Motion (SfM) 51 3D-3D registration is actually, the scale registration 3D point cloud 3D point cloud 3D point cloud Change points registration Point clouds of same scene with different size 3D point cloud 3D point cloud
  52. 52. 52 Iterative closest point (ICP) based alignment [Besl 1991]. -Need simple scenes -Need initial pose and scale -Not robust to clutters, occlusions and missing part spin images [Johnson 1998], NARF [Steder 2010], shape context [Belongie 2002], etc. Feature based alignment -Need appropriate neighborhood size 3D SIFT [Scovanner 2007], 3D SURF [Knopp 2010], etc. -Not robust to clutters, occlusions and missing part Easy data Different data Fixed scale Adaptive scale
  53. 53. 53 1. Scale estimation 2. Scale Ratio estimation Keyscale1=0.5 Keyscale2=0.1 Scale ratio=5
  54. 54.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 54
  55. 55. Bunny point cloud Width=0.001 55 Similar to each other Different to each other Width=0.1 Width=1.0 3D keypoints Spin images the minimum of similarity between spin images when the width changes. Similar to each other Keyscale Robust to clutters, occlusions and missing part Calculate similarity of collected spin images
  56. 56. 56 Decide which set of spin images are different to each other by using Contribution rate. PCA Robust to order of extracted spin images. Robust to detail
  57. 57. 57minimum 1 5 10 15 similaritysimilarity d w Similar to each other Different Similar to each other
  58. 58. minimum is not unique Finding them is not stable 58 minimumminimum similarity w
  59. 59. 59 Bunny point cloud Finding minimum is not stable similarity w
  60. 60.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 60
  61. 61. Point clouds 61 Register two plots to get scale ratio Scale ratio ICP similarity plots
  62. 62. Overlapping parts Original bunny curves 5 times larger bunny curves 62
  63. 63. Original bunny curves 5 times larger bunny curves 63
  64. 64. Scale ratio t Displaced Original bunny curves 5 times larger bunny curves 64
  65. 65. 65 Similarity estimation 3D registration Scaleratio estimation input Similarity plots Scale ratio alignment
  66. 66.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 66
  67. 67. 67 Original point cloud The number of points:207,583 Scene size: 35m x 5m
  68. 68. overlapping area 68 Created 1st point cloud Created 2nd point cloud estimate scale ratio Original point cloud
  69. 69. 69 The method provides perfect result when the overlap rate is larger than 70%. Ground truth = 1
  70. 70. Small blocks point clouds 70 Changed block
  71. 71.  Introduction ◦ Research Motivation ◦ Change Detection  3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation  3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation  3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation  Conclusions 71
  72. 72. 72 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
  73. 73. 73 Future work: 1. Improve computation speed and detection accuracy for online system. -current computation time: 20 seconds per image 2. Optimize algorithm to operate with huge size data for offline system. -current computation time: 10 minutes for 100,000 points

×