스테레오 정합, 신뢰 전파 기법에 대한 개념과 간단한 예제. [참고] J.H. Kim, and Y.H. Ko, “Multibaseline based Stereo Matching Using Texture adaptive Belief Propagation Technique." Journal of the Institute of Electronics and Information Engineers Vol. 50, No. 1, pp.75-85, 2013.

1. 2018-10-05
1
Stereo Matching Using
Belief Propagation Algorithm
ISL Lab Seminar
Han Sol Kang

2. 2018-10-05
Contents
2
Introduction
Camera
Geometry
Stereo
Matching
Belief
Propagation
Experimental
Results

3. 2018-10-05
Introduction
3
 Stereo Vision
: Stereo vision is the extraction of 3D information from digital images, such as obtained by a CCD camera. By comparing
information about a scene from two vantage points, 3D information can be extracted by examination of the relative
positions of objects in the two panels. This is similar to the biological process Stereopsis.

4. 2018-10-05
Introduction
4
 Applications
: Stereo vision is highly important in fields such as robotics, to extract information about the relative position of 3D objects in the
vicinity of autonomous systems.

5. 2018-10-05
Camera Geometry
5
 Pin-hole camera model
Pinhole
Light rays
Z
Y
X
x
C
<Mathematical model>
x'
p
x
y
f
: A simple camera without a lens

6. 2018-10-05
Camera Geometry
6
 Pin-hole camera model
Z
X
C
x'
p
x
xf
x
XZ plane
x
XxZfx :: 
ZxXfx 
Z
Xf
x x

Z
Y
C
x'
p
y
yf
x
YZ plane
y
YyZf y :: 
ZyYf y 
Z
Yf
y
y

Z
Y
X
Z)Y,(X,x 
C
x'
p
x
y
x
x
p
Z
Xf
 y
y
p
Z
Yf

Image sensor
Pin-hole
),( yx ppPrincipal point
cf. Principal point
)(x,y,z

7. 2018-10-05
Camera Geometry
7
 Epipolar geometry
X
Left view Right view
lC rC
le re
1X
2X lineepipolar
rxlx
cf. Epipolar constraint

8. 2018-10-05
Stereo Matching
8
lPMC ll LCP
f
x
Z
X l

f
x
Z
bX r


Z
f
x
X l

bZ
f
xx
bZ
f
x
Z
f
x rlrl


 ,
rl xx
bf
Z


bZ
f
x
X r

),,( ZYXP
Z
f
lx rx
),( lll yxP ),( rrr yxP
b
rClC
Left camera
lens center
Right camera
lens center
RightCameraOpticalAxis
LeftCameraOpticalAxis
Focal Length
Object Point
M N
L R
 Stereo Camera model
similar
similar
rPNC rr RCP
from a from b
a
b
Left image plane Right Image Plane
We need to disparity
information

9. 2018-10-05
Stereo Matching
9
 General idea
Left Image Right Image
: Match along the epipolar line and Find best matching value
Matching value

10. 2018-10-05
Stereo Matching
10
 Correspondence problem
• Noise
• Low-texture region
• Occlusion
• Depth-discontinuity

11. 2018-10-05
Stereo Matching
11
 Local method
SSD (Sum of Squared Difference)
 
 

M
y
N
x
rlMN ydxIyxIdyxSAD
1 1
|),(),,(|),,(
  
 

M
y
N
x
rlMN ydxIyxIdyxSSD
1 1
2
),(),,(),,(
 
 



M
y
N
x
rlMN ydxIyxI
NM
dyxMAE
1 1
|),(),,(|
1
),,(
SAD (Sum of Absolute Difference)
MAE(Mean Absolute Error)

12. 2018-10-05
Stereo Matching
12
 Global method
: Use the energy functions
)()()( dEdEdE smoothnessdata 
Belief Propagation(BP), Graph Cut(GC), Dynamic Programing(DP)…
Implementation :

13. 2018-10-05
Belief Propagation
13
 HMM (Hidden Markov Model)
Observable node(Y) : Image
Hidden node(X) : Disparity
 MRF (Markov Random Field)
: Undirected and cyclic
1) Positivity
2) Markovianity
ffP  ,0)(
)|()|( }{ pNppPp ffPffP 

14. 2018-10-05
Belief Propagation
14
 Goal
: Computes marginal probability of hidden nodes
 Attributes
: Iterative algorithm
: Message passing between neighboring hidden nodes
 Procedure
1) Select random neighboring hidden nodes ts xx ,
2) Send message stm sx txfrom to
3) Update belief about marginal probability

15. 2018-10-05
Belief Propagation
15
Probabilistic Stereo Model
Left Image Right Image Disparity
)(
)()|(
)|(
YP
XPXYP
YXP 
evidence (Data)
prior (Hypothesis)posterior
likelihood

16. 2018-10-05
Belief Propagation
16
 Likelihood
     
s
sdsFXYP ,exp| sd
Matching cost function
   
 YP
XPXYP
YXP
|
)|( 
: disparity candidate at pixel s
  |),(),(|, jdiIjiIdsF srls 
Example
Disparity(𝑥 𝑠)20 25 117 135 150 118 134 22 20 15 d=0
Left Image(Ref.) Right Image
d=1 d=2 d=3 d=4
Value(𝑦𝑠) 135 130 128 16 32

17. 2018-10-05
Belief Propagation
17
 Prior
sd
Constraint function
   
 YP
XPXYP
YXP
|
)|( 
: disparity candidate at pixel s
t
t
st t
     


s sNt
ts ddVXP
)(
,exp
  tsts ddddV ,
Example
t
t
st t
: disparity candidate at pixel ttd

18. 2018-10-05
Belief Propagation
18
 Final model
   XPXYP |
     
 
 


s sNt
ts
s
s ddVdsF ,exp,exp
   
 YP
XPXYP
YXP
|
)|( 
   
 
 


s sNt
tsst
s
sss xxyx ,, 
st
MAP maximize marginal probability (maximize belief)
s : local evidence for node
: compatibility between nodes
sx
sx and ty
t
t
st t

19. 2018-10-05
Belief Propagation
19
Observable node(Y) : Image
Hidden node(X) : Disparity
 Message Passing
mst(xt) = κmax(xs) [s(xs, ys) st(xs, xt) xkN(s)t mks(xs)]
: Message stm sx txfrom to
ms(xs) : local evidence

20. 2018-10-05
Belief Propagation
20
Observable node(Y) : Image
Hidden node(X) : Disparity
 Belief Update
: Belief
bs (xs) = κ s (xs, ys) xkN(s) mks(xs)
)( sxb

21. 2018-10-05
Belief Propagation
21
 Implementation of message, belief and disparity






 


tsk
s
xxNx
s
i
kstsstsss
x
t
i
st xmxxyxxm
)(
1
)(),(),(max)( 



tsk xxNx
skssssss xmyxxb
)(
)(),()( 
)(maxarg ks
x
MAP
s xbd
k









 


jik
i
xxNx
i
t
kijicii
x
j
t
ij xMxxxMcxM
)(
1
)(),()(min)( 








 
 jik xxNx
ikiiiii xMxMcxB
)(
)()()(
)(minarg ks
x
MAP
s xBx
k

take the negative logarithm of each equation

22. 2018-10-05
Belief Propagation
22
 Example
   
      T
ssksskssk
T
tst
xMxMxM
xM
]000[
000
000
0
321


1) Initialize
k1
 
      
      
      
2
333272
,222130
,11102
min
1
000
000
000
1
321
321
321


















ssksskssk
ssksskssk
ssksskssk
tst
xMxMxM
xMxMxM
xMxMxM
xM
 
      
      
      
3
333172
,222030
,11112
min
2
000
000
000
1
321
321
321


















ssksskssk
ssksskssk
ssksskssk
tst
xMxMxM
xMxMxM
xMxMxM
xM
2) Update
2
2
1
2
30
72
2
31
70
4
40
90
ts
k3
)( tt xM)( ss xM
)( 11 kk xM
)( 33 kk xM
}3,2,1{D
3
32
80
k2
)( 22 kk xM






 


tsk
s
xxNx
s
i
kstscss
x
t
i
st xMxxxMcxM
)(
1
)(),()(min)( 
Left image
Right image

23. 2018-10-05
Belief Propagation
23
 Example
 T
stM 4321

 
      
      
      
4
333072
,222130
,11122
min
3
000
000
000
1
321
321
321


















ssksskssk
ssksskssk
ssksskssk
tst
xMxMxM
xMxMxM
xMxMxM
xM
 T
tst xM 131211)(2

 
 
 T
ssk
T
ssk
T
ssk
xM
xM
xM
654)(
543)(
432)(
1
1
1
3
2
1









 


tsk
s
xxNx
s
i
kstscss
x
t
i
st xMxxxMcxM
)(
1
)(),()(min)( 
k1
2
2
1
2
30
72
2
31
70
4
40
90
ts
k3
)( tt xM)( ss xM
)( 11 kk xM
)( 33 kk xM
}3,2,1{D
3
32
80
k2
)( 22 kk xM

24. 2018-10-05
Belief Propagation
24
 Example
3) Calculate Belief
 
 
 
    1)(minarg,141413
141313
141222
131121




kt
x
MAP
t
T
tt
tt
tt
tt
xBxxB
xB
xB
xB
k






 
 tsk xxNx
sksssss xMxMcxB
)(
)()()(
)(minarg ks
x
MAP
s xBx
k

k1
2
2
1
2
30
72
2
31
70
4
40
90
ts
k3
)( tt xM)( ss xM
)( 11 kk xM
)( 33 kk xM
}3,2,1{D
3
32
80
k2
)( 22 kk xM

25. 2018-10-05
Experimental Results
25
 Aloe*
* middlebury 2006 stereo dataset

26. 2018-10-05
Experimental Results
26
 Lab
i=1 i=5
i=20

27. Q A