1. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
Reconstruction (of micro-objects) based on
focus-sets using blind deconvolution
Jan Wedekind
November 19th, 2001
Jan.Wedekind@stud.uni-karlsruhe.de
http://www.uni-karlsruhe.de/~unoh
University of Karlsruhe (TH) -1- Sheffield Hallam University

2. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
MINIMAN-project
Esprit Project No. 33915
Miniaturised Robot for Micro Manipulation
http://www.miniman-project.com/
Miniman III
University of Karlsruhe (TH) -2- Sheffield Hallam University

3. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
people: SHU staff
Sheffield Hallam University
microsystems & machine vision lab
http://www.shu.ac.uk/mmvl/
Prof. J. Travis B. Amavasai F. Caparrelli
no picture
A. Selvan
University of Karlsruhe (TH) -3- Sheffield Hallam University

4. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
people: UKA staff
Universit¨t Karlsruhe (TH)
a
Institute f¨ r Prozeßrechentechnik,
u
Automation und Robotik
http://wwwipr.ira.uka.de/microrobots/
J. Wedekind
University of Karlsruhe (TH) -4- Sheffield Hallam University

5. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
overview: environment
• Leica DM RXA microscope
– 2 channel illumination
– motorized z-table
(piezo-driven 0.1 µm)
– filter-module
• motorized xy-table
• Dual Pentium III with 1GHz
processors
– Linux OS
– C++ and KDE/QT
• CCD camera 768 × 576 Leica DM RXA microscope
µm
⇒ resolution up to 0.74 pixel
University of Karlsruhe (TH) -5- Sheffield Hallam University

6. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
overview: objective
• reconstruct from focus-set:
– surface
– luminosity and coloring
• identify model-parameters and quality of assembly
University of Karlsruhe (TH) -6- Sheffield Hallam University

7. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
basics: discrete fourier transform
• 1D DFT
N −1
2πxki
−
definition Fk := fx e N where i2 = −1
x=0
N −1
2πxki
1 +
⇔ fx = Fk e N
N
k=0
• 2D DFT
N −1
2πxki 2πyli
− −
analogous Fkl := fxy e N e N , i2 = −1
x,y=0
N −1
2πxki 2πyli
1 + +
⇔ fxy = Fkl e N e N
N2
k,l=0
University of Karlsruhe (TH) -7- Sheffield Hallam University

8. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
basics: 1D discrete fourier transform
image domain
* =
x x x
frequency domain
⊗ =
ω x ω
∞ ∞
∀x ∈ R : f (x) = f (x) δ(x − T n) dx
−∞ n=−∞
∞
2π 2π
⇔ ∀ω ∈ C : F (ω) = (F ⊗ λω. δ(ω − n ) )(ω)
T n=−∞
T
University of Karlsruhe (TH) -8- Sheffield Hallam University

9. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
basics: 1D discrete fourier transform
image domain
* =
x x x
frequency domain
1
sinc(x)
0.8
=
0.6
ω
⊗ 0.4
ω
0.2
0
-8 -6 -4 -2 0 2 4 6 8
w
t
∀x ∈ R : f (x) = f (x) rect( )
2T
sin(ωT )
⇔ ∀ω ∈ C : F (ω) = (F ⊗ λω. 2T )(ω)
ωT
University of Karlsruhe (TH) -9- Sheffield Hallam University

10. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: images
 
x00 ··· x0(N −1)
. .
 
..
conservative: x =  . .  ∈ RN ×N
 
. . .
 
x(N −1)0 · · · x(N −1)(N −1)
 
x00
   
x0

 x01 

   .
.

 x1  
   
. 
vector repr.: x = =  ∈ RN N
 .  

 .   x10

 .  

.

xN −1

 .
.


 
x(N −1)(N −1)
University of Karlsruhe (TH) - 10 - Sheffield Hallam University

11. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: convolution (i)
matrix repr. of convolution with a 1D-LSI-filters
 
h0 0 ··· 0
  . 

  .. .

g0  h1 h0 . . f
  .  0 
.. ..

 g1   .
 
. . 0   f1 
  
.
= ∗ . 
 
.

 .
.
 
 hK−1 h0   . 


     . 
 .. 
gK+N −1 
 0 . h1 
 fN −1
 . .
. .. .
∈R(K+N −1) . . . ∈RN
∈R(K+N −1)×N
• vectors of different size ⇒ not feasible
• matrix is clumsy and baffles mathematical approaches
University of Karlsruhe (TH) - 11 - Sheffield Hallam University

12. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: circulant matrices
 
a(0) a(N − 1) · · · a(1)

.. .. .
.

. .
 
 a(1) . 
A= 
.

. .. 

 . . a(N − 1)

a(N − 1) ··· a(1) a(0)
N −1
2πkji
without −
• eigenvalues of A: λ(k) = a(j) e N
proof: j=0
• A = FQA F −1 with
– QA = diag(λ1 , λ2 , . . . , λN ) and
– fourier-kernel F −1
University of Karlsruhe (TH) - 12 - Sheffield Hallam University

13. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: fourier kernel
2πukl i
definition of fourier-kernel: F −1 := e N
 
0 0 0 ···
 
0 1 2 · · ·
 
with U :=   and i2 = −1
0 2 4
 · · ·
. . .
..

. . .
. . . .
note: • X = DFT{x} = X = F −1 x
• x = DFT−1 {X} = x = FX where F = 1
N (F −1 )∗
• F =F
University of Karlsruhe (TH) - 13 - Sheffield Hallam University

14. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: convolution (ii)
approximated 1D convolution
     
g0 h0 hN −1 ··· h1 f
  0 
.  
.. ..
  

 g1   h1
 
. . .   f1 
. 
. ≈ .
∗
  . 
   
 .   . .. .

 .   . . hN −1   . 
  
gN −1 hN −1 ··· h1 h0 fN −1
=:g∈RN =:H∈RN ×N =:f ∈RN
g = Hf
University of Karlsruhe (TH) - 14 - Sheffield Hallam University

15. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
sparse matrices and vectors: convolution (iii)
N −1
gk = H(k−l) mod N fl
l=0
     
g0 H0 HN −1 ··· H1 f
  0 
.  
.. ..
  

 g1   H1
 
. . .   f1 
. 
. ≈ .
∗
  . 
   
 .   . .. .

 .   . . HN −1   . 
  
gN −1 HN −1 ··· H1 H0 fN −1
=:g∈RN 2 =:H∈RN 2 ×N 2 =:f ∈RN 2
⇒ g = H f is 2D convolution!
⇒ H is (block) circulant
University of Karlsruhe (TH) - 15 - Sheffield Hallam University

16. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-algorithm: description
• “incomplete” data-set y
• many-to-one transform
given:
y = g(z1 , z2 , . . . , zM ) = g(z)
• pdf pz (z; θ) and cond. pdf p(z|y; θ)
1. expectation: Determine expected
log-likelihood of complete data
U (θ, θp ) :=Ez {ln pz (z; θ)|y; θp }
EM-algorithm: = ln pz (z; θ) p(z|y; θp )dz
2. maximisation: Maximize
θp+1 = argmax U (θ, θp )
θ
3. goto step 1 until θ converges
University of Karlsruhe (TH) - 16 - Sheffield Hallam University

17. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-algorithm: overview of representations
meaning conversative sparse matrices
real fourier real fourier
N ×N N ×N N 2 ,N 2 ×N 2 N 2 ,N 2 ×N 2
sets R C R C
image x X x X
PSF h ∆ D QD
cov. image Cx Sx Λx Qx
2 2
cov. noise σv I (σv ) Λv Qv
convolution x⊗h X ∆ Dx QD X
DFT - DFT{x} - F −1 x
inv. DFT DFT−1 {X} - FX -
University of Karlsruhe (TH) - 17 - Sheffield Hallam University

18. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: derivation (i)
• incomplete/complete data-set: y ∈ RN / (x , y ) ∈ R2N
 
x
• many-to-one transform: y =: g( ) ∀y
y
• pdf of z is zero-mean normal distr.: pz (z, θ = {D, Λx , Λv }) =
 −1  
 
1x Λ Λx D
−    x x
 
2 y
 
DΛx DΛx D + Λv
 
y
1
e =:C
Λx Λx D
(2π)2N 2
DΛx DΛx D + Λv
University of Karlsruhe (TH) - 18 - Sheffield Hallam University

19. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: derivation (ii)
• conditional pdf p(z|y, θ) = p(x|y, θ) is (conditional) normal distr.:
– with mean Mx|y = Λx D (DΛx D + Λv )−1 y and
– covariance Sx|y = Λx − Λx D (DΛx D + Λv )−1 DΛx
derivation of  
2
1 1 x
−1  
ln pz (z; θ) = −N ln(2π) − ln |C| − (x , y ) C :
2 2 y
Λx Λx D
= det(Λx (DΛx D + Λv ) − DΛx Λx D )
DΛx DΛx D + Λv
= det(Λx Λv ) = |Λx ||Λv |, because Λx and Λv
are symmetric and positive definite.
University of Karlsruhe (TH) - 19 - Sheffield Hallam University

20. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: derivation (iii)
2 N2
• observe that |Λv | = (σv ) .
• |Λx | = |FQx F −1 | = |Qx | = Sx (k, l)
kl
 
A A12
using A =  11 =
A21 A22
 
(A11 − A12 A−1 A21 )−1
22 A−1 A12 (A21 A−1 A12 − A22 )−1
11 11
 
(A21 A−1 A12 − A22 )−1 A21 A−1
11 11 (A22 − A21 A−1 A12 )−1
11
 −1
Λ Λx D
−1
we get C =  x  =
DΛx DΛx D + Λv
University of Karlsruhe (TH) - 20 - Sheffield Hallam University

21. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: derivation (iv)
 
(I − D (DΛx D + Λv )−1 DΛx )−1 Λ−1
x −D Λ−1
v
C −1 =  .
−Λ−1 D
v Λ−1
v
With u Av= u∗ FQA F −1 v = (F ∗ u)∗ QA F −1 v
1 −1 ∗ −1
1 ∗
= ( 2 F u) QA F v = 2 U QA V we obtain
N N
N2 1 1
2
ln pz (z; θ) =−N ln(2π) − 2
ln(σv ) − ln Skl − 2
Y ∗ Q−1 Y
v
2 2 2N
kl
1 1
+ 2
Re{Y ∗
QD Q−1 X}
v − Qx (QD Q∗ Q−1 + Q−1 )
D v x
N 2
kl
1
− 2
X ∗ (QD Q∗ Q−1 + Q−1 )X
D v x
2N
University of Karlsruhe (TH) - 21 - Sheffield Hallam University

22. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: derivation (v)
Replacing the remaining diagonal matrices with DFT-arrays yields:
N2 1 1
2
ln pz (z; θ) =−N ln(2π) − 2
ln(σv ) − ln Skl − 2
Y∗ Y(σv )−1
2
2 2 2N
kl
1 ∗
1
+ 2
Re{Y ∆(σv )−1 X}
2
− Sx (∆∆∗ (σv )−1 + Sx )
2 −1
N 2
1
− 2
X∗ (∆∆∗ (σv )−1 + Sx )X
2 −1
2N
The last steps are:
• substitute X and X∗ ... X with their expected values.
⇒ U (θ, θk )
δU (θ, θk )
• determine argmax by setting derivative to zero.
θ δθ
University of Karlsruhe (TH) - 22 - Sheffield Hallam University

23. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
EM-Algorithm: implementation
The resulting iteration step:
(p+1) (p) 1
• Sx (k, l) = Sx|y (k, l) + 2
|Mx|y (k, l)|2
N
∗
1 Y (k, l)Mx|y (k, l)
• ∆(p+1) (k, l) =
N2 (p) 1
Sx|y (k, l) + 2
|Mx|y (k, l)|2
N
2
1 (p+1) 2 (p) 1
• σv = kl |∆ (k, l)| Sx|y (k, l) + |Mx|y (k, l)|2 +
N2 N 2
1
|Y (k, l)|2 − 2Re Y ∗ (k, l)∆(p+1) (k, l)Mx|y (k, l)
N2
University of Karlsruhe (TH) - 23 - Sheffield Hallam University

24. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
results: deblur picture
conditional mean after 10. iteration
University of Karlsruhe (TH) - 24 - Sheffield Hallam University

25. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
results: statistic and psf
log-DFT
University of Karlsruhe (TH) - 25 - Sheffield Hallam University

26. Reconstruction (of micro-objects) based on focus-sets using blind deconvolution
result: comparison
Jan Wedekind
address: . . . . . . . . . . . . . . . . . . . . . . . . . . Scheffelstr. 65, D-76135 Karlsruhe
email: . . . . . . . . . . . . . . . . . . . . . . . . . . jan.wedekind@stud.uni-karlsruhe.de
www: . . . . . . . . . . . . . . . . . . . . . . . http://www.uni-karlsruhe.de/~unoh
pgp: . . . . . . . . . . . . . . . . . EE FA AF 15 D8 ED 11 4A 5A 76 35 5F 2D 20 C4 E8
University of Karlsruhe (TH) - 26 - Sheffield Hallam University