This document summarizes research on using deep learning to achieve isotropic super-resolution for fluorescence microscopy. It describes how the method can address anisotropy issues in confocal and light-sheet fluorescence microscopy by learning a mapping between low and high-resolution volumes without needing matched data or priors. Simulation results and experiments on real microscopy data demonstrate how the method enhances resolution isotropically and corrects artifacts in both modalities.

1. Deep learning enables reference-free
isotropic super-resolution
for volumetric fluorescence microscopy
Peter Park
Research Scientist
BISPL - BioImaging, Signal Processing, and Learning Lab.
Graduate School of AI
KAIST, Korea

2. Index
1. Problem formulation – how deep learning can address anisotropy in
fluorescence microscopy.
2. Proof of concept – simulations studies
3. Super-resolution in real-world data – in CFM
4. Blind deconvolution in real-world data – in LSFM
5. Artifact correction – in LSFM
CFM: confocal fluorescencemicroscopy, LSFM: lightsheetflourescencemicroscopy

3. Lateral plane: high resolution
Axial plane: low resolution
In fluorescence microscopy, how you look determines how sharp the image is.
*Mouse cortical regions imaged with confocalfluorescence microscopy
Lateral
Axial
Lateral
20 μm

4. How do we learn if we have no matching
data nor any priors?

5. Lateral plane: high resolution
Axial plane: low resolution
Learning the mapping b/w distributions -> optimal transport approach
X
Y
Z
Low Res. High Res.
G
F
Lateral plane
Axial plane

6. Implementation of the intuition
X
Y
Z
Low Res. High Res.
G
F
Lateral plane
Axial plane
X
Z
Y
Anisotropic original Isotropic reconstruction Anisotropic reconstruction
X
Z
Y
G F
Cycle-consistency loss
X
Z
Y
( , , ) ( , , )
DX
( , , )
( , , )
DY
Strategy: train G and F by training discriminators, and :
DX DY

7. Implementation of the idea
X
Z
Y
Anisotropic original Isotropic reconstruction Anisotropic reconstruction
X
Z
Y
G F
Cycle-consistency loss
X
Z
Y
( , , ) ( , , )
DX
( , , )
( , , )
DY
Strategy: train G and F by training discriminators, and :
DX DY
+
+
UNET
DLG
DLG
OR +
+
UNET
DY
(2)
DY
(1)
DX
(1)
DX
(2)
, , , :
G:
F:
PatchGAN
Network structures

8. Proof of concept: simulation

9. Proof of concept: simulation
X
Y
Z
X
Y
Z
x
z

10. Simulation results - visuals
x
z

11. Simulation results – large scale quantification
y x
z
y x
z
y x
z
0
1
Input Network Ground-truth
PNSR (dB) / MS-SSIM: 16.94 dB / 0.70 19.03 dB / 0.86

12. Simulation results – Across different conditions
Std. of axial Gaussian blurring Axial subsampling rate
PSNR
(dB)
MS-SSIM
PSNR
(dB)
MS-SSIM

13. Simulation results – calculation of mismatches with ground-truth
Input Network
FWHM
Mismatch
(pixels)
0
2
6
10
14
Input
Output
Ground-truth
Distance
Normalized
intensity
* 317 objects

14. How it worked? – restoration of information in frequency domain
0
255
FFT

15. How it worked? – recovering the blurring PSF
0
1
Blurring PSF
X
Z
Y
Anisotropic original Isotropic reconstruction Anisotropic reconstruction
X
Z
Y
G F
Cycle-consistency loss
X
Z
Y
𝐵𝑙𝑢𝑟𝑟𝑖𝑛𝑔 𝑃𝑆𝐹 ≅ 𝐹(𝛿 𝑥 )
0
0.4
Estimated PSF
-> Albeit with the scaling mismatch, 3D geometry of the blurring PSFs matches.
-> This suggests the model is interpretable.

16. Super-resolution in real-world data:
confocal fluorescence microscopy

17. 1270 μm
8
0
0
μ
m
9
3
0
μ
m
z
y
x
Lateral plane: high resolution
Axial plane: low resolution
In our test case, anisotropy was severe.
*Mouse cortical regions imaged with confocalfluorescence microscopy
Lateral
Axial
Lateral

18. Our model blindly restores the degraded axial resolution.
*Mouse cortical regions imaged with confocalfluorescence microscopy
10
5
0
0.15
0.45
Position (μm)
Signal
intensity
20 μm
Lateral
Axial
G

19. • Mouse cortex imaged with CFM

20. Our model blindly resolves blurred image regions.
*Mouse cortical regions imaged with confocalfluorescence microscopy
50 μm
0 5 15 25
0
0.5
1
Input Network 90°-rotated
Normalized
intensity
Distance (μm)
10 μm

21. The trained network also recovers the suppressed details.
*Mouse cortical regions imaged with confocalfluorescence microscopy
50 μm
10 μm
20 μm
Input Network 90°-rotated

22. The resolution enhancement was consistent across the entire sample space.
*Mouse cortical regions imaged with confocalfluorescence microscopy
z
x
y
z
x
y
Input
Network
10 μm

23. The enhancement translates to more detailed reconstruction of neuronal morphologies.
*Mouse cortical regions imaged with confocalfluorescence microscopy
0.05
0.64
Raw
0.05
0.64
Super-resolved
Traced neurites
Traced neurites
**Neurites traced with NeuroGPS-Tree andhuman correction.
** **

24. The method works for other biological samples - astrocytes.
*Rat cortical regions imaged with confocalfluorescence microscopy
20 μm
50 μm
XZ XZ
X
Z
YZ YZ
y
Z
XY
x
y
z

25. The method works for other biological samples – blood vessel.
*Ratcortical regions imaged with confocalfluorescence microscopy
Input Network
0
1

26. Blind deconvolution in real-world data:
light-sheet fluorescence microscopy

27. Blind deconvolution on experimentally acquired PSFs
1
0.08
x y
z
Deconvolution
~300 x
1
0.08
x y
z
Input
~300 x
Isotropy
achieved!

28. Blind deconvolution on semi-synthetically blurred image
25 μm 10 μm
Blurring in Z-axis

29. Blind deconvolution on semi-synthetically blurred image
25 μm 10 μm

30. Blind deconvolution on semi-synthetically blurred image
Input Network
20
24
28
32
PSNR
(dB)
0.86
Input Network
0.88
0.90
0.92
0.94
MS-SSIM
120
110
100
90
80
BRISQUE
Input Network Ground-truth

31. Artifact correction in real-world data:
light-sheet fluorescence microscopy

32. X
Y
Z
XY XZ YZ
In a poorly calibrated Open-top LSFM system, anisotropy is highly irregular across the sample space.

33. In a poorly calibrated Open-top LSFM system, anisotropy is highly irregular across the sample space.
Lateral image (shown as a reference)
Axial under-sampling + sample vibration from stage drift
Doubling effect by asynchronization between sweeps of excitation
source and detection path.

34. Our method addresses these artifacts which are not simply corrected by PSF-deconvolution.
x
z
50 μm
10 μm

35. The artifact correction in 3D provides better signal-to-background contrast.
y
z
x
y
z
x
XZ XZ
0.18
0.02
YZ
YZ
Raw Super-resolved
25μm