Recommended
More Related Content
Similar to Deep learning enables reference-free isotropic super-resolution for fluorescence microscopy
Similar to Deep learning enables reference-free isotropic super-resolution for fluorescence microscopy (20)
Recently uploaded
Recently uploaded (20)
Deep learning enables reference-free isotropic super-resolution for fluorescence microscopy
- 1. Deep learning enables reference-free isotropic super-resolution for volumetric fluorescence microscopy Peter Park Research Scientist BISPL - BioImaging, Signal Processing, and Learning Lab. Graduate School of AI KAIST, Korea
- 2. Index 1. Problem formulation – how deep learning can address anisotropy in fluorescence microscopy. 2. Proof of concept – simulations studies 3. Super-resolution in real-world data – in CFM 4. Blind deconvolution in real-world data – in LSFM 5. Artifact correction – in LSFM CFM: confocal fluorescencemicroscopy, LSFM: lightsheetflourescencemicroscopy
- 3. Lateral plane: high resolution Axial plane: low resolution In fluorescence microscopy, how you look determines how sharp the image is. *Mouse cortical regions imaged with confocalfluorescence microscopy Lateral Axial Lateral 20 μm
- 4. How do we learn if we have no matching data nor any priors?
- 5. Lateral plane: high resolution Axial plane: low resolution Learning the mapping b/w distributions -> optimal transport approach X Y Z Low Res. High Res. G F Lateral plane Axial plane
- 6. Implementation of the intuition X Y Z Low Res. High Res. G F Lateral plane Axial plane X Z Y Anisotropic original Isotropic reconstruction Anisotropic reconstruction X Z Y G F Cycle-consistency loss X Z Y ( , , ) ( , , ) DX ( , , ) ( , , ) DY Strategy: train G and F by training discriminators, and : DX DY
- 7. Implementation of the idea X Z Y Anisotropic original Isotropic reconstruction Anisotropic reconstruction X Z Y G F Cycle-consistency loss X Z Y ( , , ) ( , , ) DX ( , , ) ( , , ) DY Strategy: train G and F by training discriminators, and : DX DY + + UNET DLG DLG OR + + UNET DY (2) DY (1) DX (1) DX (2) , , , : G: F: PatchGAN Network structures
- 8. Proof of concept: simulation
- 9. Proof of concept: simulation X Y Z X Y Z x z
- 10. Simulation results - visuals x z
- 11. Simulation results – large scale quantification y x z y x z y x z 0 1 Input Network Ground-truth PNSR (dB) / MS-SSIM: 16.94 dB / 0.70 19.03 dB / 0.86
- 12. Simulation results – Across different conditions Std. of axial Gaussian blurring Axial subsampling rate PSNR (dB) MS-SSIM PSNR (dB) MS-SSIM
- 13. Simulation results – calculation of mismatches with ground-truth Input Network FWHM Mismatch (pixels) 0 2 6 10 14 Input Output Ground-truth Distance Normalized intensity * 317 objects
- 14. How it worked? – restoration of information in frequency domain 0 255 FFT
- 15. How it worked? – recovering the blurring PSF 0 1 Blurring PSF X Z Y Anisotropic original Isotropic reconstruction Anisotropic reconstruction X Z Y G F Cycle-consistency loss X Z Y 𝐵𝑙𝑢𝑟𝑟𝑖𝑛𝑔 𝑃𝑆𝐹 ≅ 𝐹(𝛿 𝑥 ) 0 0.4 Estimated PSF -> Albeit with the scaling mismatch, 3D geometry of the blurring PSFs matches. -> This suggests the model is interpretable.
- 16. Super-resolution in real-world data: confocal fluorescence microscopy
- 17. 1270 μm 8 0 0 μ m 9 3 0 μ m z y x Lateral plane: high resolution Axial plane: low resolution In our test case, anisotropy was severe. *Mouse cortical regions imaged with confocalfluorescence microscopy Lateral Axial Lateral
- 18. Our model blindly restores the degraded axial resolution. *Mouse cortical regions imaged with confocalfluorescence microscopy 10 5 0 0.15 0.45 Position (μm) Signal intensity 20 μm Lateral Axial G
- 19. • Mouse cortex imaged with CFM
- 20. Our model blindly resolves blurred image regions. *Mouse cortical regions imaged with confocalfluorescence microscopy 50 μm 0 5 15 25 0 0.5 1 Input Network 90°-rotated Normalized intensity Distance (μm) 10 μm
- 21. The trained network also recovers the suppressed details. *Mouse cortical regions imaged with confocalfluorescence microscopy 50 μm 10 μm 20 μm Input Network 90°-rotated
- 22. The resolution enhancement was consistent across the entire sample space. *Mouse cortical regions imaged with confocalfluorescence microscopy z x y z x y Input Network 10 μm
- 23. The enhancement translates to more detailed reconstruction of neuronal morphologies. *Mouse cortical regions imaged with confocalfluorescence microscopy 0.05 0.64 Raw 0.05 0.64 Super-resolved Traced neurites Traced neurites **Neurites traced with NeuroGPS-Tree andhuman correction. ** **
- 24. The method works for other biological samples - astrocytes. *Rat cortical regions imaged with confocalfluorescence microscopy 20 μm 50 μm XZ XZ X Z YZ YZ y Z XY x y z
- 25. The method works for other biological samples – blood vessel. *Ratcortical regions imaged with confocalfluorescence microscopy Input Network 0 1
- 26. Blind deconvolution in real-world data: light-sheet fluorescence microscopy
- 27. Blind deconvolution on experimentally acquired PSFs 1 0.08 x y z Deconvolution ~300 x 1 0.08 x y z Input ~300 x Isotropy achieved!
- 28. Blind deconvolution on semi-synthetically blurred image 25 μm 10 μm Blurring in Z-axis
- 29. Blind deconvolution on semi-synthetically blurred image 25 μm 10 μm
- 30. Blind deconvolution on semi-synthetically blurred image Input Network 20 24 28 32 PSNR (dB) 0.86 Input Network 0.88 0.90 0.92 0.94 MS-SSIM 120 110 100 90 80 BRISQUE Input Network Ground-truth
- 31. Artifact correction in real-world data: light-sheet fluorescence microscopy
- 32. X Y Z XY XZ YZ In a poorly calibrated Open-top LSFM system, anisotropy is highly irregular across the sample space.
- 33. In a poorly calibrated Open-top LSFM system, anisotropy is highly irregular across the sample space. Lateral image (shown as a reference) Axial under-sampling + sample vibration from stage drift Doubling effect by asynchronization between sweeps of excitation source and detection path.
- 34. Our method addresses these artifacts which are not simply corrected by PSF-deconvolution. x z 50 μm 10 μm
- 35. The artifact correction in 3D provides better signal-to-background contrast. y z x y z x XZ XZ 0.18 0.02 YZ YZ Raw Super-resolved 25μm
Editor's Notes
- \
- Intuition
- DLG stands for deep linear generator.
- Simulaiton involves 10,000 tubular objects.
- For subsampling variations, the axial sub-sampling method of taking i-th slice takes place after the Gaussian blurring with standard deviation of 4.
- FHWM mismatches were calculated from 317 tubular objects.
- The dotted blue line is the axial input. The solid blue line is the axial output.
- [CLICK TO SHOW THE PSNR GRAPH] The PSNR metrics were calculated for 31 ROIs of 140x140 microm, where the fluorescence distribution was deteced similarly between input and the rotated image. The network introduced a mean PSNR improvement of 2.42 dB per pair of input ROI versus output ROI.
- The neurites were traced using a state-of-the-art neuronal tracing method for instance segmentation, NeuroGPS-tree, followed by human correction. The tracing was done blindly without knowledge of their image counterparts.
- \
- The real benefit of the light sheet microscopy when it comes to testing with deconvolutional capability is that you can start from almost near-isotropy and adjust the degree of anisotropy by altering the imaging conditions, such as the thickness of the light-sheet.
- First, by taking care of the fine resolution capability of Open-top light sheet fluorescence microscopy, set the imaging condition to be isotropic as closely as possible. Then blur only the axial plane of the image volume by convolving with a Gaussian blurring in the z-axis. The gaussian blurring is set with the standard deviation of 10.
- As we already know the PSF model of this image degradation, we can also compare the results to the ground-truth (before the image degradation) and also a classical deconovlution algorithm such as Richard-Lucy deconvolution algorithm, based on this PSF. What’s interesting is, while RL-deconvolved definitely show the effect of deconvolution, it also fails to reconstruct the fine details, which are mostly captured in the high frequency information that gets low-pass filtered out when we apply the blurring in the first place. In comparison, our deep-learning-based method excels at recontructing this lost high frequency information based on the local cues, as esepcially well demonstrated in the second column of the ROIs.
- PSNR, MS-SSIM, and BRISQUE metrics are calculated for 8 XZ-plane MIP images that are in depths of 17.5 μm. BRISQUE is a non-reference-aware quantification to measure how naturally perceptible the image is. A lower score means more naturally perceptible.
- The real benefit of the light sheet microscopy when it comes to testing with deconvolutional capability is that you can start from almost near-isotropy and adjust the degree of anisotropy by altering the imaging conditions, such as the thickness of the light-sheet.
- Imaging a large-scale sample at a fine resolution, such as imaging a whole mouse brain at a sub-micrometer resolution, may introduce an aggregate of unexpected image artifacts that are not noticeable in coarser resolution. In LSM microscopy, these artifacts are often by-products of a microscope that is not properly calibrated or installed. In particular, standard OT-LSM systems require the excitation path and the imaging path to be perpendicular to each other and may introduce distortions to the image quality that are uneven between XZ plane and Y Z plane, although this anisotropy could be relaxed by tightly focused excitation.
- Our system is now set up to include multiple imaging artifacts, which include not only the blurring artifacts by the spherical aberration that is caused by the refractive index mismatch between air and immersion medium, but other artifacts that span non-uniformly across the image space: for example, image doubling artifacts by a missed synchronization between the sweeping of the excitation laser and the rolling shutter of the detection sensor, or motion blur artifacts from physical sample drifts by the motorized stage.
- ROIs show the examples of artifacts that are most likely caused the axial under-sampling and the motion blur artifacts from physical sample drifts by the motorized stage.