This document presents a study on physics-informed deep learning techniques for improving b-mode ultrasound imaging, focusing on artifact removal using various deep learning methodologies. It details the implementation of deep learning models like U-Net and CycleGAN for image quality enhancement and proposes a novel approach based on optimal transport theory. The findings suggest effective applications of these techniques for enhancing the performance of low-cost ultrasound systems.

2. Physics-Informed Deep Learning for
Efficient B-mode Ultrasound Imaging
2/28/2022
Shujaat Khan

3. About Me
I did my Ph.D. under the mentorship of Prof. Jong Chul Ye at Bio Ima
ge Signal Processing & Learning (BISPL) Lab, Dept. of Bio and Brain E
ngineeing KAIST.
My research interests include medical imaging, bioinformatics and ma
chine learning. I had a pleasure of working with highly talented peopl
e, and I managed to publish multiple national and international paten
ts, and number of good papers in reputable conference and journals i
ncluding MICCAI, ISBI, IUS, IEEE-TMI, IEEE-TUFFC, IEEE/ACM-TCBB and
IEEE-TNNLS, etc.
Based on my academic performance in 2019 I was awarded with BISP
L's best researcher award.
shujaat@kaist.ac.kr sites.google.com/view/shujaatkhan Shujaat Khan

4. Contents
I. Ultrasound Imaging (overview)
II. Deep learning for ultrasound image artifact removal
III. Physics-informed deep learning for ultrasound image artifact
removal
IV. Conclusion
4

5. I. Ultrasound Imaging (overview)
basics of ultrasound data acquisition and delay and sum (DAS) beamformer

6. Application Needs
 Better Image quality
 Reduce the number of measurement
- Ultra-fast US
- Portable US
- 3 dimensional US
Ultrasound Imaging Physics (overview) 6
Introduction

7. Acquisition modes
Focused Imaging
Planewave Imaging
Ultrasound Imaging Physics (overview) 7

8. Ultrasound Imaging Physics (overview) 8
delay-and-sum (DAS) beamforming
Time-of-Flight Correction
Depth
Scanlines
Log compressed
B-Mode Image
Envelope
RF-sum Signal
Envelope
detection
depth
Scanlines
Dynamic
Range
selection

9. II. Deep learning for ultrasound image artifact
removal
Deep learning for paired and unpaired datasets, geometry of deep learning,
data-driven cycleGAN

10. supervised learning
(paired input and target)

11. Image domain (deconvolution-filter)
Input
(DAS)
Label
(DeepBF
[1])
U-NET model is trained to mimic DeepBF (Deconvolution) results using image data.
Results (in-vivo) Results (phantom)
Input Label Output Input Label Output
Deep learning for ultrasound image artifact removal 11
[1] Khan et al, IEEE TUFFC, 2021

12. Image domain (speckle de-noising filter)
Input
(DAS)
Noise-Free
Label
[1]
U-NET model is trained to mimic noise-free results of [1] Zhu, Lei, et al, IEEE CVPR 2017
Input Label Output Label-Input Output-Input
Deep learning for ultrasound image artifact removal 12

13. unsupervised learning
(unpaired input and target)

14. Deep learning for ultrasound image artifact removal 14
Theory
The objective is to learn optimal path forA B
Zhu, Jun-Yan et al, IEEE ICCV, 2017

15. III. Physics-informed deep learning for
ultrasound image artifact removal
Image-domain learning for ultrasound artifact removal using deep neural network
Khan et al, IEEE TUFFC, 2021

16. Physics-informed deep learning for ultrasound image artifact removal 16
Variational formulation of cycleGAN
Consider an inverse problem, where a measurement 𝑦𝑦 ∈ Y from an unobserved image
𝑥𝑥 ∈ X is given by
Geometric view of unsupervised learning
To estimate 𝑥𝑥 ∈ X , here, the goal of optimal transport is to find 𝐺𝐺𝜃𝜃 and 𝐹𝐹𝜙𝜙 that minimize
the Wasserstein-1 distance between 𝜇𝜇 and 𝜇𝜇𝜃𝜃, and 𝜈𝜈 and 𝜈𝜈𝜙𝜙, respectively.
Sim, B. et al, SIAM JIS, 2020

17. Physics-informed deep learning for ultrasound image artifact removal 17
Optimal transport driven CycleGAN
According to Sim, B. et al, SIAM JIS, 2020, the primal form
of cycleGAN can be represented by a dual formulation
When the duality gap vanishes, we have
Generic CycleGAN
Physics-informed OT-CycleGAN
Deconvolution
model
Speckle
removal
model
Khan et al, IEEE TUFFC, 2021
By incorporating prior
knowledge

18. Applications of Physics-
Informed OT-CycleGAN in
ultrasound imaging

19. Physics-informed deep learning for ultrasound image artifact removal 19
Deconvolution
Improved resolution (FWHM)
Physics-informed OT-CycleGAN for
deconvolution
𝑦𝑦 = ℎ ∗ 𝑥𝑥 +N
Deconvolution model
Point spread function
(PSF)
Tissue reflectivity
function (TRF)
RF-Image
[1] Khan et al, IEEE TUFFC, 2021
[1]

20. Physics-informed deep learning for ultrasound image artifact removal 20
Speckle denoising
Physics-informed OT-CycleGAN
Multiplicative noise model for
speckle denoising
[1] Zhu, Lei, et al, IEEE CVPR 2017
[1]

21. low-contrast/low-resolution high-contrast/high-resolution
Physics-informed deep learning for ultrasound image artifact removal 21
Image quality enhancement of low-cost portable ultrasound system
low-cost
machine
high-end
machine
Geometric view of unsupervised learning

22. low-contrast
/low-resolution
high-contrast
/high-resolution
high-contrast
/low-resolution
Contrast
enhancement
Network
Super-resolution
Network
Physics-informed deep learning for ultrasound image artifact removal 22
Proposed method
Target-quality

23. - fixed weights
high-contrast/high-resolution
high-contrast/low-resolution
Unsupervised training Supervised training
:high-contrast low-resolution
:low-contrast/ low-resolution
Physics-informed deep learning for ultrasound image artifact removal 23
Proposed Self-Consistent CycleGAN

24. Axial
depth(mm)
36
27
18
9
0
45
Phantom
in
vivo
Lateral length(mm)
0 20
10 30 38.2
(a) Input image (b) histogram equalization (c) standard CycleGAN (d) our method
A B
C
Physics-informed deep learning for ultrasound image artifact removal 24
Results

25. Proposed
Alpinion
ECUBE
12R
High
quality
images
NPUS050
low-quality
images
phantom in vivo
Physics-informed deep learning for ultrasound image artifact removal 25
Results

26. IV. Conclusion
Key points and potential uses

27. 27
Conclusion
• Herein, some novel deep learning-based approaches are presented for the re
construction of high-quality B-mode ultrasound image.
• Unlike black-box approaches, proposed approach was derived based on the ri
gorous formulation using optimal transport theory and physics-informed deep
learning.
• The proposed methods can be applied for various US image enhancement ap
plications, providing an important platform for ultrasound artifact removal.

28. Thank You

29. Acknowledgements
• I would like to thank Prof. Jong Chul Ye and Jaeyong Huh for their guidance
and support necessary for this research.