The document discusses deep learning applications for medical image analysis, including for diagnosis, surgical planning and guidance, and risk assessment. Specifically, it presents examples of using deep learning for tasks like classification, segmentation, detection, and pose estimation using medical images from modalities like ultrasound, X-ray, and video. Challenges in the field include limited datasets, variability in medical images, and privacy concerns, but deep learning methods are able to learn features directly from data to help with complex medical image analysis problems.

1. Deep learning for medical image analysis:
from diagnosis to intra-operative guidance
Oct 29th, 2021
Sara Moccia, PhD
The Biorobotics Institute
Department of Excellence in Robotics & AI
Scuola Superiore Sant’Anna (Pisa)
sara.moccia@santannapisa.it

2. Connections with DLI
Meetup: Prof. Emanuele Frontoni, head of
the VRAI lab at Università Politecnica delle
Marche
Master Deep Learning Italia

3. • BSc in Biomedical Engineering @POLIMI (IT)
• MSc in Biomedical Engineering @POLIMI (IT)
• Visiting PhD @ DKFZ, Heidelberg (DE)
• PhD in Biomedical Engineering @IIT/POLIMI (IT)
• PostDoc @UNIVPM (IT) – from 2018
• Visiting Researcher @ UMINHO, Braga (PT) – 2019
• Assistant Professor @ SSSA (IT) – 2021
• Adjunct Professor @UNIVPM (IT) - 2021
A little bit of myself

4. A little bit of myself
• BSc in Biomedical Engineering @POLIMI (IT)
• MSc in Biomedical Engineering @POLIMI (IT)
• Visiting PhD @ DKFZ, Heidelberg (DE)
• PhD in Biomedical Engineering @IIT/POLIMI (IT)
• PostDoc @UNIVPM (IT) – from 2018
• Visiting Researcher @ UMINHO, Braga (PT) – 2019
• Assistant Professor @ SSSA (IT) – 2021
• Adjunct Professor @UNIVPM (IT) - 2021

5. A little bit of myself
• BSc in Biomedical Engineering @POLIMI (IT)
• MSc in Biomedical Engineering @POLIMI (IT)
• Visiting PhD @ DKFZ, Heidelberg (DE)
• PhD in Biomedical Engineering @IIT/POLIMI (IT)
• PostDoc @UNIVPM (IT) – from 2018
• Visiting Researcher @ UMINHO, Braga (PT) – 2019
• Assistant Professor @ SSSA (IT) – 2021
• Adjunct Professor @UNIVPM (IT) - 2021

6. A little bit of myself

7. My team

8. Deep learning: a branch of artificial
intelligence
Deep learning relies on the ability of artificial networks to automatically
learn features (i.e., image characteristics) able to describe image content
Sara Moccia, sara.moccia@santannapisa.it

9. Deep learning: a branch of artificial
intelligence
Deep learning relies on the ability of artificial networks to automatically
learn features (i.e., image characteristics) able to describe image content
Sara Moccia, sara.moccia@santannapisa.it

10. Why deep learning for medical-image
analysis?
• Big data
• High complexity
• High variability (patients, surgeons,
devices, …)
à Deep learning learns rule from
data! We do not have to write
computationally-expensive
equations!
Sara Moccia, sara.moccia@santannapisa.it
[Maier-Hein et al., Nat. Biomed. Eng, 2017]

11. Different opportunities
Sara Moccia, sara.moccia@santannapisa.it
[Litjens et al., Med. Image Anal., 2017]

12. A huge variety of medical images
Sara Moccia, sara.moccia@santannapisa.it
• Ionizing / not-ionizing radiation
• Pre-operative / intra-operative
• Anatomical / functional
• Contrasted tissue (bones, vessels,
muscles, organs, …)
• Signal-to-noise ratio
• Depth of the contrasted tissues
• Static / dynamic
• …
[Litjens et al., Med. Image Anal., 2017]

13. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: classification
Tumor staging [Moccia et al., 2017]
Image tagging with tissues in the
field of view [Moccia et al., 2018]

14. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: segmentation
Scar tissue segmentation [Moccia et
al., 2019]
Surgical instrument segmentation
and generation [Colleoni, Moccia et al.,
2019]

15. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: detection
Polyp detection
Lung-nodule detection

16. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: informative frame
selection
Laryngoscopy [Moccia et al., 2019]
Ultrasound in rheumatology
[Fiorentino, Moccia et al., 2020]

17. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: pose estimation
Preterm-infants’ pose estimation
[Moccia et al., 2020]
Pose estimation in subjects using
smart walkers [Palermo, Moccia et al., 2021]

18. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: workflow analysis
Surgical workflow analysis
Surgical-skill analysis

19. Sara Moccia, sara.moccia@santannapisa.it
Different opportunities: image registration
CT/MRI registration
Pre/ intra-operative
registration

20. • Originally, low rate of digitalization/ storing
• No common repositories (few multi-center studies)
• Few datasets available
• Manual annotation requires high expertise
• Few rules (e.g., ontology) to define annotation vocabulary
• Data privacy, protection, GDPR, informed consent, …
• Ethical issues
• ….
Sara Moccia, sara.moccia@santannapisa.it
But a slow(er) evolution (than natural images)

21. Sara Moccia, sara.moccia@santannapisa.it
But a slow(er) evolution (than natural images)
~ million of images
~1000 classes
> 300k images
~ 80 classes

22. Sara Moccia, sara.moccia@santannapisa.it
But a slow(er) evolution (than natural images)
Few datasets available for the
medical field..
Luckily some International
Organizations (e.g., MICCAI,
ISBI, …) organize challenges!
Common size ~100 patients!
https://grand-challenge.org/challenges/

23. Sara Moccia, sara.moccia@santannapisa.it
In collaboration with UCL, we
have recently organized a
Grand Challenge.
Topic
Intra-operative segmentation
of fetoscopy videos.
But a slow(er) evolution (than natural images)

24. A gap between clinics and DL research
Sara Moccia, sara.moccia@santannapisa.it
Only few commercially-available
solutions
GI Genius™ intelligent endoscopy
module.
The first-to-market,
deep learning, computer-aided
polyp detection system.
99.7% sensitivity [Hassan et al.,
2020]

25. Screening and diagnosis
Surgical planning
Risk assessment
Robot-assisted surgery
A closer look to some specific applications

26. Screening and diagnosis
Surgical planning
Risk assessment
Robot-assisted surgery
A closer look to some specific applications

27. • Median nerve is a major peripheral nerve of the upper limb, with
both motor and sensory function
• Carpal Tunnel Syndrome (CTS): median nerve entrapment at
carpal tunnel level
• CTS diagnosis: generally based on patient clinical history and
symptoms
ULTRASOUND IMAGING: support tool to identify CTS
causes (e.g., flexor tendon tenosynovitis, wrist
synovitis, or crystal deposits)
Clinical measurement: median nerve cross-sectional
area and motion
Ultrasound rheumatological-image analysis

28. US challenges à - Intra- and inter- observer variability
- Operator dependency
- Poor image to noise ratio
- Possible presence in the images of shadows
and intensity inhomogeneities
- Different scanning protocols and various US
systems
- Morphological variability
- Easy to visualize at the proximal carpal
tunnel, but not at the middle or distal
carpal tunnel
Median nerve
challenges à
Ultrasound rheumatological-image analysis

29. US images dataset acquired at the Rheumatology Unit of “Carlo
Urbani” Hospital, in Jesi (Ancona, Italy)
151 US images from 53 patients
36 patients (95 images) à
TRAINING SET
9 patients (25 images) à
VALIDATION SET
8 patients (31 images) à
TESTING SET
US images acquisition:
- transverse scans at the carpal tunnel proximal inlet with the
same working system with at 6–18 MHz linear probe
- US images with size 606x468 pixels
- Median nerve contours manually annotated by expert
sonographers
Automatic median-nerve analysis
Di Cosmo, Mariachiara, et al. "Learning-Based Median Nerve Segmentation from Ultrasound Images for Carpal Tunnel Syndrome
Evaluation.” 43rd International Conference of the IEEE Engineering in Medicine and Biology Society (in press)

30. Mask R-CNN
Median Nerve
segmentation
- Input images size: 512x512 pixels
- Data augmentation on-the-fly à scale: [0.8, 1.2] rotation : [-10, 10] translation: [-20, 20] shearing : [-2, 2]
- Transfer learning with COCO dataset weights, initializing all layers of the model except for input layers of the network heads
- 256 anchors per image, with varying size (32, 64, 128, 256 and 512) and aspect ratios (1:1, 2:1, 1:2)
- Architectural changes: two more transposed layers inserted at the segmentation head
Backbone
(ResNet101 and Feature
Pyramidal Network)
Region
Proposal
Network
Segmentation head
Classification scores : C
(Median Nerve – Background)
Bounding box coordinates: 4*C
Fully
Connected
layers
Classification and bounding box regression heads
MASKS
US IMAGES
Roi Align
Mask for each of C classes
Automatic median-nerve analysis
Di Cosmo, Mariachiara, et al. "Learning-Based Median Nerve Segmentation from Ultrasound Images for Carpal Tunnel Syndrome
Evaluation.” 43rd International Conference of the IEEE Engineering in Medicine and Biology Society (in press)

31. RoiAlign
14x14x256
14x14x256
28x28x2
28x28x256
MASK28
RoiAlign
14x14x256
14x14x256
56x56x2
28x28x256 56x56x256
MASK56
RoiAlign
14x14x256
14x14x256
112x112x256 112x112x2
28x28x256
56x56x256
PROPOSED MODEL
Segmentation – increased mask resolution ablation study
Automatic median-nerve analysis
Di Cosmo, Mariachiara, et al. "Learning-Based Median Nerve Segmentation from Ultrasound Images for Carpal Tunnel Syndrome
Evaluation.” 43rd International Conference of the IEEE Engineering in Medicine and Biology Society (in press)

32. GT MASK
US IMAGE MASK28 MASK56 PROPOSED MODEL
Automatic median-nerve analysis
Di Cosmo, Mariachiara, et al. "Learning-Based Median Nerve Segmentation from Ultrasound Images for Carpal Tunnel Syndrome
Evaluation.” 43rd International Conference of the IEEE Engineering in Medicine and Biology Society (in press)

33. Evaluation Metrics (Mean ± Std)
MeanAP
(IoU@70)
Precision
(IoU@70)
Recall
(IoU@70)
DSC
0,903 ± 0,296 0,903 ± 0,296 0,903 ± 0,296 0,836 ± 0,279
- Median nerve always
identified in test set,
except for only 3 test
images from same subject
- Small number of false
positive and false negative
predictions thanks to the
increased output
resolution of
segmentation head
- Limited dataset size
- Still US images
- Misdetection associated
with high morphological
variability of nerve
section
Automatic median-nerve analysis
Di Cosmo, Mariachiara, et al. "Learning-Based Median Nerve Segmentation from Ultrasound Images for Carpal Tunnel Syndrome
Evaluation.” 43rd International Conference of the IEEE Engineering in Medicine and Biology Society (in press)

34. Head circumference (HC) is one of the basic biometric
parameters used to assess fetal well-being. It is manually
delineate by doctors every trimester of pregnancy during US
scan examination
Ι
ΙΙ
ΙΙΙ
Trimester
Ultrasound fetal-image analysis
Fiorentino, Mariachiara, et al. “A regression framework to head-circumference delineation from
US fetal images." Computer Methods and Programs in Biomedicine 198 (2021).

35. Developing an algorithm for automatic HC delineation
Grand Challenge HC18
TRAIN
999 images
800 x 540 pixel
pixel range [0.052,0.326] mm
TEST
335 images
800 x 540 pixels
https://hc18.grand-challenge.org/
AIM and dataset

36. Methods
Fiorentino, Mariachiara, et al. “A regression framework to head-circumference delineation from
US fetal images." Computer Methods and Programs in Biomedicine 198 (2021).

37. [1]
[2]
[3]
[4]
[5]
[1] T. L. van den Heuvel et al, Automated measurement of fetal head circumference using 2D ultrasound images, PloS One 13 (8) (2018) e0200412.
[2] Z. Sobhaninia, et al. Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning, in: 41st Annual In- ternational Conference of the IEEE Engineering in
Medicine and Biology Society, IEEE, 2019, pp. 6545–6548
[3] Z. Sobhaninia et al, Localization of fetal head in ultrasound images by multiscale view and deep neural networks, Available online at: arXiv preprint arXiv:1911.00908 (2019)
[4] Y. Rong et al, Deriving external forces via convolutional neural networks for biomedical image segmentation, Biomedical Optics Express 10 (8) (2019) 3800–3814.
[5] B. Al-Bander, et al,Improving fetal head contour detection by object localisation with deep learning, in: Annual Conference on Medical Image Understanding and Analysis, Springer, 2019, pp. 142–150
Results

38. Screening and diagnosis
Surgical planning
Risk assessment
Robot-assisted surgery
A closer look to some specific applications

39. • Birth before 37 gestational weeks
• Preterm births account for 11.1 % of the world’s births
[WHO, 2020]
Long-term complications
• Delayed language, cognitive deficits, behavioral and
motor disorders [Tucker et al., 2004]
Clinical need:
• Crucial necessity of continuously monitoring preterm
infants’ movement for evaluating the evolution of long-
term complications [Zuzarte et al., 2019]
Preterm birth

40. An intelligent crib for preterm infants
consists of a system that integrates
monitoring devices within a unique
framework:
Biometric
parameters
Cry analysis
Bilirubin analysis Motion analysis
The SINC project (EU-funded regional
projects)

41. Actual monitoring procedure:
Clinicians’ visual inspections at the crib-side in the NICUs
Drawbacks: discontinuous, inter-clinician-variable, time-consuming
Computer-based assisted methodologies:
Wearable sensors [Redd et al., 2019]
Drawbacks: pain, itch, hindering infants’ spontaneous motility
Camera sensors [Tsuji, et al., 2019, Marchi et al., 2020]
Drawbacks: semi-automatic approach, RGB-based
Movement monitoring solutions

42. Our solution

43. Our solution

44. CNNs
Detection
CNN
Regression
CNN
Spatial
Spatio-
temporal
Challenge:
Developing the first
system in the
literature for analyzing
depth frames and clips
to automatically and
continuously monitor
the movement of a
preterm infants’ limbs
in NICU
Moccia, Sara, et al. "Preterm infants’ pose estimation with spatio-temporal features."
IEEE Transactions on Biomedical Engineering 67.8 (2019): 2370-2380.
Infants’ limb pose estimation

45. M1-2D) Spatial-feature-based deep-learning
pipeline
Non maximum
suppression and bipartite
graph matching
Moccia, Sara, et al. "Preterm infants’ pose estimation with spatio-temporal features."
IEEE Transactions on Biomedical Engineering 67.8 (2019): 2370-2380.

46. Non maximum
suppression and bipartite
graph matching
Moccia, Sara, et al. "Preterm infants’ pose estimation with spatio-temporal features."
IEEE Transactions on Biomedical Engineering 67.8 (2019): 2370-2380.
M2-3D) Spatio-temporal feature-based deep-
learning pipeline

47. Frame
resolution
640 x 480
pixels
Length of
each
video
180 s
1000
annotated
frames per
infant
16 depth
videos
of 16 preterm
infants
Migliorelli, Lucia, et al. "The babyPose dataset."
Data in brief 33 (2020): 106329.
The babyPose dataset - v1

48. Migliorelli, Lucia, et al. "The babyPose dataset."
Data in brief 33 (2020): 106329.
The babyPose dataset - v1

49. Right arm Left arm Right leg Left leg
M1-2D RMSD 11.73 10.54 11.03 11.50
M2-3D RMSD 9.76 9.29 8.90 9.20 M1-2D M2-3D
Moccia, Sara, et al. "Preterm infants’ pose estimation with spatio-temporal features."
IEEE Transactions on Biomedical Engineering 67.8 (2019): 2370-2380.
Results – 2D vs 3D

50. Detection
CNN
Challenge:
Developing TwinEDA: a sustainable deep
learning algorithm for extending preterm
infants’ movement monitoring in scenarios
where computational resources are not
guaranteed
Innovative content with respect to the
previous work:
• Keeping almost unaltered the
performance of the M1-2D-detection
while drastically reducing the number
of trainable parameters, memory
requirements and testing times.
• Deploying the model on the Nvidia
Jetson Nano
Migliorelli, Lucia, et al. "TwinEDA: a sustainable deep-learning approach for limb-joint detection
in preterm infants’ depth images."
IEEE Journal of Biomedical and Health Informatics (submitted)
Infants’ limb pose estimation

51. M1-2D-detection M4-TwinEDA EDANet [Lo et al., 2019]
Migliorelli, Lucia, et al. "TwinEDA: a sustainable deep-learning approach for limb-joint detection
in preterm infants’ depth images."
IEEE Journal of Biomedical and Health Informatics (submitted)
M4-
TwinEDA
pipeline

52. Frame
resolution
640 x 480
pixels
Length of
each
video
180 s
1000
annotated
frames per
infant
27 depth
videos
of 27 preterm
infants
The babyPose dataset – version 2

53. EDANet
M4-TwinEDA
M1-2D detection
Network name Parameters
Testing on
Nvidia Jetson
M4-TwinEDA 2.66 M 20 FPS
M1-2D-detection 15.5 M
EDANet 620 k
Migliorelli, Lucia, et al. "TwinEDA: a sustainable deep-learning approach for limb-joint detection
in preterm infants’ depth images."
IEEE Journal of Biomedical and Health Informatics (submitted)
Results

54. Generating new data

55. Our solution
State of the art
Generating new data

56. From the clinician's side

57. Screening and diagnosis
Surgical planning
Risk assessment
Robot-assisted surgery
A closer look to some specific applications

58. Twin to Twin Transfusion Syndrome
Twin-to-Twin Transfusion Syndrome (TTTS), due to the presence of abnormal vascular anastomoses in the
monochorionic placenta can produce uneven blood flow between the fetuses.
of monochorionic
pregnancies
15%
risk of perinatal mortality of
one or both fetuses
90%

59. Twin to Twin Transfusion Syndrome
Limited field of view, low fetoscopic image quality and high inter-subject variability make the membrane
identification a challenging task.
Texture Paucity
Illumination variability
Colour variability

61. Twin to Twin Transfusion Syndrome
Adversarial Training Shape constraint
3D Convolution
Dense unit
Instance norm
Temporal consistency
Ease gradient flowing
Illumination adaptation
Casella, Moccia, et al. "A shape-constraint adversarial framework with instance-normalized
spatio-temporal features for inter-fetal membrane segmentation." Medical Image Analysis 70
(2021): 102008.

62. Twin to Twin Transfusion Syndrome
DSC 0.878 Accuracy 0.931
Casella, Moccia, et al. "A shape-constraint adversarial framework with instance-normalized
spatio-temporal features for inter-fetal membrane segmentation." Medical Image Analysis 70
(2021): 102008.

63. Screening and diagnosis
Surgical planning
Risk assessment
Robot-assisted surgery
Future steps

64. Future steps
Sara Moccia, sara.moccia@santannapisa.it
Our main current research directions:
• On-the-edge deployment à DL algorithms deployed on the imaging
device
• Semi-supervised learning à Attenuate the issue of poorly-available
manually-annotated datasets
• Distributed learning à no need to bring data out of the hospital

65. Future steps
Sara Moccia, sara.moccia@santannapisa.it
DL algorithms can potentially help clinicians by providing decision
support and context awareness
A strong collaboration between clinicians and engineers is needed
A look on ethics for AI

66. The aim of the Guidelines is to promote Trustworthy AI.
Trustworthy AI has three components, which should be met throughout the system's entire life
cycle:
(1) It should be lawful, complying with all applicable laws and regulations
(2) It should be ethical, ensuring adherence to ethical principles and values
(3) It should be robust, both from a technical and social perspective since, even with good
intentions, AI systems can cause unintentional harm.
Each component in itself is necessary but not sufficient for the achievement of Trustworthy AI.
Ideally, all three components work in harmony and overlap in their operation. If, in practice,
tensions arise between these components, society should endeavor to align them.
More info here:
• https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai
• https://ec.europa.eu/digital-single-market/en/high-level-expert-group-artificial-intelligence
Ethics guidelines for trustworthy AI

67. Ethics guidelines for trustworthy AI

68. Future steps
Sara Moccia, sara.moccia@santannapisa.it
DL algorithms can potentially help clinicians by providing decision
support and context awareness
A strong collaboration between clinicians and engineers is needed
A look on ethics for AI
The best is yet to come J

69. Deep learning for medical image analysis:
from diagnosis to intra-operative guidance
Oct 29th, 2021
Sara Moccia, PhD
The Biorobotics Institute
Department of Excellence in Robotics & AI
Scuola Superiore Sant’Anna (Pisa)
sara.moccia@santannapisa.it