A presentation of my MSc in Mathematical Sciences thesis at the African Institute of Mathematical Sciences (AIMS), Rwanda. This presentation explores the application of Deep Transfer Learning towards the diagnosis and classification of traditional pneumonia and pneumonia induced from COVID-19 using chest X-ray images.
Similar to Transfer Learning for the Detection and Classification of traditional pneumonia and pneumonia induced by the COVID-19 from Chest X-ray Images
Hierarchical matrices for approximating large covariance matries and computin...Alexander Litvinenko
Similar to Transfer Learning for the Detection and Classification of traditional pneumonia and pneumonia induced by the COVID-19 from Chest X-ray Images (20)
Transfer Learning for the Detection and Classification of traditional pneumonia and pneumonia induced by the COVID-19 from Chest X-ray Images
1. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning for the Detection and
Classification of traditional pneumonia and
pneumonia induced by the COVID-19 from Chest
X-ray Images
Yusuf Brima
Supervised by
Dr. Marcellin Atemkeng
Dr. Stive Roussel Tankio Djiokap
August 9, 2021
3. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
Figure 1: Map for Coronavirus-related incidence rate across the globe reported
to Johns Hopkins University on June 17, 2021 (source: Johns Hopkins
University).
4. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
I There are various molecular and serologic assays to test Severe
Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)
5. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
I There are various molecular and serologic assays to test the Severe
Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)
I Reverse Transcriptase-Polymerase Chain Reaction (RT–PCR) is the
laboratory standard for the SARS-CoV-2 testing.
6. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
I There are various molecular and serologic assays to test the Severe
Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)
I Reverse Transcriptase-Polymerase Chain Reaction (RT–PCR) is the
laboratory standard for the SARS-CoV-2 testing.
I RT-PCR has very high falsely negative rate.
7. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
I There are various molecular and serologic assays to test the Severe
Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)
I Reverse Transcriptase-Polymerase Chain Reaction (RT–PCR) is the
laboratory standard for the SARS-CoV-2 testing.
I RT-PCR has very high falsely negative rate.
I RT-PCR testing is very time-consuming. and presents a slew of
laboratory logistical challenges.
8. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Problem
Figure 2: Map of Coronavirus-related confirmed deaths per 100,000 population
across the globe with a total of 3,861,121 as reported to Johns Hopkins
University on June 17, 2021 (source: Wikipedia).
9. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Objectives
I To detect and classify traditional pneumonia and pneumonia induced
by the SARS-CoV-2 virus using Chest X-ray scans.
10. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Objectives
I To detect and classify traditional pneumonia and pneumonia induced
by the SARS-CoV-2 virus using Chest X-ray scans.
I For safe, accurate, less cumbersome and timely diagnosis.
11. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Research Objectives
I To detect and classify traditional pneumonia and pneumonia induced
by the SARS-CoV-2 virus using Chest X-ray scans.
I For safe, accurate, less cumbersome and timely diagnosis.
I Using Deep Transfer Learning
12. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
Figure 3: Hierarchy of Learning in Intelligent Machines.
13. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
UNKNOWN TARGET FUNCTION
TRAINING SAMPLES
LEARNING
ALGORITHM
HYPOTHESIS SET
FINAL HYPOTHESIS
Figure 4: A framework for supervised learning [1].
14. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
General Learning Setting
I A function estimation model such that we have a generator of
random vector x from a probability distribution P(x) which is
unknown.
I A process that maps the vector x to the output vector y according
to an unknown conditional probability distribution P(y|x)
P(y, x) = P(y|x)P(x). (1)
I Given a learning setting T ,
T := {H, P(Z), L} , (2)
where H ⊂ YX
, the hypotheses space of learnable models; P(Z) is
the probability measure of examples, that is:
Z := {(x1, y1), (x2, y2), . . . , (xm, ym)}
15. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
General Learning Setting
I L is the loss function such that:
L : H × Z → R
LCE = −
n
X
i=1
yi log(fθ(xi )),
Lmse =
1
2
n
X
i=1
(yi − fθ(xi ))
2
,
I Risk Functional R:
R(θ) =
Z
L(y, fθ(x))dP(x, y), ∀θ ∈ Θ. (3)
16. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
General Learning Setting
T is a learnable setting if the corresponding hypotheses space H is
learnable, if H is a VC dimension n, we therefore say T has a VC
dimension n.
f ∗
= min
fθ∈H
E[L(y, f (x))]
R̂m(θ) =
1
m
m
X
i=1
L(yi , fθ(xi )) (4)
The Empirical Risk Minimization (ERM) Induction Principle posits that
as m, the number of training samples gets larger,
R̂m(θ)
m→∞
= R(θ)
17. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
Generalization Error bound
The non-convexity of the loss objective makes deep learning a Hadamard
ill-posed problem. From a statistical learning theory standpoint, these
networks has a Generalization Error bound GE(θ) as stated thus:
GE(θ) = |R(θ) − R̂m(θ)|
18. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
Learning Representations and Task
Given K layers and an input vector x ∈ Rd
Let k = 1 . . . K and a non-linear activation function φ(.)
Thus, the transformation at layer k is: xk = φk (xk−1W k
) where
W k
= φk−1(xk−1W k−1
).
Generally, a deep neural network is:
Φ(x, W 1
, . . . , W K
) = φK (φK−1(. . . φ2(φ1(x, W 1
)W 2
) . . .)W K−1
)W K
),
φ(.) can be:
φ(x) = tanh(x),
φ(x) = max{0, x},
φ(x) =
1
1 + e−x
,
and many more.
19. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Learning Background
Learning Representations and Task
Given {(xi , yi )}
N
i=1, xi ∈ Rd
and yi ∈ {0, 1} for a classification and
yi ∈ R for regression.
Therefore,
Φ∗
(W) = argmin
{W k }K
k=1
L(Y , Φ(x, W 1
, . . . , W K
)) + λΘ(W 1
, . . . , W K
),
where λ > 0, and
Θ(W) =
K
X
k=1
||W k
||2
.
21. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
Formal Definition
D := {X, P(X)}
X = {x1, x2, x3, . . . , xn}, ∀xi ∈ X
Formal Definition
For domain D, a task is defined as:
T := {Y, P(Y |X)}
Y = {y1, y2, y3, . . . , yn}, ∀yi ∈ Y
23. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
The goal of Transfer Learning
Given
f : X 7→ Y where f ∈ H
∼
X = argmin
fθ∈H
{L(f (XSi
) 6= YSi
)}
And
RDT
:= P(η(XT ) 6= yT |
∼
X)
f ∗
= argmin
f ∈H
{RDT
(fθ(XT ), YT ,
∼
X)}
24. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
Input Layer ℝ¹²
∈ Hidden Layer ℝ¹²
∈ Output Layer ℝ⁴
∈
Output
Input
Conv-1
Conv-2
Conv-3
Conv-
4
...
Conv-n
Standard Convolutional Neural Network Architecture
1
2
...
2
n
Figure 5: Standard Convolutional Neural Network.
25. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
Convolution Operation for 2D
s(i, j) = (K ∗ I)(i, j) =
X
m
X
n
I(i + m, j + n)K(m, n),
Convolution Dimension
O =
W − K + 2P
S
+ 1.
27. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
Data Augmentation
zooming/flipping
/rotation etc
CNN feature
extraction layers
of ResNet50
Training the
Dense layers 1
2
3
4
Output
Input Covid-19
Lung
Opacity
Normal
(Healthy)
Viral
Pneumonia
Figure 7: The schematic represents the proposed system model.
28. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Transfer Learning
Data Augmentation
zooming/flipping
/rotation etc
CNN feature
extraction layers
of ResNet50
Training the
Dense layers 1
2
3
4
Output
Input Covid-19
Lung
Opacity
Normal
(Healthy)
Viral
Pneumonia
Figure 8: Deep Transfer Learning Stages
29. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
Normal Lung_Opacity COVID Viral Pneumonia
Class Type - Diagnosis
48.2%
28.4%
17.1%
6.4%
Number of Sample X-Ray Images per Class
Figure 9: A histogram of the distribution of the X-Ray Images per Class. The
total dataset is 18,865.
30. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
0 1 2 3 4 5 6
Min
0.0
0.2
0.4
0.6
0.8
Density
Images Colour Min Value Distribution by Class
Class
Lung_Opacity
Viral Pneumonia
Normal
COVID
(a)
0 50 100 150 200 250
Mean
0.000
0.002
0.004
0.006
0.008
0.010
Density
Image Color Mean Value Distribution by Class
Class
Lung_Opacity
Viral Pneumonia
Normal
COVID
(b)
160 180 200 220 240 260
Max
0.000
0.005
0.010
0.015
0.020
0.025
0.030
0.035
0.040
Density
Images Colour Max Value Distribution by Class
Class
Lung_Opacity
Viral Pneumonia
Normal
COVID
(c)
Figure 10: We present the min, mean and max RGB color intensity
distributions for the four X-ray image classes.
31. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
Figure 11: X-ray image format where the upper right zoomed illustration
indicates the RGB color channels.
32. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
Formal Definition
x =
1
Ic IhIw
Ic
X
i
Ih
X
j
Iw
X
k
xijk (8)
where Ic is the number of color channels, Ih is the height of the image
and Iw is the width of the image.
σ =
v
u
u
u
t
1
Ic IhIw
Ic IhIw
X
i
Ic
X
j
Ih
X
k
Iw
X
l
xjkl − x
2
(9)
33. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
COVID
Lung_Opacity
Normal
Viral Pneumonia
(a)
COVID
Lung_Opacity
Normal
Viral Pneumonia
(b)
Figure 12: A comparison illustrating a plot of the 3 colors channels (Left plot)
and single channel in (Right plot).
34. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
25 50 75 100 125 150 175 200 225
ImageChannelColourMean
20
40
60
80
100
Image
Channel
Colour
Standard
Deviation
MeanandStandardDeviationofImageSamples
Class
Lung_Opacity
ViralPneumonia
Normal
COVID
(a)
25 50 75 100 125 150 175 200 225
ImageChannelColourMean
20
40
60
80
100
Image
Channel
Colour
Standard
Deviation
MeanandStandardDeviationofImageSamples-10%ofData
(b)
Figure 13: A side-by-side comparison of the dataset clusters using image mean
and standard deviation.
35. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Dataset
25 50 75 100 125 150 175 200 225
Mean
20
40
60
80
100
Standard
Deviation
Lung_Opacity
25 50 75 100 125 150 175 200 225
Mean
ViralPneumonia
25 50 75 100 125 150 175 200 225
Mean
Normal
25 50 75 100 125 150 175 200 225
Mean
COVID
Figure 14: Individual class distributions for COVID-19 (far Left) to Healthy
(normal case, far Right). From the graph, Normal (healthy) and Lung Opacity
images have a similar cluster formation and pixel intensity distribution.
36. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Simulation Environment
I NVIDIA 537K80s, T4s, P4s and P100s Graphic Processing Unit
(GPU)
I Keras API (TensorFlow)
I Google Colaboratory (Colab) (Python 3.8x kernel)
37. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
Metrics
Accuracy =
TP
TP + FP + TN + FN
Sensitivity (r/sn) =
TP
TP + FN
Specificity (sp) =
TN
TN + FP
Precision (p) =
TP
TP + FP
F1 Score =
2
1
r + 1
p
= 2
rp
r + p
FPR =
FP
FP + TN
= 1 − sp
38. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
0 20 40 60 80 100
Epoch
0.0
0.2
0.4
0.6
0.8
loss
VGG19 Loss
Train loss
Validation loss
(a)
0 20 40 60 80 100
Epoch
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Accuracy
VGG19 Accuracy
Train accuracy
Validation accuracy
(b)
Figure 15: VGG-19 model was trained for 100 epochs.
39. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
Model Correct classification Incorrect classification
VGG-19 1988 127
DenseNet-121 1972 143
ResNet-50 1985 130
Table 1: A summary of total images classified correctly and incorrectly by
VGG-19, DenseNet-121, and ResNet-50 using a total test dataset of 2,115
images. Amongst the three models, VGG-19 demonstrated high accuracy of
XCR image classification with only 127 misclassifications.
45. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
References Dataset description Method Accuracy
[2]
5,184 chest X-ray images
that comprised 184 COVID-19 cases and 5000 normal cases
ResNet18 + ResNet50 +SqueezeNet
+ DenseNet-121
98%
[3]
18,567 X-ray images (COVID-19 = 140, normal = 8851
and Pneumonia = 9576)
ResNet-101 + ResNet-152 96.1%
[4] 320 images (COVID-19 = 160 and normal = 160)
Transfer learning with CNN
networks (Inceptionv3 and ResNet50)
99.01%
[5] 6926 images (COVID-19 = 2589 and normal = 4337) CNN 94.43%
[6] 5090 chest X-ray images (COVID-19 = 1979 and normal = 3111)
Fusion features (CNN+HOG)
+ VGG19 pre-train model
99.43%
Proposed
COVID-19 = 3616, Normal= 10192 ,
Lung Opacity = 6012, and Viral Pneumonia = 1345 images
ResNet-50V2
DenseNet-121
VGG-19
93.80%
93.24%
94.0%
Table 2: Comparative survey of literature results.
46. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
Figure 21: Activation map of ResNet-50 layer 48 before fine-tuning.
47. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Results
Figure 22: Activation map of ResNet-50 layer 48 after fine-tuning.
48. Problem Objectives Learning Methodology Dataset Simulation Results Conclusion References
Conclusion
I RT-PCR is error-prone and less accurate
I COVID-19 detection from Chest X-ray Images is a promising
diagnostic method.
I It is a fast, accurate and feasible solution especially for
asymptomatic carriers.
49. References
[1] Y. S. Abu-Mostafa, M. Magdon-Ismail, and H.-T. Lin, Learning
from data. AMLBook New York, NY, USA: 2012, vol. 4.
[2] S. Minaee, R. Kafieh, M. Sonka, S. Yazdani, and G. J. Soufi,
“Deep-covid: Predicting covid-19 from chest x-ray images using deep
transfer learning,” Medical image analysis, vol. 65, p. 101 794, 2020.
[3] N. Wang, H. Liu, and C. Xu, “Deep learning for the detection of
covid-19 using transfer learning and model integration,” in 2020
IEEE 10th International Conference on Electronics Information and
Emergency Communication (ICEIEC), IEEE, 2020, pp. 281–284.
[4] H. Benbrahim, H. Hachimi, and A. Amine, “Deep transfer learning
with apache spark to detect covid-19 in chest x-ray images,”
Romanian Journal of Information Science and Technology, vol. 23,
S117–S129, 2020.
[5] L. Duran-Lopez, J. P. Dominguez-Morales, J. Corral-Jaime,
S. Vicente-Diaz, and A. Linares-Barranco, “Covid-xnet: A custom
deep learning system to diagnose and locate covid-19 in chest x-ray
images,” Applied Sciences, vol. 10, no. 16, p. 5683, 2020.
50. References
[6] M. Ahsan, M. Based, J. Haider, M. Kowalski, et al., “Covid-19
detection from chest x-ray images using feature fusion and deep
learning,” Sensors, vol. 21, no. 4, p. 1480, 2021.