The following presentation summarizes our work (Emad Aghajanzadeh, Diba Aminshahidi, Navid Ghassemi, and I) associated with ImageCLEF2021. It was an AI-medical competition where we were supposed to present models which are able to classify/predict the tuberculosis type based on a chest CT scan. The full text can be found at: https://bit.ly/3TnSuy3

1. Classification of Tuberculosis Type on CT Scans of
Lungs using a fusion of 2D and 3D Deep
Convolutional Neural Networks
Emad Aghajanzadeh, Behzad Shomali, Diba Aminshahidi, Navid Ghassemi
CLEF2021 Working Notes
09/22/2021

2. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
2
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

3. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
3
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

4. Our team
Emad Aghajanzadeh Behzad Shomali Diba Aminshahidi Navid Ghassemi
Computer Engineering
Undergraduate Student
Computer Engineering
Undergraduate Student
Computer Engineering
Alumna
Computer Engineering
Graduate Student
4
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

5. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
5
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

6. Ranking
The best run ranked:
• 4th based on the Kappa (value of 0.181)
• 3rd based on the Accuracy (value of 0.404)
6
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

7. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
7
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

8. Task / data
• Classification of CT scans based on their TB type
• 5 existing classes
• 917 training, 421 test images
8
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

9. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
9
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

10. Preprocessing
• HU values are scaled to be in range [0, 1]
• Keep 50 middle slices
• Resize slices to 100x100
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 10

11. Preprocessing
There were two challenges while training:
• Small dataset
• Data duplication / augmentation
• Rotate by a degree in range [-5, 5]
• Zoom by a ratio of 1.25
• Resize to original size
• Imbalanced dataset
• Variable penalties
• Remove some samples
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 11

12. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 12

13. Proposed methods (settings)
• Adam Optimizer (default parameters)
• Train/validation split ratio of 0.2
• Batch size:
• 32 for 2D CNNs
• 8 for 3D CNNs
• Loss function
• Combination of Cross-entropy and weighted Kappa
• Change the multipliers over time
13
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

14. Proposed methods (2D CNN)
14
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

15. Proposed methods (2D CNN)
• Assign each CT label to all its slices
• Feed each slice to 2D CNN separately
• A vector of 50 labels for each CT
• Pick the most appearing label
• Pick the label with highest probability
15
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

16. Proposed methods (3D CNN)
16
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

17. Proposed methods (3D CNN)
• Input: 100x100x50
• Capture the spatial information
17
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

18. Proposed methods (2D CNN + RNN)
18
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

19. • Used the best-trained 2D CNN
• Feeding the:
• Predicted labels by the 2D CNN to RNN
• Features extracted by the last layer of the 2D CNN to RNN
• The results were almost the same to 2D CNN
Proposed methods (2D CNN + RNN)
19
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

20. Proposed methods (Fusion of 2D & 3D CNN with RNN)
20
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

21. Proposed methods (Fusion of 2D & 3D CNN with RNN)
• 3D CNN can capture the spatial information
• 2D CNN can better extract features in each slice
• Concatenate the features extracted by each model
• Feed the feature vector to RNN
• The result was similar to that of the 3D CNN
• Implies the more contribution of the 3D model
21
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

22. Proposed methods (Fusion of 2D & 3D + manual)
By investigating the confusion matrix, we found out:
• 3D models can better classify the last 3 classes
• 2D models is good at classifying the first 2 classes
• Select the final prediction manually
• Have a list of predicted labels for each CT scan
• Select based on the models with lower variance
22
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes

23. Proposed methods (results)
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 23
Evaluation result on 2D CNN
Evaluation result on 3D CNN
Final evaluation result on test data

24. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 24

25. Conclusion & Future works
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 25
• Proposed different approaches
• 2D CNN didn’t perform well
• Apply a smarter voting mechanism
• Define a fixed size window which moves through the labels vector
• Use more complicated data augmentation approaches
• Train a separate binary classification

26. Contents
• Our team
• Ranking
• Task / data
• Preprocessing
• Proposed methods
• Conclusion & Future works
• Questions
Classification of Tuberculosis Type on CT Scans of Lungs using a fusion of 2D and 3D Deep Convolutional Neural Networks CLEF2021 Working Notes 26

27. Thanks for your
consideration
{emad.aghajanzadeh, behzad.shomali, d.aminshahidi, navidghassemi}@mail.um.ac.ir