This document summarizes research on using generative adversarial networks (GANs) to augment medical image datasets. Specifically, it discusses using CycleGAN to generate additional images of rare disease classes, like tubular adenoma and sessile serrated adenoma, by transforming normal images. Evaluations found generated images were realistically classified by pathologists and inclusion of generated data improved classification performance of models trained on real data. In summary, GANs show promise for addressing limited rare class image data by generating synthetic augmented images.

1. MEDICAL IMAGE DATA AUGMENTATION
USING GANS
Dr. Hazrat Ali
Umea University, Sweden
27 May 2020
Hazrat Ali, MT-FoU, Umea University 1

2. ACKNOWLEDGMENTS AND CREDITS
The slides are based on Jerry Wei et al.,. Figures are
copied from the paper for demonstration purpose.
Copyrights are held by authors/publishers.
[1] Jerry Wei, et al., Generative Image Translation for Data Augmentation in Colorectal Histopathology
Images, Machine Learning for Health Workshop at NeurIPS 2019.
Hazrat Ali, MT-FoU, Umea University 2

3. TO TRAIN DEEP LEARNING MODELS
 Expectation
Positive
(Disease)
Normal
(No disease)
 Reality
Positive
(Disease)
Normal
(No disease)
Hazrat Ali, MT-FoU, Umea University 3

4. TO TRAIN DEEP LEARNING MODELS
 Expectation
Positive
(Disease)
Normal
(No disease)
 Reality
Positive
(Disease)
Normal
(No disease)
P
Normal
(No disease)
Hazrat Ali, MT-FoU, Umea University 4

5. TO TRAIN DEEP LEARNING MODELS
We need large datasets
Hazrat Ali, MT-FoU, Umea University 5

6. For rare diseases, training a deep learning model
can be very challenging
Increase the data by augmentation techniques
DataAugmentation
Rotation
Flip
Color jittering
Hazrat Ali, MT-FoU, Umea University 6

7. USING GENERATIVE MODELS
 Generate completely new images
 Hypothesis: The generation may work better if we can use data from one
class to generate data of the other class.
Hazrat Ali, MT-FoU, Umea University 7

8. USING GENERATIVE MODELS
 Generate completely new images
 Hypothesis: The generation may work better if we can use data from one
class to generate data of the other class
 Consider it a transformation
 And not base the generation on random noise input.
X Y
Hazrat Ali, MT-FoU, Umea University 8

9. USING GENERATIVE MODELS: CASE STUDY
 Normal colonic mucosa images to adenomatous preneoplastic
polyps images
 Use cycleGAN
 X: Normal Colonic Mucosa Images (Many)
 Y: Tubular ademona or sessile serrated adenoma (Few)
X Y
Hazrat Ali, MT-FoU, Umea University 9

10. MODEL DESIGN
Figure 1: Process for generating synthetic histopathology images of rare colorectal polyp classes. Path-Rank-Filter (i-ii)
enhances the adenomatous features in generated images by filtering the training data for CycleGAN for only images
with strong adenomatous features. Image: Jerry Wei et al., 2019 Hazrat Ali, MT-FoU, Umea University 10

11. KEY POINTS
 Filtering through Resnet
 GANs for transformation
 Evaluation through 4 pathologists
 Evaluations through resnet classifier (Can the augmented data improve the
classifier training?)
Hazrat Ali, MT-FoU, Umea University 11

12. KEY POINTS
 For training of GAN, use images with more prominent features
 Text from Jerry Wei et al.,
Hazrat Ali, MT-FoU, Umea University 12

13. DATA DISTRIBUTION
Normal vs Adenomatous images
 HP: Hyperplastic polp (Benign)
 NO: Normal colonic mucosa (Benign)
 TVA: Tubulovillous/villous Adenoma
 TA: Tubular Adenoma
 SSA: Sessile Serrated Adenoma
Hazrat Ali, MT-FoU, Umea University 13

14. DATA DISTRIBUTION
Normal vs Adenomatous images
 HP: Hyperplastic polp (Benign)
 NO: Normal colonic mucosa (Benign)
 TVA: Tubulovillous/villous Adenoma
 TA: Tubular Adenoma (only 15%)
 SSA: Sessile Serrated Adenoma (only 3%)
Hazrat Ali, MT-FoU, Umea University 14

15. USING GENERATIVE MODELS: CASE STUDY
 Normal colonic mucosa images to adenomatous preneoplastic
polyps images
 Use cycleGAN
 X: Normal Colonic Mucosa Images (Many)
 Y: Tubular ademona or sessile serrated adenoma (Few)
X Y
Hazrat Ali, MT-FoU, Umea University 15

16. USING GENERATIVE MODELS: CASE STUDY
 Normal colonic mucosa images to adenomatous preneoplastic
polyps images
 Use cycleGAN
 X: Normal Colonic Mucosa Images (9054 images)
 Y: Tubular ademona or sessile serrated adenoma (9054 images)
X Y
Hazrat Ali, MT-FoU, Umea University 16

17. EVALUATIONS
Hazrat Ali, MT-FoU, Umea University 17
Figure 3: CycleGAN’s generated images for different values of alpha. For instance, alpha =1/4 means
that the top 25% of images with the highest output probabilities from a ResNet were used to train
CycleGAN. TA: tubular adenoma, TVA: tubulovillous/villous adenoma, SSA: sessile serrated
adenoma. For TA and TVA, adenomatous features were enhanced at smaller values. Jerry Wei et al.,

18. EVALUATIONS
Evaluation by 4 pathologists – Tubular Adenoma Images
 Present 100 real and 100 generated (fake) images, p < 0.05 is statically significant
Hazrat Ali, MT-FoU, Umea University 18
Imagefrom:JerryWeietal.,
No. of pathologists
correctly predicting it

19. EVALUATIONS
Evaluation by 4 pathologists – Sessile Serrated Adenoma Images
 Present 100 real and 100 generated (fake) images, p < 0.05 is statically significant
Hazrat Ali, MT-FoU, Umea University 19
Imagefrom:JerryWeietal.,

20. EVALUATION
Does it improve the machine learning model?
 Train a Resnet classifier with and without generated data
Hazrat Ali, MT-FoU, Umea University 20
ResNet
X
Contains
augmented
data

21. EVALUATION
Hazrat Ali, MT-FoU, Umea University 21
Figure 5: A: AUCs of ResNets trained on real images with synthetic images from different generative models
given as additional training data. B: AUCs of ResNets trained without real images and with synthetic images
from different generative models as the only available training data. In both experiments, the ResNet that was
trained with CycleGAN’s synthetic images had the highest AUC.
Imagefrom:JerryWeietal.,

22. GENERATIVE ADVERSARIAL NETWORKS
 GAN
 G1(xa) = xb
Hazrat Ali, MT-FoU, Umea University 22
Imagefrom:XinYietal.,

23. GENERATIVE ADVERSARIAL NETWORKS
 CycleGAN
 G1(xa) = xb
 G2(xb) = xa
Hazrat Ali, MT-FoU, Umea University 23
Imagefrom:XinYietal.,

24. USEFUL RESOURCES
 Xin Yi et al., Generative Adversarial Network in Medical Imaging: A Review,
https://arxiv.org/abs/1809.07294
 Jerry Wei, et al., Generative Image Translation for Data Augmentation in Colorectal Histopathology
Images, Machine Learning for Health Workshop at NeurIPS 2019
 J. Zhu et al., "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks,"
2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 2242-2251.
 Talha Iqbal, Hazrat Ali, “Generative Adversarial Network for Medical Images (MI-GAN)”, Journal of
Medical Systems, vol 42, November 2018
 The GAN zoo: https://github.com/hindupuravinash/the-gan-zoo
Hazrat Ali, MT-FoU, Umea University 24