MICCAI2018 presentation about why image translation (via distribution matching) should not be used for direct interpretation. ArXiv paper here: https://arxiv.org/abs/1805.08841

1. 1
Image translation (via distribution matching) should
not be used for direct interpretation.
Our statement:

2. Losses like in CycleGAN just match distributions
Training Data Model Output
2

3. They are very good at distribution matching
[CycleGAN, Zhu 2017]
[Karras, 2018]
3

4. But a bias in training data can lead to incorrect translation
T1 Transformed
Everyone is
so healthy!
T1 Real
Real
Image
Image
Translation/
Synthesis
Source
Image
4

5. Even with all the training data in
the world today.
Time t Time t+1
There will be new diseases
tomorrow that are out of distribution.
5

6. The generator learns
to match the target
distribution to fool the
discriminator
Generator/
Translator
Fake
Discriminator
Real or Fake ?
Fake Real
Real
What is image translation via distribution matching?
6

7. Model Breakdown
7

8. ● should produce examples in
● can be anything non-finite, like a Gaussian
● No guarantee mapping maintains phenotypes
Optimizing
GAN CycleGAN CondGAN L1
Optimizing
8

9. ● Add a reconstruction loss regularizer for the function
● Loss term still matches distribution
● No guarantee mapping maintains phenotypes
OptimizingOptimizing
GAN CycleGAN CondGAN L1
Optimizing
9
Cycle
Loss!
a b

10. ● is given paired examples allowing detection of what to preserve
● still plays a role in what learns
● No guarantee mapping maintains phenotypes
Optimizing
GAN CycleGAN CondGAN L1
Optimizing
10

11. ● should produce examples in
● Pixel-wise loss
● No guarantee mapping maintains phenotypes
Optimizing
GAN CycleGAN CondGAN L1
11

12. Visual Evaluation
12

13. CycleGAN results
% training data with tumor
13

14. 14

15. Example with a tumor from the holdout set
15

16. Quantitative Evaluation
16

17. healthytumorPredicted as a:TestdataishealthyTestdatahastumor
17

18. healthytumorPredicted as a:TestdataishealthyTestdatahastumor
18

19. 19
Image translation (via distribution matching) should
not be used for direct interpretation.
Our statement:

20. 1. How to guarantee image translation? (I doubt it)
2. Where should distribution matching be used in medical imaging?
a. Data augmentation (for classification, segmentation, registration)
b. Better features (for unsupervised learning)
c. To correct model predictions [Zhang MICCAI 2017]
Where do go from here?
20

21. Limitations
● We test only a subset of loss terms which compose most methods
● The synthetic BRATS 2013 data had tumors added to healthy brains (in real
data the entire brain is sick)
21

22. Pr. Yoshua
Bengio, PhD
Francis Dutil Martin Weiss
Tristan Sylvain Margaux Luck, PhD Assya Trofimov Vincent Frappier, PhD
Joseph Paul Cohen,
PhD
Shawn Tan
Sina Honari

23. Thanks!
23

24. Distribution Matching Losses Can Hallucinate Features
in Medical Image Translation
See us at poster M-60
https://arxiv.org/abs/1805.08841
Joseph Paul Cohen Margaux Luck Sina Honari
24