The document discusses learning from unpaired data using deep learning techniques. It describes how collecting paired training data can be expensive, while unpaired data is easier to obtain. Several methods for learning from unpaired data are summarized, including unsupervised neural machine translation using dual models and shared encoders, CycleGAN for image-to-image translation using adversarial and cycle consistency losses, and unsupervised image captioning using object detectors and image descriptions. Applications to tasks like image dehazing and artifact reduction in medical images using disentanglement networks are also covered. The document concludes that learning from unpaired data can reduce data collection costs while achieving promising results.

1. 2020/12/14
1
Learning with Unpaired Data
Jiebo Luo
University of Rochester
Keynote at ICMLA 2020
December 14, 2020
Introduction
● The recent successes by deep learning have been largely driven by
annotated big data

2. 2020/12/14
2
Introduction
● Learning from more or less data
Learning from
Unpaired Data
Introduction
● The mapping from an input to a structured output is desired in ML
French→English
Une fusillade a eu lieu a`
l’aeroport international de
Los ´ Angeles.
There was a shooting in Los
Angeles International Airport.
Horse→Zebra Image→Caption
a black and white cat sitting
in a bathroom sink

3. 2020/12/14
3
Introduction
● The mapping from an input to a structured output is desired in ML & applications
Image Enhancement Image Dehazing Artifact Reduction
Introduction
● Usually, paired training data are collected to train such a mapping model
● Collecting paired data is expensive
Collecting paired data Possible Expensive
Machine Translation ✅ 💲
Horse→Zebra ❌
Image Captioning ✅ 💲
Image Enhancement ✅ 💲💲
Image Dehazing ❓ 💲💲💲
Artifact Reduction ❓ 💲💲💲

4. 2020/12/14
4
Introduction
● Unpaired data are much easier to collect
○ No annotation needed, or
○ Light-weight annotation
Introduction
● The new challenge: how to align the unpaired data
○ Reconstruction loss
○ GAN loss
Domain 1 Domain 2

5. 2020/12/14
5
Introduction
● Going into details
○ Machine Translation
○ Horse→Zebra
○ Image Captioning
○ Image Dehazing
○ Artifact Reduction
Unsupervised Neural Machine Translation (NMT)
● Key elements
○ Dual structure (same model for L1-L2 and L2-L1)
○ Shared encoder (same for L1/L2)
○ Fixed embeddings in the encoder (pre-trained cross-lingual embeddings vs. random initialization)
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. "Unsupervised neural machine translation." arXiv preprint arXiv:1710.11041
(2017).

6. 2020/12/14
6
On-the-fly back-translation
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. "Unsupervised machine translation using monolingual corpora
only." arXiv preprint arXiv:1711.00043 (2017).
Results
● BLEU scores on newstest2014
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. "Unsupervised neural machine translation." arXiv preprint arXiv:1710.11041
(2017).

7. 2020/12/14
7
Cycle GAN
● It is (nearly) impossible to collect paired images in some cases
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired image-to-image translation using cycle-consistent adversarial
networks." In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017.
Cycle GAN
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired image-to-image translation using cycle-consistent adversarial
networks." In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017.

8. 2020/12/14
8
Cycle GAN
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired image-to-image translation using cycle-consistent adversarial
networks." In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017.
● Adversarial Loss
● Cycle Consistency Loss
● Full Objective
Cycle GAN
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired image-to-image translation using cycle-consistent adversarial
networks." In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017.

9. 2020/12/14
9
Unsupervised Image Captioning
a. Supervised
b. Novel object
c. Domain adaptation
d. Pivot
e. Semi-supervised
f. Unsupervised
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.
Unsupervised Image Captioning
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.

10. 2020/12/14
10
Unsupervised Image Captioning
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.
Unsupervised Image Captioning
● Images are from MSCOCO dataset
● We crawl a large-scale image description corpus consisting of 2 million
natural sentences from Shutterstock to facilitate unsupervised image
captioning
● Object detectors trained on the OpenImage dataset are used
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.

11. 2020/12/14
11
Unsupervised Image Captioning
● Results
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.
Unsupervised Image Captioning
● Results
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. "Unsupervised image captioning." In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 4125-4134. 2019.

12. 2020/12/14
12
Image Dehazing
Atmosphere scattering model:
I(x) is the observed hazy image, J(x) is the scene radiance (haze-free image),
t(x) is the medium transmission map, A is the global atmospheric light.
Goal: recover J(x), t(x) and A from I(x).
Xitong Yang, Zheng Xu, and Jiebo Luo. "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training." In
AAAI, pp. 7485-7492. 2018.
Image Dehazing
● Uses synthetic hazy image pairs in
previous methods
● Generates realistic haze-free images
using only unpaired supervision in the
proposed method
Xitong Yang, Zheng Xu, and Jiebo Luo. "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training." In
AAAI, pp. 7485-7492. 2018.

13. 2020/12/14
13
Image Dehazing
● Synthetic datasets
○ D-HAZY dataset
○ NYU-Depth dataset
● Real image datasets
○ HazyCity dataset
○ Crawled photos of Beijing
○ Annotating hazy or not
○ 845 natural hazy images
○ 1891 haze-free images
Xitong Yang, Zheng Xu, and Jiebo Luo. "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training." In
AAAI, pp. 7485-7492. 2018.
Image Dehazing
● Results
Xitong Yang, Zheng Xu, and Jiebo Luo. "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training." In
AAAI, pp. 7485-7492. 2018.

14. 2020/12/14
14
Image Dehazing
● Results
Xitong Yang, Zheng Xu, and Jiebo Luo. "Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training." In
AAAI, pp. 7485-7492. 2018.
Artifact Reduction
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
Model
LossUnpaired
clinical data
● Existing models require paired data
for training, which are synthesized
● When applied to clinical data,
models trained on synthesized data
generalize poorly
● We aim to learn metal artifact
reduction (MAR) directly from
clinical data without using paired
data
input output
artifact-free

15. 2020/12/14
15
Artifact Disentanglement
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
decodingencoding
artifact reduced!
: artifact space
: content space
with artifact
bones, soft tissues,
lesions, implants, etc.
dark shadings,
streaks, noises, etc.
Artifact Disentanglement Network
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.

16. 2020/12/14
16
Artifact Disentanglement Network
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
content
encoder
content
decoder
Adversarial
Loss
artifact
reduced
with
artifact
without
artifact
Artifact Disentanglement Network
content
encoder
artifact
encoder
artifact &
content
decoder
Adversarial
Loss
with artifact
artifact
transferred
without
artifact
with
artifact
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.

17. 2020/12/14
17
Artifact Disentanglement Network
artifact &
content
decoder
self-
reconstruction
Reconstruction
Loss (L1)
artifact
encoder
content
encoder
with
artifact
with
artifact
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
Artifact Disentanglement Network
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
content
encoder
content
decoder
self-
reconstruction
Reconstruction
Loss (L1)
without
artifact
without
artifact

18. 2020/12/14
18
Artifact Disentanglement Network
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
-
-
Artifact Consistency
Loss (L1)
Artifact Disentanglement Network
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
artifact
reduced
Self-reduction
Loss (L1)
artifact
transferred
without
artifact
with
artifact
without
artifact

19. 2020/12/14
19
Quantitative Results on Synthetic Data
● Results
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.
Qualitative Results on Clinical Data
Haofu Liao, Wei-An Lin, S. Kevin Zhou, and Jiebo Luo. "ADN: Artifact disentanglement network for unsupervised metal artifact reduction." IEEE
Transactions on Medical Imaging 39, no. 3 (2019): 634-643.

20. 2020/12/14
20
Conclusions
● Learning with unpaired data can greatly reduce the effort in collecting data.
● Some level of explicit disentanglement of compounding components is the
key to improved performance beyond using cycle consistency.
● For machine translation and image captioning, the performance of unpaired
method is still lower than fully supervised methods.
● Compared with data synthesis, unpaired learning can achieve promising
results on many machine learning tasks.
Open Issues and Future Directions
● Learning with the combination of unpaired data and small paired data to
achieve better performance
● Going beyond text and image modalities, for example, to video and sound
modalities

21. 2020/12/14
21
Acknowledgments
Yang Feng (Google Cloud) Haofu Liao (AWS)
https://www.yangfeng.name http://liaohaofu.com
Visual Intelligence and Social Multimedia Analytics