This document summarizes several papers on semi-supervised learning methods published between 2017-2019. It describes techniques such as consistency regularization, which encourages consistent predictions from unlabeled data after augmentation; entropy minimization, which reduces the entropy of label distributions; and adversarial training, which adds random perturbations to make models smooth. Papers discussed include MixMatch, Mean Teachers, Virtual Adversarial Training, and S4L.

1. Yamato OKAMOTO
2019/12/15
Semi-Supervised Learning
(Survey)

2. Papers
method title year
consistency regularization MixMatch: A Holistic Approach to Semi-
Supervised Learning
2019
entropy minimization
temporal ensembles
Temporal Ensembling
for Semi-Supervised Learning manifold
2017
student-teacher model
Mean teachers are better role models:
Weight-averaged consistency targets
improve semi-supervised deep learning
results
2017
adversarial perturbation
Virtual Adversarial Training: A
Regularization Method for Supervised and
Semi-Supervised Learning
2019
self-supervision S4L: Self-Supervised Semi-Supervised
Learning
2019
consistency regularization
data filtering Unsupervised data augmentation for
consistency training
2019
consistency regularization

3. MixMatch: A Holistic Approach to Semi-Supervised
Learning (2019) *by Goodfellow
consistency regularization:
• a classifier should output the same class distribution for an unlabeled
example even after it has been augmented.
entropy minimization:
• to reduce the entropy of the label distribution, using the common
approach of adjusting the “temperature”
https://arxiv.org/pdf/1905.02249.pdf

4. Temporal Ensembling
for Semi-Supervised Learning manifold (2017)
consistency regularization under different dropout conditions:
• encourages consistent network output between two realizations of the
same input stimulus, under two different dropout conditions.
https://arxiv.org/pdf/1610.02242.pdf

5. Mean teachers are better role models:
Weight-averaged consistency targets improve
semi-supervised deep learning results (2017)
consistency regularization under different dropout conditions:
• define the consistency cost J as the expected distance between
the prediction of the student model and teacher model (for the same
data but applied different noise)
• After the weights of the student model have been updated, the teacher
model weights are updated as an average of the student weights.
• A training step with an unlabeled example would be similar
https://papers.nips.cc/paper/6719-mean-teachers-are-better-role-models-weight-averaged-consistency-targets-improve-semi-supervised-deep-learning-results.pdf

6. Virtual Adversarial Training: A Regularization Method
for Supervised and Semi-Supervised Learning (2019)
smoothness of the conditional label distribution given input:
• The loss of adversarial training by applying random perturbation
can be considered as a negative measure of the local smoothness
of the current model at each input data point x
• its reduction would make the model smooth at each data point.
https://arxiv.org/pdf/1704.03976.pdf

7. S4L: Self-Supervised Semi-Supervised Learning (2019)
Supervised-Learning
• predict image labels
Self-supervised-Learning
• predict which rotation degree was applied to rotated images
https://arxiv.org/pdf/1905.03670.pdf

8. Unsupervised data augmentation for
consistency training (2019)
Data Filtering
• a common technique for detecting out-of-domain data
• Use rained model to infer the labels of data in a large out-of-domain
dataset and pick out examples that the model is most confident about.
Unsupervised Data Augmentation
https://arxiv.org/pdf/1904.12848.pdf