This is my presentation slide at AAAI 2021

1. Semi-supervised Learning with
Variational Bayesian Inference and
Maximum Uncertainty Regularization
Kien Do, Truyen Tran, Svetha Venkatesh
Applied AI Institute (A2I2), Deakin University, Australia
1

2. Introduction
• Many big systems nowadays need a lot of labeled data to learn well.
• However, manual label annotation is expensive and time consuming.
• Semi-supervised learning (SSL) mitigates the need for labels by
leveraging similar patterns in unlabeled data to improve classification.
• Recent SOTA methods for SSL are mainly based on consistency
regularization.
2

3. Consistency Regularization for SSL
3

4. Two types of perturbation
4
data perturbation weight perturbation
Existing CR-based methods focus mainly on data perturbation

5. Some well-known CR based methods
• Pi-model:
• Mean Teacher:
5
is the exponential moving average of

6. Can we achieve a better perturbation of data?
• Under weak data perturbation, is often close to .
The classifier can only learn a locally smooth mapping from to .
• We want to be: i) not too close to , and ii) difficult for the
classifier to predict correctly.
• We choose to be a maximum uncertain (w.r.t. ) virtual point:
6

7. Approximating
• Recall that defined as follows:
• However, optimizing the above objective is difficult since it usually
has multiple local minima. To address this problem, we approximate
by optimizing the first-order Taylor expansion of :
where is the gradient of at .
7

8. Approximating (cont.)
• We can also approximate using projected gradient descent. The
update formula at step t+1 is given by:
• Solving the above equations give us:
8

9. Maximum Uncertainty Regularization
• The maximum uncertainty regularization (MUR) loss is defined as:
where is the maximum uncertain virtual point.
9

10. Weight Perturbation via Variational Bayesian
Inference
• Unlike data perturbation, weight perturbation is not straightforward
• We need some way to generate random weights
Variational Bayesian Inference (VBI) is a principled way to do that
• VBI objective:
10
Force weights to match the prior
Ensure faithful reconstruction

11. Consistency under Weight Perturbation
• The consistency loss under weight perturbation is given below:
where is the mean of .
11

12. Final Objective
The final objective when combining weight perturbation (via VBI) and
data perturbation (via MUR) is given by:
where can be an arbitrary consistency regularization based
method like Pi-model, Mean Teacher or ICT.
12

13. Results on CIFAR-10/100 and SVHN
13

14. Ablation Study
14
Different coefficient values of ( )

15. Ablation Study (cont.)
15
Performance with different radiuses Random perturbation vs. MUR

16. Visualization of most uncertain samples
16

17. Conclusion
• We have proposed two new consistency regularization based
methods: MUR and CWP
• MUR finds the most uncertain virtual point and forces its class
prediction to be similar to that of .
• CWP leverages Variational Bayesian Inference to perturb weights and
forces a noisy classifier to produce consistent outputs.
• Both MUR and CWP lead to better performances on SSL.
17

18. 18
Thank you for your attention!