Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Self-ensembling for visual domain adpation

157 views

Published on

NIP2017

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Self-ensembling for visual domain adpation

  1. 1. SELF-ENSEMBLING FOR VISUAL DOMAIN ADAPTATION ICLR 2018
  2. 2. MEAN TEACHERS ARE BETTERROLE MODELS Weight-averaged consistency targets improve semi-supervised deep learning results ANTTI TARVAINEN & HARRI VALPOLA
  3. 3. HOW THE MEAN TEACHER WORKS
  4. 4. dog dog cat horse horse dog cat cat cat horse dog horse cat horsecatdog
  5. 5. model’s prediction The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog cat horse
  6. 6. The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog cat horse model’s prediction
  7. 7. The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog cat horse model’s prediction
  8. 8. The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog dog cat horse
  9. 9. The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog cat horse
  10. 10. The truelabel pulls predictions to its direction. SUPERVISED LEARNING dog dog cat horse
  11. 11. WHAT ABOUT EXAMPLES WITHUNKNOWN LABELS? ??? dog cat horse
  12. 12. WHAT ABOUT EXAMPLES WITHUNKNOWN LABELS?dog How should we adjust the prediction? ??? cat horse
  13. 13. dog ??? cat horse We need two predictions: a student and a teacher.
  14. 14. dog We need two predictions: a student and a teacher. Then, hopefully, the student can learn something from the teacher. ??? cat horse
  15. 15. dog So how to do this? Two ways. ??? cat horse
  16. 16. dog We distort the student’s input, making its task more challenging. 1. MAKE THE TASK HARDER. cat horse
  17. 17. dog 1. MAKE THE TASK HARDER. cat horse We distort the student’s input, making its task more challenging. And then we train the harder task to predict the easier tasks’ output.
  18. 18. dog We maintain an exponential moving average of weights to create a better teacher. 2. MAKE THE TEACHER BETTER. ??? cat horse
  19. 19. dog We maintain an exponential moving average of weights to create a better teacher. A mean teacher. 2. MAKE THE TEACHER BETTER. ??? cat horse
  20. 20. dog We maintain an exponential moving average of weights to create a better teacher. A mean teacher. Then we let the student learn these better predictions. 2. MAKE THE TEACHER BETTER. ??? cat horse
  21. 21. dog Combining these two ways works even better. ??? cat horse
  22. 22. dog The student and the teacher improve each other in a virtuous cycle. ??? cat horse
  23. 23. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  24. 24. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  25. 25. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  26. 26. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  27. 27. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  28. 28. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  29. 29. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  30. 30. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  31. 31. dog ??? cat horse The student and the teacher improve each other in a virtuous cycle.
  32. 32. HOW TO STARTUSING THEMEAN TEACHER
  33. 33. θ classification cost prediction dog Take a supervised model. label input dog
  34. 34. θ θ’ classification cost prediction dog Make a copy of it. label input dog student teacher
  35. 35. θ θ’ exponentia l moving average classification cost prediction dog Update teacher weights after each training step. label input dog student teacher
  36. 36. θ θ’ exponentia l moving average classification cost consistency cost prediction dog Add a cost between the two predictions. label input dog student teacher
  37. 37. θ θ’ exponentia l moving average classification cost consistency cost prediction noise noise dog Maybe add some noise. label input dog student teacher
  38. 38. θ θ’ exponentia l moving average classification cost consistency cost prediction noise noise dog Start using it for semi-supervised learning. label input dog student teacher
  39. 39. RESULTS
  40. 40. 50000 images truck 4000 labeled
  41. 41. 0 2 4 12 10 8 6 testerrorrate 10,6 6,3 Virtual Adversaria l Training Mean Teacher (ResNet) using all labels 2,9 stateof theart CIFAR-10 WITH4000 LABELS
  42. 42. IMAGENET 2012 WITH10%OF THELABELS 1280000 images 10% labeled brambling
  43. 43. 0 40 30 20 10 top-5validation errorrate using all the labels Variational Auto-Encoder Mean Teacher (ResNet-18) 3,8 stateof the art 35,2 19,8 IMAGENET 2012 WITH10%OF THELABELS
  44. 44. 0 10 20 30 40 top-5validation errorrate using all the labels 35,2 9,1 Variational Auto-Encoder Mean Teacher (ResNet-152) (theseresults included in the Arxiv version of the paper) 3,8 stateof the art IMAGENET 2012 WITH10%OF THELABELS
  45. 45. WHY TO USE THEMEAN TEACHER
  46. 46. WHY MEAN TEACHER 1. It is easy to add to your model. 2. It can be adapted to different situations. 3. It gives good results.

×