Multi-Task Learning in Deep Neural Networks.pptx

1. Multi-Task Learning in
Deep Neural Networks
An Overview

2. • By Sebastian Ruder, Insight Centre for Data Analytics, Dublin.
• published in Jun 2017
• Cited by 1849
• Multitask learning (1997) By Rich Caruana
• His thesis on Multi-Task Learning helped create interest in a new subfield of
machine learning called Transfer Learning.

3. Traditional Machine Learning single task:
typically care about optimizing for a particular metric, whether this is a score on a
certain benchmark. In order to do this, we generally train a single model or an
ensemble of models to perform our desired task. We then fine-tune and tweak
these models until their performance no longer increases.

4. Multi-task learning (MTL):
is a machine learning approach in which we try to learn multiple tasks
simultaneously, optimizing multiple loss functions at once. Rather than training
independent models for each task, we allow a single model to learn to complete all
of the tasks at once.

6. Real world implementation
Tesla Auto Pilot
Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and
Prediction

21. MTL methods for Deep Learning
• Hard parameter sharing
• Soft parameter sharing

24. Why MTL work?
• Implicit data augmentation
• Attention focusing
• Eavesdropping
• Representation bias
• Regularization

25. By:
• Alex Kendall
• Yarin Gal
• Roberto Cipolla
Multi-Task Learning Using Uncertainty to Weigh Losses
for Scene Geometry and Semantics

26. Dataset
• https://www.cityscapes-dataset.com/
• 5 000 annotated images with fine annotations (examples)
• 20 000 annotated images with coarse annotations (examples)
• 30 classes
• 50 cities
• Several months (spring, summer, fall)

28. Example

31. Conclusion
• From natural language processing and speech recognition to computer vision and
drug discovery, multi-task learning (MTL) has led to success in a variety of machine
learning applications.
• This work aims to assist ML practitioners in implementing MTL by explaining how it
works and offering advice for selecting relevant auxiliary activities.
• Our understanding of tasks – their similarity, relationship, hierarchy, and benefit for
MTL – is still restricted, and we need to study them more them more to acquire a
deeper grasp of MTL's deep neural network generalization capabilities.

32. References
• Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv
preprint arXiv:1706.05098.
• Caruana, R. (1997). Multitask learning. Machine learning, 28(1), 41-75.
• Andrej Karpathy. (2019, 3 December). Tesla Autopilot and Multi-Task Learning for
Perception and Prediction. [Video]. YouTube.
https://www.youtube.com/watch?v=IHH47nZ7FZU
• Abu-Mostafa, Y. S. (1990). Learning from hints in neural networks. Journal of
Complexity, 6(2), 192–198. https://doi.org/10.1016/0885-064X(90)90006-Y
• Kendall, A., Gal, Y., & Cipolla, R. (2018). Multi-task learning using uncertainty to weigh
losses for scene geometry and semantics. In Proceedings of the IEEE conference on
computer vision and pattern recognition (pp. 7482-7491).