1.
Multi-Task Learning for NLP
2017/04/17 Parsing Group
Motoki Sato

2.
What is Multi-task?
l Single task
2
l Multi task
Model 1
Input
(sentence)
POS
(task1)
Model 2
Input
(sentence)
Chunking
(task2)
Model
Input
(sentence)
POS
(task1)
Chunking
(task2)

3.
Multi-task learning Paper (1)
3
l (Søgaard, 2016) ACL 2016 short.
l Tasks:
– POS (low level task)
– Chunking (high level task)

4.
Multi-task learning Paper (2)
4
l (Hashimoto, 2016) arxiv.
l Tasks (many tasks):
– POS, Chunking, Dependency parsing,
– Semantic relatedness, Textual entailment

5.
Dataset
5
(Søgaard, 2016) (Hashimoto, 2016)
POS Penn Treebank Penn Treebank
Chunking Penn Treebank Penn Treebank
CCG Penn Treebank -
Dependency parsing - Penn Treebank
Semantic relatedness - SICK
Textual entailment - SICK

6.
(Søgaard, 2016)
(Søgaard, 2016)
POS Low level task
Chunking High level task
CCG High level task
6
Input Words and Predict Tag Examples:

7.
Multi-task for Vision?
l Cha Zhang, et al. “Improving Multiview Face Detection with Multi-Task Deep
Convolutional Neural Networks”
7
Share hidden layers
(shared representation)

8.
Multi-task for NLP?
l Collobert, et al. “Natural Language Processing (Almost) from Scratch”
8
Share
hidden
layers
Individual
layer for
each task

9.
(Søgaard, 2016) Outermost ver.
9
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
w0 w1 w2 w3
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
POS
Tag
Chunk
Tag
POS
Tag
Chunk
Tag
… …
3-th layer
2-th layer
1-th layer
Previous multi-task learning shared hidden layers,
Share
hidden
layers

10.
(Søgaard, 2016) lower-layer ver.
10
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
w0 w1 w2 w3
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Chunk
Tag
Chunk
Tag
… …
3-th layer
2-th layer
1-th layer
Previous multi-task learning shared hidden layers,
POS
Tag
POS
Tag
POS
Tag
POS
Tag

11.
Experiments
11
Low-level
task
High-level task
Single task
Multi task
It is consistently better to have POS supervision at
the innermost rather than the outermost layer.

12.
(Søgaard, 2016) Domain Adaptation
l What is domain adaptation?
12
Source
Trained
Model
Trained
Model
Target
(ex.) News domain (ex.) Twitter domain

13.
(Søgaard, 2016) Source Training
13
Source
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
w0 w1 w2 w3
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Chunk
Tag
Chunk
Tag
… …
3-th layer
2-th layer
1-th layer
POS
Tag
POS
Tag
POS
Tag
POS
Tag
WSJ newswire

14.
(Søgaard, 2016) Target Training
14
Target
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
w0 w1 w2 w3
Bi-LSTM Bi-LSTM Bi-LSTM Bi-LSTM
Chunk
Tag
Chunk
Tag
… …
3-th layer
2-th layer
1-th layer
Re-train
POS at
Target
Domain
POS
Tag
POS
Tag
POS
Tag
broadcast, weblogs domain
No Chunk
training at
Target
Domain

15.
Domain Adaptation Experiments
15
High-level task supervision in the source domain,
lower-level task supervision in the target domain.

16.
(Hashimoto, 2016)
16

17.
(Hashimoto, 2016)
17

18.
(Hashimoto, 2016)
18

19.
Training Loss for Multi Task Learning
l In (Hashimoto, 2016),
19
L2-norm regularization term
The embedding parameter after training the final
task in the top-most layer at the previous training
epoch.

20.
Dataset
20
(Søgaard, 2016) (Hashimoto, 2016)
POS Penn Treebank Penn Treebank
Chunking Penn Treebank Penn Treebank
CCG Penn Treebank -
Dependency parsing - Penn Treebank
Semantic relatedness - SICK
Textual entailment - SICK
Since (Søgaard, 2016) uses same dataset (same
input), they can use the sum of loss for multi-tasks.

21.
Catastrophic Forgetting
l “Overcoming Catastrophic Forgetting in Neural Networks”, James
Kirkpatrick, Raia Hadsell, et al. https://arxiv.org/abs/1612.00796
l https://theneuralperspective.com/2017/04/01/overcoming-catastrophic-
forgetting-in-neural-networks/
21