EMNLP reading group @TMU

1. Deep Multi-Task Learning with
Shared Memory
Pengfei Liu, Xipeng Qiu, Xuanjing Huang
EMNLP2016 reading group
presenter: ryosuke miyazaki

2. Abstract
Due to the large number of parameters neural models need a large-scale
corpus.
→ unsupervised pre-training is effective
Multi-task learning also improve the final performance.
This paper propose LSTM with external memory for multi-task learning.

3. Model: ME-LSTM
Key vector, Erase vector, Add vector

4. Model: ME-LSTM
Reading operation
K segment, M dimensions per one segment
,

5. Model: ME-LSTM
Deep Fusion strategy

6. Model: ME-LSTM
Writing operation

7. Two architectures
ARC-1 ARC-2

8. Training
Task-specific output layer
Linear combination of cost function
λm is the weights for each task m

9. Experiment: text classification

10. Result: Movie

11. Result: Product

12. Analysis: Visualize deep fusion gate
Sentiment score
Dimensions of
deep fusion gate gt
Activate → black

13. Analysis: Visualize deep fusion gate

14. Conclusion
・ This paper propose two deep architectures for multi-task
learning.
・ They design an external memory to store the knowledge
by related tasks.
・ Deep fusion strategy enabling the model to give shared
information.