Deep contextualized word representations introduces ELMo (Embeddings from Language Models), word representations that model the complex characteristics of word use and how they vary across linguistic contexts. Evaluating ELMo on six NLP tasks establishes new state-of-the-art results. ELMo improves performance by adding contextualized representations from a bidirectional language model to the input and output layers of task-specific models.

1. Deep contextualized word
representations
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer
arXiv(NAACL2018)
Presenter: M2 Tomoya Ogata

2. Abstract
• Introduce word representation that models
• complex characteristics of word use (e.g., syntax and
semantics)
• how these uses vary across linguistic contexts
• The representation improve the state of the art
across six challenging NLP problems
1

3. CoVe (McCann et al., 2017)
2

4. Previous work
• different layers of deep biRNNs encode different
types of information
bi-LSTMbi-LSTM bi-LSTM
bi-LSTMbi-LSTM bi-LSTM
3

5. Bidirectional language models
Given a sequence of N tokens
LSTM LSTM LSTM
LSTM LSTM LSTM
• forward LM
• backward LM
• bi-directional LM
(𝑡1, … , 𝑡 𝑁)
4

6. ELMo
𝑠 𝑡𝑎𝑠𝑘
:softmax-normalized weigh
𝛾 𝑡𝑎𝑠𝑘
:scalar parameter
5

7. Apply ELMo to a task
fix the weight of biLM
RNN RNN
Word Embedding ELMo
tokens
contextualized
representation
ELMo
6

8. Evaluation
• the performance of ELMo across a diverse set of six benchmark NLP tasks
• simply adding ELMo establishes a new state-of-the-art result
7

9. L2 regularization
A small λ is preferred in most cases with ELMo
8

10. Where to include ELMo?
Input Layer
Output Layer
tokens
(𝑡1, … , 𝑡 𝑁)
9

11. What information is captured by
the biLM’s representations?
GloVe: concentrated in the sports-related senses of “play”
biLM: disambiguate both the part of speech and word sense
10

12. Word sense disambiguation
higher overall performance at the second layer
compared to the first Deep layer contains
semantic information
11

13. POS tagging
first biLM layer are higher than the top layer
Shallow layer contains
semantic information
12

14. Sample efficiency
After adding ELMo, a 98% relative decrease in the number of updates
needed to reach the same level of performance
13

15. Visualization of learned weigths
At the input layer, the task model favors the first biLSTM layer.
The output layer weights are relatively balanced.
14

16. Fine tuning
Fine tuning the biLM on task specific data typically resulted in
significant drops in perplexity.
15

17. Conclusion
• showing large improvements when applying ELMo
to a broad range of NLP tasks
• using all layers improves over-all task performance
16