This document discusses ELMo (Embeddings from Language Models), which are deep contextualized word representations from a bidirectional language model (biLM) pre-trained on a large text corpus. ELMo uses character convolutions and biLSTMs to produce word vectors that model both syntax and semantics, capturing polysemy. It then describes how ELMo vectors can be used as general purpose features for many NLP tasks like question answering and semantic role labeling and achieves state-of-the-art results on several benchmarks.

1. Deep contextualized
word representations
arXivtimes
Kamujun

2. • Abstract
• ELMo
•
•
•
•

3. Abstract
• (syntax semantics) (polysemy)
(ELMo:Embeddings
from Language Models)
• bidirectional language model
(biLM)
• NLP

4. ELMo

5. ELMo:Embeddings from
Language Models
•
biLM
biLSTM
cat
biLSTMbiLSTM
biLSTM biLSTMbiLSTM
small is
ELMo

6. biLM: Bidirectional language
models
•
•
•biLSTM biLSTMbiLSTM
•

7. •
ELMo
biLSTM biLSTMbiLSTM
biLSTM biLSTMbiLSTM
tk

8. ELMo
•
L2
heavy
weight
light
weight
:ELMoγ task
stask
: (softmax)

9. ELMo

10. • ELMo
RNN
ELMo Word Embedding
RNN
ELMo hidden layer output
ELMo ELMo

12. • 6 SOTA ELMo
ELMo

14. L2
•
ELMo L2
1

15. ELMo
• ELMo
SQuAD SNLI biRNN ELMo
SRL

16. ELMo
• GloVe
• biLM (ELMo)

17. • ELMo

18. • ELMo biLSTM
biLSTM

19. • (syntax semantics) (polysemy)
(ELMo:Embeddings
from Language Models)
• bidirectional language model
(biLM)
• NLP