Transfer learning in NLP has become important for business. Contextual word embeddings like ELMo and BERT provide context-dependent representations of words that address issues with earlier static embeddings. While shallow embeddings are easy to train and small in size, they cannot represent different meanings of words or handle out-of-vocabulary words. Contextual embeddings provide context-aware representations but require more complex neural architectures, while transformer-based models like BERT can be used for embeddings or fine-tuning but are very large in size and require significant computing resources to train.

1. Transfer learning in NLP
What has changed and why is it important for business?
Jakub Nowacki, PhD
Lead Machine Learning Engineer @ Sotrender
Trainer @ Sages

2. Transfer Learning
https://medium.com/@pierre_guillou/understand-how-works-resnet-without-talking-about-residual-64698f157e0
c
😭😎

3. Embeddings (Word2vec, FastText etc.)
https://towardsdatascience.com/word-embedding-with-word2vec-and-fasttext-a209c1d3e12c

4. So what is wrong with that?
[0.0, 0.0, …, 0.0]

5. Contextualized word-embeddings
http://jalammar.github.io/illustrated-bert/

6. Language model
https://medium.com/@plusepsilon/the-bidirectional-language-model-1f3961d1fb27
A statistical language model is a probability distribution over
sequences of words. Given such a sequence, say of length m, it
assigns a probability P(w_1,..., w_m) to the whole sequence.
Wikipedia: https://en.wikipedia.org/wiki/Language_model

7. ELMo
http://jalammar.github.io/illustrated-bert/

8. ELMo
http://jalammar.github.io/illustrated-bert/

9. LSTM vs Transformer
https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04
https://colah.github.io/posts/2015-08-Understanding-LSTMs/

10. BERT
http://jalammar.github.io/illustrated-bert/

11. BERT
http://jalammar.github.io/illustrated-bert/

12. Transfer learning in NLP
http://jalammar.github.io/illustrated-bert/

13. NLP’s ImageNet moment
http://ruder.io/nlp-imagenet/

14. Cyberbullying
https://kidshelpline.com.au/teens/issues/cyberbullying

15. PolEval 2019 Cyberbullying
http://poleval.pl/tasks/task6
Precision = 0.5
Recall = 0.5522
F1-score = 0.5248 (balanced)
Accuracy = 0.866
🏆
4th place
(theoretically, since we
didn’t take part)Word
embeddings
(FastText)
Flair
embeddings
(forward)
Flair
embeddings
(backward)
Stacked
embeddings
BiLSTM
(with dropouts)
Linear
Document
Harmful?

16. PolEval 2019 Cyberbullying
http://poleval.pl/tasks/task6
@anonymized_account Czyżby Madryt brał przykład z
Warszawy?
@anonymized_account @anonymized_account No to
Skończmy k**** z tym wersalem w j****** szczujni

17. The pros and cons
Shallow embeddings
(Word2Vec, FastText etc.)
Pros:
• Easy to train
• Small
• A lot of existing models
Cons:
• Same embedding for
different meaning
• May have issues with
inflection
• May have issues with
out-of-vocabulary (OOV)
words
Contextualized Embeddings
(ELMo, Flair etc.)
Pros:
• Embedding based on the
context
• Moderate size and
training speed
• Existing models
• No OOV problem
Cons:
• Require extra network
architecture
• LSTMs are rather slow
• Should be used along
with shallow embeddings
Transformer-based models
(e.g. BERT etc.)
Pros:
• Task-agnostic model
• Can be used as
embeddings or tuned
• Existing models
• Faster than LSTMs
• No OOV problem
Cons:
• Can be really large
• Hard to tune and even
harder to train (TPUs
almost a must)
• Multilingual versions are
very large
https://lilianweng.github.io/lil-log/2019/01/31/generalized-language-models.html

18. Thank you!
Questions?