Allganize AI seminar - GPT3 and PET

1. All Tasks are Language Modeling
GPT-3 and PET
AI model team
Minwoo Park
minwoo@allganize.ai
AI Seminar - Paper review

2. Abstract
2
https://arxiv.org/pdf/2009.07118.pdf

3. GPT-3
3
https://arxiv.org/pdf/2005.14165.pdf

4. GPT-3: Introduction
4
https://arxiv.org/pdf/2005.14165.pdf
Recent NLP paradigm - Finetuning a pre-trained LM to downstream tasks
- has led substantial progress on many challenging NLP tasks
- entirely removing the need for task-specific architectures.
However,
- needs large amount of task-specific datasets
- needs task-specific fine-tuning
Humans do not require large supervised datasets to learn.
A brief directive / a tiny number of demonstrations is often sufficient.

5. GPT-3
5
My daughter successfully predicted the next word with one-shot learning.

6. GPT-3: LM Meta-learning
6

7. GPT-3: Larger model
7

8. GPT-3: Few-shot Inference
8

9. Traditional fine-tuning
9

10. GPT-3: Performance
10

11. GPT-3 shows how powerful language models can be.
Drawbacks:
- It requires a gigantic LM to work well, making it UNUSABLE in real-world.
- It does not scale to more than a few examples as the context window of most LMs is limited to a
few hundred tokens.
GPT-3: Summary
11

12. PET
12
Pattern Exploiting Training
Reformulating tasks as cloze
questions with regular gradient-based
finetuning.

13. PET: Semi-supervised Knowledge Distillation
13
1. Various patterns are used for
finetuning language models.
2. The ensemble of trained
language models annotates
unlabeled data.
3. A classifier is trained on the
so-obtained soft labeled
dataset.

14. PET: Without Knowledge Distillation
Without knowledge distillation
using unlabeled dataset, PET
performs even better.
But n*k times larger than the
distilled model. (k=number of
PVPs, n=number of LMs per each
PVP)

15. PET: Patterns
15

16. iPET
16
Some patterns perform worse than others -> The training set for final model may contain many
falsely labeled examples.

17. PET vs. GPT-3
17

18. PET: Summary
18
Small LM is possible to achieve few-shot performance similar to GPT-3.

19. 👍
- Unsupervised / semi-supervised
learning
- No / few labeled data
- Text in text out using Language
model
Conclusion: Future of NLP
19
👎
- Supervised learning
- Large amount of labeled data
- Task specific fine-tuning

20. All Tasks Are Language Modeling
20

21. Tool: Next Word Prediction
21
https://github.com/renatoviolin/next_word_prediction

22. Reference
Language Models are Few-Shot Learners
- https://arxiv.org/pdf/2005.14165.pdf
Exploiting Cloze Questions for Few Shot Text Classification and
Natural Language Inference
- https://arxiv.org/pdf/2001.07676.pdf
It’s Not Just Size That Matters: Small Language Models Are Also
Few-Shot Learners
- https://arxiv.org/pdf/2009.07118.pdf

23. minwoo@allganize.ai
Thank you
23