Leveraging pre-trained models for Natural language understanding

1. Leveraging Pre-Trained
Language Models for
Natural Language
Understanding

2. HELLO!
I am Pooja Bhojwani.
I am a Senior Data Scientist
at Scotiabank, Toronto.
2

3. Workshop Assistance
Dhanush Dharmaretnam
Senior Data Scientist at
SGSCO Labs.
3

4. AGENDA
4
1. Introduction
2. Setup and Guidelines
3. Language Modelling
4. Application
5. Let’s do some hands-on!

5. Introduction
5
Source: https://www.youtube.com/watch?v=V8qrVleGY5U

6. AGENDA
6
1. Introduction
2. Setup and Guidelines
3. Language Modelling
4. Application
5. Let’s do some hands-on!

7. Setup and Guidelines
Setup
● Google Colab:
Download notebooks from
here:
● Please turn your videos off
to save bandwidth and
mute your microphones
Questions?
https://join.slack.com/t/tmls-nlpworksh
op/shared_invite/zt-j6fjxg6b-IeuLIa_Yp
LrYO4WbQVXcxg
Please make sure to ask all questions
on slack under the ‘questions’ channel.
Done with hands-on?
Go ahead and let us know in
‘classroom-poll’ channel.
7

8. AGENDA
8
1. Introduction
2. Setup and Guidelines
3. Language Modelling
4. Application
5. Let’s do some hands-on!

9. Reference:http://web.stanford.edu/class/cs224n/
(Note: To read more about language models, this course is highly recommended)
9

10. Evolution of Language Modelling
Reference: https://www.freshgravity.com/evolution-of-natural-language-processing/
10

11. Why focus on pre-trained models?
AI21 Labs estimated cost to include hyperparameter tuning and multiple runs for each
setting:
● $2.5k – $50k (110 million parameter model)
● $10k – $200k (340 million parameter model)
● $80k – $1.6m (1.5 billion parameter model)
11

12. Common Language models
12 Image Reference: http://jalammar.github.io

13. Transformer
13 Reference: http://jalammar.github.io/illustrated-transformer/

14. Self Attention
14 Reference: http://jalammar.github.io/illustrated-transformer/

15. BERT: Bi-directional Encoder
Representation from Transformer
● Trained Transformer Encoder Stack
15
Reference: http://jalammar.github.io/illustrated-bert/

16. BERT Training: Masked LM
16 Reference: https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html

17. BERT Training: Next Sentence
Reference: https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html17

18. AGENDA
18
1. Introduction
2. Setup and Guidelines
3. Language Modelling
4. Application
5. Let’s do some hands on

19. NLP Tasks
▪ Named Entity Recognition
▪ Question Answering
▪ Sentiment Analysis
▪ Text Summarization
▪ Text Generation
▪ Machine Translation
▪ Predict Missing Word
▪ Conversation
19

20. Named Entity Recognition
What is it?
▪ Extracting words or strings of interest and categorizing it
20

21. Fine Tuning BERT for NER
Reference: https://www.groundai.com/project/towards-lingua-franca-named-entity-recognition-with-bert/2
21

22. Question Answering
Citation: https://paperswithcode.com/task/question-answering
22

23. Fine Tuning Bert for Q and A
Read more here: https://laptrinhx.com/question-answering-with-a-fine-tuned-bert-3648357004/23

24. Sentiment Analysis
Reference: https://opendatahub.io/news/2019-09-04/sentiment-analysis-blog.html
24

25. Fine Tuning BERT for Sentiment
Analysis
Reference: https://opendatahub.io/news/2019-09-04/sentiment-analysis-blog.html
Try fine tuning: https://skimai.com/fine-tuning-bert-for-sentiment-analysis/
25

26. NLP Tasks
▪ Named Entity Recognition
▪ Question Answering
▪ Sentiment Analysis
▪ Text Summarization
▪ Text Generation
▪ Machine Translation
▪ Predict Missing Word
▪ Conversation
26

27. Text Summarization
Reference (Read more about it):
https://medium.com/lsc-psd/a-bert-based-summarization-model-bertsum-88b1fc1b3177
27

28. Text Generation
https://app.inferkit.com/demo
28

29. Machine Translation
Applications:
● Cross-lingual translation
● Text to text
● Speech to text
● Text to speech
29

30. NLP Tasks
▪ Named Entity Recognition
▪ Question Answering
▪ Sentiment Analysis
▪ Text Summarization
▪ Text Generation
▪ Machine Translation
▪ Predict Missing Word
▪ Conversation
30

31. Predicting missing word
31

32. Conversation
Reference:
https://www.information-age.com/chatbots-ai-conversa
tion-123479584/
https://research.aimultiple.com/business-chatbot/
32

33. AGENDA
33
1. Introduction
2. Setup and Guidelines
3. Language Modelling
4. Application
5. Let’s do some hands on

34. Preprocessing for Fine Tuning
Reference: http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/
34

35. THANKS!
Any questions?
You can find us at:
poojabhojwani10@gmail.com
dhanush.ai1990@gmail.com
35