Uploaded byPOOJA BHOJWANI
Pre-Trained-Language-Models-for-NLU
This document provides an overview of leveraging pre-trained language models for natural language understanding. It introduces BERT and how it is trained using masked language modeling and next sentence prediction. It then discusses how BERT and other pre-trained models can be fine-tuned for downstream tasks like named entity recognition, question answering, sentiment analysis, text summarization, text generation, machine translation, predicting missing words, and conversation modeling. The document concludes with an overview of preprocessing text for fine-tuning these models.
More Related Content
Similar to Pre-Trained-Language-Models-for-NLU
Pre-Trained-Language-Models-for-NLU
- 1. Leveraging Pre-Trained Language Modelsfor Natural Language Understanding
- 2. HELLO! I am PoojaBhojwani. I am a Senior Data Scientist at Scotiabank, Toronto. 2
- 3. Workshop Assistance Dhanush Dharmaretnam SeniorData Scientist at SGSCO Labs. 3
- 4. AGENDA 4 1. Introduction 2. Setupand Guidelines 3. Language Modelling 4. Application 5. Let’s do some hands-on!
- 5.
- 6. AGENDA 6 1. Introduction 2. Setupand Guidelines 3. Language Modelling 4. Application 5. Let’s do some hands-on!
- 7. Setup and Guidelines Setup ●Google Colab: Download notebooks from here: ● Please turn your videos off to save bandwidth and mute your microphones Questions? https://join.slack.com/t/tmls-nlpworksh op/shared_invite/zt-j6fjxg6b-IeuLIa_Yp LrYO4WbQVXcxg Please make sure to ask all questions on slack under the ‘questions’ channel. Done with hands-on? Go ahead and let us know in ‘classroom-poll’ channel. 7
- 8. AGENDA 8 1. Introduction 2. Setupand Guidelines 3. Language Modelling 4. Application 5. Let’s do some hands-on!
- 9. Reference:http://web.stanford.edu/class/cs224n/ (Note: To readmore about language models, this course is highly recommended) 9
- 10. Evolution of LanguageModelling Reference: https://www.freshgravity.com/evolution-of-natural-language-processing/ 10
- 11. Why focus onpre-trained models? AI21 Labs estimated cost to include hyperparameter tuning and multiple runs for each setting: ● $2.5k – $50k (110 million parameter model) ● $10k – $200k (340 million parameter model) ● $80k – $1.6m (1.5 billion parameter model) 11
- 12. Common Language models 12Image Reference: http://jalammar.github.io
- 13.
- 14. Self Attention 14 Reference:http://jalammar.github.io/illustrated-transformer/
- 15. BERT: Bi-directional Encoder Representationfrom Transformer ● Trained Transformer Encoder Stack 15 Reference: http://jalammar.github.io/illustrated-bert/
- 16. BERT Training: MaskedLM 16 Reference: https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html
- 17. BERT Training: NextSentence Reference: https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html17
- 18. AGENDA 18 1. Introduction 2. Setupand Guidelines 3. Language Modelling 4. Application 5. Let’s do some hands on
- 19. NLP Tasks ▪ NamedEntity Recognition ▪ Question Answering ▪ Sentiment Analysis ▪ Text Summarization ▪ Text Generation ▪ Machine Translation ▪ Predict Missing Word ▪ Conversation 19
- 20. Named Entity Recognition Whatis it? ▪ Extracting words or strings of interest and categorizing it 20
- 21. Fine Tuning BERTfor NER Reference: https://www.groundai.com/project/towards-lingua-franca-named-entity-recognition-with-bert/2 21
- 22.
- 23. Fine Tuning Bertfor Q and A Read more here: https://laptrinhx.com/question-answering-with-a-fine-tuned-bert-3648357004/23
- 24.
- 25. Fine Tuning BERTfor Sentiment Analysis Reference: https://opendatahub.io/news/2019-09-04/sentiment-analysis-blog.html Try fine tuning: https://skimai.com/fine-tuning-bert-for-sentiment-analysis/ 25
- 26. NLP Tasks ▪ NamedEntity Recognition ▪ Question Answering ▪ Sentiment Analysis ▪ Text Summarization ▪ Text Generation ▪ Machine Translation ▪ Predict Missing Word ▪ Conversation 26
- 27. Text Summarization Reference (Readmore about it): https://medium.com/lsc-psd/a-bert-based-summarization-model-bertsum-88b1fc1b3177 27
- 28.
- 29. Machine Translation Applications: ● Cross-lingualtranslation ● Text to text ● Speech to text ● Text to speech 29
- 30. NLP Tasks ▪ NamedEntity Recognition ▪ Question Answering ▪ Sentiment Analysis ▪ Text Summarization ▪ Text Generation ▪ Machine Translation ▪ Predict Missing Word ▪ Conversation 30
- 31.
- 32.
- 33. AGENDA 33 1. Introduction 2. Setupand Guidelines 3. Language Modelling 4. Application 5. Let’s do some hands on
- 34. Preprocessing for FineTuning Reference: http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/ 34
- 35. THANKS! Any questions? You canfind us at: poojabhojwani10@gmail.com dhanush.ai1990@gmail.com 35