Download as PDF, PPTX
![Self-Attention Layer
• Inputs: X = [x1, x2, … xm]
• Parameters: WQ, WK, WV (don’t share parameters)
28
This is called single-head self-attention](https://image.slidesharecdn.com/5bert-220609073059-7de45f84/85/5_BERT-pdf-28-320.jpg)
Uploaded byFEG
5_BERT.pdf
This document provides an introduction and overview of sequence-to-sequence (seq2seq) models, transformer models, attention mechanisms, and BERT for natural language processing. It discusses applications of seq2seq models like language translation and text summarization. Key aspects covered include the encoder-decoder architecture of seq2seq models, how attention improves seq2seq by allowing the model to focus on relevant parts of the context, and the transformer architecture using self-attention rather than recurrent layers. BERT is introduced as a bidirectional transformer model pre-trained on large unlabeled text that achieves state-of-the-art results for a range of NLP tasks. Code examples and homework suggestions are also provided.
5_BERT.pdf
- 1.
- 2. About me • Education •NCU (MIS)、NCCU (CS) • Work Experience • Telecom big data Innovation • AI projects • Retail marketing technology • User Group • TW Spark User Group • TW Hadoop User Group • Taiwan Data Engineer Association Director • Research • Big Data/ ML/ AIOT/ AI Columnist 2
- 3.
- 4. Code • Download code •https://github.com/orozcohsu/ntunhs_2022_01.git • Folder/file • 20220523_inter_master/run.ipynb 4
- 5. Code 5 Click button Open itwith Colab Copy it to your google drive Check your google drive
- 6. Seq2seq • Seq2seq isa family of machine learning approaches used for natural language processing (NLP) • Applications include • language translation • image captioning • conversational models • text summarization 6
- 7. Seq2seq • Seq2seq turnsone sequence into another sequence • sequence transformation • The primary components are one encoder and one decoder network • It often applies a recurrent neural network (RNN) or more often LSTM or GRU to avoid the problem of vanishing gradient 7
- 8. Seq2seq • The contextfor each sentence is the output from the previous step • The encoder turns each sentence into a corresponding hidden vector containing its context • The decoder reverses the process, turning the vector into an output item, using the previous output as the input context 8 Ref: https://en.wikipedia.org/wiki/Seq2seq
- 9.
- 10.
- 11. Attention for Seq2seq •Attention tremendously improves Seq2seq model • With attention , Seq2seq model does not forget context input • With attention, the decoder knows which part of context needs to be noticed • Consuming lots of computation 11
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20. Attention for Seq2seq 20 Ref:Attention and Augmented Recurrent Neural Networks (distill.pub)
- 21. Attention for Seq2seq 21 Ref:Attention and Augmented Recurrent Neural Networks (distill.pub)
- 22. Attention is allyou need 22 Ref: 1706.03762.pdf (arxiv.org) Publish 2017
- 23. Transformer model • Transformeris a kind of seq2seq model (encoder/decoder) • Transformer is NOT RNN • Transformer only has attention and dense layers (No RNN) • Very good performance in language translation • Transformer 23
- 24.
- 25.
- 26.
- 27.
- 28. Self-Attention Layer • Inputs:X = [x1, x2, … xm] • Parameters: WQ, WK, WV (don’t share parameters) 28 This is called single-head self-attention
- 29.
- 30.
- 31. Transformer model 31 Multi-head self-attention (Sameinput X, concatenation output of self-attention) Each word has 512-dim word embedding Input vs output are same 512-dim
- 32. Transformer model 32 One Blockof Decoder From encoder output
- 33.
- 34. Transformer model • Languagetranslation • Google translation • http://www.manythings.org/anki/ • Auto text generation • Demo – InferKit 34
- 35.
- 36.
- 37. BERT • Bidirectional EncoderRepresentation from Transformers (BERT) • BERT = Encoder of transformer • Learning from a large amount of text without annotation • For Chinese inputs, using character is better than vocabulary 37 Encoder
- 38. BERT 38 Google’s BERT -NLP and Transformer Architecture That Are Reshaping AI Landcape (neptune.ai) RNN based
- 39. Transformer model 39 (beta) DynamicQuantization on BERT — PyTorch Tutorials 1.11.0+cu102 documentation Small BERT: 24 layers Big BERT: 48 layers BERT (from scratch or transfer learning) (A classifier)
- 40. BERT • BERT isgood for NLU process • Text classification • Sentiment analysis 40
- 41. Exercise • Try torun this code 41 ktrain01 (Day master only) ktrain02 (International master class only)
- 42. Homework • Try touse BERT and build a text classification model • Prepare a multi-labels dataset 42 Hint: For multi-labels text classification, the amount of dataset is the key of performance ktrain_multi-labels_bad_performance