3. Basic Concepts about the new BERT
linguistic model
● Today, most advanced text models use transformers to teach how to
represent text.
● Ease of use - one output layer to existing neural architecture to obtain state-
of-art accuracy in several NLP tasks
● 2 categories of NLP tasks:
○ Holistic
○ Tokenized
● Masked Language Models
● 2 stages of BERT model training
● Performed very well on GLUE, SQuAD, and SWAG (natural language
understanding tasks)
4. BERT Retraining Methodology for Text
Problems
2 groups of methodologies:
● The use of pretrained models (transfer learning)
● Multitasking Learning
When adapting BERT to specific word processing tasks,
a special retraining technique is required. 3 types of
techniques:
1. Further pre-training
2. Retraining strategies
3. Multitasking Learning
“Catastrophic Forgetting”
Basic BERT model:
● An encoder with 12 transformer blocks,
12 attention areas, and a textual
representation dimension of 768.
● 512 tokens input, and outputs its vector
representation
● SEP, CLS tokens
5. improving the
subject-specific
classification of
texts using BERT
Traditional text embedding models
represent tokens as an embedding
Problems: ambiguity, subject-
specificity
General Universal Text Model, pre-
trained on a large corpus of general
purpose texts
Research is ongoing, potential as a
universal text model has not yet been
revealed