The document discusses BERT (Bidirectional Encoder Representations from Transformers), a powerful pre-trained language model developed by Google in 2018 for natural language processing tasks. It describes BERT's transformer architecture with an encoder stack and its pre-training using a masked language modeling task. The document then explains how BERT can be fine-tuned on a downstream text classification task by adding a classification layer to the encoder output and using a softmax layer to classify texts into categories.

1. BERT MODEL FOR TEXT
CLASSIFICATION
Presented by,
Mokshadayini K S
TECHNICAL SEMINAR PRESENTATION 2021-22
Under the guidance of
Dr. Ravi Kumar V

2. TABLE OF
CONTENT
• What is BERT?
• Introduction
• Architecture
• Working
• Text Classification

3. INTRODUCTION
 In 2018, a powerful Transformer-based machine learning
model, namely, BERT was developed by Jacob Devlin and
his colleagues from Google for NLP applications.
 BERT is a very good pre-trained language model which
helps machines learn excellent representations of text
with respect to context in many natural language tasks
and thus outperforms the state-of-the-art.

4. BERT-
Bidirectional
Encoder
Representations
from
Transformers

5. BERT is basically an Encoder stack of transformer architecture. A
transformer architecture is an encoder-decoder network that uses self-
attention on the encoder side and attention on the decoder side.
BERTBASE has 12 layers in the Encoder stack while BERTLARGE has 24
layers in the Encoder stack.
These are more than the Transformer architecture described in the
original paper (6 encoder layers). BERT architectures (BASE and LARGE)
also have larger feedforward-networks (768 and 1024 hidden units
respectively), and more attention heads (12 and 16 respectively) than
the Transformer architecture suggested in the original paper.
It contains 512 hidden units and 8 attention heads. BERTBASE contains
110M parameters while BERTLARGE has 340M parameters.

6. BERT WORKING

7. Masked Language Model:
• In this NLP task, we replace 15% of words
in the text with the [MASK] token.
• The model then predicts the original words
that are replaced by [MASK] token.
• Beyond masking, the masking also mixes
things a bit in order to improve how the
model later for fine-tuning because [MASK]
token created a mismatch between training
and fine-tuning.
• In this model, we add a classification layer
at the top of the encoder input. We also
calculate the probability of the output using
a fully connected and a softmax layer.

8.  pre-trained BERT model for a binary text
classification task.
 In-text classification, the main aim of
the model is to categorize a text into one
of the predefined categories or labels.
TEXT CLASSIFICATION USING BERT

9. This model takes CLS token as input first, then it is
followed by a sequence of words as input. Here CLS is a
classification token.
It then passes the input to the above layers. Each layer
applies self-attention, passes the result through a
feedforward network after then it hands off to the next
encoder.
The model outputs a vector of hidden size (768 for BERT
BASE). If we want to output a classifier from this model
we can take the output corresponding to CLS token.

10. TEXT CLASSIFICATION

11. THANK YOU