Pytorch Tutorial for End-to-end sequence labeling via bi-directional LSTM-CNN's-CRF

1. State-of-the-Art Named Entity
Recognition Framework
Anirudh Ganesh
Jayavardhan P Reddy
Speech and Language Processing
(CSE 5525) Spring 2018
Final Presentation
Prof. Eric Fosler-Lussier

2. Introduction
● The main goal with this tutorial is to develop a state of the art Named Entity
Recognizer.
● Along with this we also want the student to be able to understand and
implement a deep learning approach to solve a given problem.
● The tutorial is also structured in such a way as to provide the student with a
decent hands on experience in replicating a journal publication, given the
data and the model used.

3. Importance
● This is important since deep learning is gaining widespread traction for most
modern machine learning applications especially NLP.
● Replication of the results in such deep learning research publication is critical
for accelerated research growth.
● This is one crucial point that we wanted to tackle, that is replicability of deep
learning based publications and studies, because we feel that there is a huge
shortage of such work.

4. Resources used
● PyTorch
● NumPy
● Jupyter Notebook
● Python 3.5
● CoNLL 2003 dataset for NER
● Paper: End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF by
Xuezhe Ma, Eduard Hovy; https://arxiv.org/abs/1603.01354
● F1 score : 91.21

5. Main parts of the Architecture:
● Data Preparation
● Convolutional Neural Network (CNN) Encoder for Character Level
representation
● Bi-directional Long Short Term Memory (LSTM) for Word-Level Encoding
● Conditional Random Fields (CRF) for output decoding

6. Data Preparation
The paper*
uses the English data from CoNLL 2003 shared task
● Tag Update:
In the paper, the authors use the tagging Scheme ( BIOES ) rather than BIO (which
is used by the dataset). So, we need to first update the data to convert tag scheme
from BIO to BIOES. (Beginning, Inside, Outside, End, Single or Unit Length)
● Mappings:
Create mapping for Words-to-ids, Tag-to-ids, characters-to-ids
* “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF”, Xuezhe Ma and Eduard H. Hovy, CoRR (2016) abs/1603.01354

7. Word Embedding
● Using Pre-Trained Embeddings:
The paper uses Global Vectors (GloVe) 100 dimension vectors trained on the (
Wikipedia 2014 + Gigaword 5 ) corpus containing 6 Billion Words.
● Word embedding Mapping
Create mapping for words in vocabulary to word embeddings

8. Model Details (CNN Encoder for Character Level representation)
● Convolution layer on top that generates
spatial coherence across characters
● Maxpool extracts meaningful features out
of our convolution layer
● This now gives us a dense vector
representation of each word.
● This representation will be concatenated
with the pre-trained GloVe embeddings
using a simple lookup
Figure Illustrating the Character Embedding CNN layer.
(Adapted from the paper)

9. Model Details (Bi-LSTM for Word-Level Encoding)
● The word-embeddings that we generated in
our previous layer, we feed to a bi-directional
LSTM model
● The forward layer takes in a sequence of word
vectors and generates a new vector based on
what it has seen so far in the forward
direction
● This vector can be thought of as a summary
of all the words it has seen
● The backwards layer does the same but in
opposite direction
Figure Illustrating the Sequence labelling LSTM layer.
(Adapted from the paper)

10. Model Details (CRF Layer)
● Even if we capture some information from
the context thanks to the bi-LSTM, the
tagging decision needs to take advantage of
this.
● Since NER is heavily influenced by
neighboring tagging decisions.
● This is why we apply CRFs over traditional
softmax
● Given a sequence of words and a sequence
of score vectors, a linear-chain CRF defines
a global score such that it generates
sentence level likelihoods for optimal tags.
Figure Illustrating Conditional Random Fields (CRF) for
sequence tagging.
(Adapted from the paper)

11. Computing Tags
Recall that the CRF computes a conditional probability. Let y be a tag sequence
and x an input sequence of words. Then we compute maximum likelihood as,
Viterbi decode is basically applying dynamic programming to choosing our tag
sequence

12. Closing Comments
Sample Output from the given model

13. Experience of presenting to friends
● Initial few drafts
○ Too much time preprocessing data
○ PyTorch API
● Changes
○ Detailed comments explaining each step and the intuition behind it.
○ Detailed comments for the PyTorch functions
● Final Draft:
○ Takes a little longer than the intended time if the student solving it
doesn’t have sufficient background in deep learning and PyTorch.

14. Thank You!