Find the code on: https://github.com/anmold07/Graphical_Models/tree/master/CRF%20Learning Probabilistic Graphical Models (PGMs) provides a general framework to model dependencies among the output variables. Among the family of graphical models include Neural Networks, Markov Networks, Ising Models, factor graphs, Bayesian Networks etc, however, this project considers linear chain Conditional Random Fields to learn the inter-dependencies among the output variables for efficient classification of handwritten word recognition. Such models are capable of representing a complex distribution over multivariate distributions as a product of local factor functions. Find all the relevant code on: https://github.com/anmold-07/Graphical_Models

1. Modeling, Learning and Inference in
Linear-Chain Conditional Random
Fields (CRFs)
Anmol Dwivedi
RIN: 661982035
ECSE 6810: Probabilistic Graphical Models
Instructor: Dr. Qiang Ji

2. Outline
• Modeling Linear-Chain CRFs (LC-CRF) using relevant feature functions
• Challenges of parameter learning in LC-CRFs and equations
• Inference algorithms for LC-CRF
• Experiments and Results
All the above with application to hand-written word recognition*

3. • Consider the task of inferring hand-written words.
• Let denote observed characters of a word (input word is an image)
• Let denote labels to respectively.
• One approach is to train a Neural Network/logistic regression model for
each separately.
• However, makes the framework agnostic to dependencies present among
variables

4. • Solution: Linear-Chain CRFs
• Essentially an undirected graphical model over nodes
Choice for ?

5. • is modeled using Log-linear models with user defined features
Choice for feature function set ?

6. • Conditional features
• 26 x 32 x 2 possible
conditional feature
parameters
• Input feature for is 8x4
image with 0’s and 1’s.
• Pairwise features
• 26 x 26 possible
pairwise feature
parameters
• Singleton features
• 26 possible singleton
feature parameters
A total of (26 x 32 x 2) + (26 x 26) + (26) = 2366 parameters to learn from data!

7. • Given the discussed log-linear model and features, the goal is to learn all the
2366 parameters form training data.
Parameter Learning in CRFs: Training Phase
• Consider the training dataset we seeks to maximize
the conditional log likelihood.

8. Expected Data feature counts
Summation of indicator functions
Expected model feature counts

9. Inference in CRFs: Training Phase
A
B
C
D
Compute Beliefs
Compute all messages
Construct Junction Tree

10. After computing all messages and beliefs for training examples
Normalize beliefs to obtain in order to perform gradient
update on training data.
Multiply any two unnormalized messages into a clique
and marginalize to obtain the partition function
On new test data, perform max-sum inference using the same
beliefs and messages in order to infer the hand-written word.
Junction tree Max Sum belief propagation
Junction tree Sum Product belief propagation

11. 0 100 200 300 400 500 600 700 800 900 1000
Stochastic Gradient Descent Iterations
0
10
20
30
40
50
60
70
80
90
NegativeConditionalLog-Likelihood
CRF Training

12. Linear Chain CRF Character Level Accuracies: 61%
Test Dataset consists of 80 examples.
Linear Chain CRF Word Level Accuracies: 31%
Logistic regression Character Level Accuracies: 50.4%
Logistic regression CRF Word Level Accuracies: 12.5%

14. Thank You