This document provides an introduction to machine learning techniques for natural language processing. It discusses how machines can be trained to predict labels, reason, understand language, and generate text. Some challenges of language like ambiguity and subjectivity are also covered. Classical NLP techniques like using term frequencies and probabilistic models are compared to modern deep learning approaches that use word embeddings to represent text, recurrent neural networks to encode sentences, and attention mechanisms. Applications mentioned include text summarization, classification of toxic language, and analyzing language over time.

1. AI MADE EASY

2. An Introduction to
Machine Learning in
Natural Language Processing
The use of different techniques from computer science to
understand and manipulate human language and speech.
Isar Nejadgholi
isar@imrsv.ai

3. What do we want machines to
learn?
• Prediction or labeling
• Reasoning
• Understanding
• Language generation

4. LANGUAGE IS HARD
Irony and sarcasm
Elizabeth told Amanda
that she had a problem.
Reference resolution
The tank was full of water
I saw the military tank.
Lexical ambiguity
visiting relatives can
cause problems.
Syntactic (structural) ambiguity
I bought an iPhone a few
days ago. It was such a nice
phone. The touch screen
was really cool. The battery
life was not long, though and
my mother thought the
phone was too expensive.
Subjectivity of annotations
Tell me something
I don’t know.

5. APPLICATIONS
Natural
Language
Processing

6. Classical Natural Language Processing
Computational
Linguistics
Statistical and
Probabilistic
ML

7. TEXT REPRESENTATION IN CLASSICAL NLP
Term frequency vector
How to get a dense and informative representation?
… …

8. CLASSICAL NLP PIPELINE
• Frequency based representation
• Converting sparse representation to dense
vectors using SVD-based methods
• Statistical inference, probabilistic models,
similarity metrics
• Knowledge base of lexical rules and
word relations
• Character pattern matching

9. TOOLS

10. NLP Meets Deep Learning
Recent advances in NLP

12. DEEP LEARNING NLP MODEL
Embed, encode, attend, predict: The new deep learning formula for state-of-the-art NLP models
November 10, 2016, Matthew Honnibal

13. EMBED: WORD REPRESENTATION WITH WORD EMBEDDINGS
INPUT PROJECTION
OUTPUT
w(t)
w(t-2)
w(t+2)
w(t+1)
w(t-1)
Pretrained embeddings:
• Word2vec
• Glove
• Fasttext

14. REASONING WITH WORD VECTORS
Pair 1 Pair 2
China - Chinese Sri Lanka - Sri Lankan
Colombian - FARC Somalian - Alshabab
Roma - Hungarian Bahai - Iranian
Palestinian - Hamas Lebanon - Hezbollah
PRRA - Preremoval RPD - post hearing
Man is to woman as king is to ______?
Meaning(king) - meaning(man) + meaning(woman) = ?
(Mikolov et al. 2013)
Analogies in Legal Embedding

15. IMRSV WORD SUMMARIZER
Probabilistic PCA

16. ENCODE: SENTENCE REPRESENTATION WITH
RNNs
• Converts concatenation of word vectors (or bag of words) to a more meaningful sentence matrix
• Gets meaning of word in the context using memory and sequential modeling
• Can be bidirectional
• Popular models: GRU, LSTM, BiLSTM
• Can be used as decoder to generate text

17. Context vector or matrix
Predict
ATTEND: TEXT REPRESENTATION WITH
ATTENTION
• Reduces sentence matrix to sentence vector
• Without attention this conversion is done by averaging or max pooling
• Attention is a nonlinear weighted averaging of sentence matrix and context and learns what to keep

19. UNBALANCED AND MULTI-LABEL CLASSIFICATION PROBLEM

20. DATA PREPARATION
Cleaning
• Removing punctuations
• Autocorrect a set of critical words:
a$$ clowns -> ass clowns
sh!t -> shit
5hit -> shit
b l o o d y -> bloody
Augmentation with translation
• en -> fr -> en
• en -> de -> en
• en -> sp -> en

21. STRUCTURE
OF TOXICITY
CLASSIFIER

22. VIOLENCE IN ISIS FANBOY’S TWEETS

23. VIOLENT TWEETS OVER TIME
Explosion in Pakistan
ISIS commander killed
Explosion in Turkey
Numberofviolenttweets

24. WHERE DO I GET THE RIGHT DATA?
• Text is everywhere, but it is messy
• Volume, variety and velocity
• Annotation is subjective and expensive
Get messy!
Learn to clean!
Learn to augment!
Try to develop an intuition about your data!

25. Questions?