This document provides an overview of natural language processing (NLP) and the use of deep learning for NLP tasks. It discusses how deep learning models can learn representations and patterns from large amounts of unlabeled text data. Deep learning approaches are now achieving superior results to traditional NLP methods on many tasks, such as named entity recognition, machine translation, and question answering. However, deep learning models do not explicitly model linguistic knowledge. The document outlines common NLP tasks and how deep learning algorithms like LSTMs, CNNs, and encoder-decoder models are applied to problems involving text classification, sequence labeling, and language generation.

1. Using Deep Learning for Natural Language Processing (DNLP)
Yogesh Kulkarni
Introduction to NLP
Use case: Bank Call Center
Calling the Call Center
ˆ Calling to an IVR (Integrated voice response)
ˆ A prerecorded menu selection.
ˆ “Please press 1 for Account Details, Please press 2 for . . . ”
ˆ till it comes to your option.
ˆ towards end, somewhere, given access to a person to talk
to.
Boring? Annoying?
(Ref: Deep Learning and NLP A-Z - Kirill Eremenko)
Instead, how about typing/saying your query directly and getting
the answer right away?
Solution
Chatbots
ˆ Which problem of IVR it is solving?
ˆ Advantages?
ˆ Disadvantages?
ˆ Gaining popularity . . .
ˆ Many platforms
ˆ Companies in Pune?
The Giants are at it . . .
(Ref: Deep Learning and NLP A-Z - Kirill Eremenko)
ˆ Chatbots or QA systems, predominantly voice based,
ˆ Underlying processing is primarily Natural Language Pro-
cessing (NLP).
ˆ You can have your own chatbot, specific to you!!
ˆ NLP is the core skill needed.
Why so much popularity?
Chatbots are:
ˆ Autonomous and Always Available
ˆ Drive Conversation
ˆ Able to handle millions of requests, scalable.
Its hard to master language, and thus NLP.
NLP is AI-complete
ˆ “The most difficult problems in AI manifest themselves in
human language phenomena.”
ˆ Use of language is the touchstone of intelligent behavior.
ˆ Test for Intelligence - Turing Test
ˆ Alan Turing (1950) proposed a test of a machine’s capabil-
ity to perform human-like conversation.
Turing Test
A human judge engages in a natural language conversation with
two other parties, one a human and the other a machine; if the
judge cannot reliably tell which is which, then the machine is said
to pass the test.
Early Conversational Programs
ˆ ELIZA (by Joseph Weizenbaum), 1966.
ˆ A psychotherapist, but NO real understanding;
ˆ Simple pattern-matching to respond to user input to
canned responses

2. Loebner Prize
ˆ In 1990, Hugh Loebner started Turing Test competition
ˆ $100,000 will be awarded to the first bot that judges can-
not distinguish from a real human in a Turing test that
includes text, visual, and auditory input.
ˆ Nobody has won the grand prize yet.
ˆ 2016 (and 2013) year-wise top winner - Mitsuku.
https://www.facebook.com/mitsukubot
Why can’t we win the Grand Prize? What are the challenges?
Why Language is hard? What is Language?
What is Language?
Language Types
Natural Language
Artificial Language
Differences?
Language, simplistically
ˆ A vocabulary consists of a set of words
ˆ A text is composed of a sequence of words from a vocabu-
lary
ˆ A language is constructed of a set of all possible texts
NLP
ˆ NLP is Natural Language Processing, ie processing Natural
Langauge for some end-purpose in mind.
ˆ Inspite of usage of Natural Language for thousands of years,
why are we not able to process it well?
NLP Challenges
Paraphrasing
Paraphrasing: Different words/sentences express the same mean-
ing
ˆ Season of the year: Fall/Autumn
ˆ Book delivery time
– When will my book arrive?
– When will I receive my book?
Ambiguity
Ambiguity: One word/sentence can have different meanings
ˆ Fall
– The third season of the year
– Moving down towards the ground or towards a lower
position
ˆ The door is open
– Expressing a fact
– A request to close the door
Syntax and ambiguity
“I saw the man with a telescope.” - Who had the telescope?
Semantics
The astronomer loves the star.
ˆ Star in the sky
ˆ Celebrity
NLP Applications

3. Grammar
Spell and Grammar Checking
ˆ Checking spelling and grammar
ˆ Suggesting alternatives for the errors
Word Prediction
Word Prediction: Predicting the next word that is highly probable
to be typed by the user
ˆ Mobile typing
ˆ Search Engines
Information Retrieval
Information Retrieval: Finding relevant information to the user’s
query
Text Categorization
Text Categorization: Assigning one (or more) pre-defined cate-
gory to a text
Text Categorization
Summarization
Summarization: Generating a short summary from one or more
documents, sometimes based on a given query
Question answering
Question answering: Answering questions with a short answer
Question answering
Question answering: IBM Watson in Jeopardy

4. Information Extraction
Information Extraction: Extracting important concepts from
texts and assigning them to slot in a certain template
Information Extraction
Information Extraction: Includes named-entity recognition
Machine Translation
Machine Translation: Translating a text from one language to
another
Sentiment Analysis
Sentiment Analysis: Identifying sentiments and opinions stated
in a text
Sentiment Analysis
Restaurant/hotel recommendation, Product reviews
Sentiment Analysis
Text analytics in financial services
NLP in today’s time
Trends:
ˆ An enormous amount of information is now available in
machine readable form as natural language text (newspa-
pers, web pages, medical records, financial filings, product
reviews, discussion forums, etc.)
ˆ Conversational agents are becoming an important form of
human-computer communication
ˆ Much of human-human interaction is now mediated by
computers via social media
Collectively, this means that copious data is available to be used
in the development of NLP systems.
Level of difficulties
ˆ Easy (mostly solved)
– Spell and grammar checking
– Some text categorization tasks
– Some named-entity recognition tasks
ˆ Intermediate (good progress)
– Information retrieval
– Sentiment analysis
– Machine translation
– Information extraction
ˆ Difficult (still hard)

5. – Question answering
– Summarization
– Dialog systems
Langauge Representation:
How to make text
computable?
Document Representation & Language Model
ˆ How to represent a document?
ˆ Make it computable
ˆ How to infer the relationship among documents or identify
the structure within a document?
ˆ Knowledge discovery
ˆ Language Model and N-Grams
Bag-of-Words representation
Term as the basis for vector space
ˆ Doc1: Text mining is to identify useful information.
ˆ Doc2: Useful information is mined from text.
ˆ Doc3: Apple is delicious.
What are Word Vectors/Embeddings?
ˆ Word Embeddings are the texts converted into numbers
ˆ There may be different numerical representations of same
text.
ˆ Many Machine Learning algorithms and almost all Deep
Learning Architectures are incapable of processing strings
or plain text in their raw form.
ˆ They require numbers as inputs to perform any sort of job,
be it classification, regression etc. in broad terms.
ˆ So, for the computer to be able to ”understand” a vector
representation of a word is required.
Different types of Word Vectors
ˆ (Traditional) Frequency based Embedding:
– One-hot
– Count Vector
– TF-IDF Vector
– Co-Occurrence Vector
ˆ (Modern) Prediction based Embedding:
– Word2vec (Google)
– Global Vector Representations (GloVe) (Stanford)
One Hot
One-hot: Suppose our vocabulary has only five words: King,
Queen, Man, Woman, and Child. We could encode the word
‘Queen’ as:
No meaningful comparison possible.
Good Vector Representation
ˆ To have ”Semantic” (meaning-wise) representation, the
Similar words should be close to each other in the hyper
dimensional space.
ˆ Non-similar words should be far apart from each other in
the hyper dimensional space.
Good Vector Representation
ˆ Traditional One Hot Encoding:
– Apple = [1, 0, 0]
– Orange = [0, 1, 0]
– Plane = [0, 0, 1]
ˆ Very few cells participate in the representation.
Word2Vec
Word2vec (Google): a distributed representation of a word is used
and not sparse like One-Hot.
Represent in some abstract way the ‘meaning’ of a word.

6. Word Distributed Representation - Word2Vec
ˆ All vector cells participate in representing each word.
ˆ Words are represented by real valued dense vectors of sig-
nificantly smaller dimensions (e.g. 100 - 1000).
ˆ Intuition: consider each vector cell as a representative of
some feature.
Word Representations Comparison
Traditional Method - Bag of
Words Model
ˆ Uses one hot encoding
ˆ Each word in the
vocabulary is rep-
resented by one bit
position in a HUGE
vector.
ˆ For example, with a
vocabulary of 10000
words, and ”Hello” is
the 4th word in the
dictionary: 0 0 0 1 0 0
. . . . . . . 0 0 0 0
ˆ Context information is
not utilized
Modern - Word Vectors
ˆ Stores each word in
as a point in space,
represented by a vec-
tor of fixed number of
dimensions (generally
300)
ˆ Unsupervised, built
just by reading huge
corpus
ˆ For example, ”Hello”
might be represented
as : [0.4, −0.11, 0.55,
0.3 . . . 0.1, 0.02]
ˆ Context information is
utilized
Examples
Vectors for King, Man, Queen, & Woman:
Examples
Gender relation:
Plural relation:
Examples
Word pair relationships:
Examples
Country-capital city relationship:
The Power of Word2Vecs
ˆ They provide a fresh perspective to ALL problems in NLP,
and not just solve one problem.
ˆ Technological Improvement
ˆ Rise of deep learning since 2006 (Big Data + GPUs + Work
done by Andrew Ng, Yoshua Bengio, Yann Lecun and Geoff
Hinton)

7. ˆ Application of Deep Learning to NLP - led by Yoshua Ben-
gio, Christopher Manning, Richard Socher, Tomas Mikalov
ˆ The need for unsupervised learning . (Supervised learning
tends to be excessively dependent on hand-labeled data and
often does not scale)
NLP Activities: How to
process text?
Document/Section splitting
Document/Section splitting: Splitting a text into sections
Sentence splitting
Sentence splitting: Splitting a text into sentences
Tokenization
Tokenization
ˆ Process of breaking a stream of text up into tokens ( =
words, phrases, symbols, or other meaningful elements)
ˆ Typically performed at the “word” level
ˆ Not easy: Hewlett-Packard, U.S.A., in some languages
there is no “space” between words!
Stemming
Stemming
ˆ Reduces similar words to a given “stem”
ˆ E.g. detects, detected, detecting, detect : detect (stem).
ˆ Usually set of rules for suffix stripping
ˆ Most popular for English: Porter’s Algorithm
ˆ 36% reduction in indexing vocabulary (English)
ˆ Linguistic correctness of resulting stems not necessary (sen-
sitivities : sensit)
Lemmatization
Lemmatization
ˆ Uses a vocabulary and full morphological analysis of words
ˆ Aims to remove inflectional endings only
ˆ Return the base or dictionary form of a word, which is
known as the lemma.
ˆ E.g. saw : see, been, was : be
Part-of-speech tagging
Part-of-speech tagging: Assigning a syntatic tag to each word in
a sentence
Parsing
Parsing: Building the syntactic tree of a sentence
Parsing
((DaimlerChryslersshares)NP (rose(threeeights)NUMP (to22)P P −NU

8. Syntax Tree
Syntax: Sample English grammar
Named-entity recognition
Named-entity recognition: Identifying pre-defined entity types in
a sentence
Topic modelings
Topic modeling: Identifying structures in the text corpus
Word embeddings
Word embeddings: Compute a vector representing the distributed
representation for every word
Introduction to Deep NLP
Use of Deep Learning in
NLP
Let’s discuss
ˆ What is Deep Learning?
ˆ Why is it important?
ˆ How to apply Deep Learning to Natural Language Process-
ing?
(Ref (next few slides): Deep Learning for Natural Langauge Processing - Sihem Romd-
hani)
Machine Learning
(Ref: Deep Learning for Natural Language Processing - Sihem Romdhani)
Hand crafted features are needed in Machine Learning. E.g. for
Spam Detection, features could be presennce of BIG $ amounts,
FROM country, etc.
Deep Learning
(Ref: Deep Learning for Natural Language Processing - Sihem Romdhani)
Hand crafted features are NOT needed in Deep Learning. E.g.
for Object Detection, CNNs are come up with own features like,
edges, parts, etc.

9. Reasons for Applying Deep Learning to NLP
Automatic Representation Learning
(Ref: A not-so-short introduction to Deep Learning NLP - Francesco Gadaleta, PhD)
Reasons for Applying Deep Learning to NLP
Learning from unlabeled data
ˆ Typical traditional Machine Leaning based NLP requires
labeled training data.
ˆ DL based methods like Skip Gram, CBOW generate
word2vec by making labels from unlabeled data.
Reasons for Applying Deep Learning to NLP
Human language is seqeuntial and contextual.
(Ref: A not-so-short introduction to Deep Learning NLP - Francesco Gadaleta, PhD)
RNNs serve the purpose well.
Traditional NLP under threat?
ˆ Deep learning models have taken NLP by storm, achieving
superior results across many applications.
ˆ Many DL approaches do not model any linguistic knowl-
edge. They view language as a sequence of strings.
ˆ Is this the end of NLP as a separate discipline?
NLP
ˆ Rule based systems (since 1960s): Regex
ˆ Machine Learning (since late 1980s): Naive Bayes, SVM,
HMM
ˆ Deep Learning (since 2000)
The Promise of Deep NLP
(Ref: Deep Learning for NLU - Dr. David Talby)
Deep NLP Opportunities
(Ref: Deep Learning and NLP A-Z - Kirill Eremenko)
Deep NLP Algorithms
LSTM for sequence labelling
(Ref: Deep Learning for NLP - Yves Peirsman)
Application: named entity recognition
Encoder-Decoder Architecture
(Ref: Deep Learning for NLP - Yves Peirsman)
Applications: machine translation, text summarization, dialogue
modelling, etc

10. Traditional Named Entity Recognition
The problem
(Ref: Deep Learning for NLU - Dr. David Talby)
Traditional Named Entity Recognition
Conditional Random Fields (CRFs), “Classic” machine learning
approach
(Ref: Deep Learning for NLU - Dr. David Talby)
Deep Named Entity Recognition
LSTM
(Ref: Deep Learning for NLU - Dr. David Talby)
Deep Named Entity Recognition
Bi-LSTM
(Ref: Deep Learning for NLU - Dr. David Talby)
Summary of DL algos for NLP
(Ref: Engineering Intelligent NLP Applications Using Deep Learning Part 2 Saurabh
Kaushik)
Conclusion
ˆ Deep learning has simplified feature engineering in many
cases (it certainly hasn’t removed it)
ˆ Less feature engineering is leading to more complex ma-
chine learning architectures
ˆ Most of the time, these model architectures are as specific
to a given task as feature engineering used to be.
ˆ The job of the data scientist will stay sexy for a while (keep
your fingers crossed on this one).
What next?
ˆ Coursera : Dr Radev’s NLP course (https://www.coursera.org/l
language-processing)
ˆ Course: Deep NLP By Richard Sochar (Stanford)
ˆ Book: Natural Language Processing with Python

11. References
References
Many publicly available resources have been refereed for making
this presentation. Some of the notable ones are:
ˆ Introduction to Natural Language Processing - Dr. Mari-
ana Neves, SoSe 2016
ˆ Machine Learning for Natural Language Processing - Tra-
ian Rebedea, Stefan Ruseti - LeMAS 2016 - Summer School
ˆ CSC 594 Topics in AI - Natural Language Processing - De
Paul
ˆ Deep Learning for Natural Language Processing - Sihem
Romdhani
ˆ Notebooks and Material @ https://github.com/rouseguy/DeepL
Copyleft « Send suggestions to yogeshkulkarni@yahoo.com