This document provides an overview of deep learning in natural language processing (NLP). It discusses traditional approaches like convolutional neural networks (CNNs) and recurrent neural networks (RNNs) that are used for tasks like sentiment analysis, machine translation, and question answering. It also covers innovative approaches like reinforcement learning, unsupervised learning, and memory augmented networks. Real-world applications of NLP are mentioned, such as search engines, voice assistants, translation, and sentiment analysis of social media. Challenges in NLP like the curse of dimensionality and evaluation are also briefly discussed.

1. History of Deep
Learning in NLP
Wen Li, Fangya Tan

2. Overview
• What is NLP for: Computer science discipline focuses on the analysis of human
languages
• NLP Application & Challenges
• Traditional Approaches: CNN, RNN
• Innovative Approaches: Video
• Q & A

3. What Is NLP
• Natural language processing (NLP) deals with building computational
algorithms to automatically analyze and represent human language.
• Aim: produce a condensed prestation of an input text that captures the core
meaning of the original text. Extractive vs Abstractive
• 7 Problems: Text Classification, Language Modeling, Speech Recognition, Caption Generation,
Machine Translation, Document Summarization, Question Answering
• Challenge: curse of dimensionality, multiple documentation, evaluation

4. Deep Learning in NLP
Application
Why should we care?
NLP in Daily Life:
• Google Search Engine
• Voice Technology: Amazon Alexa, Apple Siri
• Youdao voice translation
• Auto Email detection
• Marketing (Father’s day)
NLP in Future:
• Social Media (including emoji)
• Sentimental analysis: spam vs Non-spama

5. Model 1:
Convolutional Neural Network:
represents a feature function that is
applied to constituting words or n-
grams to extract higher-level features
sentiment analysis, machine translation, and
question answering, among other tasks.
Basic Steps:
1. tokenize: Sentence into words,
matrix of d dimension
2. Filter for feature map
3. Produce Final sentence
Challenge: Long distance dependency

6. Model 2:
Recurrent Neural Network
• The main strength of an RNN is
the capacity to memorize the results
of previous computations
• Inputs of arbitrary length so as to
create a proper composition of the
input.
• Tasks:
Machine Translation, Image
Captioning, Language Modeling
RNN is effective at processing
sequential information

7. Innovative Approaches Basic Traditional Approaches:
• RNN: recursive Neural Network
• Reinforcement Learning
• Unsupervised Learning
• Deep Generative Models
• Memory Augmented Network
Mutli-Modal Methods
Computer teaches themselves to
Recognize Cats
• Watch video of a generic cat or a specific
cat
• The feeling of petting a cat’s soft fur,
meow
• The letters ‘c’, ‘a’ and ‘t’
• Sometimes-selfish and largely
independent creatures

8. Reference:
• https://medium.com/dair-ai/deep-learning-for-nlp-an-overview-of-recent-
trends-d0d8f40a776d
• https://en.wikipedia.org/wiki/Curse_of_dimensionality
• https://arxiv.org/abs/1708.02709
• http://ruder.io/4-biggest-open-problems-in-nlp/
Thanks!