This talk covers the fundamental building blocks of neural network architectures and how they’re used to tackle problems in modern natural language processing. Topics include an overview of language vector representations, text classification, named entity recognition, and sequence-to-sequence modeling approaches. Dr. Brown emphasizes the shape of these types of problems from the perspective of deep-learning architectures, which will help attendees successfully identify the most applicable neural network techniques to new problems they encounter.

1. Deep Learning and Modern Natural
Language Processing
Zachary Brown, Lead Data Scientist, S&P Global

2. Outline
• Neural Methods for Natural Language Processing
• Shapes of Natural Language Processing Tasks
• Perceptron Text Classification
• Local vs. Global Text Representations
• Contextual Representations and Sequence Modeling
2

3. Neural Methods for Natural Language
Processing

4. Neural Methods for Natural Language
Processing
• Natural Language Processing (NLP) has moved largely to neural
methods in recent years
4

5. Neural Methods for Natural Language
Processing
• Natural Language Processing (NLP) has moved largely to neural
methods in recent years
5
• Traditional NLP builds on years of research into
language representation
• Theoretical foundations can lead to model rigidity
• Tasks often rely on manually generated and
curated dictionaries and thesauruses
• Built upon local word representations

6. Neural Methods for Natural Language
Processing
• Natural Language Processing (NLP) has moved largely to neural
methods in recent years
6
• Few to no assumptions need to be made
• Active area of research; most open-source
• Ability to learn global and contextualized
word representations
• Purpose-built model architectures

7. Neural Methods for Natural Language
Processing
• Natural Language Processing (NLP) has moved largely to neural
methods in recent years
7
• Few to no assumptions need to be made
• Active area of research; most open-source
• Ability to learn global and contextualized
word representations
• Purpose-built model architectures

8. Shapes of Natural Language
Processing Tasks

9. Shapes of Natural Language Processing
Tasks
• A general task in natural language processing often takes the form:
9

10. Shapes of Natural Language Processing
Tasks
• For binary classification processes (relevance), our target is a single
number, often interpreted as a probability
10

11. Shapes of Natural Language Processing
Tasks
• For multi-class classification processes (type of text), our target is a
set of probabilities, one for each of the output classes
11

12. Shapes of Natural Language Processing
Tasks
• For sequential classification (e.g. LM, NER, POS) the target is a
probability of each class for each element in the input
12

13. Shapes of Natural Language Processing
Tasks
• In a traditional machine learning pipeline, vectorization (feature
engineering) process is often a (very) time consuming process
13
80-90%

14. Shapes of Natural Language Processing
Tasks
• A relatively small proportion of time is spent on the actual modeling
14
10-20%

15. Shapes of Natural Language Processing
Tasks
• Neural networks allow us to develop purpose-built architectures to
solve tasks, that learn the appropriate vectorization for a task
15
100%

16. Perceptron Text Classification

17. Perceptron Text Classification
• To introduce the shape of information as it flows through a neural
network, we'll first look at a network that only handles classification
17

18. Perceptron Text Classification
• For the vectorization, we'll assume that we've converted our text
into a vector using a count-based method like tf-idf
18
tf-idf

19. Perceptron Text Classification
• A perceptron is one of the simplest neural network architectures,
and is a good fit for this task
19

20. Perceptron Text Classification
• A perceptron is one of the simplest neural network architectures,
and is a good fit for this task
20
input
hidden
(linear)
activation
output

21. Perceptron Text Classification
• The hidden layer represents the weights that will be optimized by
the deep learning framework.
21
input
hidden
(linear)
activation
output
weights

22. Perceptron Text Classification
• If we want to change our task to multiclass classification, we can
simply change the size of our hidden layer (+ minor mods)
22

23. Perceptron Text Classification
• The result of this is that we now have a matrix of weights to optimize
23
weights

24. Local vs. Global Text Representations

25. Local vs. Global Text Representations
• Let's look back to the problem of creating a vector representation for
our text
25
tf-idf

26. Local vs. Global Text Representations
• Further, let's only consider the task of how we'd represent single
words or tokens as vectors
26
dog

27. Local vs. Global Text Representations
• Traditional approaches to word representations treat each word as
a unique entity
27

28. Local vs. Global Text Representations
• Traditional approaches to word representations treat each word as
a unique entity
28

29. Local vs. Global Text Representations
• Traditional approaches to word representations treat each word as
a unique entity
29

30. Local vs. Global Text Representations
• Traditional approaches to word representations treat each word as
a unique entity
30

31. Local vs. Global Text Representations
• Modern approaches move to a fixed dimensional vector size, with
dense vectors
31

32. Local vs. Global Text Representations
• Modern approaches move to a fixed dimensional vector size, with
dense vectors
32

33. Local vs. Global Text Representations
• Modern approaches move to a fixed dimensional vector size, with
dense vectors
33

34. Local vs. Global Text Representations
• There are a variety of frameworks available that allow for computing
these vectors in an unsupervised way
34

35. Contextual Representations and
Sequence Modeling

36. Contextual Representations and Sequence
Modeling
• Global word representations are a fantastic starting point for many
problems in NLP, but consider the following sentence
36
I'm going to book our vacation then relax and read a good book

37. Contextual Representations and Sequence
Modeling
• Global word representations are a fantastic starting point for many
problems in NLP, but consider the following sentence
37
I'm going to book our vacation then relax and read a good book

38. I don't really hate horror movies, but I hate comedies
Contextual Representations and Sequence
Modeling
• Global word representations are a fantastic starting point for many
problems in NLP, but consider the following sentence
38

39. I don't really hate horror movies, but I hate comedies
Contextual Representations and Sequence
Modeling
• Global word representations are a fantastic starting point for many
problems in NLP, but consider the following sentence
39
Context
Matters

40. Contextual Representations and Sequence
Modeling
• For modeling tasks where word ordering and context matter,
sequential models are often used. These tasks often take the
following shape:
40

41. Contextual Representations and Sequence
Modeling
• Recurrent neural networks are a type of neural network architecture
that naturally handles modeling sequential data
41

42. Contextual Representations and Sequence
Modeling
• This type of network generates a new output vector for each input in
a sequence, and also feeds that same information forward
42

43. Contextual Representations and Sequence
Modeling
• By feeding the information forward, each subsequent output vector
has contextual information encoded from the preceding words
43

44. Contextual Representations and Sequence
Modeling
• This type of architecture can be used to build language models,
where the task is to predict the next word in the sequence
44

45. Contextual Representations and Sequence
Modeling
• It can also be used for problems like named entity recognition
45
animalo o o o animal

46. Contextual Representations and Sequence
Modeling
• By taking the final vector in the sequence, you can perform tasks
like sentiment classification
46
positive

47. Contextual Representations and Sequence
Modeling
• For all of these different types of tasks, a network similar to the
perceptron can be placed at the end to carry out the final
classification of each word
47

48. Contextual Representations and Sequence
Modeling
• For all of these different types of tasks, a network similar to the
perceptron can be placed at the end to carry out the final
classification of each word, or the classification of the whole
sequence
48

49. Contextual Representations and Sequence
Modeling
• In a similar manner, these individual elements can be combined in a
variety of ways to tackle very complex tasks
49

50. Contextual Representations and Sequence
Modeling
• In a similar manner, these individual elements can be combined in a
variety of ways to tackle very complex tasks
50

51. Contextual Representations and Sequence
Modeling
• In a similar manner, these individual elements can be combined in a
variety of ways to tackle very complex tasks
51

52. Contextual Representations and Sequence
Modeling
• In a similar manner, these individual elements can be combined in a
variety of ways to tackle very complex tasks
52

53. Thank you.