.

1. Copyright Notice
These slides are distributed under the Creative Commons License.
DeepLearning.AI makes these slides available for educational purposes. You may not use or distribute
these slides for commercial purposes. You may make copies of these slides and use or distribute them for
educational purposes as long as you cite DeepLearning.AI as the source of the slides.
For the rest of the details of the license, see https://creativecommons.org/licenses/by-sa/2.0/legalcode

2. deeplearning.ai
Vector Space
Models

3. Outline
● Vector space models
● Advantages
● Applications

4. Why learn vector space models?
What is your age?
How old are you?
Where are you heading?
Where are you from?
Different meaning Same Meaning

5. Vector space models applications
Machine Translation
• You eat cereal from a bowl
• You buy something and someone else sells it
Information Extraction Chatbots

6. Fundamental concept
“You shall know a word by the company it keeps”
Firth, 1957
(Firth, J. R. 1957:11)

7. • Representation that captures relative meaning
• Represent words and documents as vectors
Summary

8. deeplearning.ai
Word by Word
and Word by
Doc.

9. Outline
• Co-occurrence Vector representation
• Relationships between words/documents

10. Word by Word Design
Number of times they occur together within a certain distance k
I like simple data
I prefer simple raw data
data
simple
k=2
2
n
raw like I
1 1 0

11. Word by Document Design
Number of times a word occurs within a certain category
Entertainment
data
Entertainment Economy
Machine
Learning
Economy
Machine
Learning
500 6620 9320
Corpus
film 7000 4000 1000

12. Vector Space
Measures of “similarity:”
Angle
Distance
ML
Economy
Entertainment
data
film
1000 5000 10000
1000
5000
10000
data
Entertainment Economy ML
6620 9320
film 4000 1000
500
7000

13. Summary
● W/W and W/D, counts of occurrence
● Vector Spaces Similarity between words/documents

14. deeplearning.ai
Euclidean
Distance

15. Outline
● Euclidean distance
● N-dimension vector representations comparison

16. Euclidean distance
Corpus B: (9320,1000)
Corpus A: (500,7000)
ML
Entertainment
data
film
1000 5000 10000
1000
5000
10000

17. Euclidean distance
Corpus B: (9320,1000)
Corpus A: (500,7000)
data
film
1000 5000 10000
1000
5000
10000
ML
Entertainment

18. Euclidean distance for n-dimensional vectors
data boba ice-cream
AI 6 0 1
drinks 0 4 6
food 0 6 8
Norm of ( -
)

19. Euclidean distance in Python
# Create numpy vectors v and w
v = np.array([1, 6, 8])
w = np.array([0, 4, 6])
# Calculate the Euclidean distance d
d = np.linalg.norm(v-w)
# Print the result
print("The Euclidean distance between v and w is: ", d)
The Euclidean distance between v and w is: 3

20. ● Straight line between points
Summary
● Norm of the difference between vectors

21. deeplearning.ai
Cosine
Similarity:
Intuition

22. Outline
● Problems with Euclidean Distance
● Cosine similarity

23. d1
Euclidean distance vs Cosine similarity
Agriculture corpus (20,40)
The cosine of the angle
between the vectors
d2
Euclidean distance: d2 < d1
α β
Angles comparison: β >
α
Food corpus (5,15) History corpus (30,20)
eggs
disease
10 20 30
10
20
30
40

24. Summary
● Cosine similarity when corpora are different sizes

25. deeplearning.ai
Cosine
Similarity

26. Outline
● How to get the cosine of the angle between two vectors
● Relation of this metric to similarity

27. Previous definitions
Vector norm
Dot product

28. Cosine Similarity
β
Agriculture corpus (20,40)
History corpus (30,20)
eggs
disease
10 20 30
10
20
30
40

29. Cosine Similarity
eggs
disease
10 20 30
10
20
30
40
eggs
disease
10 20 30
10
20
30
40
β
Similar
Dissimilar

30. Summary
● Cosine Similarity
● Cosine Similarity gives values between 0 and 1

31. deeplearning.ai
Manipulating
Words in
Vector Spaces

32. Outline
● How to use vector representations

33. Manipulating word vectors
USA Washington
DC
Russia

34. Manipulating word vectors
[Mikolov et al, 2013, Distributed Representations of Words and Phrases and their Compositionality]
Washington (10,5)
Moscow (9,3)
Tokyo(8.5,
2)
Ankara(8.5, 0.9)
Turkey (3,1)
USA (5,6)
Russia (5,5)
Japan (4,3)
(10,4)
Washington - USA =
Russia + =
Moscow

35. Summary
• Use known relationships to make predictions

36. deeplearning.ai
Visualization
and PCA

37. Outline
● Some motivation for visualization
● Principal Component Analysis

38. Visualization of word vectors
How can you visualize if your representation
captures these relationships?
oil & gas
town & city
oil 0.20 … 0.10
gas 2.10 … 3.40
city 9.30 … 52.1
town 6.20 … 34.3
d > 2

39. Visualization of word vectors
oil 0.20 … 0.10
gas 2.10 … 3.40
city 9.30 … 52.1
town 6.20 … 34.3
oil 2.30 21.2
gas 1.56 19.3
city 13.4 34.1
town 15.6 29.8
d = 2
d > 2
PCA

40. village
town
city
country
gas
oil
petroleum
happy
joyful
sad
Visualization of word vectors

41. Principal Component Analysis
Uncorrelated
Features
Dimensionality
Reduction

42. Summary
• Original Space Uncorrelated features Dimension
reduction
• Visualization to see words relationships in the vector space

43. deeplearning.ai
PCA
Algorithm

44. Outline
● How to get uncorrelated features
● How to reduce dimensions while retaining as much information as
possible

45. Principal Component Analysis
Uncorrelated
Features
Dimensionality
Reduction

46. PCA algorithm
Eigenvector: Uncorrelated features for your data
Eigenvalue: the amount of information retained by each
feature

47. PCA algorithm
Eigenvectors Eigenvalues
Mean Normalize
Data
Get Covariance
Matrix
Perform SVD

48. PCA algorithm
Eigenvectors Eigenvalues
Dot Product to
Project Data
Percentage of
Retained Variance

49. Summary
• Eigenvectors give the direction of uncorrelated features
• Eigenvalues are the variance of the new features
• Dot product gives the projection on uncorrelated features