This document provides an overview of Word2Vec, a neural network model for learning word embeddings developed by researchers led by Tomas Mikolov at Google in 2013. It describes the goal of reconstructing word contexts, different word embedding techniques like one-hot vectors, and the two main Word2Vec models - Continuous Bag of Words (CBOW) and Skip-Gram. These models map words to vectors in a neural network and are trained to predict words from contexts or predict contexts from words. The document also discusses Word2Vec parameters, implementations, and other applications that build upon its approach to word embeddings.

1. WORD2VEC
M.Javad Hasani
1

2. Outline • Goal
• History
• Word Embedding
• Introduction toWord2Vec
• CBOW
• Skip-Gram
• Parameters
• Implementations
• Other usecases
2

3. When? Who?
• Word2vec was created by a team of
researchers led by Tomas Mikolov at Google.
• Embedding vectors created using the
Word2vec algorithm have many advantages
compared to earlier algorithmssuch as :
latent semantic analysis.
2013
3

4. Goal:
Reconstruct linguistic
contexts of words
context
words
Word2Vec
Target
Word
WordWord2Vec
Context
words
4

5. Tasks
(WATER – WET ) + FIRE = FLAMES
(PARIS - FRANCE) + ITALY = ROME
(WINTER - COLD) + SUMMER = WARM
(KING - MAN) +WOMAN = QUEEN
5

6. Why vector
space?
similar
distributions
similar
meanings
6

7. Vector space: word embeddings
7

8. word
embedding
A technique to turn
words into numbers
to use by many of the machine learning
algorithms
8

9. One-hot
vector
simple word representation
• Vector length is equal to dictionary size
• Any vector has one non-zero element
9

10. Types of
Word
Embeddings
Frequency
based
CountVector
TF-IDF
Vector
Co-
Occurrence
Vector
Prediction
based
CBOW
Skip – Gram
10

11. What is
word2vec?
11

12. What is
word2vec?
• Word2vec is a combination of two
techniques
– CBOW(Continuous bag of words)
– Skip-gram model.
• Both of these map word(s) to
word(s).
• learn weights which act as word
vector representations.
Skip-
gram
CBOW
12

13. How it
works?
1. Both input word wi and the output word wj are one-hot
encoded into binary vectors x and y of size V.
2. First, the multiplication of the binary vector xx and the
word embedding matrix W of size V×N gives us the
embedding vector of the input word wi: the i-th row of
the matrix W.
3. The multiplication of the hidden layer and the word
context matrix W′ of size N×W produces the output
one-hot encoded vector y.
13

14. Embedding
matrix X x W=v
14

15. Training Samples
By
sibling window
15

16. CBOW
(Continuous Bag of words)
Skip-gram
Syntactic relation Semantic relation
16

17. Loss
Functions
Full Softmax
Hierarchical Softmax
Cross Entropy
Noise Contrastive Estimation (NCE)
Negative Sampling (NEG)
17

18. Softmax
Full Hierarchical
18

19. Parametrization • Sub-sampling
– High frequency words often provide little information.
• Dimensionality
– Quality of word embedding increases with higher
dimensionality.
– But after reaching some point, marginal gain will
diminish.
– Typically, the dimensionality of the vectors is set to be
between 100 and 1,000.
• Context window
– The recommended value is 10 for skip-gram and 5 for
CBOW.
19

20. Result
https://ronxin.github.io/wevi/ 20

21. Variants models class
• documents to vector spaceDoc2vec
• There are a lot of noisy text and informal
language structure.tweet2vec
• dealing with item and user similarity is at heart
of lot of recommendation algorithmsitem2vec
• this embedding technique tries to marry best of
both worlds, word2vec and LDALda2vec
21

22. Implementation
s
23

23. Thanks
24