This document provides an overview of Word2Vec network structure for generating word embeddings. It discusses how Word2Vec uses a neural network model like CBOW or Skip-gram to represent words as dense vectors based on their surrounding context words in a corpus. The Word2Vec neural network is trained to learn vector representations that are useful for performing analogy-style questions like "King - Man + Woman = Queen". The document also briefly outlines techniques like hierarchical softmax and negative sampling that allow Word2Vec training to scale to large corpora.

1. Word2Vec Network Structure Explained
Presented by: Subhashis Hazarika (The Ohio State University)
(Visualization Seminar Study)

2. Related Work
• Efficient Estimation of Word Representations in Vector Space – Mikolov et al.
2013
• Distributed Representations of Words and Phrases and their Compositionality –
Mikolov et al. 2013
• Linguistic Regularities in Continuous Space Word Representations – Mikolov et
al. 2013
• Implementation : https://code.google.com/archive/p/word2vec/ - Mikolov et al.
• word2vec Parameter Learning Explained – Rong 2014
• word2vec Explained: Deriving Mikolov et al’s Negative Sampling Word-
Embedding Method – Goldberg and Levy 2014

3. Word Embedding
• Atomic Word Representation:
• 1-of-N / one-hot vector encoding
King :
Queen:
Man:
Women:
Child:
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1

4. Word Embedding
• Distributed Word Representation (word2vec):
• 1 word  high-dimensional vector
King :
Queen:
Man:
Women:
Child:
0.99 0.99 0.05 0.7
0.99 0.05 0.93 0.6
0.05 0.99 0.05 0.4
0.02 0.01 0.99 0.5
0.4 0.43 0.45 0.2

5. Word Embedding
• Reasoning with word vectors:
• Xapple – Xapples ≈ Xcar – Xcars , Xfamily – Xfamilies ≈ Xcar – Xcars
• Addressing analogy:

6. Word Embedding
• Vector composition to answer questions like “ King – Man + Woman = ? ”
(Queen!)

7. Word Embedding

9. Word2Vec overview

10. Context-based Representation
• Word is represented by context in use.
I eat an apple every day.  eat | apple
Sometimes I like to eat orange as well.  eat | orange
I like to drive my own car to work.  drive | car

11. Context-based Representation Models
• Continuous Bag-of-words model (CBOW):
I eat an apple every day.  eat, an, every, day | apple
• Skip-gram (SG):
I eat an apple every day.  apple | eat, an, every, day

12. Word2Vec Neural Net Structure

13. Basic Neuron Structure

14. Basic Neuron Structure : Training Process

15. Multilayer Neural Network

16. Multilayer Neural Network : Training

17. Word2Vec Network

18. Word2Vec Network

19. Online demo

20. Intuition

21. Context-based Representation Models
• Continuous Bag-of-words model (CBOW):
I eat an apple every day.  eat, an, every, day | apple
• Skip-gram (SG):
I eat an apple every day.  apple | eat, an, every, day

23. Training Scalability

24. Hierarchical Softmax

25. Hierarchical Softmax

26. Hierarchical Softmax

27. Negative Sample

28. Interpreting Word Embedding Models
Neural Word Embedding as Implicit Matrix Factorization: Levy & Goldberg, NIPS 2014

29. Limitations
• Word Ambiguity
• Debuggability
• Sequence

30. Paragraph Vector / Doc2Vec
• Similar network structure
Distributed Representation of Sentences and Documents: Le & Mikolov 2014

31. Thank You