The document discusses various models and techniques in data science and text mining, particularly focusing on vector representations of words and paragraphs using models like word2vec, skip-gram, and CBOW. It explains concepts like softmax regression, hierarchical softmax, and negative sampling, as well as the paragraph vector model (paragraph2vec) for representing variable-length text. Performance comparisons highlight that while paragraph2vec offers higher accuracy in classification tasks, vector averaging is faster, and clustering provides a balance of accuracy and speed.

1. Science in Data
Science
Science in Text Mining
by
Tanay Chowdhury
Data Scientist,Zurich North America

2. Word2vec model
• Creates the vector representation of every word using one-hot
encoding.
• The displacement vector (the vector between two vectors)
describes the relation between two words.
• Comparing displacement vectors helps find similarity.
Eg: vqueen−vking=vwoman−vman.

3. Skip gram model
It’s a simple neural network with a single hidden layer.

4. • For training words w1,w2, . . . ,wt proper position of output word wo for
input word wI is found by minimizing the loss function
• Weight update equation using stochastic gradient descent.
• Then back propagation for learning rate is
Skip gram model

5. Skip gram model
If we now re-compute the probability of each word given the input
word queen, we see that the probability of king given queen goes up
Softmax
regression
Weight
update Backpropagation
Example available at : https://github.com/fbkarsdorp/doc2vec/blob/master/doc2vec.ipynb

6. cbow model

7. cbow model
• If the context C comprises multiple words, instead of copying the
input vector we take the mean of their input vectors as our
hidden layer:
• The update functions remain the same except that for the
update of the input vectors, we need to apply the update to
each word in the context C:
•

8. Softmax
• Softmax regression (or multinomial logistic regression) is a generalization
of logistic regression to the case where we want to handle multiple
classes.
• In text classification every word in vocabulary is a label, making it a
multiclass problem.
• Hence using softmax regression, we can compute the posterior
probability P(wO|wI)

9. Hierarchical Softmax
• It consists of a binary Huffman tree based on word frequencies,
with each word wo as leaf.
• Probability of the path from root to the word causes
normalization.
• One N-way normalization is replaced by a sequence of O(logN)
local (binary) normalizations, as it is all about traversing subtree
with higher probability.
“.”makes
the result -1
or +1

10. Negative Sampling
• The objective is to distinguish the target word from draws from
the noise distribution using logistic regression, where there are k
negative samples for each data sample.
• References
• Gutmann and Hyvarinen 2012 . Noise-contrastive estimation of unnormalized statistical models, with
applications to natural image statistics.
• Mnih and The 2012. A fast and simple algorithm for training neural probabilistic language models.

11. Vectoring paragraph
• Paragraph Vector model attempts to learn fixed-length continuous
representations from variable-length pieces of text.
• Some of the attempts made in that direction are paragraph2vec,vector
averaging and clustering.
• Reference : Le & Mikolov (2014)

12. Pragraph2vec

13. Pragraph2vec
• At each step h is computed by concatenating or averaging a
paragraph vector d with a context of word vectors C
• The weight update functions are the same as in Word2Vec
except that paragraph vectors also gets updated.

14. Pragraph2vec
• Vector representation of a paragraph with different length
sentence.
Paragraph
w/17
different
words

15. Pragraph2vec
Paragraph
vector of
dimension
17x3
Document
vector of
dimension
5x3
Softmax
regression
Weight
update

16. Vector Averaging
• Vector averaging is the simplest technique to get the vector
representation for a sentence.
• It calculates the sum of all word vectors and divides that with
number of words.
• It’s considered fastest but not so accurate in result.

17. Clustering method
• It creates predetermined number of cluster of words.
• Synonymous words (mainly adjectives) ends up in same cluster.
• Rather than using bag of words, it uses bag of centroids
(counting the occurrence of a word in a certain cluster)then.
• Reference : https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-3-more-fun-with-word-vectors

18. Performance comparison
• Paragraph2vec is generally the most accurate one when used in
classification algorithm.
• Vector averaging is the fastest but clustering is considered to be
decent in terms of accuracy and speed.

19. Questions