![credits: [x]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/75/introduction-to-word-embeddings-with-python-15-2048.jpg?cb=1668181651)
![here comes word2vec
Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al: [paper]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/75/introduction-to-word-embeddings-with-python-24-2048.jpg?cb=1668181651)
![word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling
Word-Embedding Method, Goldberg et al, 2014 [arxiv]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/75/introduction-to-word-embeddings-with-python-38-2048.jpg?cb=1668181651)
![deep walk:
DeepWalk: Online Learning of Social
Representations [link]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/75/introduction-to-word-embeddings-with-python-67-2048.jpg?cb=1668181651)
![predicting hashtags
interesting read: #TAGSPACE: Semantic
Embeddings from Hashtags [link]](https://image.slidesharecdn.com/introductiontowordembeddingswithpython1-150915104728-lva1-app6892/75/introduction-to-word-embeddings-with-python-69-2048.jpg?cb=1668181651)
Introduction to word embeddings with Python
•4 likes•3,932 views
Download to read offline
Report
My talk at Moscow Data Fest, 2015
Recommended
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec👋 Christopher Moody
21.5K views•161 slides
More Related Content
What's hot
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec👋 Christopher Moody
81.2K views•161 slides
What's hot(20)
Similar to Introduction to word embeddings with Python
Similar to Introduction to word embeddings with Python(20)
Recently uploaded
Recently uploaded(20)
Introduction to word embeddings with Python
- 1. Introduction to word embeddings Pavel Kalaidin @facultyofwonder Moscow Data Fest, September, 12th, 2015
- 5. лойс
- 6. годно, лойс лойс за песню из принципа не поставлю лойс взаимные лойсы лойс, если согласен What is the meaning of лойс?
- 7. годно, лойс лойс за песню из принципа не поставлю лойс взаимные лойсы лойс, если согласен What is the meaning of лойс?
- 8. кек
- 9. кек, что ли? кек))))))) ну ты кек What is the meaning of кек?
- 10. кек, что ли? кек))))))) ну ты кек What is the meaning of кек?
- 12. simple and flexible platform for understanding text and probably not messing up
- 13. one-hot encoding? 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
- 14. co-occurrence matrix recall: word-document co-occurrence matrix for LSA
- 15. credits: [x]
- 16. from entire document to window (length 5-10)
- 17. still seems suboptimal -> big, sparse, etc.
- 18. lower dimensions, we want dense vectors (say, 25-1000)
- 19. How?
- 22. lots of memory?
- 23. idea: directly learn low- dimensional vectors
- 24. here comes word2vec Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al: [paper]
- 25. idea: instead of capturing co- occurrence counts predict surrounding words
- 26. Two models: C-BOW predicting the word given its context skip-gram predicting the context given a word Explained in great detail here, so we’ll skip it for now Also see: word2vec Parameter Learning Explained, Rong, paper
- 28. CBOW: several times faster than skip-gram, slightly better accuracy for the frequent words Skip-Gram: works well with small amount of data, represents well rare words or phrases
- 29. Examples?
- 36. Wwoman - Wman = Wqueen - Wking classic example
- 38. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method, Goldberg et al, 2014 [arxiv]
- 39. all done with gensim: github.com/piskvorky/gensim/
- 40. ...failing to take advantage of the vast amount of repetition in the data
- 41. so back to co-occurrences
- 42. GloVe for Global Vectors Pennington et al, 2014: nlp.stanford. edu/pubs/glove.pdf
- 43. Ratios seem to cancel noise
- 44. The gist: model ratios with vectors
- 45. The model
- 49. recall:
- 51. Restoring symmetry, part 2
- 52. Least squares problem it is now
- 53. SGD->AdaGrad
- 54. ok, Python code
- 56. two sets of vectors input and context + bias average/sum/drop
- 57. complexity |V|2
- 65. Abusing models
- 67. deep walk: DeepWalk: Online Learning of Social Representations [link]
- 68. user interests Paragraph vectors: cs.stanford. edu/~quocle/paragraph_vector.pdf
- 69. predicting hashtags interesting read: #TAGSPACE: Semantic Embeddings from Hashtags [link]
- 70. RusVectōrēs: distributional semantic models for Russian: ling.go.mail. ru/dsm/en/
- 72. corpus matters
- 73. building block for bigger models ╰(*´︶`*)╯
- 74. </slides>