GDG Tbilisi 2017. Word Embedding Libraries Overview: Word2Vec and fastText
•
2 likes•618 views
This presentation about comparing different word embedding models and libraries like word2vec and fastText, describing their difference and showing pros and cons.
More Related Content
What's hot
What's hot (20)
Similar to GDG Tbilisi 2017. Word Embedding Libraries Overview: Word2Vec and fastText
Similar to GDG Tbilisi 2017. Word Embedding Libraries Overview: Word2Vec and fastText (20)
Recently uploaded
Recently uploaded (20)
GDG Tbilisi 2017. Word Embedding Libraries Overview: Word2Vec and fastText
- 1. RUDOLF EREMYAN MACHINE LEARNING SOFTWARE ENGINEER WORD EMBEDDING LIBRARIES OVERVIEW: Word2Vec && fastTEXT CONTACTS: EREMYAN.RUDOLF@GMAIL.COM HTTPS://WWW.LINKEDIN.COM/IN/RUDOLFEREMYAN/
- 2. Ti bot for TBC bank
- 3. B Bot for TBC Insurance
- 4. Word Embedding? Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with much lower dimension. https://en.wikipedia.org/wiki/Word_embedding
- 6. Chatbots
- 7. Ranking
- 10. How represent word as vector?
- 11. One-hot-encoding “I love cake hate pizza”
- 12. Problems of one-hot-encoding 1.One-hot vectors are high-dimensional and sparse 2.Feature vector grows with the vocabulary size 3.Semantic and syntactic information are lost
- 13. WORD2VEC by Google Efficient Estimation of Word Representations in Vector Space by Mikolov, Corrado, Dean, Chen 2013 NAACL https://arxiv.org/pdf/1301.3781.pdf
- 14. word2vec
- 15. word2vec
- 16. word2vec
- 18. word2vec
- 19. word2vec
- 23. word2vec. skip-gram model One-hot vector is almost all zeros… what’s the effect of that?
- 24. word2vec. skip-gram model One-hot vector is almost all zeros… what’s the effect of that?
- 27. WORD2VEC LIMITATIONS??? 1. Doesn’t take into account the internal structure of words (bad for morphologically rich languages) 2.Out-of-Vocabulary cases for unseen words
- 28. fastText by Facebook Enriching Word Vectors with Subword Information by Mikolov , Bojanowski, Grave, A. Joulin 2016 https://arxiv.org/abs/1607.04606
- 29. fastText The main goal of the Fast Text embeddings is to take into account the internal structure of words while learning word representations – this is especially useful for morphologically rich languages, where otherwise the representations for different morphological forms of words would be learnt independently. The limitation becomes even more important when these words occur rarely. N = 2 N = 4 student => ['st', 'tu', 'ud', 'de', 'en', 'nt'] student => ['stud', 'tude', 'uden', 'dent']
- 30. fastText fastText differs from word2vec only in that it uses char n-gram embeddings as well as the actual word embedding in the scoring function to calculate scores and then likelihoods for each word, given a context word. In case char n-gram embeddings are not present, this reduces (at least theoretically) to the original word2vec model. This can be implemented by setting 0 for the max length of char n-grams for fastText.
- 32. fastText vs word2vec comparision Syntactic test set Semantic test set
- 33. fastText vs word2vec comparision
- 34. fastText vs word2vec comparision
- 35. fastText vs word2vec comparision
- 37. • https://www.datascience.com/resources/notebooks/word- embeddings-in-python • http://mccormickml.com/2016/04/19/word2vec-tutorial-the- skip-gram-model/ • https://rare-technologies.com/fasttext-and-gensim-word- embeddings/ • https://www.gavagai.se/blog/2015/09/30/a-brief-history-of- word-embeddings/ LINKS
- 38. QUESTIONS?