Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Word embeddings et leurs applications (Meetup TDS, 2016-06-30)

299 views

Published on

Les récentes avancées dans le domaine du traitement du langage ont permis de repousser la qualité des applications présentes dans notre quotidien : traduction, analyse de sentiments, reconnaissance vocale, dialogue homme-machine (Chatbots). Camille nous invite à comprendre une structure de représentation du langage assez puissante : les word embeddings. Cela consiste à représenter le sens des mots dans un espace vectoriel continu présentant des propriétés remarquables ; on peut par exemple déterminer des analogies entre les mots : reine = roi – home + femme.

Published in: Science
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Word embeddings et leurs applications (Meetup TDS, 2016-06-30)

  1. 1. Word embeddings et leurs applications Toulouse Data Science 30/06/2016 Camille Pradel 1
  2. 2. Plan • Représentations symboliques du sens des mots • Représentations vectorielles basées sur la similarité distributionnelle • Co-occurences et réduction de dimensions • Réseaux de neurones • Propriétés remarquables • Applications
  3. 3. Représentations symboliques du sens des mots • WordNet : 117 659 synsets (synonym set), un groupe de mots interchangeables, dénotant un sens ou un usage particulier 1. car, auto, automobile, machine, motorcar -- (4-wheeled motor vehicle; usually propelled by an internal combustion engine; he needs a car to get to work) 2. car, railcar, railway car, railroad car -- (a wheeled vehicle adapted to the rails of railroad; three cars had jumped the rails) 3. car, gondola -- (car suspended from an airship and carrying personnel and cargo and power plant) 4. car, elevator car -- (where passengers ride up and down; the car was on the top floor) 5. cable car, car -- (a conveyance for passengers or freight on a cable railway; they took a cable car to the top of the mountain)
  4. 4. • Relations sémantiques entre les synsets • Relations d'hyperonymie/hyponymie car, auto, automobile, machine, motorcar -> motor vehicle, automotive vehicle -> vehicle -> conveyance, transport -> instrumentality, instrumentation -> artifact, artefact -> object, physical object -> entity, something Représentations symboliques du sens des mots
  5. 5. • Relations sémantiques entre les synsets • Relation de méronymie/holonymie car, auto, automobile, machine, motorcar HAS PART: accelerator, accelerator pedal, gas pedal, gas, throttle, gun HAS PART: air bag HAS PART: auto accessory HAS PART: automobile engine HAS PART: automobile horn, car horn, motor horn, horn Représentations symboliques du sens des mots
  6. 6. • Limites • Manque de nuances Adept = expert = good = practiced = proficient = skillful • Pas à jour wicked, badass, nifty, crack, ace, wizard, genius, ninjia • Subjective • Chère à construire • Compromis couverture VS exhaustivité • Difficile de déduire une mesure de similarité entre les mots Représentations symboliques du sens des mots
  7. 7. Représentations basées sur la similarité distributionnelle Hypothèse distributionnelle: les mots qui apparaissent dans des contextes similaires ont tendance à être similaires. Le chat s’allonge sur le paillasson. Le chaton s’allonge sur le paillasson. Le chien s’allonge sur le paillasson. Mon chat m’a griffé. Mon chaton m’a griffé. garantie pour conclure un prêt avec une autre banque, dans un autre pays. "Le blanchisseur centrale européenne perdra sa crédibilité. Cette banque centrale, enfin, sera contrainte de serrer des emprunts en marks. Voici que cette banque fait maintenant ouvertement part de sa Philosophical Investigation Wittgenstein, L 1953 Sur le plan thématique: Sur le plan sémantique:
  8. 8. Matrices de co-occurrences I like deep learning I like NLP I enjoy flying • Limites: • Augmente en taille avec le vocabulaire • Consomme de l’espace de stockage • Vecteurs creux -> modèles peu robustes
  9. 9. Réduction de dimensions • Enregistrer les informations les plus importantes dans un nombre réduit de dimensions (en général entre 25 et 1000) • Analyse en composante principale (PCA) • Décomposition en valeur singulière (SVD) • …
  10. 10. Réseaux de neurones • SVD • Complexité quadratique -> passage à l’échelle compliqué • Difficile de prendre en compte de nouveaux documents  apprendre directement des vecteurs de mots de dimension réduite • Learning representations by back-propagating errors (Rumelhart et al., 1986) • A neural probabilistic language model (Bengio et al., 2003) • NLP (almost) from Scratch (Collobert & Weston, 2008) • word2vec (Mikolov et al. 2013)
  11. 11. Réseaux de neurones • Utilisation d’un réseau de neurones pour apprendre une tâche fictive • Prédire le contexte d’un mot • Prédire un mot en fonction de son contexte Démo: ronxin.github.io/wevi/ Efficient estimation of word representations in vector space Mikolov, T., Chen, K., Corrado, G. & Dean, J. 2013
  12. 12. Représentation naïve sous forme vectorielle chaque mot est considéré comme une symbole atomique One-hot encoding Chien [ 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 ] Bateau [ 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … 0 0 0 ] Hôtel [ 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 … 0 0 0 ] Motel [ 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 … 0 0 0 ] Taille du vocabulaire Réseaux de neurones
  13. 13. An improved model of semantic similarity based on lexical co-occurrence Rohde, D. L., Gonnerman, L. M., and Plaut, D. C. 2006 Emergence de motifs syntaxiques Propriétés remarquables
  14. 14. An improved model of semantic similarity based on lexical co-occurrence Rohde, D. L., Gonnerman, L. M., and Plaut, D. C. 2006 Emergence de motifs sémantiques Propriétés remarquables
  15. 15. Propriétés remarquables • Relations linéaires • Analogies syntaxiques apple − apples ≈ car − cars ≈ family − families • Analogies sémantiques shirt − clothing ≈ chair − furniture king − man ≈ queen − woman Efficient estimation of word representations in vector space Mikolov, T., Chen, K., Corrado, G., & Dean, J. 2013
  16. 16. GloVe: Global Vectors for Word Representation Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014 Capture du genre Propriétés remarquables
  17. 17. GloVe: Global Vectors for Word Representation Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014 Superlatifs Propriétés remarquables
  18. 18. GloVe: Global Vectors for Word Representation Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014 Compagnie - CEO Propriétés remarquables
  19. 19. Applications Distances, analogies, OK. Quoi d’autre?
  20. 20. Applications • Identifier l’intrus dans une liste https://github.com/dhammack/Word2VecExample math shopping reading science eight six seven five three owe nine breakfast cereal dinner lunch england spain france italy greece germany portugal australia
  21. 21. Applications • Dice utilise des modèles de vecteurs de mots pour rapprocher des mots-clés liés • Analytics -> Business Intelligence Implementing Conceptual Search in Solr using LSA and Word2Vec - Simon Hughes http://fr.slideshare.net/lucidworks/implementing-conceptual- search-in-solr-using-lsa-and-word2vec-presented-by-simon- hughes-dicecom
  22. 22. Applications A Word is Worth a Thousand Vectors - Chris Moody http://multithreaded.stitchfix.com/blog/2015/03/11/word-is- worth-a-thousand-vectors/ ITEM_3469 pregnant ITEM_3469 ITEM_3469 ITEM_3469
  23. 23. Applications • Embeddings à plus grande échelle • Traduction statistique • Robots conversationnels • Description d’image et de vidéos https://vimeo.com/146492001
  24. 24. Sources "pédagogiques" • CS224d: Deep Learning for Natural Language Processing - Richard Socher • http://cs224d.stanford.edu/syllabus.html • https://www.youtube.com/watch?v=T8tQZChniMk (Lecture 2) • word2vec parameter learning explained - Xin Rong • http://arxiv.org/abs/1411.2738 • Grounding distributional semantics in the visual world - Marco Baroni • http://clic.cimec.unitn.it/marco/publications/lectures/marco-grounding-ds-vl-2015.pdf • Understanding Dimensionality Reduction- Principal Component Analysis And Singular Value Decomposition - Priya Rana • http://hpc-asia.com/understanding-dimensionality-reduction-principal-component-analysis-and-singular- value-decomposition/ • A Beginner’s Guide to word2vec AKA What’s the Opposite of Canada? - Will Critchlow • https://www.distilled.net/resources/a-beginners-guide-to-word2vec-aka-whats-the-opposite-of-canada/ • Page Wikipedia sur WordNet • https://fr.wikipedia.org/wiki/WordNet

×