Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Poincare embeddings for Learning Hierarchical Representations

19,929 views

Published on

Talked at ABEJA Paper Reading Meetup July 4, 2017

Published in: Technology

Poincare embeddings for Learning Hierarchical Representations

  1. 1. Poincaré Embeddings for Learning Hierarchical Representations July 4, 2017 Tatsuya Shirakawa ABEJA Inc.
  2. 2. CONFIDENTIAL Tatsuya Shirakawa
  3. 3. CONFIDENTIAL Today’s Paper Paper Stats • Guys from FAIR • Sumitted to arXiv at May 26, 2017 https://arxiv.org/abs/1705.08039 • Sumitted to NIPS2017? Key Contributions • Introducing hyperbolic geometry (Poincaré disk model) into word/graph embeddings paradigm • Automatically capture hierarchical structure of data • Achieved incredible better results than previous works.
  4. 4. CONFIDENTIAL 1. Problems 2. Hyperbolic Geometry 3. Poincaré Embeddings (and Some Incredible Results) Agenda 4
  5. 5. CONFIDENTIAL Problems 5
  6. 6. CONFIDENTIAL Find good representation(embedding) of items such that underlying hierarchical relation structure are well reconstructed The Problem
  7. 7. CONFIDENTIAL Embed nouns in WordNets such that related nouns are close in embedded space Taxonomy Embedding 7 http://www.nltk.org/book_1ed/ch02.html
  8. 8. CONFIDENTIAL Embed nodes in given graph such that missing links are well-reconstructed Graph Link Prediction 8 http://ml.cs.tsinghua.edu.cn/~jiaming/publications/
  9. 9. CONFIDENTIAL Back Theory 9
  10. 10. CONFIDENTIAL • Geometry with negative curvature • Many models (realizations): - Poincaré half space model - Poincaré disk model … each is isometric Hyperbolic Geometry 10
  11. 11. CONFIDENTIAL • Defined on upper half space with metric • Distance btw points is Poincaré Half Space Model 11
  12. 12. CONFIDENTIAL 12 Tree representation in H https://arxiv.org/abs/1006.5169 • Tree structure is well represented in Poincaré half space
  13. 13. CONFIDENTIAL • A realization of hyperbolic geometry • Defined on equipped with metric of • Distance btw points is Poincaré Disk Model 13 M.C. Escher's Circle Limit III, 1959
  14. 14. CONFIDENTIAL (for simplicity: 2-dim, identify as ) Relation to Poincaré Half Space Model 14 https://arxiv.org/abs/1006.5169
  15. 15. CONFIDENTIAL • Euclidean space is too narrow to embed hierarchical (tree) structures Why not Euclidean Space? 15 Surface Area / # of leaf nodes Volume / # of nodes Euclidean Ball O(R^n) O(R^n) b-ary tree O(b^R) O(b^R) ※ R=radius of ball or depth of tree
  16. 16. CONFIDENTIAL • b-array tree can be interpreted as discrete analogue of Poincaré disk Why Hyperbolic Space? 16
  17. 17. CONFIDENTIAL • Hyperbolic space is far more appropriate than Euclidean space to represent hierarchical structure • Many equivalent models - Poincaré half space model - Poincaré disk model … Conclusion Here 17
  18. 18. CONFIDENTIAL • R. Kleinberg, “Geographic routing using hyperbolic spaces”, 2007 • M. Boguna et al., “Sustaining the internet with hyperbolic mapping”, 2010 • P. D. Hoff et al., “Latent space approaches to social network analysis”, 2016 • A. B. Adcock et al., “Tree-like structure in large social and information networks’, 2013 • D. Krioukov et al., “Hyperbolic geometry of complex networks”, 2010 Prior Works around hyperbolic geometry applications 18
  19. 19. CONFIDENTIAL Poincaré Embeddings 19
  20. 20. CONFIDENTIAL 1. Parametrize each item in Poincaré ball 2. Optimize them by Riemannian optimization under metric of Proposed Method
  21. 21. CONFIDENTIAL 1. Compute stochastic (Euclidean) gradient 2. Correct metric 3. Apply GD 4. Project onto space Riemannian SGD 21
  22. 22. CONFIDENTIAL Embed nouns in WordNets such that related nouns are close in embedded space Taxonomy Embedding 22 http://www.nltk.org/book_1ed/ch02.html
  23. 23. CONFIDENTIAL Maximize Reconstruction setting: - D is full relations Prediction setting - D is subset of full relations Objective Function 23 randomly chosen 10 negative samples
  24. 24. CONFIDENTIAL Result 24
  25. 25. CONFIDENTIAL 25
  26. 26. CONFIDENTIAL Embed nodes in given graph such that missing links are well-reconstructed Graph Link Prediction 26 http://ml.cs.tsinghua.edu.cn/~jiaming/publications/
  27. 27. CONFIDENTIAL Minimize the cross entropy of probability Objective Function 27
  28. 28. CONFIDENTIAL Result 28
  29. 29. CONFIDENTIAL • Poincaré embeddings automatically capture hierarchical structure from data • Riemannian SGD provides the way to optimize Poincaré embeddings • Achieved quite good results on word/graph embedding tasks Summary 29

×