Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

「知識」のDeep Learning

20,547 views

Published on

2015/06/04にPFIセミナーで発表した、知識表現に対する表現学習に関するスライドです

Published in: Technology
  • Be the first to comment

「知識」のDeep Learning

  1. 1. Deep Learning 1 Preferred Infrastructure (@unnonouno) 2015/06/04 PFI
  2. 2. !  1 !  !  NLP 12014-
  3. 3. NLP 1YANS !  YANS 19 !  140 !  !  !  YANS 13 !  !  "
  4. 4. Knowledge Representation
  5. 5. 1. 3. 4. 2.
  6. 6. 1. 3. 4. 2.
  7. 7. 1. 3. 4. 2. RNN
  8. 8. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] !  RNN !  !  !  LSTM
  9. 9. Long Short-Term Memory (LSTM) !  !  1 ! 
  10. 10. RNN1LSTM !  ! 
  11. 11. !  !  argmaxy P(y|x) = argmaxy P(x|y) P(y)
  12. 12. 1. 3. 4. 2.
  13. 13. [ 15] !  1 !  -‐‑‒
  14. 14. -‐‑‒ [ 15] !  Pydata Tokyo
  15. 15. 1. 3. 4. 2.
  16. 16. -‐‑‒ !  !  ! 
  17. 17. Knowledge Representation
  18. 18. !  EMNLP2014 Bordes Weston 1 part 2 [Bordes&Weston14]
  19. 19. !  [ 15]
  20. 20. Q: Deep Learning Representation Learning -‐‑‒ A: Deep
  21. 21. 推論を導けるような知識の表現、およびその方法 を開発する人工知能研究の領域である。
  22. 22. !  !  !  1
  23. 23. 1 !  2 t t 3 !  t t !  RDF (x, r, y) x yr
  24. 24. 1Knowledge Base
  25. 25. !  New York NY
  26. 26. !  !  t t !  t t !  t t
  27. 27. -‐‑‒ !  -‐‑‒ NLP
  28. 28. !  {(xi, ri, yi)}: !  x, y: t t !  r: t t !  x, y !  r
  29. 29. !  !  1 argmax ∑i f(xi, ri, yi)
  30. 30. Distance model (Structured Embedding) [Bordes +11] !  e !  Rleft, Rright !  !  f f(x, r, y) = || Rleft(r) e(x) – Rright(r) e(y) ||1
  31. 31. TransE model [Brodes+13] !  r r !  f(x, r, y) = || e(x) + r – e(y) ||2 2
  32. 32. TransE model x y r
  33. 33. TransE model 1 TransE !  TransM: !  TransH:
  34. 34. TransM model [Fan+14] !  r !  wr r x, y f(x, r, y) = wr|| e(x) + r – e(y) ||2 2
  35. 35. TransH model [Wang+14] !  TransE
  36. 36. Bilinear model !  r !  f(x, r, y) = e(x)T Wr e(y)
  37. 37. Neural Tensor Network (NTN) [Socher+13] f(x, r, y) = ur tanh(e(x)Wre(y) + V1 r e(x) + V2 r e(y) + br) !  r ! 
  38. 38. [Yang+15] !  2 !  Bilinear
  39. 39. Q: -‐‑‒ A: [Nickel+11]
  40. 40. Link prediction !  t t 1 !  QA !  -‐‑‒ !  t t (e1, r, e2) (e1, r, ? )
  41. 41. TransH TransE [Bordes&Weston14]
  42. 42. !  ! 
  43. 43. [Weston+13] !  x y r !  [Bordes&Weston14]
  44. 44. [Weston+13] !  !  -‐‑‒ [Bordes&Weston14]
  45. 45. [Bordes&Weston14]
  46. 46. Link prediction 1QA !  Link prediction ! 
  47. 47. QA [Bordes+14] !  !  q t f(q) g(t) !  f(q), g(t) q, t
  48. 48. Memory networks: [Weston+15] !  I: I(x) !  G: mi = G(mi, I(x), m) !  O: o = O(I(x), m) !  R: r = R(o)
  49. 49. I: O: R: G:
  50. 50. !  !  !  ! 
  51. 51. !  Deep Learning -‐‑‒
  52. 52. -‐‑‒ !  !  -‐‑‒ !  -‐‑‒ !  -‐‑‒ !  !  factoid !  -‐‑‒ ! 
  53. 53. RNN [Peng&Yao15] !  RNN ! 
  54. 54. !  !  !  !  !  !  !  ! 
  55. 55. 1/4 !  [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S. Khudanpur. Recurrent neural network based language model. Interspeech 2010. !  [ 15] . . 2015. !  [ 15] . NLP Introduction based on Project Next NLP. PyData.Tokyo Meetup #5, 2015. !  [Bordes&Weston14] A. Bordes, J. Weston. Embedding Methods for Natural Language Processing. EMNLP2014 tutorial. !  [ 15] . . JSAI 2015 .
  56. 56. 2/4 !  [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio. Learning structured embeddings of knowledge bases. AAAI2011. !  [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. NIPS 2013. !  [Fan+14] M. Fan, Q. Shou, E. Chang, T. F. Zheng. Transition-based Knowledge Graph Embedding with Relational Mapping Properties. PACLIC 2014. !  [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen. Knowledge Graph Embedding by Translating on Hyperplanes. AAAI 2014.
  57. 57. 3/4 !  [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng. Reasoning With Neural Tensor Networks for Knowledge Base Completion. NIPS 2013. !  [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng. Embedding Entities and Relations for Learning and Inferenece in Knowledge Bases. ICLR 2015. !  [Nickel+11] M. Nickel, V. Tresp, H. P. Kriegel. A Three-Way Model for Collective Learning on Multi-Relational Data. ICML 2011.
  58. 58. 4/4 !  [Bordes+14] A. Bordes, J. Weston, N. Usunier. Open Question Answering with Weakly Supervised Embedding Models. ECML/PKDD 2014. !  [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015. !  [Peng&Yao15] B. Peng, K. Yao. Recurrent Neural Networks with External Memory for Language Understanding. arXiv:1506.00195, 2015.

×