2015/06/04にPFIセミナーで発表した、知識表現に対する表現学習に関するスライドです

1. Deep Learning
1 Preferred Infrastructure
(@unnonouno)
2015/06/04 PFI

2. ! 1
!
!
NLP 12014-

3. NLP 1YANS
! YANS 19
! 140
!
!
! YANS 13
!
! "

4. Knowledge Representation

5. 1.
3.
4.
2.

6. 1.
3.
4.
2.

7. 1.
3.
4.
2.
RNN

8. Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
! RNN
!
!
! LSTM

9. Long Short-Term Memory (LSTM)
!
!
1
!

10. RNN1LSTM
!
!

11. !
!
argmaxy P(y|x) = argmaxy P(x|y) P(y)

12. 1.
3.
4.
2.

13. [ 15]
!
1
! -‐‑‒

14. -‐‑‒ [ 15]
! Pydata Tokyo

15. 1.
3.
4.
2.

16. -‐‑‒
!
!
!

17. Knowledge Representation

18. ! EMNLP2014 Bordes Weston
1 part 2 [Bordes&Weston14]

19. ! [
15]

20. Q: Deep Learning
Representation Learning
-‐‑‒
A: Deep

21. 推論を導けるような知識の表現、およびその方法
を開発する人工知能研究の領域である。

22. !
!
! 1

23. 1
! 2 t t 3
! t t
! RDF
(x, r, y)
x yr

24. 1Knowledge Base

25. ! New York NY

26. !
! t t
! t t
! t t

27. -‐‑‒
!
-‐‑‒
NLP

28. ! {(xi, ri, yi)}:
! x, y: t t
! r: t t
! x, y
! r

29. !
!
1
argmax ∑i f(xi, ri, yi)

30. Distance model (Structured Embedding) [Bordes
+11]
! e
! Rleft, Rright
!
! f
f(x, r, y) = || Rleft(r) e(x) – Rright(r) e(y) ||1

31. TransE model [Brodes+13]
! r r
!
f(x, r, y) = || e(x) + r – e(y) ||2
2

32. TransE model
x
y
r

33. TransE model
1 TransE
! TransM:
! TransH:

34. TransM model [Fan+14]
! r
! wr r x, y
f(x, r, y) = wr|| e(x) + r – e(y) ||2
2

35. TransH model [Wang+14]
! TransE

36. Bilinear model
! r
!
f(x, r, y) = e(x)T Wr e(y)

37. Neural Tensor Network (NTN) [Socher+13]
f(x, r, y) = ur tanh(e(x)Wre(y)
+ V1
r e(x) + V2
r e(y) + br)
! r
!

38. [Yang+15]
! 2
!
Bilinear

39. Q:
-‐‑‒
A: [Nickel+11]

40. Link prediction
! t t 1
! QA
! -‐‑‒
! t t
(e1, r, e2) (e1, r, ? )

41. TransH
TransE
[Bordes&Weston14]

42. !
!

43. [Weston+13]
! x y r
!
[Bordes&Weston14]

44. [Weston+13]
!
!
-‐‑‒
[Bordes&Weston14]

45. [Bordes&Weston14]

46. Link prediction 1QA
! Link prediction
!

47. QA [Bordes+14]
!
! q t
f(q) g(t)
! f(q), g(t) q, t

48. Memory networks:
[Weston+15]
! I: I(x)
! G: mi = G(mi, I(x), m)
! O: o = O(I(x), m)
! R: r = R(o)

49. I:
O:
R:
G:

50. !
!
!
!

51. ! Deep
Learning -‐‑‒

52. -‐‑‒
!
! -‐‑‒
! -‐‑‒
! -‐‑‒
!
! factoid
! -‐‑‒
!

53. RNN [Peng&Yao15]
! RNN
!

54. !
!
!
!
!
!
!
!

56. 1/4
! [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S.
Khudanpur.
Recurrent neural network based language model.
Interspeech 2010.
! [ 15] .
.
2015.
! [ 15] .
NLP Introduction based on Project Next NLP.
PyData.Tokyo Meetup #5, 2015.
! [Bordes&Weston14] A. Bordes, J. Weston.
Embedding Methods for Natural Language Processing.
EMNLP2014 tutorial.
! [ 15] .
.
JSAI 2015 .

57. 2/4
! [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio.
Learning structured embeddings of knowledge bases.
AAAI2011.
! [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston,
O. Yakhnenko.
Translating Embeddings for Modeling Multi-relational Data.
NIPS 2013.
! [Fan+14] M. Fan, Q. Shou, E. Chang, T. F. Zheng.
Transition-based Knowledge Graph Embedding with Relational
Mapping Properties.
PACLIC 2014.
! [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen.
Knowledge Graph Embedding by Translating on Hyperplanes.
AAAI 2014.

58. 3/4
! [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng.
Reasoning With Neural Tensor Networks for Knowledge Base
Completion.
NIPS 2013.
! [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng.
Embedding Entities and Relations for Learning and Inferenece
in Knowledge Bases.
ICLR 2015.
! [Nickel+11] M. Nickel, V. Tresp, H. P. Kriegel.
A Three-Way Model for Collective Learning on Multi-Relational
Data.
ICML 2011.

59. 4/4
! [Bordes+14] A. Bordes, J. Weston, N. Usunier.
Open Question Answering with Weakly Supervised
Embedding Models.
ECML/PKDD 2014.
! [Weston+15] J. Weston, S. Chopra, A. Bordes.
Memory Networks.
ICLR 2015.
! [Peng&Yao15] B. Peng, K. Yao.
Recurrent Neural Networks with External Memory
for Language Understanding.
arXiv:1506.00195, 2015.