1. The document discusses knowledge representation and deep learning techniques for knowledge graphs, including embedding models like TransE, TransH, and neural network models. 2. It provides an overview of methods for tasks like link prediction, question answering, and language modeling using recurrent neural networks and memory networks. 3. The document references several papers on knowledge graph embedding models and their applications to natural language processing tasks.

1. Deep Learning
1 Preferred Infrastructure
(@unnonouno)
2015/06/04 PFI

2. ! 1
!
!
NLP 12014-

3. NLP 1YANS
! YANS 19
! 140
!
!
! YANS 13
!
! "

4. Knowledge Representation

5. 1.
3.
4.
2.

6. 1.
3.
4.
2.

7. 1.
3.
4.
2.
RNN

8. Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
! RNN
!
!
! LSTM

9. Long Short-Term Memory (LSTM)
!
!
1
!

10. RNN1LSTM
!
!

11. !
!
argmaxy P(y|x) = argmaxy P(x|y) P(y)

12. 1.
3.
4.
2.

13. [ 15]
!
1
! -‐‑‒

14. -‐‑‒ [ 15]
! Pydata Tokyo

15. 1.
3.
4.
2.

16. -‐‑‒
!
!
!

17. Knowledge Representation

18. ! EMNLP2014 Bordes Weston
1 part 2 [Bordes&Weston14]

19. ! [
15]

20. Q: Deep Learning
Representation Learning
-‐‑‒
A: Deep

21. 推論を導けるような知識の表現、およびその方法
を開発する人工知能研究の領域である。

22. !
!
! 1

23. 1
! 2 t t 3
! t t
! RDF
(x, r, y)
x yr

24. 1Knowledge Base

25. ! New York NY

26. !
! t t
! t t
! t t

27. -‐‑‒
!
-‐‑‒
NLP

28. ! {(xi, ri, yi)}:
! x, y: t t
! r: t t
! x, y
! r

29. !
!
1
argmax ∑i f(xi, ri, yi)

30. Distance model (Structured Embedding) [Bordes
+11]
! e
! Rleft, Rright
!
! f
f(x, r, y) = || Rleft(r) e(x) – Rright(r) e(y) ||1

31. TransE model [Brodes+13]
! r r
!
f(x, r, y) = || e(x) + r – e(y) ||2
2

32. TransE model
x
y
r

33. TransE model
1 TransE
! TransM:
! TransH:

34. TransM model [Fan+14]
! r
! wr r x, y
f(x, r, y) = wr|| e(x) + r – e(y) ||2
2

35. TransH model [Wang+14]
! TransE

36. Bilinear model
! r
!
f(x, r, y) = e(x)T Wr e(y)

37. Neural Tensor Network (NTN) [Socher+13]
f(x, r, y) = ur tanh(e(x)Wre(y)
+ V1
r e(x) + V2
r e(y) + br)
! r
!

38. [Yang+15]
! 2
!
Bilinear

39. Q:
-‐‑‒
A: [Nickel+11]

40. Link prediction
! t t 1
! QA
! -‐‑‒
! t t
(e1, r, e2) (e1, r, ? )

41. TransH
TransE
[Bordes&Weston14]

42. !
!

43. [Weston+13]
! x y r
!
[Bordes&Weston14]

44. [Weston+13]
!
!
-‐‑‒
[Bordes&Weston14]

45. [Bordes&Weston14]

46. Link prediction 1QA
! Link prediction
!

47. QA [Bordes+14]
!
! q t
f(q) g(t)
! f(q), g(t) q, t

48. Memory networks:
[Weston+15]
! I: I(x)
! G: mi = G(mi, I(x), m)
! O: o = O(I(x), m)
! R: r = R(o)

49. I:
O:
R:
G:

50. !
!
!
!

51. ! Deep
Learning -‐‑‒

52. -‐‑‒
!
! -‐‑‒
! -‐‑‒
! -‐‑‒
!
! factoid
! -‐‑‒
!

53. RNN [Peng&Yao15]
! RNN
!

54. !
!
!
!
!
!
!
!

56. 1/4
! [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S.
Khudanpur.
Recurrent neural network based language model.
Interspeech 2010.
! [ 15] .
.
2015.
! [ 15] .
NLP Introduction based on Project Next NLP.
PyData.Tokyo Meetup #5, 2015.
! [Bordes&Weston14] A. Bordes, J. Weston.
Embedding Methods for Natural Language Processing.
EMNLP2014 tutorial.
! [ 15] .
.
JSAI 2015 .

57. 2/4
! [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio.
Learning structured embeddings of knowledge bases.
AAAI2011.
! [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston,
O. Yakhnenko.
Translating Embeddings for Modeling Multi-relational Data.
NIPS 2013.
! [Fan+14] M. Fan, Q. Shou, E. Chang, T. F. Zheng.
Transition-based Knowledge Graph Embedding with Relational
Mapping Properties.
PACLIC 2014.
! [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen.
Knowledge Graph Embedding by Translating on Hyperplanes.
AAAI 2014.

58. 3/4
! [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng.
Reasoning With Neural Tensor Networks for Knowledge Base
Completion.
NIPS 2013.
! [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng.
Embedding Entities and Relations for Learning and Inferenece
in Knowledge Bases.
ICLR 2015.
! [Nickel+11] M. Nickel, V. Tresp, H. P. Kriegel.
A Three-Way Model for Collective Learning on Multi-Relational
Data.
ICML 2011.

59. 4/4
! [Bordes+14] A. Bordes, J. Weston, N. Usunier.
Open Question Answering with Weakly Supervised
Embedding Models.
ECML/PKDD 2014.
! [Weston+15] J. Weston, S. Chopra, A. Bordes.
Memory Networks.
ICLR 2015.
! [Peng&Yao15] B. Peng, K. Yao.
Recurrent Neural Networks with External Memory
for Language Understanding.
arXiv:1506.00195, 2015.