The document presents research on using recurrent neural networks and path-based relation representations to complete knowledge bases and address misrelations. It introduces previous translation-based models like TransE, TransH, and TransR. It then describes the proposed Path-TransE (PTransE) model, which learns relation compositions through RNNs to infer relations not explicitly in the training data. The research team evaluated PTransE on Freebase datasets, finding it outperformed previous methods with a hit rate of 53.1% for raw queries and 86.6% for filtered queries, demonstrating the effectiveness of using path representations and LSTMs for knowledge base completion.

1. Final Report:
Construction through Deep RNN
Representation Learning
of Knowledge Bases
KoreaUniversity,
DepartmentofComputerScience&Radio
CommunicationEngineering
MASSIVEDATAMANAGEMENT
Professor JaewooKang
1
2015010661
2015011155
2016010646
MinhwanYu
YonghwaChoi
BumsooKim

2. MASSIVE DATA MANAGEMENT Presentation format
Contents
01.Introduction
02.PTransE
1-1.Whatdowewant?
1-3.TransH
2-1.Whatisdifferent?
2-2.RelationPathRepresentation
2-3.RecurrentNeuralNetwork
2
03.Ouralgorithm
3-1.Objectfunction
3-2.Activationfunction
3-3.LongShortTermMemory
04.Evaluation
4-1.Results
4-2.Comparison&Analysis
1-4.TransR
1-5.PTransE
1-2.TransE

3. MASSIVE DATA MANAGEMENT Presentation format
Introduction
1-2.TransE
1-3.TransH
3
1-4.TransR
1-5.PTransE
1-1.Whatdowewant?

4. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
4
1-1. What do we want?
Large-Scale
KnowledgeBases
Freebase
DBpedia
Yago
Realworld triples
KnowledgeBase(KB)

5. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
5
1-1. What do we want?
Large-Scale
KnowledgeBases
Freebase
Realworld triples
∴KBsareveryincomplete
Misrelation
1.Norelationfound
2.Wrongrelationfound
DBpedia
Yago

6. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
6
1-1. What do we want?
How doweaddress mis-relations?

7. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
7
1-1. What do we want?
Q.Howdoweaddressmisrelation?
A1. Addexternalsourcesfor
completion
Manualandintuitional
Requirestoomuchlabor
Inefficient,time-consuming
PerformancedropsasKBgetslarger

8. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
8
1-1. What do we want?
Q.Howdoweaddressmisrelation?
A1. Addexternalsourcesfor
completion
A2. Referencingothertuples
forcompletion
Powerful&Efficient
Largelyexpandable
MorepowerfulasKBgetslarger

9. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
9
1-1. What do we want?
Wetriedout…
TransE
TransH
TransR
PTransE
CompletionofKBbyaddressingmisrelationsthrough
co-entity,co-relationreference

10. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
10
1-2. TransE
relation#1Boy Girl
relation#1Queen
TransE
Representingdirectrelationswith
differencesbetweenentityvectors
Simple&Effective
reference:TransE(Bordesetal.NIPS2013)

11. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
11
1-2. TransE
TransE
Simple&Effective
relation#1Boy Girl
relation#1Queen King
Representingdirectrelationswith
differencesbetweenentityvectors
reference:TransE(Bordesetal.NIPS2013)

12. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
12
1-2. TransE
However…
TransEcan’trepresentmorethan
onerelationshipbetweenentities.
Inrealworld,weconstructmany
relationshipswithmanysubjects.

13. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
13
1-3. TransH
TransH
AbletorepresentM-to-Mrelations
Representingprojectedrelationswith
differencesbetweenentityvectors
Entitiesandrelationshavedifferentcharacteristics
However,theyarerepresentedinthesamespace
reference:TransH(Wangetal.AAAI2014)

14. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
14
1-4. TransR
Abletorepresentdifferentcharacteristics
betweenentitiesandrelations
Representingprojectedentitiesand
mappingthemintorelationspace‘r’
TransR
However,thesewereallnotgoodenoughfor
detectingandaddressingmisrelations!!
reference:TransR(AAAI2015)

15. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
15
1-5. PTransE
PTransE
Representingrelationsthroughcompositionof
relationsbetweenentityvectors
Widelyexpandable&powerfulmethod
h
t
e1
r1
r2
r=r1⋅r2
r1:father r1:father
reference:emnlpprocessing

16. MASSIVE DATA MANAGEMENT Presentation format
1. Introduction
16
1-5. PTransE
PTransE
Representingrelationsthroughcompositionof
relationsbetweenentityvectors
Widelyexpandable&powerfulmethod
h
t
e1
r1
r2
r=r1⋅r2
r:grandfather
reference:emnlpprocessing

17. MASSIVE DATA MANAGEMENT Presentation format
PTransE
2-2.RelationPathRepresentation
2-3.RecurrentNeuralNetwork
17
2-1.Whatisdifferent?

18. MASSIVE DATA MANAGEMENT Presentation format
2. PTransE
18
2-1. What is different?
PTransE
LearnsonlyonesinglematrixEachrelationneedsitsown
projectionmap.
TransR
50,000relationsneeds50,000
enormousprojectionmaps!
Learnsn-steprelationalpaths
Learnsrelationsthatevendon’t
haveenoughtrainingdataIfarelationdoesn’thaveenough
trainingdata,itwillsufferfromlow
performance.
Learnsrelationsonlyinthetraining
dataset.
Learnsrelationsthatarenotinthe
trainingdatasetthroughrelation
paths.(Zero-shotKBinference)

19. MASSIVE DATA MANAGEMENT Presentation format
2. PTransE
19
2-2. Relation Path Representation
MicrosoftisbasedinSeattle.
WhichcountryisMicrosoftlocatedin?
Microsoft Seattle
IsBasedInCountryIn
???
RelationNOTFOUND!(Misrelation)

20. MASSIVE DATA MANAGEMENT Presentation format
2. PTransE
20
2-2. Relation Path Representation
‘Microsoft’isbasedin‘Seattle’.
‘Seattle’islocatedinstate‘Washington’
Microsoft Seattle Washington USA
IsBasedIn
‘Washington’islocatedincountry‘USA’
StateIn CountryIn

21. MASSIVE DATA MANAGEMENT Presentation format
2. PTransE
21
2-2. Relation Path Representation
‘Microsoft’isbasedin‘Seattle’.
‘Seattle’islocatedinstate‘Washington’
Microsoft USA
‘Washington’islocatedincountry‘USA’
CountryLocatedIn
‘Microsoft’islocatedin‘USA’.

22. MASSIVE DATA MANAGEMENT Presentation format
2. PTransE
22
2-3. Recurrent Neural Network
Microsoft Seattle Washington
IsBasedIn StateIn CountryIn
USA
CountryLocatedIn
RNN
RNN
Learningamatrix 𝑾𝑾𝒓𝒓 for
constructingcomposition
vector
StateLocatedIn
p= 𝒇𝒇( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟑𝟑;𝒄𝒄𝟒𝟒 ) = 𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟑𝟑;𝒄𝒄𝟒𝟒 )
𝐜𝐜𝟑𝟑 = 𝒇𝒇( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟏𝟏;𝒄𝒄𝟐𝟐 )
𝒄𝒄𝟏𝟏 𝒄𝒄𝟐𝟐
𝒄𝒄𝟒𝟒
reference:CompositionalVectorSpaceModelsforKnowledgeBaseInference

23. MASSIVE DATA MANAGEMENT Presentation format
Our Algorithm
3-2.Activationfunction
3-3.LongShortTermMemory
23
3-1.Objectivefunction

24. MASSIVE DATA MANAGEMENT Presentation format
3. Our Algorithm
24
3-1. Objective function
Me Father Grandfather
FatherOf FatherOf
RNN
Compositionvector
𝐩𝐩 = 𝒇𝒇( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟏𝟏;𝒄𝒄𝟐𝟐 )
𝒄𝒄𝟏𝟏 𝒄𝒄𝟐𝟐
Me Grandfather
GrandfatherOf
Update 𝑾𝑾𝒓𝒓

25. MASSIVE DATA MANAGEMENT Presentation format
3. Our Algorithm
25
3-1. Objective function
Compositionvector
𝐩𝐩 = 𝒇𝒇( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟏𝟏;𝒄𝒄𝟐𝟐 )GrandfatherOf
n-steprelationvector
𝒍𝒍𝒍𝒍𝒍𝒍𝒍𝒍
Updating 𝑾𝑾𝒓𝒓 …
𝒍𝒍𝒍𝒍𝒍𝒍𝒍𝒍BackPropagation
ForwardPropagation 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑
−𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝒕𝑼𝑼𝑼𝑼𝑼𝑼𝑼𝑼𝑼𝑼𝑼𝑼
𝐖𝐖𝐫𝐫

26. MASSIVE DATA MANAGEMENT Presentation format
3. Our Algorithm
26
3-2. Activation function
However,NeuralNetworkisbasicallyalinearoperation
𝐩𝐩 = 𝒇𝒇( 𝑾𝑾𝒓𝒓 𝒄𝒄𝟏𝟏;𝒄𝒄𝟐𝟐 )
Applyinganon-linearaftereach
operationsinordertolearnnon-
lineardecisionboundary
tanhistheconventionaldefault
non-linearactivationfunctionfor
RNNmodels.

27. MASSIVE DATA MANAGEMENT Presentation format
3. Our Algorithm
27
3-2. Activation function
Then,what’swrongwithtanh?
Whenbackpropagatinggradients
inordertoupdatethematrix,
thesegradientstendtoconvergeto
0andeliminatesthegradientflow.
Therefore,nogradientflows
backward,andtheparameterstays
unmodified.
GradientVanishingProblem

28. MASSIVE DATA MANAGEMENT Presentation format
3. Our Algorithm
28
3-3. Long Short Term Memory
AddressesGradientVanishingproblems
HandleLongtermdependencies
ForgetGate

29. MASSIVE DATA MANAGEMENT Presentation format
Evaluation
4-2.Comparison&Analysis
29
4-1.Results

30. MASSIVE DATA MANAGEMENT Presentation format
4. Evaluation
30
4-1. Results
WeusedFB15K,andFB40Kdatasets
FB15K FB40K
# Relations 1,345 1,336
# Entities 14,951 39,528
# Training set 483,142 370,648
# Validation set 50,000 67,948
# Test set 59,071 96,678

31. MASSIVE DATA MANAGEMENT Presentation format
4. Evaluation
31
4-2. Comparisons & Analysis
Hit 10 (RAW) Hit 10 (FILTER)
TransE 34.9% 47.1%
TransH 45.7% 64.4%
TransR 43.8% 65.5%
Hit 10 (RAW) Hit 10 (FILTER)
PTransE(RNN)
(Original model)
50.6% 82.2%
PTransE(LSTM)
(Our model)
53.1% 86.6%
PTransE,RNNcomposition
Baseline

32. MASSIVE DATA MANAGEMENT Presentation format
4. Evaluation
32
4-2. Comparisons & Analysis
Hit 10 (RAW) Hit 10 (FILTER)
TransE 34.9% 47.1%
TransH 45.7% 64.4%
TransR 43.8% 65.5%
Hit 10 (RAW) Hit 10 (FILTER)
PTransE(RNN)
(Original model)
50.6% 82.2%
PTransE(LSTM)
(Our model)
53.1% 86.6%
PTransE,RNNcomposition
(+9.3%)
Baseline

33. MASSIVE DATA MANAGEMENT Presentation format
4. Evaluation
33
4-2. Comparisons & Analysis
Hit 10 (RAW) Hit 10 (FILTER)
TransE 34.9% 47.1%
TransH 45.7% 64.4%
TransR 43.8% 65.5%
Hit 10 (RAW) Hit 10 (FILTER)
PTransE(RNN)
(Original model)
50.6% 82.2%
PTransE(LSTM)
(Our model)
53.1% 86.6%
PTransE,RNNcomposition
(+21.1%)
Baseline

34. MASSIVE DATA MANAGEMENT Presentation format
4. Evaluation
34
4-2. Comparisons & Analysis
Hit 10 (RAW) Hit 10 (FILTER)
PTransE(ADD)
(Original model)
51.8% 83.4%
PTransE(ADD)
(Our model)
52.1% 84.1%
PTransE(MUL)
(Original model)
47.4% 77.7%
PTransE(MUL)
(Our model)
47.1% 77.2%
PTransE(RNN)
(Original model)
50.6% 82.2%
PTransE(LSTM)
(Our model)
53.1% 86.6%
PTransE,Compositioncomparison

35. Thank you for your attention!
35