2019/09/05 LGCNS AI Tech Talk for NLU (feat.KorQuAD) - 서울대학교 김대식님 - accepted for 2019 ACL - Textbook Question Answering (TQA) with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension

1. Daesik Kim*, Seonhoon Kim*, Nojun Kwak

2. 김대식

3. Textbook QA?
The most realistic QA problem !

4. Textbook QA?
The most complicated QA problem !

5. Context
Question
Input
Answer
Output
Question
Answer
Input
Context
Output
https://tenor.com/view/johnny-johnny5-johnnyfive-reading-scan-gif-12961036
Skimming step

6. … and the title is …
Q What is the title of this paper?
https://media-assets.bookbub.com/wp-content/uploads/
2015/12/cute-baby-skim-reading-gif.gif
Context
Solving step

7. PROBLEM

8. Q Plates move over Earth’s surface
because of _________
a) conduction within the crust.
b) radiation from the inner core.
c) subduction in the outer core.
d) convection within the mantle.
http://www.grygla.k12.mn.us/uploads/2/3/7/1/23718473/earth_sci_-_ch._6_-_plate_tectonics.pdf
Context

9. Q Plates move over Earth’s surface
because of _________
a) conduction within the crust.
b) radiation from the inner core.
c) subduction in the outer core.
d) convection within the mantle.
Q What is the southern most point
of the ring of fire?
a) South sandwich trench
b) Japan trench
c) Aleutian trench
d) Kurile trench
http://www.grygla.k12.mn.us/uploads/2/3/7/1/23718473/earth_sci_-_ch._6_-_plate_tectonics.pdf

10. (Kembhavi et al., 2017)
single sentence / multiple sentences within a paragraph
question/context diagram, and one paragraph

11. METHODS

12. Overall Framework

13. Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
…
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4

14. Q & a#1 & C#2 Correct? Wrong?
Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Top-1
Tf-idf score
…
model

15. Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Top-1
Tf-idf score
…
Q & a#1 & C#2 Correct? Wrong?
Q & a#2 & C#1 Correct? Wrong?
model

16. Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Top-1
Tf-idf score
…
Q & a#1 & C#2 Correct? Wrong?
Q & a#2 & C#1 Correct? Wrong?
Q & a#3 & C#2 Correct? Wrong?
model

17. Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Top-1
Tf-idf score
…
Q & a#1 & C#2 Correct? Wrong?
Q & a#2 & C#1 Correct? Wrong?
Q & a#3 & C#2 Correct? Wrong?
Q & a#4 & C#9 Correct? Wrong?
model

18. Self-supervised comprehension
Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
… Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Top-1
Tf-idf score
Q & a#1 → C#2 (Top-1 tf-idf score)
→ C#4 (Top-2 tf-idf score)
→ C#1 (Top-3 tf-idf score)
→ C#7 (Top-4 tf-idf score)
→ C#5 (Top-5 tf-idf score)

19. Q & a#1 & C#2 Correct
Q & a#1 & C#4 Wrong
Q & a#1 & C#1 Wrong
Q & a#1 & C#7 Wrong
Q & a#1 & C#5 Wrong
Self-supervised comprehension
Context
Paragraph #1
Paragraph #2
Paragraph #3
Paragraph #9
Paragraph #10
Q What is the question?
a) answer option #1
b) answer option #2
c) answer option #3
d) answer option #4
Q & a#1 & C#2 Correct? Wrong?
Q & a#1 → C#2 (Top-1 tf-idf score)
→ C#4 (Top-2 tf-idf score)
→ C#1 (Top-3 tf-idf score)
→ C#7 (Top-4 tf-idf score)
→ C#5 (Top-5 tf-idf score)
Top-1
Tf-idf score…
m
model

20. Self-supervised comprehension

21. Multi-modal context graph understanding: visual context
rabbit
connects to
fox
mouse
ladybird
snake
14 objects
22 stages
…
diagram parsing from UDPnet
(Kim et al., 2018)
parsed info from UDPnet
& OCR info from TQA
build context graph
of diagrams
rabbit
fox
mouseladybird
14
objects
22
stages
context matrix 𝑪 𝒅
adjacency matrix 𝑨 𝒅

22. Multi-modal context graph understanding: textual context
… and the title is …
Context
Q What is the title of this paper?
anchor node
dependency
parsing
filtering by
anchor
nodes
context matrix 𝑪 𝒕
adjacency matrix 𝑨𝒕
3
1
1
1
1
2
2

23. Multi-modal context graph understanding
Question
Answer
word representation
(word embedding;
char representation;
anchor flag)
BiLSTM
Q repr. : 𝒉 𝒒
A repr. : 𝒉 𝒂
visual Context graph
textual Context graph
GCN
GCN GCN
attention
fused representation (f-GCN1) fused representation (f-GCN2)
…
max pooling
𝑯 𝒄
𝟏
𝑯 𝒄
𝟐
𝑨 𝒅
𝑨 𝒕
𝑨 𝒕
𝑨 𝒅
, 𝑨 𝒕
: adjacency matrix
Question & Answer & Context Correct? Wrong?model

24. Multi-modal problem solving
𝑘𝑡ℎ input for the solving RNN
𝒄𝒐𝒏𝒄𝒂𝒕(𝒉 𝒒; 𝒉 𝒂; 𝑨𝒕𝒕 𝒒; 𝑨𝒕𝒕 𝒂; 𝑨𝒕𝒕 𝒒
𝒒𝒅
; 𝑨𝒕𝒕 𝒂
𝒒𝒅
)
BiLSTM
FC
ෝ𝒚 𝟏 ෝ𝒚 𝒌…
𝒌 𝒕𝒉
𝑨𝒕𝒕 𝒒 = 𝒂𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏(𝒉 𝒒, 𝑯 𝒄)
𝑨𝒕𝒕 𝒂 = 𝒂𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏(𝒉 𝒂, 𝑯 𝒄)
𝑨𝒕𝒕 𝒒
𝒒𝒅
= 𝒂𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏(𝒉 𝒒, 𝑯 𝒒𝒅)
𝑨𝒕𝒕 𝒂
𝒒𝒅
= 𝒂𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏(𝒉 𝒂, 𝑯 𝒒𝒅)
only for the diagram question

25. RESULTS

26. Experiments: quantitative results
Model Text T/F Text MC Text All Diagram All
Random 50.10 22.88 33.62 24.96 29.08
MemN+VQA (Kembhavi et al., 2017) 50.50 31.05 38.73 31.82 35.11
MemN+DPG (Kembhavi et al., 2017) 50.50 30.98 38.69 32.83 35.62
BiDAF+DPG (Kembhavi et al., 2017) 50.40 30.46 38.33 32.72 35.39
Challenge - - 45.57 35.85 40.48
IGMN (Li et al., 2018) 57.41 40.00 46.88 36.35 41.36
Our full model w/o visual context 62.32 49.15 54.35 36.61 45.06
Our full model w/ f-GCN2 62.22 48.76 54.11 37.72 45.52
Our full model 62.73 49.54 54.75 37.61 45.77

27. Experiments: quantitative results
Model Text T/F Text MC Text All Diagram All
IGMN (Li et al., 2018) 57.41 40.00 46.88 36.35 41.36
Our full model w/o visual context 62.32 49.15 54.35 36.61 45.06
w/o SSOC(VAL) 60.82 49.08 53.72 36.53 44.72
w/o SSOC(TR+VAL) 60.72 46.34 52.02 36.57 43.93
w/o f-GCN & SSOC(TR+VAL) 58.62 44.77 50.24 35.20 42.36
Our full model w/ f-GCN2 62.22 48.76 54.11 37.72 45.52
w/o SSOC(VAL) 62.63 48.43 54.03 37.32 45.28
w/o SSOC(TR+VAL) 61.42 46.67 52.49 36.71 44.22
w/o f-GCN & SSOC(TR+VAL) 58.72 45.16 50.51 35.67 42.74
Our full model 62.73 49.54 54.75 37.61 45.77
w/o SSOC(VAL) 62.22 48.82 54.11 37.47 45.39
w/o SSOC(TR+VAL) 60.02 46.86 52.06 36.61 43.97
w/o f-GCN & SSOC(TR+VAL) 58.72 45.16 50.51 35.67 42.74
SSOC : Self-Supervised Open-set Comprehension

28. Experiments: quantitative results
Model Text Diagram All
Our full model w/o SSOC 52.06 36.61 43.97
w/o anchor flag (q) 49.29 35.78 42.21
w/o anchor flag (a) 43.24 31.50 37.09
w/o anchor flag (q & a) 42.64 31.72 36.92

29. Experiments: qualitative results
… lithosphere and asthenosphere are
layers based on physical properties .
the outermost layer is the lithosphere .
the lithosphere is the crust and the
uppermost mantle . in terms of physical
properties , this layer is rigid , solid ,
and brittle . it is easily cracked or
broken . below the lithosphere is the
asthenosphere . the asthenosphere is
also in the upper mantle . this layer is
solid , but it can flow and bend . a
solid that can flow is like silly putty ..…
Q what layer is directly below the crust ?
a) mantle
b) core
c) inner layer
d) space
asthenosphere
crust
…
…
uppermost
lithosphere
mantle
below
layer
outer
middle
layer
mantle
crust

30. Experiments: qualitative results
… runoff carved channels in the soil in
figure 19.1 . running water causes most
soil erosion , but wind can carry soil
away too . what humans do to soil
makes it more or less likely to be eroded
by wind or water . human actions that
can increase soil erosion are described
below .…
Q the main cause of soil erosion is ____
a) wind .
b) ice wedging .
c) abrasion .
d) running water .
causes
water
running
…
…
erosion
soil

31. Experiments: qualitative results
… the dense , iron core forms the center
of the earth . scientists know that the
core is metal from studying metallic
meteorites and the earths density .
seismic waves show that the outer core
is liquid , while the inner core is solid .
movement within earths outer liquid iron
core creates earths magnetic field . these
convection currents form in the outer
core because the base of the outer core
is heated by the even hotter inner core…
Q convection currents occur in the inner core .
a) true
b) false
form
core
currents
outer convection
these
in
……

32. Conclusion & Further work
Multi-modal understanding for TQA problems
more paragraphs
visual information
Self-supervised open-set comprehension
without Q-A-Context triple
State-of-the-art performance
still a lot of room for improvement