발표일: 2018.1. 발표자: 서홍석(POSTECH 박사과정) Visual dialog is a task of answering a series of inter-dependent questions given an input image, and often requires to resolve visual references among the questions. This problem is different from visual question answering (VQA), which relies on spatial attention (a.k.a. visual grounding) estimated from an image and question pair. We propose a novel attention mechanism that exploits visual attentions in the past to resolve the current reference in the visual dialog scenario. The proposed model is equipped with an associative attention memory storing a sequence of previous (attention, key) pairs. From this memory, the model retrieves the previous attention, taking into account recency, which is most relevant for the current question, in order to resolve potentially ambiguous references. The model then merges the retrieved attention with a tentative one to obtain the final attention for the current question; specifically, we use dynamic parameter prediction to combine the two attentions conditioned on the question. Through extensive experiments on a new synthetic visual dialog dataset, we show that our model significantly outperforms the state-of-the-art (by ~16 % points) in situations, where visual reference resolution plays an important role. Moreover, the proposed model achieves superior performance (~ 2 % points improvement) in the Visual Dialog dataset, despite having significantly fewer parameters than the baselines.

1. Visual Reference Resolution using
Attention Memory for Visual Dialog
Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, Leonid Sigal
{hsseo, bhhan}@postech.ac.kr {andreas.lehrmann, lsigal}@disneyresearch.com
NIPS 2017

2. Visual Dialog
• Similar to Visual Question Answering (VQA)
• Answering sequential questions grounded to a visual input.
• Ambiguous expressions due to previous references.
• Attention models dominate VQA
Q: How many people are on wheelchairs?
A: Two.
Q: What are their genders?
A: One male and one female.
Q: Which one is holding a racket?
A: The woman.
A: Two.
Q: What are their genders?
A: One male and one female.
Q: Which one is holding a racket?
A: The woman.
Ambiguous expressions due to earlier question referring.
Motivation
§ Attention models:
Dominant approach for answering independent questions (VQA).
§ Ambiguous expressions:
Ambiguous visual reference without understanding dialog contexts.
Q: What is sitting on the handle bar of a bicycle? Q: What are their colors?
?
A: Two.
Q: What are their genders?
A: One male and one female.
Q: Which one is holding a racket?
A: The woman.
Ambiguous expressions due to earlier question referring.
Motivation
§ Attention models:
Dominant approach for answering independent questions (VQA).
§ Ambiguous expressions:
Ambiguous visual reference without understanding dialog contexts.
Q: What is sitting on the handle bar of a bicycle? Q: What are their colors?
?
2

3. Attention-based Image Encoding
• Extracting features from relevant regions of image by
attention based on input question (𝑞) and history (𝐻).
3

4. Visual Reference Resolution
• Obtain tentative attention 𝜶 𝑡
tent
and retrieved attention 𝜶 𝑡
mem
using
• Question embedding 𝒄 𝑡
• Image feature map 𝒇
• Memory 𝑴 𝑡 = {(𝜶 𝜏, 𝒌 𝜏)|0 ≤ 𝜏 ≤ 𝑡 − 1}
…
…
attention retrieval
𝜶 𝜏 𝒌 𝜏
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
tentative attention
𝒇
𝜶 𝑡
tent
𝑴 𝑡
4

5. Visual Reference Resolution
• Final attention 𝜶 𝑡 is computed by dynamically combining
• Tentative attention 𝜶 𝑡
tent
• Retrieved attention 𝜶 𝑡
mem
…
…
attention retrieval
𝜶 𝜏 𝒌 𝜏
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
tentative attention
𝒇
𝜶 𝑡
tent
𝑾DPL
𝒄 𝑡
𝜶 𝑡
dynamic combination
𝒄 𝑡
𝑴 𝑡
5

6. Tentative Attention Computation
• Attention score is computed
at every spatial location.
• By inner-product of image
feature and question
embedding
• As in many VQA models
…
…
attention retrieval
𝜶 𝜏 𝒌 𝜏
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
tentative attention
𝒇
𝜶 𝑡
tent
𝑾DPL
𝒄 𝑡
𝜶 𝑡
dynamic combination
𝒄 𝑡
𝑴 𝑡
6

7. • Retrieval from key-value
memory
• Sequential preference
Attention Retrieval from memory
…
𝒌0
𝒌1
𝒌 𝑡−1
…
…
𝜶0
𝜶1
𝜶 𝑡−1
𝜷 𝑡
∑ ∑
𝑾mem
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
keysattentions
…
…
attention retrieval
𝜶 𝜏 𝒌 𝜏
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
tentative attention
𝒇
𝜶 𝑡
tent
𝑾DPL
𝒄 𝑡
𝜶 𝑡
dynamic combination
𝒄 𝑡
𝑴 𝑡
and
⇒
7

8. Dynamic Combination of Attentions
• Local merge using a
convolution layer.
• Global merge using
dynamic parameter layer
from (Noh et al., 2016).
• Weights 𝑾DPL
(𝒄 𝑡) are
dynamically predicted
from question
embedding.
…
…
attention retrieval
𝜶 𝜏 𝒌 𝜏
𝒄 𝑡
𝜶 𝑡
mem
𝒌 𝑡
mem
tentative attention
𝒇
𝜶 𝑡
tent
𝑾DPL
𝒄 𝑡
𝜶 𝑡
dynamic combination
𝒄 𝑡
𝑴 𝑡
8

9. MNIST Dialog
• Synthetic visual dialog dataset that highlights model’s
ability resolving visual references.
• Designed to contain ambiguous expressions and strong inter-
dependency among questions in a dialog.
9

10. Results on MNIST Dialog
• Attention based models perform better.
• The proposed model outperforms baselines.
• Use of memory component stabilizes the performance at
later steps of dialog.
10

11. Qualitative Result on MNIST Dialog
• Semantically reasonable use of retrieved attention from
attention memory.
11

12. Parameter Analysis
• Memory addressing coefficients with/without sequential
preference.
• t-SNE plots of dynamically predicted weights.
12

13. Results on Visual Dialog
• Proposed model outperforms previous works with a fewer
number of parameters.
13

14. Conclusion
• We proposed a novel method for visual dialog resolving
visual references through attention memory.
• The proposed method can retrieve the attention of
ambiguous expressions from the memory to resolve the
reference of the current target.
• Memory retrieval process allows the model to obtain
semantically reasonable attention from attention history.
• Our model shows the state-of-the-art performance on both
synthetic and real datasets.
14

15. Visual Reference Resolution using
Attention Memory for Visual Dialog
Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, Leonid Sigal
{hsseo, bhhan}@postech.ac.kr {andreas.lehrmann, lsigal}@disneyresearch.com
NIPS 2017
Visit http://cvlab.postech.ac.kr/research/attmem/ for more information