Expressing and understanding dialogue act: Is it an explicit or an implicit process? It must be implicit because: - The age of acquisition - Reaction time Additional evidence - People tend to give postdictive explanations that are not always correct for what they did.

1. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
Natsuki OKA
Faculty of Information and Human Sciences
Kyoto Institute of Technology

2. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
Situation
Partner’s action
Automatic processing
implicit ToM
Own action
Post hoc explanation
explicit ToM

3. Dialogue Act (DA)
• A dialogue act (DA) represents the meaning of an
utterance at the illocutionary force level (Austin
1962) such as a question, a request, and a
greeting.
• In Japanese, DAs can be estimated almost exactly
by focusing only on sentence-final particles which
are one type of function words.
• We have been trying to build a robot which learns
response to a speech by estimating DA.
→https://arxiv.org/abs/1810.09949

4. It looks yummy. (I want to confirm that
we both think it looks yummy.)
Oishi sou da ne.
It looks
yummy. It looks
yummy.
https://arxiv.org/abs/1810.09949

5. It’s yummy. (I want to confirm that we
both believe we both feel yummy.)
Oishii ne. yum
my
yum
myyum
my
yum
my
Explicit ToM (postdictive reconstruction):
explicit representation of the percept or belief of others
nested
A picture of two people eating ice cream (omitted)
A picture of two people
eating ice cream
(omitted)
A picture of two people
eating ice cream
(omitted)

6. It’s yummy. (I want to inform you that
it’s yummy.)
Oishii yo.
? yummy
A picture of one person eating ice cream and the other looking at it (omitted)

7. It’s yummy. (I want to confirm that we
both believe you feel yummy.)
Oishii ne.
yummy
yummy
Photograph of a mother feeding food to her baby (omitted)
Photograph
of a baby

8. You've grown up. (I want to confirm
that we both think you’ve grown up.)
Ookiku natta ne.
Picture of an adult talking with a boy (omitted)
baby
boy
baby
boy

9. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
• It must be implicit because:
1. The age of acquisition
2. Reaction time
• Additional evidence
1. People tend to give postdictive explanations that
are not always correct for what they did.

10. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
• It must be implicit because:
1. The age of acquisition
2. Reaction time
• Additional evidence
1. People tend to give postdictive explanations that
are not always correct for what they did.

11. Explicit ToM
• Four-year-olds: Pass
• Three-year-olds: Fail
• My own belief ≠
Sally’s belief

12. Age of acquiring Japanese words that
express the mental attitude of speakers
(Watamaki, 2016)
Age in month
20-23 ne (sympathy)
24-27 no
yo (information delivery)
te
tai
28-31 ka (question)
da
you/u

13. Fig. 1 Logical structure of events in Experiment 1. (omitted)
Ágnes Melinda Kovács et al. Science 2010;330:1830-1834
Implicit or spontaneous ToM:
Ball detection task
Participants were instructed
to press a button as soon as
they detected the ball.
Participants were instructed
to press a button when the
agent left the scene.

14. Fig. 2 (A) Results of Experiment 1 (adults; agent present in the last scene). (omitted)
Ágnes Melinda Kovács et al. Science 2010;330:1830-1834
Adults automatically compute and store the beliefs of other agents (implicit ToM: no
distinction among the owners of beliefs?).

15. Violation of the agent’s expectation -> Looking time
https://science.sciencemag.org/content/330/6012/1830/tab-figures-data
movie (omitted)

16. The mere presence of social agents is sufficient to
automatically trigger online belief computations not
only in adults, but also in 7-month-old infants.
Fig. 3 Results of Experiments 4 to 7. (omitted)
Ágnes Melinda Kovács et al. Science 2010;330:1830-1834
Infants computed the agent’s belief and looked
longer when this belief was not confirmed.
violation of expectation paradigm

17. Brain activation for spontaneous and explicit
false belief tasks overlaps: new fMRI evidence
on belief processing and violation of
expectation
Lara Bardi, Charlotte Desmet, Annabel Nijhof, Jan R.
Wiersema, Marcel Brass
Social Cognitive and Affective Neuroscience, Volume 12,
Issue 3, March 2017, Pages 391–
400, https://doi.org/10.1093/scan/nsw143
Published: 01 October 2016

18. During both the spontaneous and the explicit task,
a question appeared randomly in 18 trials after the
end of the movie. … In the spontaneous version, the
question was: ‘Did Buzz have a blue cap?’ … In the
explicit version, the question was: ‘Did Buzz think
the ball was behind the screen?’
Figure omitted

19. Fig. 3. Left panel. Cluster of activation in the PPC for the contrast false
belief > true belief (irrespective of the task) during the belief formation
phase. Right panel. Clusters of activation in the MPFC for the participant’s
violation of expectation (participant positive content prediction > negative
outcome in green) and the agent’s violation of expectation (agent positive
content prediction > negative outcome in blue) (irrespective of the task) in the
outcome phase.
Figure omitted

21. Figure omitted

22. * p=.03
Graph showing experimental results (omitted)

23. Do 18-month-olds really attribute mental states to
others? A critical test.
Senju A, Southgate V, Snape C, Leonard M, Csibra G.
• https://journals.sagepub.com/doi/suppl/10.1
177/0956797611411584
picture omitted

24. Results
• First gaze
– Opaque condition: 14/18 = left window (p=.031)
– Trick condition: 6/18 = left window (p=.238)
– First gaze ← opaque/trick (p=.018)
• Looking time
Differential Looking Score = (left-right)/(left+right)
– Opaque condition: above zero (p=.03)
– Trick condition: n.s. (p=.15)
– DLS ← opaque/trick (p=.012)

25. Mirror neuron
Figures omitted

26. The age of acquisition
1 2 3 4
Implicit
ToM
Explicit
ToM
Ne
0 5
Mirror
neuron
system?
Distinction
between
you/me?
YoImplicit
ToM
…

27. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
• It must be implicit because:
1. The age of acquisition
2. Reaction time
• Additional evidence
1. People tend to give postdictive explanations that
are not always correct for what they did.

28. There are a few postdictive perceptual phenomena
known, in which a stimulus presented later seems
causally to affect the percept of another stimulus
presented earlier. …
… By extending the list of postdictive phenomena to
memory, sensory-motor and higher-level cognition,
one may note that such a postdictive reconstruction
may be a general principle of neural computation,
ranging from milliseconds to months in a time scale,
from local neuronal interactions to long-range
connectivity, in the complex brain. …

29. Benjamin Libet’s findings on postdictive process,
and backward referral (Libet, B.(2004). MindTime)
Figure omitted

30. Libet’s functional account of the backward referral
in the real world (Libet, B.(2004). MindTime)
Implicit and fast sensory-motor pathways
A boy runs
to the road
Brake
Aware of the boy
Report of
awareness
Backward referral

31. The alternation latency
in dialogue is often less
than 500 milliseconds,
which is the time
necessary for the
contents of the
conversation partner's
speech to be conscious.
Table3 Duration of switching pauses (omitted)

32. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
• It must be implicit because:
1. The age of acquisition
2. Reaction time
• Additional evidence
1. People tend to give postdictive explanations that
are not always correct for what they did.

33. RP precedes the reportable time (W) for appearance of
the subjective experience of 'wanting' or intending to act.
Figures omitted

34. Participants made choices between presented face pairs on the basis of
attractiveness, while the experimenter covertly manipulated the relationship
between choice and outcome. Participants offered introspectively derived reasons
for why they chose the way they did.
TED: Do you really know why
you do what you do? | Petter
Johansson
picture ｏmitted

35. post hoc construction
The participants often came to prefer the alternative.
“She is radiant. I would
rather have approached
her at a bar than the
other one. I like
earrings!”
Pictures omitted

36. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
• It must be implicit because:
1. The age of acquisition
2. Reaction time
• Additional evidence
1. People tend to give postdictive explanations that
are not always correct for what they did.

37. Expressing and understanding dialogue act:
Is it an explicit or an implicit process?
Situation
Partner’s action
Automatic processing
implicit ToM
Own action
Post hoc explanation
explicit ToM

38. Generating ne/yo without explicit ToM:
No distinction among the owner of percept, but the information source is distinguished?
Audio-visual
system
Mirror neuron system
Somatosensory
system
- Action of self
- Sensor of self
- Action of others
- Appearance of others
yo is associated
with activated MNS
only through the
blue path.
ne yo
ne is associated
with activated MNS
through the yellow
path.
Percept: yummy
Taste
oishii

39. Interpreting ne/yo without explicit ToM:
No distinction among the owner of percept, but the information source is distinguished?
Audio-visual
system
Mirror neuron system
Somatosensory
system
- Action of self
- Sensor of self
- Action of others
- Appearance of others
ne: confirm the
contents by
activating MNS
through the yellow
path;
yo ne
yo: confirm the
contents by
activating MNS
only through the
blue path;
Percept: yummy
Taste
oishii

40. Cognitive Interaction Design
A Model-based Understanding of Communication
and its Application to Artifact Design
https://www.cognitive-interaction-design.org/english-digest-1/
The mental model of others, which is used for understanding and predicting
partners’ actions under certain situations, plays an important role in human
communication. In fact, we sometimes feel a gap in the conversation with a
stranger, since we do not have such a mental model of others at first. This kind of
phenomenon, namely the communication facilitation by the mental model of
others, is expected to find in the interactions between humans and companion
animals/artifacts as well as in human-human communication.
This research project is aiming at establishing a new academic field which we call
“Cognitive Interaction Design (CID)”. We strongly believe that the cognitive
construction of “mental model of others” is a key to establish the design and
implementation of artifacts that can adapt themselves to human users naturally
and persistently. To this end, we focus especially on the common cognitive
process among human-human communication as well as the interactions
between humans and animals/artificial agents. Then we implement the mental
model of others at an algorithmic level and utilize it for the design of artifacts.

41. Baker, Chris & Jara-Ettinger, Julian & Saxe,
Rebecca & B. Tenenbaum, Joshua. (2017).
Rational quantitative attribution of beliefs,
desires and percepts in human mentalizing.
Nature Human Behaviour. 1. 0064.
DOI: 10.1038/s41562-017-0064.
mentalize: To understand the behavior of others as a product of their mental state

42. Machine Theory of Mind
Neil C. Rabinowitz, Frank Perbet, H. Francis
Song, Chiyuan Zhang, S.M. Ali Eslami, Matthew
Botvinick
arXiv:1802.07740v2

43. レポート課題
1. 他者の発話意図の理解が意識的に行われ
ているか、自動的に行われているかに関す
るあなたの見解（仮説）を記せ。
2. その見解（仮説）が正しいかどうかを確かめ
る方法を、できるだけ具体的に、できるだけ
詳細に記せ。
（A4で1ページ程度を標準とする）