Spatial Dual-path Model - Object-based pointers and visual heuristics support meaning - Infant physical/social cognition (e.g., old-goals have approach information) - Thematic relations (e.g., caused motion = approach + separation) - Language uses these spatial mechanisms and representations - Eyes move when we understand language (e.g., eye tracking in the visual world) - Spatial biases and errors in development (e.g. FROM passives) - Powerful generalization abilities have been used to argue for innate language knowledge - Critics have argued that domain-general statistical learning can explain language abilities - Language evolved on top of preexisting visual/spatial mechanisms

1. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 1 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Linking vision and language:
From infant cognition to eye-
tracking in the visual world
Franklin Chang and Andrew Jessop —- University of Liverpool

2. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 2 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
VisionandLanguage
How are vision and language linked?·
Meaning mediated account-
2/24

3. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 3 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Evidenceforalinkbetweenvisionandlanguage
Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)
Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)
·
CM I put the doll here -> SP the doll goes here (Bloom, 1993)
Two perspectives on the same visual scene: Child or Doll
-
-
·
Normal BY passives 受け身: I was caught by you
Children never hear caught from in input
Spatial similarity between Agents and Sources
-
-
-
The girl is running from the man (man = SOURCE 起点)
the girl is chased by the man (man = AGENT 動作主)
-
-
3/24

4. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 5 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
LinkingVisual-spatialcognitionandLanguage
Standard Approach: Vision and language are separate modules/systems
Alternative Approach: Language is built on pre-existing spatial mechanisms
·
Concepts mediate between vision and language-
·
Use infant/adult visual cognition to motivate a rich set of spatial mechanisms and
representations
Show how these can be linked to thematic roles in language
Present a connectionist model that uses these representations to explain eye-tracking in
the visual world
-
-
-
5/24

5. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 6 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Thematicroles(主題役割)
Agent 動作主 = doer of action (child in I put the doll here)
Patient 被動者 = affected by action (doll)
Theme 主題 = moved by action (doll in the doll goes here)
How do children figure out if the doll is patient or theme?
Existing theories use abstract features (e.g., volition 意志, sentience 感覚性; Dowty, 1991)
·
·
·
·
Reality: Child moves the doll (child is agent)
Imagination: Doll is moving by itself (doll is theme)
-
-
·
These features are not defined clearly
Vision-based relational features are needed
-
-
6/24

6. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 7 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
AVision-basedTheoryofThematicRoles
Thematic roles in language make use of mechanisms that developed for object tracking
Vision has evolved powerful mechanisms for tracking objects
·
These two views are consistent with two tigers
Object tracking might show that there is only one tiger
-
-
·
Spatial pointers support object tracking (Pylyshyn & Storm, 1988)- 7/24

7. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 8 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
SpatialIndexes/Objectfiles(Pylyshyn&Storm,1988)
Participants see initial scene with identical crosses
Some crosses are highlighted as targets
Crosses move which participants keeps attention in center of screen
Cross is highlighted at test and participant responds whether it is one of the targets
·
·
·
·
8/24

8. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 9 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
SpatialPointersandVisualheuristics
Wolf is the agent of chasing (sheep are patients)
Approach visual heuristic
Visual heuristics can be used to identify thematic roles
·
Wolf moves towards sheep (approach)
Wolf and sheep are identical circles (object tracking)
-
-
·
Angle of motion (Gao et al., 2009)
Goal-object motion (Woodward, 1998)
-
-
·
Innate (infants for approach, Luo, 2011)-
9/24

9. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 10 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Separation,Distance,andPrimacyHeuristic
Separation heuristic
Distance heuristic
Primacy heuristic
·
Reverse of Approach Heuristic-
·
Represents the distance between pairs of objects in the scene
Lew, Bremner, and Lefkovitch (2000) found that 8.5 month old infants encode a hidden
toy’s position between two landmarks after displacement to a new location
-
-
·
Represents the order of motion of objects in scenes
Bullock & Gelman (1979) showed two events where balls went down a ramp, but one
ball went down before a jack-in-the-box appeared and another ball went down
afterwards. Children labeled the first ball as the cause.
-
-
10/24

10. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 11 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
AConnectionistModelofLanguageAcquisition
Simple recurrent network (Elman 1990)·
11/24

11. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 12 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Dual-pathmodel(Chang2002)
Simple recurrent network learns sequencing constraints
Message: Fast changing weights between roles and concepts
·
·
12/24

12. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 13 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
SpatialDual-pathmodel
Roles are replaced with spatial pointers/indexes (e.g., P1)
Visual heuristics are input to SRN and help in identifying thematic roles
·
·
13/24

13. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 14 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
ExplainingEye-trackinginProduction
Griffin & Bock (2000) eye-tracking during sentence production·
Look at subject before they produce head noun-
14/24

14. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 15 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Eyetrackingduringnounproduction
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"
produced
·
15/24

15. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 16 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
OutputMessageSupportsPredictiveEyemovements
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"
produced
·
16/24

16. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 17 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
OutputMessageSupportsPredictiveEyemovements
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"
produced
·
17/24

17. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 18 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
ExplainingEye-trackinginComprehension
Altmann (2004) Blank Screen Paradigm·
The boy will eat the cake
Screen is blank when eye-tracking is performed
-
-
18/24

18. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 19 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Internalobjecttrackingisneededtoexplainblankscreenstudies
Input and output messages are set when picture is visible·
19/24

19. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 20 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
InputMessageSupportsReactiveEyemovements
Attention is shifted to corresponding input pointer, even when screen is blank·
20/24

20. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 21 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
InputMessageSupportsReactiveEyemovements
Attention is shifted to corresponding input pointer, even when screen is blank·
21/24

21. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 22 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
InputMessageSupportsReactiveEyemovements
Attention is shifted to corresponding input pointer, even when screen is blank·
22/24

22. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 23 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Errorsandstructuralchanges
Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)
Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)
·
CM I put the doll here -> SP the doll goes here (Bloom, 1993)
Changing thematic roles: AGENT=CHILD PATIENT=DOLL -> THEME=DOLL
Spatial similarity in visual heuristics creates these changes in the model
-
-
-
·
Normal BY passives: I was caught by you
Changing AGENT=CHILD -> SOURCE=CHILD
Spatial similarity between agents and sources create these errors in the model
-
-
-
23/24

23. 7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 24 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
SpatialDual-pathModel
Object-based pointers and visual heuristics support meaning
Language uses these spatial mechanisms and representations
Powerful generalization abilities have been used to argue for innate language knowledge
·
Infant physical/social cognition (e.g., old-goals have approach information)
Thematic relations (e.g., caused motion = approach + separation)
-
-
·
Eyes move when we understand language (e.g., eye tracking in the visual world)
Spatial biases and errors in development (e.g. FROM passives)
-
-
·
Critics have argued that domain-general statistical learning can explain language
abilities
Language evolved on top of preexisting visual/spatial mechanisms
-
-
24/24