1. Incrementality and Anticipation in a
Scaleable Network Model of Linguistic
Competence and Performance
Marshall R. Mayberry, III
Matthew W. Crocker
Saarland University
2. Goal of Computational Psycholinguistics
• Incremental models of language understanding
• Competence
– Transparent semantic representation
– Wide coverage
– Accuracy
• Performance
– Incremental interpretation and ambiguity resolution
– On-line behavior (e.g., RTs, ERPs, and eye-tracking)
– Adaptation to context
– Robustness
4. Connectionist Language Modelling
• General architecture with simple memory
• Parsimonious account of language processing
• Sentence mapped into a case-role representation
• Incremental development of interpretation with revision
5. BUT …
• Fixed output with prespecified role assemblies
• Multiple case-role frames? Binding?
• Embedded clauses? Multiple Phrases?
• Cognitively appealing but difficult to scale up.
6. INSOMNet
Incremental Nonmonotonic Self-Organization of
Meaning Network (Mayberry, 2003)
• Based on Simple Recurrent Network (Elman, 1990)
• Generates explicit compositional representations
• Incremental sentence processing
• Scaled up to Redwoods Treebank/VerbMobil (English)
NEGRA (German)
10. John saw Mary leave
Change in subcategorization frame for saw causes a
different map node to be used.
11. Scaling Up Without Dumbing Down
• LinGO Redwoods Treebank (Oepen et al, 2002)
– 5000 sentences from VerbMobil (Wahlster, 2000) -
dialogues of face-to-face business travel arrangements
– Rich semantic annotation based on Minimal Recursion
Semantics (Copestake et al, 2001)
– 71.6% recovery of highly-activated dependents
• NEGRA (Skut et al, 1997)
– 20000 sentences of German newspaper text
– Syntactic annotation based on dependency grammar
– 62% accuracy on sentences of less than 15 words
12. INSOMNet Summary
Model of comprehension
Wide coverage
Explicit semantic representations constructed
incrementally
1. Preferred interpretation
2. Nonmonotonic revision/reanalysis
3. Anticipation
4. Demonstrate observed on-line behavior
5. Adaptation to diverse constraints
6. Modelling Comprehension in Visual Worlds
13. Visual Worlds
• Attention in visual scene reveals interpretation
– Anticipation (Altmann & Kamide, 1999)
– Rapid integration of diverse linguistic constraints
• Case and Argument structure (Kamide et al, 2003)
• Plausibility (Altmann & Kamide, 1999)
• Prosody (Weber et al, 2003)
• Scene itself can influence interpretation
– Preferred interpretation (Tanenhaus et al, 1995)
– Establish relationships
• Referential Context (Tanenhaus et al, 1995)
• Depicted Actions (Knoeferle et al, 2003)
14. German
• Flexible constituent order
– Second element must be verb or auxiliary
– Subject or object can occur before or after the verb
• Case-marking often marks subject or object
Die Prinzessin wäscht gleich der Pirat.
(“The princessOBJ washes soon the pirateSUBJ ”)
• Facilitates the study of incremental interpretation
of thematic roles
15. Linguistic Constraints on Anticipation
• Case and Plausibility (Kamide et al, 2003)
Der Hase frisst gleich den Kohl. / Den Hasen frisst gleich der Fuchs.
(“The hareSUBJ eats soon the cabbageOBJ”) / (“The hareOBJ eats soon the foxSUBJ”)
• Argument Structure (Scheepers et al, submitted)
Der Hase interessiert gleich den Fuchs. / Den Hasen interessiert gleich der Kohl.
(“The hareSUBJ interests soon the foxOBJ”) / (“The hareOBJ interests soon the cabbageSUBJ”)
16. Influence of Visual Context
Knoeferle, Crocker, Scheepers, and Pickering (to appear)
• SVO: Die Prinzessin malt offensichtlich den Fechter.
(“The princessSUBJ paints soon the fencerOBJ”)
• OVS: Die Prinzessin wäscht offensichtlich der Pirat.
(“The princessOBJ washes soon the pirateSUBJ”)
17. Empirical Results
• Attention to scene shows
– Scene information rapidly integrated with
utterance to anticipate upcoming arguments.
(Kamide et al, 2003)
• Influence from scene shows
– Depicted actions allow early role
disambiguation even in absence of selectional
restrictions. (Knoeferle et al, to appear)
18. Modelling Incremental Interpretation
and Anticipation
• Three experiments with complementary properties
– Plausibility
– Depicted actions
– Ambiguity
• Training data consists of all combinations of referents such
that actual experimental sentences are held out.
• Scene information is provided as context for half of
training data. Testing with and without scene.
– Demonstrate anticipatory behavior of empirical studies.
– Time course of ambiguity resolution should show
correspondence to gaze fixations.
19. Simulations
• Two simulations trained on data from three experiments
• Small-scale
– 16 verbs
– 24 nouns
– 48 non-experimental nouns
– All sentences have matching scene
– 50% trained with and without scene
• Large-scale
– 80 verbs
– 120 nouns
– 240 non-experimental nouns
– Fillers with extra sentence constructions and words
– Non-matching scenes
34. Small-scale Simulation
• Close to 100% accuracy with and without
scene information
– End of sentence
– Proper selection of plausible arguments and
linking in first experiment
– Early disambiguation using scene in second
• Demonstrates model’s potential to adapt to
and use scene information when available.
35. Large-Scale Simulation
• Scaling up to complete experiments
– 15% of sentences match empirical behavior.
– Typically agent and patient roles are confused.
• Only a small portion of architectural
parameters have been explored so far.
– How best to encode scene?
– Sizes of scene and output maps?
– Relative strengths of links?
– How much noise to add?
36. Conclusions
• Incremental Nonmonotonic Sentence Processing
• Semantic comprehension with soft constraints
– Cognitive model of sentence processing
– Parallel, nonmonotonic interpretations
• Scales up to realistic language
• Shows promise for modelling fine-grained
adaptive behavior to context
37. Linguistic Constraints on Anticipation
• Case and Plausibility
(Kamide et al, 2003)
– Der Hase frisst gleich den
Kohl.
– Den Hasen frisst gleich
der Fuchs.
• Argument Structure
(Scheepers et al, submitted)
– Der Hase interessiert
gleich den Fuchs.
– Den Hasen interessiert
gleich der Kohl.
39. Semantic Frame Encoder/Decoder
• SRN output is map of encoded semantic frames
• Dedicated links for specific components
– Verb frames have WORD, TENSE, AGT, and PAT links
– Noun frames have WORD and GENDER links
41. Semantic Self-Organization
• Frame Modulator Map
– Provides graded Frame Node selection
– Trained through back-propagation
– Target is the Target Indicator Map
• Target Indicator Map
– Only used during training
– Self-organized using compressed frames
– Selects Frame Nodes to encode semantic frames
42. INSOMNet
Putting It All Together
• Sequence Processor reads in sentence incrementally
• Frame En/Decoder represents semantic frames
• Frame Selector models graded frame selection
43. Human Sentence Processing
• Sentence interpretation is constructed incrementally
• Many factors involved (e.g., experience, plausibility)
• Ambiguity is pervasive
– How is preferred interpretation determined?
