Workshop on Language Grounding for Robotics - W17-2801

1. Overview Grounding
Learns
games
Learns fetch
and deliver
tasks
ROSie
Rosie
Motivation
How does
human
processing
work?
How can we
make robots
understand
us?
Lucia
Agent Embedding
Soar Agent
World Model
Ontology
Input
Words
Action
Messages
Comprehender
Pick up the
green sphere
on the stove.
Agent
Operations
Translator
ECG
Grammar
Files Grammar
Rules
Grammar
Rules
Context
Rules
Context
Rules
Soar Agent
World Model
Ontology
Input
Words
Action
Messages
Comprehender
Pick up the
green sphere
on the stove.
Agent
Operations
Translator
ECG
Grammar
Files Grammar
Rules
Context
Rules
Example Parse
PICK
Pick
Action
Descriptor
action pick-up1
PickUp
UP
up
THE
the
GREEN
green
Property
Descriptor
color green1
SPHERE
sphere.
RefExpr
Entity
block sphere1
Reference
Descriptor
object
block sphere1green1 large1
Transitive
Command
ActOnIt
large-green-sphere1
Pick up
the green
sphere.
Recognize
Evoke
Ground
Interpret
ActOnIt
action: @A1001
object: large-green-sphere1
Theoretical Background
Comprehender
How human language processing works
Psychology
How to model cognition
Computer Science
How to represent language knowledge
Linguistics
Soar
Incremental,
single path
processing
Embodied
Construction
Grammar (ECG)
Pick up
the green
sphere.
Embodied Construction
Grammar (ECG)
construction TransitiveCommand
subcase of Imperative
constructional
constituents
verb: ActionVerb
object: RefExpr
meaning: ActOnIt
constraints
self.m.action <--> verb.m
self.m.object <--> object.m
schema Action
roles
action
direction
location
schema ActOnIt
subcase of Action
roles
object
OntologyComprehension State
ECG Grammar
World Model
Benchmark Samples
World Model
{"id":"obj-006",
"handle":"large-green-sphere1",
"item_type":"object",
"properties":{
"category":"block",
"color":"green1",
"shape":"sphere1",
"size":"large1"}}
Gold Standard
{"id":"S-061",
"text":"Pick up the green block on the stove.",
"message":{
"type":"command",
"action":"action-006",
"object":"obj-004"}}
Ontology
"categories":[
{"id":"category-001","handle":"block"},
{"id":"category-002","handle":"location"},
{"id":"category-003","handle":"object"},
"colors":[
{"id":"color-001","handle":"red1"},
{"id":"color-002","handle":"orange1"},
{"id":"color-003","handle":"yellow1"},
{"id":"color-004","handle":"green1"},
{"id":"color-005","handle":"blue1"},
Interactive Task
Learning
Fetch a stapler.
What is the goal?
The goal is that the stapler is in the starting
location.
I cannot find the stapler. Can you help?
Recall the stapler in a location.
(Rosie goes to the location and brings the stapler
back.)
Deliver the box to the main office.
What is the goal?
The goal is that the box is in the office.
What do I do next?
Pick up the box.
What do I do next?
Put down the box.
What do I do next?
You are done.
Deliver the package to Alice.
What is the goal?
The goal is that Alice is holding the package.
(Rosie picks up the package.)
I cannot find Alice. Can you help?
Think of an office for Alice.
(Rosie goes to her office and gives her the
package.)
Grounding Language for Interactive Task Learning
Peter Lindes, Aaron Mininger, James R. Kirk, and John E. Laird
University of Michigan
http://www.aclweb.org/anthology/W17-2801
green1
rectangle1
medium1
orange1
triangle1
large1
red1
triangle1
small1
green1
sphere1
large1
green1
rectangle1
large1
stove pantry
sink garbage
on
on
Pick up the green block on the stove.
TC
RE
PP
PU RE
RE
Action
Descriptor
RefDesc
verb object
TransitiveCommand
ActionVerb
PICK UP
PickUp
spec noun
SpecNoun
spec noun
SpecPropNoun
prop
refExpr prepPhrase
RefExprPrepPhrase
RefExpr
UPPICK
THE STOVE BLOCK GREEN ON
Specifier CommonNoun Property Preposition
prep object
PrepPhrase
ActOnIt
red1
green1
blue1
color
block1
category
shape
rectangle1
triangle1
sphere1
location
pantry
stove
sink
garbage
stove pantry
sink garbage
action
pick-up1
put-down1
initiate-
pick-up1
initiate-
put-down1
Action
Descriptor
Property
Descriptor
ActOnIt
Stove
Referent
Descriptor
Referent
Descriptor
Referent
Descriptor
Relation
Block
Command
action: initiate-pick-up1
object: medium-green-rectangle1
Interpret