Peter Lindes - 2017 - Grounding Language for Interactive Task Learning
•
0 likes•27 views
This document describes an AI system that understands natural language commands by parsing sentences into semantic representations using Embodied Construction Grammar. The system uses an ontology, world model, and grammar rules to interpret sentences like "Pick up the green block on the stove" and take appropriate actions in a simulated world. It aims to model how humans understand language by incrementally processing words and representing the meaning, context and intent of commands.
Recommended
More Related Content
More from Association for Computational Linguistics
More from Association for Computational Linguistics (20)
Recently uploaded
Recently uploaded (20)
Peter Lindes - 2017 - Grounding Language for Interactive Task Learning
- 1. Overview Grounding Learns games Learns fetch and deliver tasks ROSie Rosie Motivation How does human processing work? How can we make robots understand us? Lucia Agent Embedding Soar Agent World Model Ontology Input Words Action Messages Comprehender Pick up the green sphere on the stove. Agent Operations Translator ECG Grammar Files Grammar Rules Grammar Rules Context Rules Context Rules Soar Agent World Model Ontology Input Words Action Messages Comprehender Pick up the green sphere on the stove. Agent Operations Translator ECG Grammar Files Grammar Rules Context Rules Example Parse PICK Pick Action Descriptor action pick-up1 PickUp UP up THE the GREEN green Property Descriptor color green1 SPHERE sphere. RefExpr Entity block sphere1 Reference Descriptor object block sphere1green1 large1 Transitive Command ActOnIt large-green-sphere1 Pick up the green sphere. Recognize Evoke Ground Interpret ActOnIt action: @A1001 object: large-green-sphere1 Theoretical Background Comprehender How human language processing works Psychology How to model cognition Computer Science How to represent language knowledge Linguistics Soar Incremental, single path processing Embodied Construction Grammar (ECG) Pick up the green sphere. Embodied Construction Grammar (ECG) construction TransitiveCommand subcase of Imperative constructional constituents verb: ActionVerb object: RefExpr meaning: ActOnIt constraints self.m.action <--> verb.m self.m.object <--> object.m schema Action roles action direction location schema ActOnIt subcase of Action roles object OntologyComprehension State ECG Grammar World Model Benchmark Samples World Model {"id":"obj-006", "handle":"large-green-sphere1", "item_type":"object", "properties":{ "category":"block", "color":"green1", "shape":"sphere1", "size":"large1"}} Gold Standard {"id":"S-061", "text":"Pick up the green block on the stove.", "message":{ "type":"command", "action":"action-006", "object":"obj-004"}} Ontology "categories":[ {"id":"category-001","handle":"block"}, {"id":"category-002","handle":"location"}, {"id":"category-003","handle":"object"}, "colors":[ {"id":"color-001","handle":"red1"}, {"id":"color-002","handle":"orange1"}, {"id":"color-003","handle":"yellow1"}, {"id":"color-004","handle":"green1"}, {"id":"color-005","handle":"blue1"}, Interactive Task Learning Fetch a stapler. What is the goal? The goal is that the stapler is in the starting location. I cannot find the stapler. Can you help? Recall the stapler in a location. (Rosie goes to the location and brings the stapler back.) Deliver the box to the main office. What is the goal? The goal is that the box is in the office. What do I do next? Pick up the box. What do I do next? Put down the box. What do I do next? You are done. Deliver the package to Alice. What is the goal? The goal is that Alice is holding the package. (Rosie picks up the package.) I cannot find Alice. Can you help? Think of an office for Alice. (Rosie goes to her office and gives her the package.) Grounding Language for Interactive Task Learning Peter Lindes, Aaron Mininger, James R. Kirk, and John E. Laird University of Michigan http://www.aclweb.org/anthology/W17-2801 green1 rectangle1 medium1 orange1 triangle1 large1 red1 triangle1 small1 green1 sphere1 large1 green1 rectangle1 large1 stove pantry sink garbage on on Pick up the green block on the stove. TC RE PP PU RE RE Action Descriptor RefDesc verb object TransitiveCommand ActionVerb PICK UP PickUp spec noun SpecNoun spec noun SpecPropNoun prop refExpr prepPhrase RefExprPrepPhrase RefExpr UPPICK THE STOVE BLOCK GREEN ON Specifier CommonNoun Property Preposition prep object PrepPhrase ActOnIt red1 green1 blue1 color block1 category shape rectangle1 triangle1 sphere1 location pantry stove sink garbage stove pantry sink garbage action pick-up1 put-down1 initiate- pick-up1 initiate- put-down1 Action Descriptor Property Descriptor ActOnIt Stove Referent Descriptor Referent Descriptor Referent Descriptor Relation Block Command action: initiate-pick-up1 object: medium-green-rectangle1 Interpret