This document summarizes research into the intersection of language, artificial intelligence (AI), and virtual reality (VR). It discusses three key points: 1) How robots are learning language and building common ground through studies of word meanings. 2) The promise of VR/AR for creating shared understanding and directing attention during language learning. 3) How VR impacts the brain's representation of space and time based on mismatches between visual and other sensory inputs. The research aims to advance social technologies through collaborative work between computer scientists, roboticists, psychologists and other experts.

1. www.itee.uq.edu.au/research/co-innovation
Co-Innovation Research ITEE
Language in social technologies:
what happens when AI meets VR?
Janet Wiles
Co-Innovation Group
ARC Centre of Excellence for the Dynamics of Language &
School of Information Technology & Electrical Engineering
University of Queensland
SIGGRAPH Brisbane, 17 November 2019

2. Overview
1. Robots learning language
Studies in grounding
2. Language in AI: the promise of VR/AR
Common ground: learning new words, words in context;
shared understanding; directing attention
3. VR and the brain
Conversation: checking back, interactions, speed

3. 3
A family of child-friendly robots for
research into the effects of social
robots on learning from tablets.
Design: ongoing, iterative, immersive
UQ Social robots
A bio-inspired robot for research at the
intersection of neurorobotics,
neuroscience and embodied cognition.
Design: D Ball; Programming: S Heath
OPIEiRat

4. Lingodroids: Could a robot
understand language?
Scott Heath
Ruth Schulz and
the Lingodroids
team

5. CATgrounded meaning
Common ground and
the triangle of meaning
Representamen
Object Interpretant

6. Can a child teach the
robot words for colour?
Kristyn Hensby
Scott Heath
OPAL Team
OPIE
Language
games

7. purple
purple
purple
purple
INTENSITY
HUE
blue
Colour space
blue
name-this-colour?
blue
what’s-the-difference?
Scott Heath
lighter

8. gifugrounded meaning
Common ground and
the triangle of meaning
Representamen
Object Interpretant

9. yozozuku
Which word refers to space
and which to time?
Scott Heath
spacetime
Heath et al (2016)
Lingodroids:
Cross-situational
learning, IEEE
Trans CDS

10. An individual robot’s semiotic network
(based on Pierce’s semiotic triangle)
The Lingodroid sign system
Internal representation
Symbol
Referent
social
grounding
symbol grounding
private
grounding
Schultz et al (2012)
Adaptive Behavior
Representamen
Object Interpretant

11. Robot sensors and “Deixis”
4/5

12. Insights from the Lingodroid studies
• The Lingodroids are embodied and enactive
• Words are not grounded in the direct perceptions of the
robots, but rather, in the integrated navigation system: the
nodes of the map, and the relational spaces of distance
and direction
Each word has a meaning as part of a wider network (extended)
• Common ground depends on shared culture
(understanding of the activity the robots are engaged in –
which game they are playing: where-are-we vs how-far)
Culture assigns meaning to activities which then becomes
encoded when the robots name common ground
Activities that involve common ground are intrinsically
embedded in culture

13. In theory there is no
difference between theory
and practice.
In practice there is.
Saying in robotics; attributed variously to baseball
player Yogi Berra, computer scientist Jan L. A. van
de Snepscheut and physicist Albert Einstein.
iRat

14. Biological systems are
clock-free, grid-free, scale-free & symbol-free
Brain regions synchronize over a range of timescales.
Fast local coupling enables rapid communication,
slow episodes require sustained coupling.
Buzsaki&Mizuseki,NatureReviewsNeuro,2014
1 second
Time
(msecs) Function
< .1 location from stereo sound;
1 neuron spike width
10 communication in neural regions
100 social moments;
1000 cognitive moments

15. Timescales of fluid conversation
(human-to-human)
Speaker 1
200
time (milliseconds)
700
• The gap between speakers is typically 200 msec
• The time to process an utterance is 500-700 msec
• Social micro abilities provide a scaffold for these rapid exchanges
Speaker 2

16. How the brain represents space
16Drieu and Zugaro, Hippocampal sequences… Front. Cell. Neurosci., 2019
Placecells

17. How a brain
constructs
space-time
The representation of space
in the brain
Grieves & Jeffery, (2017)
Behav Proc. 135 Fig. 8
17

18. “VR breaks the laws of physics,” Mayank Mehta, UCLA
• “The rats in the virtual maze found
hidden rewards as skillfully as rats in a
real maze. … Despite this, the neural
activity in their hippocampus was highly
abnormal in virtual reality. More than half
of all neurons in the hippocampus shut
down in virtual reality and the remaining
neurons fired in a disordered fashion
• “If the vision says you are moving in
space, but the sounds and smells tell
otherwise, as is the case in virtual reality,
the neural activity becomes very strange.
• “VR … removes the consistent
relationship between different stimuli in
the world that all the animals have used
for millions of years.”
www.wired.com/2013/05/virtual-reality-spatial-sense/
Space, Time & Imagination: How Virtual Reality Changes the Brain | Mayank Mehta | TEDxUCLA

19. ITEE/Co-Innovation Group: Janet Wiles, Scott Heath, Jonathon Taufatofua, Nik Rybak, Josh Arnold,
Josh Riddell, Gautier Durantin, Ola Olsson, Carlos Ramirez, Andrew Back; Scott Frazier, Beth Cave,
Dylan, Alumni: Kris Rogers, Michael Smith, Rollsy Ponmattam Madassery, Lachlan Fitzpatrick,
Autumn Qiu Jing Chew, Nick Buckeridge, Hugh Raynor, Wilson Kong; ITEE/Robotics: Pauline Pounds;
ITEE/Interaction Design: Pete Worthy, Jason Weigel, Steve Viller, Ben Matthews, Marie Boden
School of Psychology Early Cognitive Development Centre: Kristyn Hensby, Virginia Slaughter, Mark
Nielsen, Tamara van der Zant
School of Communication and Arts/Discursis: Dan Angus, Yvonne Yu
Bond U+ITEE: Helen Chenery, Jacki Liddle, Brooke-Mai Whelan
Melbourne U: Jill Wigglesworth; Ngukurr language centre: Greg Dickson, Jackie van den Bos,
Angelina Joshua, Tamara Joshua, Grant Thompson, Jordan
UCSD Machine Perception Lab: Deborah Forster, Javier Movellan, Marian Bartlett,
Mohsen Malmir; HARLIE: David Ireland (CSIRO), Christina Atay, + Harlie team
computer scientists
mechatronics
electrical & software
engineers
roboticists
designers
psychologists
Speech pathologists
occupational
therapist
linguists
CoEDL Scholars
MPhil, PhD
THE SOCIAL TECHNOLOGIES TEAMS: LINGODROIDS, OPAL, HARLIE, DISCURSIS
j.wiles@uq.edu.au