Virtual Reality Autonomous Vehicle Simulator. Research Subject: Human–computer interaction Capstone Presentation for Arizona State University. Sponsor: Dr. Georgios Fainekos Mentors: Kangjin Kim, Joseph Campbell

1. Group 46
Virtual Reality Simulator for Human-Robot Interaction

2. Thanks To Our Sponsor!
Dr. Georgios Fainekos

3. ● Introducing autonomous vehicles to a world dominated by human drivers
gives rise to a set of unique problems
● There exists a need for ways to handle situations where human driven
vehicles and autonomous vehicles need to communicate to proceed in their
respective direction
○ An example can be one lane streets with the vehicles travelling in opposite directions -
someone needs to back up
○ Such problem can be solved easy between two humans or two robots, but the combination of
the two is more complicated
Introduction and Problem Background

4. Introduction and Problem Background
● We will be studying various scenarios that might arise between human driven
vehicles and autonomous driven vehicles and how these conflicts can be
resolved
○ Our goal is to evaluate various techniques for handling the communication during these
conflicts, and in what way they can resolve possible “deadlock” situations

5. Simulator
● Built as a Unity game to simulate conflict situations where communication
needs to take place for vehicles to proceed
● The purpose is to test and evaluate various communication strategies using
human testers
● Using full driving rig with VR headset

6. Driving Rig
https://www.playseatstore.com/evolution-black-ready-to-race-bundle.html http://fortune.com/2016/01/09/heres-what-its-like-to-use-the-oculus-rift/

7. Discretization
● Map discretization:
○ Grid, Triangulation, etc.
○ Utilizing delaunay triangulation algorithm
● Need to consider vehicle size
○ Discretized triangles/squares need to be large enough to contain a full
vehicle
● Static and dynamic map updates
○ Send discretized map as JSON
○ Update each vehicle location at each timestep
● Need to account for “inaccessible” nodes
○ Some nodes will contain buildings or other static obstacles, so these
shouldn’t be accessible for computing paths

8. Path-finding & Websocket Python-C#
● External Python program
○ Kangjing Kim (PhD student) writing Python algorithm
○ Path Finding algorithm built for resolving human vs. robot conflicts
○ Runs as separate program
● Need to communicate between Unity and Python
○ Python Program as Server, Unity Simulator (C#) as Client
○ Transmission Control Protocol (TCP) socket - One-to-One connection
○ JSON data stream containing path-finding data (graph, coordinates, path)
○ Encode/Decode to handle serialization
● Need for real time updates
○ Drive autonomous car based on received data
○ Communication handled in while not at_target
○ Receives coordinates, calculates path, sends back
○ Constant stream

9. Methodology
● Basic Requirements
○ Create a game-like simulator to study
human interaction with autonomous
vehicles in deadlock situations.
○ Build a hardware system: a driving chair,
steering wheel, pedals, and a VR headset
○ Deliver system to Polytech team for
conducting the research
model car prototype
path-finding algorithm

10. Jan
Feb
March
April
May
Assemble Team
Clearly describe
requirements and divide
requirements among
team members
Communication
Establish
communication between
Unity and Python
Virtual Reality
Build a virtual reality
environment for the AI
and human driven
vehicle
Driving Rig
Construct the driving rig
and set up Oculus Rift
environment
Testing
Test the driving rig and
debug issues
Timeline and Progress

11. References
● http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-204.pdf