The document discusses the IV4XR project, which focuses on intelligent verification and validation of extended reality (XR) systems through automation and artificial intelligence. It details the development of autonomous testing agents aimed at improving user experience and system functionality while also providing insight into pilot studies conducted in gaming and simulation environments. The framework aims to address the complexities of testing XR technologies with models for emotional prediction and cognitive assessment.

1. The future of XR: Current ecosystem and upcoming opportunities
Intelligent Verification/Validation for XR
Based Systems
Rui Prada
Instituto Superior Técnico, Universidade de Lisboa
INESC-ID
The future of XR: Current ecosystem and upcoming opportunities - May 27, 2021
https://iv4xr-project.eu
Project Ref. EU H2020-ICT-2018-3 - 856716

2. Project vision

3. Testing XR Systems
Highly interactive
Rich and complex environments
Diverse multi-modal input and output
Testing demands high human labour (user
testing)

4. Use Artificial Intelligence (AI) to
support automation of XR testing
(Autonomous Testing Agents)

5. Team
University and Research
- INESC-ID, Portugal (Coordinator)
- Utrecht University, The Netherlands
- Fondazione Bruno Kessler, Italy
- Universitat Politecnica Valencia, Spain
- Umeå University, Sweden
Industry
- Gameware Europe, UK
- Good AI, Czech Republic
- Thales (SIX and AVS), France
https://iv4xr-project.eu

6. Autonomous Testing Agents
Actively pursue testing goals
Intelligent coverage of the interaction
space
Identify potential interaction paths
Represent users with different profiles

7. Testing goals
 Testing Functional Properties
 The behaviour of the system
 Testing User Experience (UX)
 The impact on the user

8. The framework

9. iv4XR Agent-based Testing Framework

10. https://github.com/iv4xr-project
 aplib + iv4XR EDSL
 aplib:
 basic agents and runtime system
 action and tactic
 goal and goal-combinators
 iv4XR:
 test agents and testing related basic support
 extensions: pathplanning, world object model, emotion,
learning, ...
iv4XR
framework,
from a different
perspective
https://github.com/iv4xr-project

11. Interface-pattern (instead of interface) to
cater for diverging XR technologies

12. Functional Test Agents

13. Two types of FTA
 Two types of FTA’s are being
developed:
 The first type of agent makes
deliberations to choose the
appropriate strategies that will
allow him to do goal-solving.
 The second type of agent is
intended to test the
functionality of the XR system
using exploration.

14. Goal Solving Functional Test Agent

15. Fully Automated Exploring/Testing
Space Engineers
 TESTAR as iv4XR agent
 While exploring:
 We test for robustness
 To infer a model that can be used
for Model-Based Testing (MBT)
 Find the best representation of
the World Object Model (WOM)

16. b_0
b_1
b_2 b_3
d_1
d_2
d_T
agent
"b_0 -- EXPLORE --> d_1_m",
"d_1_m -- EXPLORE --> b_1",
"b_1 -- EXPLORE --> b_0",
"b_0 -- EXPLORE --> d_T_m",
"d_T_m -- EXPLORE --> d_1_m",
"d_1_m -- EXPLORE --> b_0"

17. Socio Emotional Test Agents

18. Different approaches
focusing on User eXperience

19. Automated assessment of UX
Agent
Cognitive
model
Emotional
model
Difficulty
Interaction
Policies
Personas

20. Emotional Prediction Through Machine
Learning
Predictive Model
Feed
Predicts
Expected
Emotional
Changes
Emotional
Changes
(Continuous Self
Reporting)
Training
SUT
(Observations)

21. Emotional Prediction Through Machine
Learning
 The Case Study:
 A simple top-down 2D
game
 Continuous self-
reporting after the fact
of the 3 dimensions of
the PAD emotional
model (Pleasure,
Arousal and Dominance)

22. Emotional Prediction Through Machine
Learning
 The emotional
reporting was cut into
slices, which were then
categorized as
decreasing, steady or
increasing.
 A machine learning model
based on Random Forests
was then trained to
predict these 3 classes.

23. Emotional Prediction Through Machine
Learning
Current Results (3 seconds slices):
 Arousal
 Accuracy:
 86.69 %
 Average Confusion Matrix:
 Pleasure
 Accuracy:
 90.95 %
 Average Confusion Matrix:
 Dominance
 Accuracy:
 82.07 %
 Average Confusion Matrix:
48.39 0.46 0.10
8.32 119.33 10.71
5.81 9.21 110.67
70.84 0.59 0.45
7.35 147.54 8.58
2.56 4.04 102.65
47.49 1. 0.56
12.71 153.24 13.69
6.2 5.42 89.96

24. Automated Assessment of Cognitive
Emotions (OCC Model)
Intelligent agents are deployed to generate tests based on UX test
specifications. This is achieved by deploying a Computational model of
emotion designed to provide affective processing in our intelligent agents

25. Pilot Studies

26. Three Pilot Studies
 We are making use of three pilots to test the framework:
 Space Engineers - a 3D game
 Players make use of several tools and fabrications (blocks) to obtain
resources and explore a solar system with player constructed vehicles and
machines.
 Maev - A simulation environment
 A defend/attack scenario in a nuclear powerplant. Defenders test security
measures to stop external intrusions.
 LiveSite - A monitoring system for infrastructure and construction sites
 Test the reading of sensors.
 Sensors at these sites can produce up to thousands of readings per
second. There are many virtual sensors.

27. How are we using the testing agents?
 Space Engineers requires ~20000 tests for each major
release.
 One arduous test that we are working on automating with
iv4XR is checking textures of blocks depending on their build
state.
 The Maev powerplant scenario currently requires two teams
of humans:
 Iv4XR will take control of the incursion forces, automatically
testing the efficacy of the proposed security measures.
 LiveSite buildings and critical infrastructures have a large
number of sensors that produce many readings per second.
 Iv4XR constructs tests based on their predefined relationships
and thresholds, but also investigates found errors to
determine if the readings are correct, or if there is some
anomaly (faulty/miscalibrated sensors).

28. Intelligent Verification/Validation for XR
Based Systems
https://iv4xr-project.eu
@iv4xr
https://github.com/iv4xr-project
rui.prada@gaips.inesc-id.pt
Project Ref. EU H2020-ICT-2018-3 - 856716