The document discusses research into using virtual reality (VR) and machine learning to study human attention and improve user experiences. Key areas covered include: 1. Conducting experiments using eye tracking in VR to study patterns of visual attention and gaze. 2. Developing AI-assisted navigation models using recurrent neural networks to recommend destinations in VR environments based on user walking behavior. 3. Integrating trained models into VR experiments to provide implicit and explicit navigation recommendations to participants and compare the impact on experience. 4. Suggesting future work on using VR and AI to detect and treat mental illnesses like social anxiety disorder.

1. Dr Mu Mu, 6 July 2022
Grab your attention with AI
Next steps for VR
Murtada Dohan (PhD Comp), Cleyon Johns (Game Design)
Yoana Slavova (Business Comp), Petar Stepanić (Comp)

2. • Human Computer Interaction (HCI)
• multidisciplinary field with a focus on the interaction between
humans (the users) and computer systems.
• Objective video quality assessment – assess user experience
using machine learning
• Better networking, better video streaming standard.
• Video -> Immersive media -> virtual reality
• Machine learning -> deep learning / neural networks
Background

3. •VR journey ( learning, art, AI navigation, mental
health… )
•The role of human attention in VR designs

4. VR takes a front seat
• Education, training, entertainment, advertisement, creative art,
healthcare.

5. • Motivation: Does “WOW!” translate to an
A?
• how VR could impact the learning of
science in universities.
• Comparative study
• Stonehenge VR vs PowerPoint
• Multiple choice test (recognising numerical,
textual, visual info) and interviews
• 50 students (age 18-26, 60% female)
• Results, observations, etc.
• VR not always better (some contributing
factors).
• More work on peer interactions and
interactive tools in VR.
A Comparative Study of the Learning Outcomes
and Experience of VR in Education - Yoana
Slavova
• Slavova, Y. and Mu, M., A Comparative Study of the Learning Outcomes and Experience of VR in Education, in Proceedings of the 25th
IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2018), Germany, 05/2018
• Panel, Westminster HE Forum – Technologies in higher education, https://www.westminsterforumprojects.co.uk/publication/technology-in-
higher-education-19

6. A Comparative Study of the Learning Outcomes
and Experience of VR in Education - Yoana
Slavova

7. Attention is the answer.

8. attention
visual
audio
haptic
olfactory,
taste
movement
Human attention
“Everyone knows what
attention is. It is the taking
possession by the mind, in
clear, and vivid form, of one
out of what seems several
simultaneously possible
objects or trains of thought.”
- William James
(experimental psychology) -

9. • We designed a gaze-controlled Unity VR game for this study and implemented
additional libraries to bridge raw eye-tracking data with game elements and
mechanics.
• The experimental data show distinctive patterns of fixation spans which are paired
with user interviews to help us explore characteristics of user attention.
Understanding User Attention In VR Using Gaze
Controlled Games – Murtada Dohan (MSc Dissertation)
Dohan, M. and Mu, M., Understanding User Attention In VR Using Gaze Controlled Games. In Proceedings of ACM International Conference on
Interactive Experiences for TV and Online Video (ACM TVX ’19), June 5–7, 2019, Salford (Manchester), United
Kingdom. https://doi.org/10.1145/3317697.3325118 06/2019

10. • This work indicates future avenues for content creation in this emerging field and what this
might mean for artists and art institutions to experiment with methods of exhibiting
innovative content.
Abstract Painting Practice: Expanding in a Virtual
World
Goodyear, A. and Mu, M., Abstract Painting Practice: Expanding in a Virtual World. In Proceedings of ACM International Conference on
Interactive Experiences for TV and Online Video (ACM TVX ’19), June 5–7, 2019, Salford (Manchester), United Kingdom. 06/2019

11. VR Environment
Open-source tools
oTilt Brush export tool: https://github.com/MrMMu/tiltbrushfbxexport
oUnity script: https://github.com/Murtada100/MTAP_Data_and_Experiment

12. VR Environment
Open-source tools
oTilt Brush export tool: https://github.com/MrMMu/tiltbrushfbxexport
oUnity script: https://github.com/Murtada100/MTAP_Data_and_Experiment
{‘fbxname’:FBXNAME,
‘fbxmeta’:
[{‘meshname’:MESHNAME,
‘meshmeta’:
{‘brush_name’:BRUSHNAME,
‘brush_guid’:BRUSH_GUID,
‘v’:V, #list of positions (3-tuples)
‘n’:N, #list of normals (3-tuples, or None if missing)
‘uv0’:UV0, #list of uv0 (2-, 3-, 4-tuples, or None if
missing)
‘uv1’:UV1, #see uv0
‘c’:C, #list of colors, as a uint32. abgr little-endian, rgba
big-endian
‘t’:T, #list of tangents (4-tuples, or None if missing)
‘tri’:TRI #list of triangles (3-tuples of ints)
}
},{},…]
}

13. Experiment
● The experiment involved participants freely navigating through an VR artwork
● It consists of thousands of different brushstrokes
● HTC VIVE PRO EYE
● Data collected: Walk data, and eye gaze data, hand tracking, verbal response, environment
data.
Dataset: https://doi.org/10.6084/m9.figshare.15090309.v1

14. Experiment

15. Experiment

16. Walk

17. Eye orientation
Eye Tracking Coordinate System
(ETCS).
The ETCS is a cartesian, right
handed system and the coordinate
axes are oriented

18. Eye orientation
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]

19. Eye orientation
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]

20. Gender
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in
Springer Multimedia Tools and Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]

21. Neural Network
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and
Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]

22. Experiment
Mu, M., Dohan, M., Goodyear, A., Hill, G., Johns, C., Mauthe, A., “User Attention and Behaviour in Virtual Reality Art Encounter”, in Springer Multimedia Tools and
Applications. https://arxiv.org/abs/2005.10161, 2022 [dataset]

23. Experiment
Participant 5170 commented that he loved the experiences of “going into the paint
and looking up”, which really gave him “a sense of scale”. He preferred walking to
explore the artwork more than the idea of “flying” using VR controllers because
“tethering to the ground” gave him “a point of reference”.
Participant 4475 felt that she was “in a virtual jungle” and “loved the lights shining
through the paint”. She also suggested “more pink trees” and “tropical music” in
the background.
Participant 1157 “felt a bit lost in the painting...”. The feedback reflects her
walking patterns shown in Figure 8(b) where she spent little time adventuring
inside the artwork.
Participant 4646 “really liked a glowing triangular area in the middle of nowhere”
because it reminded her of a song. She also suggested “swimming in the paint”
as a better navigation method compared to walking and flying.

24. Solving navigation

25. AI-assisted navigation
manual
Community
activities
Director’s cut
AI model
automated
Navigation
recommend
recommend
recommend
recommend
recommend

27. Recurrent Neural Network
https://colah.github.io/posts/2015-08-Understanding-LSTMs/

28. AI
Navigation

29. AI
Navigation
• Implemented based on LSTM model.
• Integrated to Unity
• Collects walking data in real-time to
predict the best destination in VR.
• Navigator was embedded using the
most common navigation techniques:
Avatars and Arrows.
• We have used 3 avatars, 2 arrows for
showing top(3) and top (2)
predictions.
• 15 participants joined the experiment.
Dataset: https://github.com/Murtada100/EyeVRDataset

31. AI
Navigation
• ML recommender can recognise
mobility pattern.
• The model is embedded in the VR
environment.
• We have employed two different
techniques to illustrate these
recommendations to the participants via
directed arrows and Avatars.
• Implicit recommendation (Live-avatars)
are less distracted than explicit
recommendation (Directed-arrows).

32. What’s next?
Attention
in VR
visual
audio
haptic
olfactory,
taste
movement

33. Autonomous VR psychiatry for the early detection and
treatment of mental illness

34. Fix social anxiety disorder? - Petar Johan Stepanić (BSc
dissertation)

35. Conclusions
Attention is the answer.
Machine learning is the way.
Mu Mu, Technology, FAST
mu.mu@northampton.ac.uk
Murtada Dohan (PhD Comp), Cleyon Johns (Game)
Yoana Slavova (Business Comp), Petar Stepanić (Comp)