2020.04.17 Seunghyeon Choe

1. Enlightening Patients with
Augmented Reality
Jaki et al., IEEE VR 2020
Presenter: Seunghyeong Choe
2020. 04. 17

2. Contents
• Motivation & Backgrounds
• Interactive Stories for AR
• AR Prototyping
• Evaluation & Results
• Limitation
• Criticism

3. Motivation
• Low health literacy (Doak et al.)
• Audio or computer-aided instruction may be helpful (51)
• Positive effects on patient education with multimedia systems (56)
• Enlightening Patients with Augmented Reality (EPAR)
 Designed for patient education
 Complementary tool of doctor’s mandatory consultation
2

4. Motivation
• Enlightening Patients with Augmented Reality (EPAR)
1. Definition of scenarios
2. Interactive Storytelling
3. 3D modeling and sound
4. Prototype mobile AR app
5. User experience and interaction evaluation
• Research Questions
 RQ1. Concerning different types of interactions with virtual objects
 RQ2. 3D augmentation, valuable or not?
 RQ3. Can interacting with 3D objects modify the story?
3

5. Backgrounds
• Educational AR
4
Benefits
(16)
Challenges
(36)
Case study
(13)
Medical
field
• Better understanding
• Long-term memory
• Better collaboration
• Attention tunneling
• Usability difficulties
• Classroom integration
• Learner differences
• Observe butterflies
• Better academic
achievement
• Heightened motivation
• Supports doctors
• AR surgical monitoring
• Bone anatomy
Only a few AR projects for patient education
Virtual Environment for Radiotherapy Training (VERT)
VIPER – Virtual Patient Education in Radiotherapy

6. Backgrounds
• Interactive Storytelling
 Computer-supported storytelling systems
• Twine: Tools for writing interactive stories
 Developing interactive stories
• Location-based Mixed and Augmented Reality storytelling (Azuma et al.)
• Urban tourism (34)
• Learning handicraft (47)
 VR + Storytelling = Effective knowledge gain (57)
5

7. Interactive Stories for AR
• Story Development
 Topic: Eye surgery of strabismus patients
• Improve their story with eye surgical expert, Anna-Maria Leinhart
 Divide in three chapter
• Introduction of surgery
• Explain functions of eye muscles
• What happens if eyes are misaligned
• Explanation of eye surgery
• Recovery and healing process
 Simplify the process not to overwhelm patients
6

8. Interactive Stories for AR
• Interaction design
 Two main goals
• Experience the story and receive information in the same time
• Users should be able to interact with AR-environment
 Vector structure (with Optional Side Branches, 40)
• Describe easier and provide a chance to look a side branch
 Interaction can change the length of the experience
• Interaction triangle
7

9. Interactive Stories for AR
• Interaction design
 Twine
• Effective for planning an interactive story
• Output an HTML page which users can look
• Click the link to go to the different part of stories
 Interaction prompt concepts
• Highlight on clickable objects
• Haptic feedback
• Audio signal
8

10. AR Prototyping
• ARCore Augmented Reality Subsystem and Unity
9
Flow Control System
• Facilitate creating and changing the story
• Acts: main 3 chapters of the story
• Sequence: User’s possible decisions
• State configuration file
ARCore Augmented Image
• Image target to use as an anchor
CSV-Logs
• For evaluation
• For future researches

11. AR Prototyping
• ARCore Augmented Reality Subsystem and Unity
10
Contents Creation
• Human eye 3D model
• Autodesk Maya 2019
Driving User Attention
• Visual (highlight)
• Haptic (phone vibration): synchronized to the highlights
• Auditive (sound): notification
Generate Speech Files
• Download only once

12. Evaluation
• Two stage usability test
 First stage evaluation
• Six experts (AR, UI, Health)
• Questionnaire and System Usability Scale
 Second stage evaluation
• 18 recruited possible end users
• Three personal profiles
• Low technical knowledge: not familiar with computer and AR
• Good technical knowledge: not familiar with AR
• High technical knowledge: familiar with both computer and AR
• 3 combinations of interaction prompts
• Questionnaire and System Usability Scale
11

13. Evaluation Results
• First stage evaluation
 Questionnaires
• Start/tracking procedure
• 2 out of 6 felt it’s not clear for them
• Main menu
• 4 out of 6 had difficulties
• Couldn’t realize that interactive 3D object can show the menu
• Textual information
• 3 out of 6 felt it was not necessary that explaining all items every time
 System Usability Score
• Median score 67.5
• Mean 72.9, mode 92.5, minimum 32.5, maximum 92.5, n=6
• Prototype was revised to increase the score
12

14. Evaluation Results
• Second stage evaluation
 System Usability Score
• Median score 80.0
• Mean 82.2, mode 77.5, minimum 52.5, maximum 97.5, n=18
• Low technical knowledge: median score 77.5
• Intermediate technical knowledge: median score 90.0
• High technical knowledge: median score 82.5
13

15. Evaluation Results
• Second stage evaluation
 Questionnaires
• RQ1. Which combination of interaction seems comfortable?
14
Visual highlight is enough sufficient as interaction prompt

16. Evaluation Results
• Second stage evaluation
 Questionnaires
• RQ2. Does the 3D augmented objects valuable?
• Participants can repeat and summarize the content quite well
• Median 4.4/5 point of the question: 3D objects are useful, reasonable, and informative.
• Median 5.0/5 point of the question: It can be a complementary tool to understanding medical
consultation.
• RQ3. How interactions with 3D objects modify the story?
• Give some questions to find out some information which need interaction
• Perceived highlights are useful but some participants did not notice the highlights
• Median 4.0/5 point of the question: The interaction with 3D objects is clear to me
• Median 4.0/5 point of the question: No difficulties in interacting with 3D objects
15

17. Limitation
• No real patients participated in the study
• Testing with real patients might have influences on some results
• In case of HMD, might have a different outcome
• Ethics approval
16

18. Criticism
• Haptic feedback
 How it works to inform 3D objects with interaction?
• Flow control with XML file
 How it works?
• Saving TTS files
17

19. Thank you😀
Any questions?