Abstract—In this paper, we investigate the validity of Mixed Reality (MR) Simulation by conducting an experiment studying the effects of the visual realism of the simulated environment on various search tasks in Augmented Reality (AR). MR Simulation is a practical approach to conducting controlled and repeatable user experiments in MR, including AR. This approach uses a high-fidelity Virtual
Reality (VR) display system to simulate a wide range of equal or lower fidelity displays from the MR continuum, for the express purpose of conducting user experiments. For the experiment, we created three virtual models of a real-world location, each with a different perceived level of visual realism. We designed and executed an AR experiment using the real-world location and repeated
the experiment within VR using the three virtual models we created. The experiment looked into how fast users could search for both physical and virtual information that was present in the scene. Our experiment demonstrates the usefulness of MR Simulation and provides early evidence for the validity of MR Simulation with respect to AR search tasks performed in immersive VR.
The effects of visual realism on search tasks in mixed reality simulations-IEEE Transaction Paper 2013
THE EFFECTS OF VISUAL REALISM ON SEARCH
TASKS IN MIXED REALITY SIMULATION
Un d e r t h e g u i d a n c e o f RAJI R. PILLAI (Asst. Professor)
Roll No. 63
Seminar on :
• Experiments in AR Domain are difficult.
- Difference in Display Systems used.
(Result are not generalizable)
- Difficult to conduct Controlled Experiments
in Unpredictable Environment Conditions.
Data Glove Display ( H. M. D. )
( CAVE )
Hint : Real Display systems can be used only in Real
Hence use, Mixed Reality Simulations.
• A concept of using High Fidelity Virtual Reality systems to
simulate environment from MR Continuum.
• Different display systems are simulated by varying the level of
FIDELITY OF MR SYSTEMS
Display Interaction Simulation
Fidelity Fidelity Fidelity
(Depends on (Depends on (Depends on
Sensory Fidelity) Interactions) Environment &
Fidelity of Objects)
• The degree to which images on simulated environment is
perceived by user in the real world.
• The concept of visual realism is very important for
understanding and verifying Fidelity of Simulations.
• In this paper, the aim is to conduct experiments in
different Simulated Environments rather than Real World
Environment by varying the level of Visual Realism in
What level of Visual Realism is required for
replicating AR Experiments?
• Search Experiments were conducted and
performance was analyzed in 3 MR Simulations
and Real World.
Level 1 Level 2 Level 3 Real World
Increasing Order of Visual Realism
VISUAL REALISM FACTORS
• Shadow Softness
• Surface Softness
Analyzing these, it was
found that Soft Shadows
and Surface Type affects
perception of Visual
CREATING VIRTUAL MODELS
• Different formats and techniques are used for
creating Virtual Models, each resulting in different
levels of Visual Realism.
Model Geometry Color Information Lighting Techniques
• Image Based
• Point Based
• Stored Polygonal
• Simple Colors
• Coloring Material
• Ray Casting
• Ray Tracing
There were 2 goals to be achieved :
1. Verify the validity of MR Simulations by checking
whether Visual Realism has any impact on Search
Task Performance. Or Simply, Does Visual Realism
2. If so, How?
• Design outdoor AR experiment involving
• Choose a Real World location for
• Use 3 Models and Simulate them for
conduction experiment indoors.
TASK & ENVIRONMENT
Virtual Models were designed using 3D Modeling Techniques.
Low Fidelity Medium Fidelity High Fidelity
• Polygons for major
• Neglected minor
• Plain color was
used but no
• Increased number
of polygons for
• Simple low
• No change in
• High polygon
• High resolution
images instead of
• Baked Lighting.
• To search for both ‘Virtual’ and ‘Physical’
information based on both ‘Virtual’ and ‘Physical’
( The most recognizable AR Experiments which
require very little training. )
They created Virtual Information on the scene such that
it must be suitable for the current environment where a
particular user might want to do it.
The high-fidelity model and the real environment, both with virtual annotations.
Information about Office
The Method :
• Created 16 questions.
• There were mix of questions that required the user to find
both virtual information and physical information based
on both virtual and physical information context.
• For each task question the participant was required to
find certain pieces of information and verbally report that
Target information refers to the exact nature of the response
required by the task question.
Physical information is inherent in the real world while virtual
information needs to be provided by the virtual icons and the text
Criteria information refers to what information the user is using to
search the scene
DISPLAY SYSTEM USED
NVis SX111 HMD
• Pointgrey USB3 Flea camera
• 102° Horizontal FOV
• 64° Vertical FOV
• 1600 x 1024 pixels resolution
• 60 FPS
+ InterSense IS900 Tracking System.
Additional Hardwares and Softwares used :
• Windows 7 PC with a Quadro 5600
• Intel Core2 2.4 GHz Duo-Core CPU
• 2 GB Memory
• WorldViz’s VR Toolkit Vizard 4.0
Dependent Variable for each task was the Search Time.
The 3-Time Question Sessions :
• Q1 – Q5 : Single item search
• Q6 – Q11 : Multiple item Search
• Q12 – Q16 : Comprehensive
Avoided ‘Learning Effect’ by giving 2 Minute breaks after each session.
• Color Vision Test for Participants.
• Training with Icons using HMD.
• Screening Participants.
ISSUES WITH REPLICATING AN
• Accuracy of Models
• Real Location undergone changes.
• Geometrical difference between CAD plan and Real
• Weather change affected lighting.
1. Difference in Task Times :
• For the analysis of the time results from our experiment,
ANOVA test was used to determine if the level of realism had
a significant effect on task time.
• In the second stage a post-hoc analysis of all task pairs was
Result of ANOVA :
A plot of the mean task times, 95 % confidence intervals, and significance P
values for each level of realism for all 16 task questions
P - Value :
• These result were reported as P-Values.
• P-Value is a measure how likely this spot will be
obtained if no real difference existed. A smaller P-Value
indicates more significant is the difference between
groups. (Small usually means 0.05).
Only four of the 16 task questions revealed a significant difference
between the realism conditions.
Q3 (a door under bridge)
Q6 (a tree) For these tasks, A Tukey post-hoc
analysis was used to determine
Q8 (trash bin) pairwise difference.
Q12 (a comprehensive search)
Q3, Q6, Q8
Observations from Tukey Post-hoc test :
• For Q3, Q6, Q8 the difference in significant only in Real
• Camera artifacts, vegetation, and lighting conditions,
affected the visibility of the real-world objects much
greater than the virtual objects.
• Cluttered environment might have increased task time
when the search criteria was relatively small.
2. Equivalence in Task Times :
• All other tasks did not reveal significant difference for level of
• For further investigation, the Two-one-sided t-test (TOST) analysis
was used to look for equivalent task time performance within the
• All other task questions produced at least one instance where
two of the realism conditions were equivalent except Q12 and
What type of courses are generally
taught in North Hall ?
Which professor has the office which is
located the furthest from their lab ?
3. Notable Task Questions :
• Q8 - the only task with results indicating any obvious
performance trend with respect to task time.
• Q12 - by far the most difficult one. the results here
reflect the difficulty of the task and the effects from
strategy more than from level of realism.
4. Task type and Visual Realism :
• Participants were quicker to respond when the
search target type was virtual information.
• Search criteria type is shown to be significant in all
four conditions while search target type has a
similar effect on the high and the real conditions.
This suggests that the high level of realism performs
similarly to the real conditions.
• The similarity of these effects support the validity of
• Clarify the exact cause of the differences between the AR
case and the simulated AR cases.
• Improving the actual AR experience for better ways to present
AR imagery to users.
• Exploring Mixed Reality Simulation for different task types (not
just search, but also browsing and annotation and interaction
tasks) and for different training environments to simulate.
• D. A. Bowman, C. North, J. Chen, N. F. Polys, P. S. Pyla, and U. Yilmaz.
Information-rich virtual environments: theory, tools, and research
agenda. In Proceedings of the ACM symposium on Virtual reality
software and technology, VRST ’03, pages 81–90, New York, NY, USA,
• M. Elhelw, M. Nicolaou, A. Chung, G.-Z. Yang, and M. S. Atkins. A gaze-based
study for investigating the perception of visual realism in
simulated scenes. ACM Trans. Appl. Percept., 5(1):3:1–3:20, Jan. 2008.
• J. L. Gabbard, J. E. Swan, II, and D. Hix. The effects of text drawing
styles, background textures, and natural lighting on text legibility in
outdoor augmented reality. Presence: Teleoper. Virtual Environ.,