This is my Master's Thesis; University: Arizona State University The idea of this thesis study is to develop and test a real time collaboration system between VR and MR. The system works similar to a Google document where two or more users can see what others are doing i.e. writing, modifying, viewing, etc. Similarly, the system developed during this study will enable users in VR and MR to collaborate in real time.

1. 1
Real Time Cross Platform
Collaboration Between VR & MR
Nayan Sateesh Seth
April 13th, 2017

2. 2
Problem Statement
• Virtual Reality and Mixed Reality are fairly
new technologies that can improve
productivity multifold
• Very little research has been performed to
connect the two technologies

3. 3
Objective of Thesis Study
• The idea of this thesis study is to develop and
test a real time collaboration system between
VR and MR
• The system works similar to a Google
document where two or more users can see
what others are doing i.e. writing, modifying,
viewing, etc

4. 4
What is VR?
• “The computer-generated simulation of a
three-dimensional image or environment that
can be interacted with in a seemingly real or
physical way by a person using special
electronic equipment, such as a helmet with a
screen inside or gloves fitted with sensors.”
[1]

5. 5
What is AR?
• “A technology that superimposes a computer-
generated image on a user's view of the real
world, thus providing a composite view” [2]

6. 6
What is MR?
• “MR, sometimes referred to as hybrid reality,
is the merging of real and virtual worlds to
produce new environments and visualizations
where physical and digital objects co-exist
and interact in real time” [3]

7. 7
Working of Multiplayer
Games

8. 8
PLATO
• PLATO worked on a centralized server. Every
user was given a stipulated amount of time to
complete a turn, after which the turn gets
passed on to the next device. Such multi-
player games are controlled via control
signals [4]

9. 9
Modem & Telephone Based Multiplayer
• Another multiplayer gaming mode in which
users with distinct computers and a modem
connect to telephone lines and eventually do
long distance multiplayer gaming.
• Restricted to two users [5]

10. 10
LAN Party
• A LAN party is a gathering of two or more
people in a house, building, or campus where
they can play multiplayer games on the same
network. [6]

11. 11
Remote Access Server Games
• Remote Access Server games represent
another mode of multiplayer gaming.
• Restricted to 16 players.
• Players with a modem can dial up to the
remote server and every player connected to
the server acts as a client who can play with
other clients connected to the server [5]

12. 12
Online Service Multiplayer
• Online Service Multiplayer Network Games
are similar to Remote Access Server Games.
• The servers are on the Internet and make
use of TCP/IP protocol to communicate.
• This method can support as many as 100
users

13. 13
• Centralized server with a network address
• Users need not seek each other
• Additional resources can be added to the
central sever to accommodate more players
• Centralized server takes input from user and
updates state of the game and intimates the
users
• Makes use of a database to store states [5]

14. 14
P2P Gaming
• In P2P gaming, every user contributed to the
memory, CPU cycles and bandwidth to
manage the shared game state
• Availability and security were two major
issues with this type of Massively Multiplayer
Games. [7]

15. 15
Distributed Multiplayer Gaming
• Distributed architecture involved use of
several severs which were geographically
distributed [8]

16. 16
Commonality Between Multiplayer Gaming
• Client-Server architecture is used
• Client connects to a server and the sends
input to server
• Server replies back with new information

17. 17
Methodology

18. 18
• How are users collaborating?
• What is the group size?
• Where is everyone located?
• When are they collaborating?
• How similar are their physical environments?
• What devices are they using? [9]

19. 19
How are Users Collaborating?
• Presentation – When a user presents content
to others but does not want others to take
control over content or modify the content
• Collaboration – When everyone collaborating
want to achieve a common goal
• Guidance – In this scenario, one of the users
helps others to solve some problem [9]

20. 20
What is the Group Size?
• In order to ensure that the collaborative
experience is smooth, the number of
participants collaborating should be known [9]

21. 21
Where is Everyone Located?
• Co-located: Users are in the same room.
• Remote: Users are in remote locations.
• Both: Users are co-located as well as in
remote locations.
• This further raises questions like
– What are the modes of communication?
– Does the experience need to be adapted to
the environment?
– What objects needs to be shared? [9]

22. 22
When are they Collaborating?
• Synchronous: Users are collaborating at the
same time.
• Asynchronous: Users are collaborating at
different times.
• Both: Some users will be collaborating
synchronously and some asynchronously.
• Using this information, application can be
programmed to handle object and
environment persistence [9]

23. 23
How Similar are their Physical
Environments?
• Similar – Environments with similar layout,
light, sound, size, etc.
• Dissimilar – Environments which do not have
similar layout, light, sound, size, etc. [9]

24. 24
What Devices are they Using?
• Different devices from different
manufacturers have different capabilities.
• Understanding the devices user use, the
developer can limit/expand the scope of the
application [9]

25. 25
Procedure

26. 26
Thesis Study Parameters
• Two users will collaborate. One user will use
VR device i.e. the HTC Vive and the other
user will use a MR device i.e. Microsoft
HoloLens
• Both users will assemble a virtual chair and
eventually achieve a common goal i.e. an
assembled chair

27. 27
• The study is conducted in the same room. So
the environment setting is same for both the
users
• Collaboration will be synchronous
• Users can communicate with each other
verbally
• Each user can assemble one object at a time

28. 28
Workflow of Photon Unity Networking
Source: Exit Games

29. 29
Client-Server Architecture
Source: Rice University

30. 30
Project Setup
• Two separate Unity3D projects are created to
avoid SDK file conflicts
• PUN plugin is installed from the asset store
• For MR,
– Main Camera’s clear flags are set to solid
color and background is set to black
– Position of camera is set to x = 0, y = 0, z = 0
– Default build type is set to Windows Universal
and Virtual reality supported option is checked

31. 31
– Based on application additional functionalities
like microphone, spatial mapping, etc, needs
to be enabled
• For VR,
– SteamVR plugin needs to be installed and the
SteamVR camera rig needs to placed in the
scene
• 3 folders namely, _Scenes, Scripts and
Resources have been created

32. 32
PUN Preferences & Script
• App ID
• Server Location
• Auto Join Lobby
• These preferences need to be passed in the
scene. Hence, a Network Manager script is
used to intimate the scene about these
preferences.

33. 33
Scene Controller
• Manages instantiation of objects in the scene
• Makes use of Even/Odd principle to support
the policy of alternating turns

34. 34
Particle System
• Along with the object instantiated, a white
mesh of the object will also be instantiated at
the proposed position where the object has to
be placed

35. 35
Game Objects
• Every object is instantiated as a network
object so that actions performed on it reflect
on the network to other users
• For obtaining network characteristics, Photon
View component is attached to each object
• A script named Network Character is
attached to observe the movements of the
object in the network

36. 36
Game Object Handling
• Users will use functionalities of the device to
control the object
• For VR user, the controller will have a laser
pointer which the user can point at the object
and use the trigger button to select the object
• For MR user, the user will use the air-tap
gesture to select the object

37. 37
• The user can move the object by moving the
VR controller or the MR headset
• The user has to place the object at the
position where the white mesh is positioned
• As soon as the user touches the white mesh,
OnTriggerEnter() function is called and the
object gets placed
• Count is incremented and Scene Controller
instantiates new object if any left

38. 38
Output

39. 39

40. 40
Demo

41. 41
Pre-Survey Results

42. 42
• 71.4% of the 21 candidates had heard about
VR/AR/MR
• 19% of the 21 candidates were unsure about
VR/AR/MR
• 9.5% of the 21 participants had never heard
about it

43. 43
• Although, 71.4% had heard about
VR/AR/MR, 47.6% of all the 21 participants
did not have any explanation for VR/AR/MR
• Only 19% of all the 21 participants gave the
correct explanation
• A few described AR as a popular application
like Pokemon Go

44. 44
24%
14%
5%
5%
52%
Devices Previously Used
HTC Vive
Google Cardboard
Galaxy Gear VR
Microsoft HoloLens
None

45. 45
Post-Survey Results

46. 46
9%
4%
87%
Devices Used In Testing
HTC Vive
Microsoft HoloLens
Both

47. 47
First Impressions
• The majority of the participants found VR and
MR as great pieces of technology.
• Most participants found VR to be comfortable
and easy to use, but there were several
participants who found MR to be confusing
and difficult.
• A few participants were surprised with the
technology

48. 48
4.47
3.55
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Usability
Device Usability
HTC Vive Microsoft HoloLens

49. 49
Best Aspect of Usability
• For the HTC Vive, participants showed strong
affinity towards its controller
• The participants also liked that the device is
immersive and has good Field of View
• For the Microsoft HoloLens, the participants
liked the Mixed Reality the most
• Other aspects which participants liked about
the Microsoft HoloLens include wireless and
gesture capabilities

50. 50
Difficult Aspect of Usability
• Participants expressed their disapproval
towards the HTC Vive’s ergonomics
• Participants were also scared of tripping due
to lack of access to the real world
• Participants disliked the FOV and color
imbalance in the Microsoft HoloLens
• Although, several participants liked the
gestures, there were far more participants
who disliked the gesture capabilities in MR

51. 51
4.28
4.21
4.47
3.95
3.6
3.7
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
Survey Audio Interview
User Interface
Overall HTC Vive Microsoft HoloLens

52. 52
UI Improvement Suggestions
• The majority of the participants suggested
addition of hints in form of text, visual
graphics or audio
• Many participants suggested content scaling
for Microsoft HoloLens
• Other suggestions included addition of
functionality, improvement in functionality,
better graphics, better use of colors, etc.

53. 53
4.42 4.44
4.78
4.1
3.6
3.8
4
4.2
4.4
4.6
4.8
5
Survey Audio Interview
Easiness of Application
Overall HTC Vive Microsoft HoloLens

54. 54
4.28
3.95
3.7
3.8
3.9
4
4.1
4.2
4.3
4.4
Survey Audio Interview
Easiness of Working with Partner
Overall

55. 55
Difficult Aspects of Collaboration
• A great number of participants said that the
partner using the Microsoft HoloLens slowed
down the chair-building activity.
• The participants also said that discovering
content in VR and MR was difficult since the
entire environment setup was new to them.
• Some participants found the FOV of Microsoft
HoloLens aggravated the difficulty of content
discovery in MR

56. 56
Easiest Aspects of Collaboration
• Although, initial coordination was difficult, the
participants found coordination to be one of
the easiest aspects of collaboration
• Majority of the participants found object
selection and placement to be easy
• Participants also liked the very fact that real-
time collaboration can be achieved in these
devices

57. 57
Future Use
• 57.1% of the 21 participants said they would
definitely use VR and MR in the future to
collaborate

58. 58
Discussions

59. 59
• Many users liked MR more than VR because
they could see the real world and the virtual
world concurrently
• Poor FOV of the MR device resulted in many
users reporting a higher affinity towards VR.
• The problem which most of the users had
with VR was the clunky HMD and the wires
that kept dangling while the users were
immersed in the VR

60. 60
• Although, the VR device provides a
chaperone, users still found themselves close
to the edge of the room-scale play area
• Several users found content discovery in VR
and MR to be difficult because of a
360˚environment
• Users found color imbalance issues in bright
lighting

61. 61
• There is a need of better standards for UI
representation in VR and MR
• Better methods for interactivity
• However, current devices are completely
capable of delivering a minimum viable
product
• Application developed is a proof of concept

62. 62
• Companies can use this idea and source
code and package it in different ways as per
application use case
• Additionally, new functionalities or application
working can be application use case
dependent

63. 63
Conclusion

64. 64
• The participants gave positive feedback
regarding collaboration in VR and MR for
futuristic use cases
• The participants found it very simple to
interact with the GameObjects in the
collaboration app

65. 65
• New techniques will have to be developed to
make applications more self-explanatory
since content discovery is not all that easy
new users
• There are several limitations in the current
generation of VR and MR devices which
restricts people from buying it and using it for
prolong periods of time

66. 66
• Most of these limitations are hardware
specific i.e. FOV, latency, display, etc
• Certain limitations are software specific like
gestures and voice in Microsoft HoloLens
• Additionally, these devices are highly
expensive and have limited applications
available

67. 67
References

68. 68
• [1] "Define Virtual Reality". [Online]. Available:
https://www.google.com/search?q=define+virt
ual+reality&oq=define+virtual+reality.
[Accessed: 07- Feb- 2017].
• [2] "Define Augmented Reality". [Online].
Available:
https://www.google.com/#q=define+augment
ed+reality. [Accessed: 07- Feb- 2017].

69. 69
• [3] "Mixed Reality", Wikipedia. [Online].
Available:
https://en.wikipedia.org/wiki/Mixed_reality.
[Accessed: 28- Mar- 2017].
• [4] D. Woodley, "PLATO: The Emergence of
Online Community", 1994. [Online]. Available:
http://www.thinkofit.com/plato/dwplato.htm.
[Accessed: 10- Feb- 2017].

70. 70
• [5] B. Gillam and S. James, Patent
US5964660 - network multiplayer game.
USPTO, 1997.
• [6] "LAN Party", Wikipedia. Wikimedia
Foundation, 2004.
• [7] B. Hopkins, W. Xu, H. Lu and B.
Knutsson, "Peer-to-peer support for
massively multiplayer games - IEEE Xplore
document", 2004. [Online]. Available:

71. 71
• [8] S. Seshan, J. Pang and A. Bharambe,
"Colyseus: A Distributed Architecture for
Online Multiplayer Games", 2006.
• [9] Microsoft Hololens, "Shared Holographic
Experiences".