Cross-Manipulation in Mixed Reality Based on a Bridge Virtual Marker Raphael Grasset∗ ¨ Julian Stadon † Mark Billinghurst‡ HIT Lab NZ University of Canterbury Private Bag 4800, Christchurch, NZA BSTRACT AR VRIn this paper we introduce a concept for transferring and manip- User Userulating objects between different interface contexts (AR and VR),thus extending the capabilities of manipulation in transitional inter-faces, which provide navigation between these different contexts.To transport objects between AR and VR environments, we propose Shared Datathe use of a virtual marker proxy, which provides a bridge betweenworlds. The concept is demonstrated with a prototype that allowsusers to transfer objects between a Massively Multiplayer OnlineWorld and a desktop Augmented Reality application. AR VRIndex Terms: H.5.1 [Information Interfaces and Presentation]:Multimedia Information Systems—Artiﬁcial, augmented, and vir- User ? Usertual realities; H.5.3 [Information Interfaces and Presentation]:Group and Organization Interfaces—Collaborative computing Data 1 Data 21 I NTRODUCTIONHybrid collaborative systems supporting a combination of Aug-mented Reality Environments, Virtual Reality Environments orWeb Interfaces have been previously explored. In this type of Figure 1: Cross-Manipulation Problem: In the top ﬁgure, two usersMixed Reality collaboration (often highly asymmetric), users can in different worlds (Augmented Reality and Virtual Reality Environ-experience a shared world such as that proposed in . In this con- ments) can manipulate similar data, such as the purple element viatext, the information and the model are generally similar, the dif- a shared data mechanism. However if the data are separated, like infering element is indeed the interface and the representation of the the bottom ﬁgure, how can objects be transferred between worlds?data (see the top of Figure 1 for a typical example). Another range of conﬁguration issues consist in cases where thedata is not shared between both worlds, each world is associatedwith its own data (illustrated at the bottom of Figure 1). In this con- the creation of a unique mixed environment rather than dealing withtext manipulating and exchanging data and content between worlds two separate environments.is intrinsically more challenging. It can be sometimes even more This brief literature survey shows that different solutions havedifﬁcult when users have no control over the architecture of the sys- been proposed for symmetric environments, but hardly any researchtem used; for example with a Massively Multiplayer Online World has been conducted with an asymmetric solution (like between an(MMO). Offering a way to combine a traditional AR system with a AR Context and a VR Context) where different type of data areVR world like a MMO is therefore attractive; thus there is a need associated with each world. In the next section we introduce ourfor techniques to transfer the data between both of these world (see approach to the problem. We focused on the scenario of an ARbottom of Figure 1). environment coupled with a VR environment for a mixed-reality Previous solutions have been proposed in the context of multiple collaboration.virtual worlds in virtual reality. For example, Kiyokawa et al.suggested the use of a window portal metaphor for remote manipu- 2 C ONCEPTlation, using the concept of a tunnel window. 2.1 Virtual Marker For Augmented Reality, Butz et al.  proposed a drag and droptechnique between a 3D AR space to a screen space within the EM- Our approach has been largely inspired by previous works in Vir-MIE system. Schmalstieg et al.  has explored the use of tangible tual Reality and interactive workspaces that use the concept of aobject and public locales to migrate applications between remote bridge. A bridge deﬁnes a dedicated space where the user can trans-physical spaces in a context of a collaborative AR scenario. Fi- fer things between worlds, spaces, and contexts. Since we considernally, MacIntyre has recently modiﬁed a client of a MMO for creat- the possible inability to modify or redesign the Virtual Reality En-ing mixed reality experiences . However, his purpose was more vironment, it is necessary to deﬁne this bridge as part of the content of the virtual world. ∗ e-mail:Raphael.Grasset@hitlabnz.org To do this, we introduce the notion of a virtual marker. Real † e-mail:Julian.Staton@hotmail.com ﬁducial markers are generally used for tracking but they also sup- ‡ e-mail:Mark.Billinghurst@hitlabnz.org port association with virtual objects using tangible interface con- cepts in an Augmented Reality Environment. Similarly, equivalent Virtual Markers can be described in the virtual world. However, in our case the presence of a virtual marker in the virtual world will provide an anchor for an object in the world, and thus a bridge into
Screen (Window of VR) 2.3 Object Coherence State The transfer of an object between both worlds implies the ability to deﬁne the state of the object in both of these worlds. For example, if Virtual Marker we transfer an object from SecondLife to the AR Base and modify an object property like scale, what should happen with the object in SecondLife, is it affected by this or is it in undeﬁned state? If we transfer the object back into SecondLife how will its properties be Real Marker affected? To resolve this issue we consider three cases, mimicking the gen- eral solution implemented in standard 3D Modeling software. A transferred object can be in three states: (1) a copy, (2) an instance or (3) the same unique object. The instance will keep a coherentFigure 2: The user can visualize both real markers and virtual mark- state when manipulated in both of the worlds. The other two casesers in the Augmented Reality Environment. won’t. 3 D ESIGN AND I MPLEMENTATION We have implemented this concept between SecondLife and a ded-the virtual environment. icated AR prototype application. This prototype is part of an art To support the transfer between both worlds, a window of the installation that will be presented for an exhibition in May, in Perth,virtual world should be available in the real world. Having this Australia.window, a user can observe the content of the virtual world and The AR system has been implemented using the osgART librarytherefore visualize virtual markers. In the user’s view, this virtual . The user is equipped with an E-Magin Z800 HMD, and a Log-marker becomes equivalent to a real marker, integrated in the AR itech Quickcam 5000 USB camera. The VR World is displayedApplication (illustrated in Figure 2). on a standard LCD screen. A 3D mesh corresponding to the vir- tual avatar of SecondLife is used to demonstrate the transfer. When the user moves the transfer tool from the AR context base next to2.2 Interactive Space the virtual marker on screen a 3D mesh is transferred from the AR environment into the on-screen SecondLife environment. To showAssuming the availability of a virtual marker, our system uses three that we can change object state while doing this, when the user hascomponents to transfer objects from the virtual world to the AR transferred the uncoloured 3D mesh from the transfer tool to theworld. We use the virtual marker, a transfer tool and a contextual base, the object is coloured and textured (and can be also animated).base. The Contextual Base provides a reference space for manip-ulating the Augmented Reality Content, while the Transfer Tool We have been confronted with different design issues for the im-provides the interface to transfer things between the Virtual World plementation of this prototype, including:and the Augmented Reality World. Window: the visibility of the virtual marker is highly dependent on the technology used to display the virtual marker. Thus the use The user can transfer elements using proximity between two of a non-reﬂective desktop screen provided the best solution. Amarkers. They can thus realize bidirectional transfers in this or- projected image is problematic since the use of the camera for ARder: Virtual Marker ↔ Transfer Tool ↔ Base. To help the user to tracking requires good ambient lighting.understand the relationship between them we augment the markers Markers: the virtual marker can be easily designed with built-inwith basic visual feedback provided by a virtual torus around each tools from SecondLife. Using the navigation mode of SecondLife,of the components (shown in Figure 3). placing a vertical marker in front of a clear background will im- [You might want to put more here about the actual mechanism prove the robustness of the AR tracking.of how the transfer is done - see what I wrote below] Coherence State: it remains challenging to support and maintain a coherent object state with SecondLife. The SecondLife API is limited, so we have only been able to implement a copy mode in VR AR our application. Window of VR Context 4 C ONCLUSION We have presented an implementation for transferring and manipu- Virtual lating objects from a VR MMO to an AR World based on using a Bridge Marker virtual marker. Acting like a bridge, the user can have the beneﬁt of both worlds to visualize and modify different objects. We have Transfer identiﬁed design issues with this type of solution and presented our Tool initial approach to resolving them. In the future we hope to inves- tigate further the problems related to maintaining a coherent data AR Context state between both worlds. Base R EFERENCES  B. Brown, I. MacColl, M. Chalmers, A. Galani, C. Randell, and A. Steed. Lessons from the lighthouse: collaboration in a shared mixedFigure 3: A Virtual transfer: the red torus is on the virtual marker, the reality system. In CHI ’03, pages 577–584, 2003.green torus on the transfer tool, and the blue on the AR base marker.  A. Butz, T. H¨ llerer, S. Feiner, B. MacIntyre, and C. Beshers. Envelop- o ing users and computers in a collaborative 3d augmented reality. In IWAR ’99: Proceedings of the 2nd IEEE and ACM International Work- shop on Augmented Reality, page 35, Washington, DC, USA, 1999.
 K. Kiyokawa and H. Takemura. A tunnel window and its variations: Seamless teleportation techniques in a virtual environment. In HCI International 2005, 2005. T. Lang, B. MacIntyre, and I. J. Zugaza. Massively multiplayer online worlds as a platform for augmented reality experiences. In IEEE VR ’08, 2008. J. Looser, R. Grasset, H. Seichter, and M. Billinghurst. Osgart - a prag- matic approach to mr. In Industrial Workshop at ISMAR 2006, 2006. D. Schmalstieg and G. Hesina. Distributed applications for collabora- tive augmented reality. In IEEE Virtual Reality 2002, 2002.