1. A Mobile Augmented Reality Environment
for Collaborative Learning and Training
ZGDV, Computer Graphics Center
Mobile Information Visualization
Abstract: The following work focuses on the requirements for a Collaborative Augmented
Reality System with the usage of Internet technologies to create new possibilities for
collaborative teaching. In most learning environments, learning is imparted through seeing
and hearing. In this paper, Collaborative AR Discussion and personal experience are also
integrated in order to achieve a quicker transmission of study material and an increase in
memory efficiency. We have analyzed the scenarios occur most frequently in service and
maintenance and also the literature of Tele-Teaching. The needed requirements were derived
and as a result integrated in a test system. This collaborative game simulates a web-based
training and teaching environment in Collaborative AR as a new approach for teachers and
trainees and describes scenarios and research perspectives for distance education and training
at the same location. The game was used as a testbed to clarify the problems associated with
AR and Collaborative Learning.
Through the integration of Multimedia, study material can be transmitted much faster. In 1965,
William Glasser described what influences a student’s ability to retain information and ideas in his book
Control Therapy in the Classroom. Respective to this thesis, people remember 10% of what they read, 20% of
what they hear, 30% of what they see, 50% of what they see and hear, 70% of what they discuss with others,
80% of what they personally experience and 95% of what they teach others (Glasser 65). With Collaborative
AR, learners communicate directly with each other in a face-to-face communication, but also interact with
remote learners located in other learning rooms. According to Glasser, and as it relates to the interactive aspect
in the learning environment of a Collaborative AR, people can remember information in a very efficient way,
because users can discuss and have experiences by interacting with the augmented world in the collaborative
learning and training environment.
Tele-presence as a multimedia aspect and the access to worldwide distributed information enable a
new way of teaching and learning. By combining learning and multimedia, a couple of net-based teaching and
training systems have already been realized. By Tele-Teaching, we mean the distribution of knowledge in a
synchronous Online Scenario or net of computers, in which the teachers and students work simultaneously. By
Tele-Training, we mean the provision of study materials on network servers. These materials are retrieved by
students asynchronously. Tele-Teaching is usually associated with university study, while Tele-Training is
more suited for further professional training in a cooperate setting (Effelsberg).
Augmented Reality (AR) has become an important part of computer graphics. “Unlike Virtual Reality
where the physical world is completely replaced with synthetic environments, in Augmented Reality environ-
ments, 3D computer graphics objects are mixed with physical objects to become part of the real world”
Billinghurst says (Billinghurst 00). Computer Supported Cooperative Work (CSCW) has also been expanded as
an identifiable interdisciplinary research field. According to Wilson, CSCW is a generic term, which combines
the understanding of the way people work in groups with the enabling technologies of computer networking,
and associated hardware, software, services and techniques (Rodden 91). By combining CSCW with
Augmented Reality, a new collaboration - Collaborative AR - became possible. Collaborative AR is defined by
Reitmayr and Schmalstieg, where co-located users can experience a shared space that is filled with real and
virtual objects (Reitmayr 01). The use of CSCW technologies in combination with Tele-Teaching and Tele-
Training opened a new application area known as Computer Supported Cooperative Learning (CSCL)
(McCONNEL 94). But it is a new idea to take CSCL into Collaborative AR. With Collaborative AR, group
discussions are possible, where each user can interact and attain personal experience. In reference to Glasser, it
is the best way to achieve a quicker transmission of study materials and an increase in memory efficiency.
Today, the ability to communicate with almost anyone, to exchange knowledge, solve problems, and
learn new things via computer networks is widespread. We know that we learn most effectively if we learn
together with others in groups. The Internet allows cooperative learning independent of time and place (IPSI).
Almost all of the existing e-learning systems do not support AR. So, one of the advantages of the
component-based System will be to support individual users in their simultaneous interaction with augmented
objects by using Collaborative AR while exchanging knowledge.
To specify the requirements for a cooperative AR learning environment, the scenarios that occur most
frequently in service and maintenance have been analyzed and two standard situations were identified:
1. cooperative interaction in a master / trainee scenario at the same place with e.g. skilled workers and a
teacher having different views to the augmented world, standing and discussing in front of a machine;
2. direct interaction with a remote expert using AR Technologies like interactive video and having the same
view as the skilled worker. The expert can interact with the skilled worker and offer support.
From these standard situations we have derived the following needed technical and interactive require-
ments. A collaborative AR system should give support and assistance to a learning group in their common
work. Multiple users can be at the same place, at the same time, interacting together directly in the augmented
world. Furthermore, group discussions are supported within the field of Augmented Reality. A common central
view is established for each participant in order to get a defined understanding of the problem and to meet the
requirements for this specific problem. Another aspect of teaching and collaboration is that an AR system has to
provide remote support with the help of Augmented Reality over great distances. However, with a mobile
aspect, the system would give multiple users access to information and help at any given time. It should clarify
technical aspects, such as how to distribute the functionality in a learning collaboration and which methods can
be used to get visual or haptic feedback. To meet the requirement of interaction by a collaborative group, a
game could be implemented using the Collaborative AR Prototype System with the advantage of 3D group
interaction and individual views for multiple users co-operating at the same place or with a connection over the
Internet to remote users. As a result, a 3D Tetris clone was implemented as a testbed for different scenarios.
The Tetris game poses a good possibility for realizing the requirements of a collaborative AR learning
environment, because it can be played at the same place or from a remote location. It is possible to login
dynamically and share the AR Tetris game with other participants. It simulates e.g. the collaboration of skilled
production workers and the connection of technicians to a remote expert for support over the Internet.
Exploring this research topic within a game presents a significant challenge.
The Tetris game is split in the functionality realized by the Tetris Application and the visualization by
the rendering component of the AR system. Both are embedded inside the web page, which displays points, the
next piece, and the augmented view of the game (Fig. 1). The Tetris application can be started by any player
and other users can login dynamically and share the AR Tetris with
these participants. To realize this aspect, the status of the game is
transmitted into the collaboration component. The first player who
starts the game has the role of the master. The other players do not
start another game, but get this status at the beginning of their
session and initialize their game with these settings. If a player
rotates a piece, the orientation of the actual piece will be set into the
system. If a translation is done, the new position of the center of the
piece is set in the system. A video server captures the picture of the
camera and provides it to the rendering component and to the
marker-based tracking system. A component was needed to handle
the communication between the main components of the base AR
system and to notify other components or other clients co-operating Figure 1: 3D AR Tetris clone
together when values are changing.
In the case of the collaborative learning environment, the values to transmit are the orientation of the
scene, the names of the virtual objects used and also the actions like translations or rotations. The values are
present on both the client and server side and are compared by way of a Push Mechanism. The Push
Mechanism has been conceived to forward a notification to the clients immediately after an event occurs. Both
client- and server-sided components can be informed via notification by the Event Mechanism. Another
3. component is responsible for controlling the collaboration. If two users are changing a value at the same time,
this component has to recognize and regulate the collision detection and the common concurrent interaction.
This Collaborative AR System can be used in a wide range of educational, e.g. in the field of
commerce, to help users in their homes by way of a furniture company consultant, or in the industrial field, to
support skilled workers standing in front of a machine by enlisting an expert. By using Collaborative AR to
transmit study materials, a Cooperation at the same place is enabled by transmission of the scene, the actions
and the orientation of the virtual objects to all participating users in the group; a remote cooperation over great
distances is also enabled, requiring additionally the camera image of one of the cooperating users. Thus,
training in a collaborative group interacting together at the same place or via distance education can be
portrayed in a lot of scenarios. Both forms of cooperation can also be transmitted to the participants
synchronously, as well as asynchronously. With an asynchronous connection, a later alteration of the
augmented world is possible through assigning and storing scene, actions and orientation to a specific frame in
the videostream. In this way, a later correction can be carried out to better portray an error or problem and to
heighten the learning experience. In contrast, with synchronous learning in groups, collision recognition in
terms of a simultaneous access of the virtual object is necessary. By implementing a Collaborative Augmented
Reality learning environment, applications in the field of Tele-Teaching and Tele-Training become both syn-
chronously and asynchronously realizable. For a remote synchronous learning environments, the remote user
can have the same model of the real world, which has to be augmented, and can interact the same way as a
collaborative user at the same site. In the case of the Tetris game, both users have the model of a pen or a
joystick for interaction and the board of the game. Another synchronous possibility is to transmit the video
images of one user to the remote user. The interaction can be done with an interactive graphic and will be sent
through the event mechanism, as well. In the case of face-to-face collaboration, a pen with markers is used for
3D interaction. This allows a very natural interaction mechanism with the game. In a later version, speech
recognition will be integrated to enable a multi-modal interface and, according to Glasser, we can improve the
effectiveness of remembering information with the aspect of speech.
But it is also possible to realize asynchronous teaching and training collaborations. We need to store
the camera images of at least one user and also the actions like translations or rotations of the scene, the
orientation of the virtual objects and the name of the scene dependent on the frame number of the images. If we
store the images of more than one user, we can also switch between the views of the users. In this asynchronous
scenario, the images are retrieved from memory, are identified by the corresponding frame number and are
superimposed with the formerly valid objects. However, the virtual objects can be altered later as to position or
orientation, or other virtual objects can be loaded. In this way, errors can be better explained and understood.
This can occur on-site in the real world by projecting the virtual objects time-dependent to the recording in the
real world with the help of a head-mounted display (HMD), or in a remote scenario, displayed on the desktop
together with the video content. Discussions concerning a particular action could be subsequently initiated and
problems better be solved. Explanations could be offered as to the point in time when something was done
wrong and how it could be made better could be visualized.
The Tetris game supports multiple users playing one game. This simulates group discussions and the
connection to a remote teacher collaborating with a student while assembling a mechanical engine. It also
handles the problem of common competing interaction. The actual field of view of other players and their
different games can be integrated into the view of a teacher. One special view can be selected and the
interaction can be switched to the selected game. This represents the scenario of a master who supervises
several trainees learning the AR system or being instructed how to handle a machine. The master can select the
view of a trainee and can give him advice and support in his augmented environment. As a result, a master can
supervise a few learners.
Another aspect is the view of common and private data by the collaboration participants dependent on
the expertise of the users. Thus, each user of the learning collaboration can customize the view to his needs or
to see different aspects of the same thing adapted to the circumstance. Converted to the Tetris game, the users
play the same game and get a private view of the game and their interactions. The common data is the next
piece and the score. The users can add and customize visual aspects to their needs like the deleted levels or the
How to visualize the result of cooperation by a learning group will be a further step in this work. If
many people play the same game remotely, a player wants to know why the piece moves to the left even though
4. he/she pushed the piece to the right. Therefore, the representation of the results of the interaction is an
important field. The best way to do this will be to use multiple indications like color, arrows and numbers (Fig.
2) or the player can get haptic feedback.
Figure 2: Visualization of results Figure 3: Collision detection and Semaphores
There are a lot of difficulties to solve in sharing virtual learning environments. Mutual exclusions have to be
implemented and collision detection is needed. Semaphores have to control the access to interaction
mechanisms. This problem can be approached in one game for two players, where two pieces are falling down
(Fig. 3). Each person can interact with one piece, gets personalized information and has to coordinate his work
with the other player while getting information when collisions are detected.
We have analyzed specific scenarios. The needed technical and interactive requirements for these
scenarios have been derived. A collaborative AR environment was suggested and realized prototypically as a
test system for a Tetris game. The system can be used in a wide range of educational settings and it allows an
asynchronous, as well as a synchronous access e.g. to a learning environment that can be used remotely for
distance education or in a collaborative group interacting together at the same place in a master – trainee
scenario. It is possible to login dynamically to the same group and share the augmented space with other
participants at any given time. It was observed that the fast event mechanism can provide the information to
other participants in real time by pushing the information. A further step for the future is to evaluate the
outcome of this cooperative game and to transfer these initial results to education applications. But until then, a
lot of experiences can be garnered and further research work completed.
Glasser, W. (1965). Control Therapy in the Classroom, Harper & Row: NY, 1986; Reality Therapy:A New
Approach to Psychiatry, Harper & Row, NY.
Wolfgang Effelsberg, Lehrstuhl für Praktische Informatik IV, Universität Mannheim, Lehren und Lernen im
Billinghurst, M., Poupyrev, I., Kato, H., May, R. (2000). Mixing Realities in Shared Space: An Augmented
Reality Interface for Collaborative Computing, ICME2000, New York, USA.
Rodden, T. (1991). A survey of CSCW systems, Interacting with computers - the interdisciplinary journal of
human-computer interaction, 3(3), pp.319-353.
Reitmayr, G., Schmalstieg, D. (2001). Mobile Collaborative Augmented Reality, ISAR2001, New York.
McCONNEL, D. (1994). Implementing Computer Supported Cooperative Learning, Kogan Page Publishing.
Fraunhofer IPSI, Integrated Publication and Information Systems, Darmstadt, http://ipsi.fhg.de/CSCL/
ARVIKA, Augmented Reality for Development, Production and Servicing, Germany, http://www.arvika.de/