1. Augmented Learning Environment using Mixed Reality Technology
National Institute of Multimedia Education
Abstract: This paper describes an augmented learning environment using mixed reality
technology. Mixed reality is similar to virtual reality technology that combines interactive three-
dimensional computer graphics with the real world. As a result, giving rise to the possibility of
expanding the functions of traditional media. The purpose of this study is to propose the augmented
learning environment. Therefore the following 3 examples of using this technology; 1) An
augmented text books, 2) a mathematics instructional material and 3) a museum display system
were developed. Furthermore, the authoring tool for developing these contents was introduced.
In the current educational media, textbooks, overhead cameras and personal computers etc. have been used
in classrooms. On the other hand, three-dimensional computer graphics (3DCG) technology used with personal
computers has been gaining popularity in recent years. Because of this situation, the potential of 3DCG technology
for advance learning is currently increasing. One example of 3DCG technology is virtual reality (VR). This is used
in various fields of application, in particular simulation and in the area of training. However, VR needs special
equipments such as HMD, tracking sensor, data glove, immersive display system etc. This means that it is difficult
to prepare such equipment for a general school, which in turn is a problem.
One solution to the problem of VR is using mixed reality (MR), which focuses on advancing the current
learning environment. MR is in between virtual and real environment (Milgram & Colquhoun 1999). MR is similar
to VR technologies. However, it differs on the point that it can merge 3DCG with the real world. In other words,
although majority of VR contents are made from only computer graphics, MR contents are made from computer
graphics in the real world. To use both virtual and real is important in the educational field because it is suitable for
The possibility of advancing the current learning environment is very high using the combination of
interactive 3DCG with the real world. As a result, it can expand the functions of traditional media.
The purpose of this study is to alter the traditional learning environment by using mixed reality with the
traditional educational media such as textbooks and overhead cameras. In order to have a better understanding of
that environment, the following examples are given; 1) augmented textbooks, 2) a mathematics instructional
material and 3) a museum display system were developed.
Mixed Reality System
Media Characteristics of Virtual Reality, Web3D and Mixed Reality
In order to apply 3DCG to education, there are some methods, for example Virtual Reality, Web3D, and
Mixed Reality. These consist of different systems and each system has each media characteristics. Fig.1-3 all
expresses the contents of the same church in France. Fig. 1 shows immersive VR system called CAVE, the operator
enters into a 3m cube, and by using stereoscopic vision, and he can have the experience of actually being in the
actual church. Fig. 2 shows the content of Web3D, which displays 3DCG on an Internet browser. Although an
immersive VR system is more realistic than Web3D, financially Web3D is m practical for e-learning. Fig. 3
shows the exterior of the same church displayed on top of a hand.
In a mixed reality environment, the displayed mage can be shown on a person’s hand therefore, the
operator can experience an animated image, which is embedded on an actual hand in a normal environment.
2. Fig.1 Immersive VR Fig.2 Web3D Fig.3 Mixed Reality
Elemental Technology of Mixed Reality
There is a display system which specializes in MR, one of them uses a half mirror (Bimber et al. 2001), and
the other uses a video see-through HMD with a built in small video camera (Uchiyama et al. 2002). The final system
uses a PDA or handheld PC wit h a video camera (Schmalstieg & Wagner 2005). All of these systems are the
combination of reality mixed with 3DCG technology.
Moreover, MR technology is a combination of reality mixed with 3DCG and it also has tracking
technology in which an animated image can follow a person’s head or hand.
In such tracking method, there are a magnetic position sensor, a gyroscope, etc., a method of calculating a
position and inclination by recognizing the image of the two -dimensional marker in the real world, the method of
detecting from the form of the real world by image recognition (Papagiannakis et al. 2005), etc. This two-
dimensional marker tracking method was used in this study .
Development and Operating Environment
ARToolKit (Kato & Billinghurst 1999) was used to develop the contents in this study. It is a software
library to create a mixed reality environment. It uses a two -dimensional marker for tracking. A key feature of using
ARToolKit, there is no special equipment needed for this technology , therefore this technology is well suited for
educational purpose. The development was do ne in the following environment.
OS: Windows 2000 Professional
Compiler: Microsoft Visual C++6.0
Graphics library: OpenGL+GLUT
Multimedia API: Microsoft DirectX 8.0 SDK
Mixed Reality library: ARToolKit 2.65
The application can be operated on Intel Pentium III 600MHz or higher with Windows2000 / XP. A high
speed graphic accelerator card works well with OpenGL, and a Web camera or a video camera connection is
Examples of Mixed Reality Contents
Printed media is the most popular media and majority of printed instructional materials includes
illustrations as additional information of text. Occasionally, this is not enough information to understand the
illustration because it is in a two-dimensional form. Fig. 4 is an “augmented textbook” that explains the inside of a
human brain. The students use a PC with a web camera to create a 3D image while looking at their textbook.
Teachers can also use it in their lecture. A university lecturer can use a plastic model of a human brain, but the
biggest difference between the model and a 3D model is that learners cannot see through the model. However they
can with the 3D model.
3. Fig.5 is another augmented textbook for midair structure. An ordinary text page is limited because of the
limitation of the page. Creating an augmented textbook using 3D animation would resolve this dilemma. For
example, it is impossible to show from the earth to the moon on one page in an ordinary textbook. However, it is
possible to explain the orbit of a communication satellite or the moon from the earth using 3D animation.
Fig.4 Augmented t extbook (human brain) Fig.5 Augmented t extbook (midair structure)
Mathematics Instructional Material
In elementary school and junior high school in Japan, the number of space figures that are taught in
mathematic classes has been decreasing. Therefore, the number of students who do not know many space figures has
been increasing in high school and higher education. To solve this problem, “mrSpaceFigures” was developed by
using MR technology for the expression and manipulation of the space figures. By using MR technology, it can be
operated as if there was a space figure in front of them.
This application software has a cut mode and a boolean operation mode. Fig.7 shows the cut mode, a
prepared space figure appears on the base marker and its size and direction are variable. The space figure is cut by
moving a paddle across a cut ting plane. The upper right window is the cut surface. When the mode is changed to the
operation mode (Fig.8), the paddle function is changed. In this mode, there are two space figures, one is on the base
card and another is on the paddle. The space figure represented on the paddle can merge with another space figure
on the base maker by boolean operation.
A usability test was conducted in a high school mathematics classroom using this application. Based on this
evaluation, some of the functions were improved. The final product of this software can be downloaded from the
URL below: < http://tkondo.nime.ac.jp/mrsf/ >.
Fig.6 System configuration of mrSpaceFigures Fig.7 Cut mode
4. Fig.8 Boolean operation mode Fig. 9 High school mathematics class
Museum Display System
There are actual skeletons of a dinosaur in museums. However, many children do not really know what a
dinosaur looked like before, for example the contour of the body or its skin color. In this system, by using a
handheld pc with camera it can be seen to replicate the look of the dinosaur overlay the actual skeleton. Fig.10 is a
picture from the National Science museum in Tokyo. Fig.11 is the display of the handheld PC, it is shown a half
transparent 3DCG and names of the body parts.
Stegosaurus Formatted: Font color: White
Formatted: Font color: White
PC Formatted: Font color: White
Marker Formatted: Font color: White
Fig.10 Configuration of museum display system Fig.11 Display of handheld PC
The purpose of this content is not only for children to have a better understanding of what dinosaurs looked
like before but also children can think scientifically. Because the skin color can be changed by some academic
theories. For example, Fig.12 is the examples of the skin color. This system has an interactive function, which asks
the children to choose a specific skin color for the certain purpose.
Fig.12 Examples of skin color of the dinosaur
5. Authoring Tool for Mixed Reality Contents
The MR contents that were introduced in this paper were developed by using ARToolKit. Knowledge of
C++ and OpenGL is required for those developments, and it is difficult for general users to develop their own
contents. Dart is an authoring tool developed by GVU Center, Georgia Institute of Technology (MacIntyre et al.
2003). Even though this is a user-friendly tool, special knowledge, such as Macromedia Director and its scripting
language is required. The authoring tool will eliminate the hassle of traditional C++ programming which needs a lot
of time and experience. Therefore, the general user will be able to spend more time of developing the content instead
Currently, an authoring tool is now under development in order to develop MR content such as augmented
t extbooks and the museum display system etc. which were introduced in this paper. By using this tool, MR contents
by a two-dimensional marker can be developed easily in a short time (Fig.13).
Fig.13 Authoring t ool for mixed reality contents Fig.14 Execution screen
To create contents, it is necessary to prepare a 3D object or 3D animation separately created with a 3D
modeling software. The fundamental function of this authoring tool makes this 3D object correspond to the selected
marker. The number of markers, the position, and the size can be specified and the position of 3D object and
zooming are possible. Furthermore, it also has the function of annotations, sounds, and hyperlinks. Fig. 13 shows the
authoring scene which assigned the hippocampus of the human brain to the marker and added an annotation of the
name. Fig. 14 shows the scene which executes this content and where the hippocampus and the annotation are
displayed on the marker.
This study focused on how MR is one way of applying 3DCG technology to education. The special feature
of MR is the ability to merge interactive 3DCG with the real world. It is possible to make it merge with the
traditional educational media in the real world by using the special feature efficiently. Accordingly, in this research,
the augmented learning environment which extends the traditional learning environment using this mixed reality
technology was proposed. This paper showed the following three applications, 1) augmented text books, 2) a
mathematics instructional material and 3) a museum display system . The following examples would have been
impossible using traditional educational media. However, using an augmented learning environment, doing the
impossible was possible. 1) In order to compare the additional information by 3DCG with a real object, make
3DCG into translucence. 2) Description is added to a real object by annotation. 3) T extures are changed and various
conditions are compared. 4) The intuitive interface by a real object is possible. Furthermore, to make augmented
learning more practical, a user-friendly authoring tool was developed to assist general users to focus on developing
contents easily. Therefore, the authoring tool under development was introduced. Even though MR technology still
has ways to go and there are technical problems, the potential and p ossibilities of improving traditional learning
environment is endless.
A part of this study was subsidized by The Ministry of Education, Culture, Sports, Science and Technology, and
Japan Society for the Promotion of Science with Grant-in-Aid for Scientific Research (No. 16650219 and
Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. (2001). TheVirtual Showcase. IEEE Computer
Graphics & Applications, vol. 21, no.6, pp. 48-55
Kato, H., and Billinghurst, M. (1999). Marker Tracking and HMD Calibration for a Video-based Augmented Reality
Conferencing System, In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality ’
99 (Iwar'99), San Francisco: 85-94
MacIntyre, B., Gandy, M., Bolter, J., Dow, S., and Hannigan, B. (2003). DART: The Designer’s Augmented Reality
Toolkit. Presented as a demo at The Second International Symposium on Mixed and Augmented Reality (ISMAR03),
Milgram, P., and Colquhoun, H.(1999). A taxonomy of real and virtual world display integration. In Y. O. H.
Tamura (Ed.), Mixed Reality: Merging Real and Virtual Worlds. pp. 5-30. Tokyo: Ohmsha / Springer-Verlag.
Papagiannakis,G.,Kim,H.,Magnenat -Thalmann,N.(2005). Believability and Presence in Mobile Mixed Reality
Environments. IEEE VR2005 Workshop on Virtuality Structures. February 2005.
Schmalstieg, D. and Wagner, D. (2005). A handheld augmented reality museum g
uide. In Proceedings of IADIS
International Conference on Mobile Learning 2005 .
Uchiyama, S. , Takemoto, K., Satoh, K., Yamamoto, H ., and Tamura, H.(2002). MR Platform: A basic body on
which mixed reality applications are built, In Proceedings of IEEE and ACM Int. Symp. on Mixed and Augmented
Reality (ISMAR 2002), pp. 246-253