Kondo(2006) Augmented Learning Environment Using Mixed Reality Technology


Published on

Kondo(2006) Augmented Learning Environment Using Mixed Reality Technology

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Kondo(2006) Augmented Learning Environment Using Mixed Reality Technology

  1. 1. Augmented Learning Environment using Mixed Reality Technology Tomotsugu Kondo National Institute of Multimedia Education Japan tkondo@nime.ac.jp Abstract: This paper describes an augmented learning environment using mixed reality technology. Mixed reality is similar to virtual reality technology that combines interactive three- dimensional computer graphics with the real world. As a result, giving rise to the possibility of expanding the functions of traditional media. The purpose of this study is to propose the augmented learning environment. Therefore the following 3 examples of using this technology; 1) An augmented text books, 2) a mathematics instructional material and 3) a museum display system were developed. Furthermore, the authoring tool for developing these contents was introduced. Introduction In the current educational media, textbooks, overhead cameras and personal computers etc. have been used in classrooms. On the other hand, three-dimensional computer graphics (3DCG) technology used with personal computers has been gaining popularity in recent years. Because of this situation, the potential of 3DCG technology for advance learning is currently increasing. One example of 3DCG technology is virtual reality (VR). This is used in various fields of application, in particular simulation and in the area of training. However, VR needs special equipments such as HMD, tracking sensor, data glove, immersive display system etc. This means that it is difficult to prepare such equipment for a general school, which in turn is a problem. One solution to the problem of VR is using mixed reality (MR), which focuses on advancing the current learning environment. MR is in between virtual and real environment (Milgram & Colquhoun 1999). MR is similar to VR technologies. However, it differs on the point that it can merge 3DCG with the real world. In other words, although majority of VR contents are made from only computer graphics, MR contents are made from computer graphics in the real world. To use both virtual and real is important in the educational field because it is suitable for hands-on training. The possibility of advancing the current learning environment is very high using the combination of interactive 3DCG with the real world. As a result, it can expand the functions of traditional media. The purpose of this study is to alter the traditional learning environment by using mixed reality with the traditional educational media such as textbooks and overhead cameras. In order to have a better understanding of that environment, the following examples are given; 1) augmented textbooks, 2) a mathematics instructional material and 3) a museum display system were developed. Mixed Reality System Media Characteristics of Virtual Reality, Web3D and Mixed Reality In order to apply 3DCG to education, there are some methods, for example Virtual Reality, Web3D, and Mixed Reality. These consist of different systems and each system has each media characteristics. Fig.1-3 all expresses the contents of the same church in France. Fig. 1 shows immersive VR system called CAVE, the operator enters into a 3m cube, and by using stereoscopic vision, and he can have the experience of actually being in the actual church. Fig. 2 shows the content of Web3D, which displays 3DCG on an Internet browser. Although an immersive VR system is more realistic than Web3D, financially Web3D is m practical for e-learning. Fig. 3 ore shows the exterior of the same church displayed on top of a hand. In a mixed reality environment, the displayed mage can be shown on a person’s hand therefore, the i operator can experience an animated image, which is embedded on an actual hand in a normal environment.
  2. 2. Fig.1 Immersive VR Fig.2 Web3D Fig.3 Mixed Reality Elemental Technology of Mixed Reality There is a display system which specializes in MR, one of them uses a half mirror (Bimber et al. 2001), and the other uses a video see-through HMD with a built in small video camera (Uchiyama et al. 2002). The final system uses a PDA or handheld PC wit h a video camera (Schmalstieg & Wagner 2005). All of these systems are the combination of reality mixed with 3DCG technology. Moreover, MR technology is a combination of reality mixed with 3DCG and it also has tracking technology in which an animated image can follow a person’s head or hand. In such tracking method, there are a magnetic position sensor, a gyroscope, etc., a method of calculating a position and inclination by recognizing the image of the two -dimensional marker in the real world, the method of detecting from the form of the real world by image recognition (Papagiannakis et al. 2005), etc. This two- dimensional marker tracking method was used in this study . Development and Operating Environment ARToolKit (Kato & Billinghurst 1999) was used to develop the contents in this study. It is a software library to create a mixed reality environment. It uses a two -dimensional marker for tracking. A key feature of using ARToolKit, there is no special equipment needed for this technology , therefore this technology is well suited for educational purpose. The development was do ne in the following environment. OS: Windows 2000 Professional Compiler: Microsoft Visual C++6.0 Graphics library: OpenGL+GLUT Multimedia API: Microsoft DirectX 8.0 SDK Mixed Reality library: ARToolKit 2.65 The application can be operated on Intel Pentium III 600MHz or higher with Windows2000 / XP. A high speed graphic accelerator card works well with OpenGL, and a Web camera or a video camera connection is necessary. Examples of Mixed Reality Contents Augmented Textbook Printed media is the most popular media and majority of printed instructional materials includes illustrations as additional information of text. Occasionally, this is not enough information to understand the illustration because it is in a two-dimensional form. Fig. 4 is an “augmented textbook” that explains the inside of a human brain. The students use a PC with a web camera to create a 3D image while looking at their textbook. Teachers can also use it in their lecture. A university lecturer can use a plastic model of a human brain, but the biggest difference between the model and a 3D model is that learners cannot see through the model. However they can with the 3D model.
  3. 3. Fig.5 is another augmented textbook for midair structure. An ordinary text page is limited because of the limitation of the page. Creating an augmented textbook using 3D animation would resolve this dilemma. For example, it is impossible to show from the earth to the moon on one page in an ordinary textbook. However, it is possible to explain the orbit of a communication satellite or the moon from the earth using 3D animation. Fig.4 Augmented t extbook (human brain) Fig.5 Augmented t extbook (midair structure) Mathematics Instructional Material In elementary school and junior high school in Japan, the number of space figures that are taught in mathematic classes has been decreasing. Therefore, the number of students who do not know many space figures has been increasing in high school and higher education. To solve this problem, “mrSpaceFigures” was developed by using MR technology for the expression and manipulation of the space figures. By using MR technology, it can be operated as if there was a space figure in front of them. This application software has a cut mode and a boolean operation mode. Fig.7 shows the cut mode, a prepared space figure appears on the base marker and its size and direction are variable. The space figure is cut by moving a paddle across a cut ting plane. The upper right window is the cut surface. When the mode is changed to the operation mode (Fig.8), the paddle function is changed. In this mode, there are two space figures, one is on the base card and another is on the paddle. The space figure represented on the paddle can merge with another space figure on the base maker by boolean operation. A usability test was conducted in a high school mathematics classroom using this application. Based on this evaluation, some of the functions were improved. The final product of this software can be downloaded from the URL below: < http://tkondo.nime.ac.jp/mrsf/ >. Fig.6 System configuration of mrSpaceFigures Fig.7 Cut mode
  4. 4. Fig.8 Boolean operation mode Fig. 9 High school mathematics class Museum Display System There are actual skeletons of a dinosaur in museums. However, many children do not really know what a dinosaur looked like before, for example the contour of the body or its skin color. In this system, by using a handheld pc with camera it can be seen to replicate the look of the dinosaur overlay the actual skeleton. Fig.10 is a picture from the National Science museum in Tokyo. Fig.11 is the display of the handheld PC, it is shown a half transparent 3DCG and names of the body parts. Stegosaurus Formatted: Font color: White Web cam Formatted: Font color: White Handheld PC Formatted: Font color: White Marker Formatted: Font color: White Fig.10 Configuration of museum display system Fig.11 Display of handheld PC The purpose of this content is not only for children to have a better understanding of what dinosaurs looked like before but also children can think scientifically. Because the skin color can be changed by some academic theories. For example, Fig.12 is the examples of the skin color. This system has an interactive function, which asks the children to choose a specific skin color for the certain purpose. Fig.12 Examples of skin color of the dinosaur
  5. 5. Authoring Tool for Mixed Reality Contents The MR contents that were introduced in this paper were developed by using ARToolKit. Knowledge of C++ and OpenGL is required for those developments, and it is difficult for general users to develop their own contents. Dart is an authoring tool developed by GVU Center, Georgia Institute of Technology (MacIntyre et al. 2003). Even though this is a user-friendly tool, special knowledge, such as Macromedia Director and its scripting language is required. The authoring tool will eliminate the hassle of traditional C++ programming which needs a lot of time and experience. Therefore, the general user will be able to spend more time of developing the content instead of programming. Currently, an authoring tool is now under development in order to develop MR content such as augmented t extbooks and the museum display system etc. which were introduced in this paper. By using this tool, MR contents by a two-dimensional marker can be developed easily in a short time (Fig.13). Fig.13 Authoring t ool for mixed reality contents Fig.14 Execution screen To create contents, it is necessary to prepare a 3D object or 3D animation separately created with a 3D modeling software. The fundamental function of this authoring tool makes this 3D object correspond to the selected marker. The number of markers, the position, and the size can be specified and the position of 3D object and zooming are possible. Furthermore, it also has the function of annotations, sounds, and hyperlinks. Fig. 13 shows the authoring scene which assigned the hippocampus of the human brain to the marker and added an annotation of the name. Fig. 14 shows the scene which executes this content and where the hippocampus and the annotation are displayed on the marker. Conclusions This study focused on how MR is one way of applying 3DCG technology to education. The special feature of MR is the ability to merge interactive 3DCG with the real world. It is possible to make it merge with the traditional educational media in the real world by using the special feature efficiently. Accordingly, in this research, the augmented learning environment which extends the traditional learning environment using this mixed reality technology was proposed. This paper showed the following three applications, 1) augmented text books, 2) a mathematics instructional material and 3) a museum display system . The following examples would have been impossible using traditional educational media. However, using an augmented learning environment, doing the impossible was possible. 1) In order to compare the additional information by 3DCG with a real object, make 3DCG into translucence. 2) Description is added to a real object by annotation. 3) T extures are changed and various conditions are compared. 4) The intuitive interface by a real object is possible. Furthermore, to make augmented learning more practical, a user-friendly authoring tool was developed to assist general users to focus on developing contents easily. Therefore, the authoring tool under development was introduced. Even though MR technology still has ways to go and there are technical problems, the potential and p ossibilities of improving traditional learning environment is endless.
  6. 6. Acknowledgements A part of this study was subsidized by The Ministry of Education, Culture, Sports, Science and Technology, and Japan Society for the Promotion of Science with Grant-in-Aid for Scientific Research (No. 16650219 and No.17300284). References Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. (2001). TheVirtual Showcase. IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55 Kato, H., and Billinghurst, M. (1999). Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System, In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality ’ 99 (Iwar'99), San Francisco: 85-94 MacIntyre, B., Gandy, M., Bolter, J., Dow, S., and Hannigan, B. (2003). DART: The Designer’s Augmented Reality Toolkit. Presented as a demo at The Second International Symposium on Mixed and Augmented Reality (ISMAR03), pp. 329-339 Milgram, P., and Colquhoun, H.(1999). A taxonomy of real and virtual world display integration. In Y. O. H. Tamura (Ed.), Mixed Reality: Merging Real and Virtual Worlds. pp. 5-30. Tokyo: Ohmsha / Springer-Verlag. Papagiannakis,G.,Kim,H.,Magnenat -Thalmann,N.(2005). Believability and Presence in Mobile Mixed Reality Environments. IEEE VR2005 Workshop on Virtuality Structures. February 2005. Schmalstieg, D. and Wagner, D. (2005). A handheld augmented reality museum g uide. In Proceedings of IADIS International Conference on Mobile Learning 2005 . Uchiyama, S. , Takemoto, K., Satoh, K., Yamamoto, H ., and Tamura, H.(2002). MR Platform: A basic body on which mixed reality applications are built, In Proceedings of IEEE and ACM Int. Symp. on Mixed and Augmented Reality (ISMAR 2002), pp. 246-253