Kevin L. Dean
                                                       Virtual Explorer:

                                                                enabled the developme...
Dean et al.   507

Figure 2. Students can interact with the immune system in multiple environments (left to right: bloo...

Figure 3. A solo space mission gone terribly wrong: Background story for the Virtual ...
Dean et al.   509

Figure 4. The immunology module. Diverse scienti c disciplines ranging from astronomy to quantum mec...

Figure 7. An optional display, keeping the user updated about the nanobot status (lef...
Dean et al.   511

                                                                    ever, makes it impossible for on...

Figure 11. The immunology module, allowing the student to select from missions that e...
Dean et al.   513

Figure 13. Results summaries, concluding each task with an update on the current status of the immun...

Figure 15. Differing scales. Depicting scales that differ by several orders of magnit...
Dean et al.   515

Figure 17. Virtual Explorer’s depiction of the bloodstream, helping to clarify issues of relative ce...

Figure 19. Full-motion video animation, supplementing the interactive real-time graph...
Dean et al.   517

Figure 21. The Virtual Explorer software in our most expansive
version, running on a four-processor ...

Figure 23. The Onyx generating six-channel video (RGBS), which is processed through R...
Dean et al.   519

                                                                     Ultimately, a much more modest ...

Figure 27. User input from a Windows PC and audio output to an SGI Indigo2 Extreme, l...
Dean et al.   521

plorer is subdivided into six threads of execution, based
upon Performer’s multiprocessing framework...

Figure 30. Software:Theater at HeinzNixdorf Museumsforum in
Paderborn, Germany.      ...
Dean et al.   523

Dean, K. L., Finn, E. M., Friesner, J. A., Naylor, B. J., Wust-     Physical and Informational Techn...
Upcoming SlideShare
Loading in …5

Virtual Explorer: Interactive Virtual Environment for Education


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Virtual Explorer: Interactive Virtual Environment for Education

  1. 1. Kevin L. Dean Virtual Explorer: Interactive Virtual Environment for Xylar S. Asay-Davis Education Evan M. Finn Tim Foley Jeremy A. Friesner Yo Imai Bret J. Naylor Sarah R. Wustner Abstract University of California, San Diego La Jolla, CA 92093-0339 The Virtual Explorer project of the Senses Bureau at the University of California, San Diego, focuses on creating immersive, highly interactive environments for edu- Scott S. Fisher cation and scienti c visualization which are designed to be educational—and excit- Telepresence Media ing, playful, and enjoyable, as well. We have created an integrated model system on San Francisco, CA USA human immunology to demonstrate the application of virtual reality to education, and we’ve also developed a modular software framework to facilitate the further Kent R. Wilson extension of the Virtual Explorer model to other elds. The system has been in- Department of Chemistry and stalled internationally in numerous science museums, and more than 7,000 individu- Biochemistry als have participated in demonstrations. The complete source code—which runs on University of California, San Diego a variety of Silicon Graphics computers—is available on CD-ROM from the authors. 1 Overview and Purpose The Senses Bureau is an undergraduate research group with a thirty-year history of innovation in computer graphics and multimedia technology for ed- ucation and scienti c visualization. We at the Bureau believe that virtual reality (VR) has excellent potential as an educational medium to supplement conven- tional techniques because it provides both greater interactivity as well as the ability to create a convincing sense of immersion in the computer-generated environment that is beyond what is possible with conventional textbook- and blackboard-based educational approaches. Many topics in science education involve processes that occur simulta- neously on multiple time and length scales that are dif cult to accurately repre- sent, perceive, and visualize with traditional static media. Examples can be found in complex elds such as immunology, astronomy, relativistic dynamics, quantum mechanics, and rainforest ecology. We wanted to create a system that would be suitable for a diverse target audience that includes several types of educational venues, such as high school, college, and university institutions, museums and other public places, and independent student use. Although we do not feel that 3-D graphics technology can entirely replace conventional classroom teaching techniques, we are convinced that properly implemented virtual environments can serve as valuable supplemental teaching and learning resources to augment and reinforce traditional methods. The Virtual Explorer project employs a two-tiered approach to demonstrat- Presence, Vol. 9, No. 6, December 2000, 505–523 ing VR’s potential for scienti c visualization, as well as to creating interactive ©2001 by the Massachusetts Institute of Technology virtual environments for education. First, we’ve developed a proof-of-concept Dean et al. 505
  2. 2. 506 PRESENCE: VOLUME 9, NUMBER 6 enabled the development of user-interface technologies that can immerse a student in these interactive learning environments. It seems that the capabilites of these new technologies facilitate learning through a process of self- paced exploration and discovery, in contrast to the more traditional approach of instruction and memorization. Through the interactive exploration of immersive envi- ronments, a student can engage in a curriculum that is based on learning by doing, as well as encountering subject matter in contexts that are more meaningful. Several attempts to develop immersive learning envi- ronments predate the use of computational technolo- gies, and two of the most memorable speci cally relate Figure 1. Concept view of the Virtual Explorer theater in the to the human body. A surviving example is the walk- development lab. through scale model of the human heart at the Franklin Institute science museum in Philadelphia, Pennsylvania. Since the 1950s, visitors can explore the giant chambers of the heart surrounded by a soundtrack of booming Virtual Explorer system to demonstrate and study the heartbeats. Later, in the 1970s, neurosurgeon David potential applications and bene ts of an integrated VR Bogen and artist David Macaulay developed a detailed installation in an educational arena. This prototype in- proposal for a thirty-story replica of the human brain as stallation currently runs our example module, which a new museum for San Jose, California. Bogen intended focuses on human immunology (Figure 1). Second, we the structure as an important learning environment for have created a modular software framework and toolkit medical students studying neuroanatomy as well as for for the further development of virtual reality for educa- the general public (Bogen, 1972). tion based on the Virtual Explorer model. We envision For many decades, interactive real-time graphics have numerous applications for the Virtual Explorer as a visu- been used for training applications that require the ac- alization tool in diverse scienti c elds and hope that quisition of speci c skill sets for unique missions or pur- this toolkit (which is available from the authors in full poses (such as the control of a variety of aircraft, auto- source code version for a wide variety of Silicon Graph- mobiles, or ships). But its use for more-general ics computers) will provide others with the means to educational applications hasn’t been explored until the expand upon our work. recent development of lower-cost hardware platforms and powerful software tools. Recent research efforts that examine the use of virtual environment technology in 2 Background education include: In the past thirty years, many research and com- c Science Space—This joint research effort between mercial efforts have investigated the application of new George Mason University and NASA’s Johnson media technologies to education. In particular, the de- Space Center is developing a series of “virtual real- velopment of computer-based interaction with educa- ity microworlds” for teaching science concepts and tional material has enabled the development of learning skills through the use of an interactive virtual labo- environments that can be personalized to better match ratory con guration. Current modules include individual vocabularies, styles, and speci c needs. More NewtonWorld, MaxwellWorld, and PaulingWorld recently, advances in interactive computer graphics have (Dede, 1996; Salzman, 1999).
  3. 3. Dean et al. 507 Figure 2. Students can interact with the immune system in multiple environments (left to right: blood vessel, cell surface, lymph node). c Zengo Sayu—This immersive, interactive virtual ment that features compelling visual and interactive quali- environment is designed to teach Japanese preposi- ties and that has been designed to be attractive to students tions to students who have no prior knowledge of who have been raised in an age of computer games and the Japanese language. In one con guration, stu- music videos. An entertaining background story, whose dents can hear digitized speech samples represent- plot is set on an isolated spacecraft, captures the user’s ing the Japanese name of many virtual objects and imagination with a fantastic setting and expands the mis- their relative spatial location when touched by the sion beyond its immunological content (Figure 3). user in the virtual environment. The system was After selecting the immunology module (Figure 4), developed at the Human Interface Laboratory at the student is presented with a brief computer-animated the University of Washington (Rose, 1996). movie that describes an ill-fated mission into deep c Anatomic Virtualizer—This interactive, immersive, space. Returning with samples of a dangerous off-world virtual environment for teaching anatomy at the bacteria, the transport ship USS Archon suffers an ex- university level was developed at the Learning Re- plosion caused by an unnoticed fuel leak in the propul- sources Center in the School of Medicine, Univer- sion system. This explosion allows the bacteria to escape sity of California, San Diego (Hoffman, 1999). and to contaminate the ship’s air supply, resulting in the infection of the pilot, the ship’s sole crew member. He possesses only minimal medical knowledge, and the 3 The Mission ship’s supply of antibiotics has proven useless against The Virtual Explorer allows students to interac- this foreign pathogen. Being an accomplished engineer, tively explore the immune system at both the cellular however, the pilot has been able to modify the remote- and molecular scales, and at more familiar time and controlled nanobots that are normally used for repairing length scales, while still retaining a sense of overall sys- the ship’s computers for operation within his own body temic scale. Students are free to explore realistic virtual (Figure 5). Online references, a helpful “ship’s com- environments that include blood vessels, cell surfaces, puter” character, and virtual tools are available to assist and lymph nodes, while carrying out detailed missions the student-pilot in completing the mission. For exam- and several series of assigned tasks (Figure 2). We seek ple, the nanobot is equipped with several tools that aid to provide students with a means not only to explore the pilot in carrying out this unique mission, including the structure, appearance, and function of various com- monoclonal antibody-based protein dye jets for identify- ponents of the immune system, but also with a tool for ing different types of white blood cells, a remote probe gaining an understanding of the interactions among that allows the pilot to explore cell surfaces at the mo- these components. lecular scale, a vacuum for collecting bacterial samples, We present immunology in a rich, game-like environ- and protein dye jets (Figure 6).
  4. 4. 508 PRESENCE: VOLUME 9, NUMBER 6 Figure 3. A solo space mission gone terribly wrong: Background story for the Virtual Explorer’s immunology module.
  5. 5. Dean et al. 509 Figure 4. The immunology module. Diverse scienti c disciplines ranging from astronomy to quantum mechanics are also candidates for the Virtual Explorer. Figure 6. Virtual tools. Such tools, including a bacterial sample Figure 5. Remote-controlled nanobots. These nanobots provide a collection vacuum shown here, assist students in performing assigned vehicle allowing students to interact with the immune system at tasks. microscopic scales. Additionally, the nanobot’s outer hull can be dynami- game-like appeal is all heightened by challenges such as cally modi ed so that it can emulate cell surfaces and func- nite resources (for example, the number of times the pro- tionality. Fortunately, in addition to its quirky personality, tein dye jets can be red), damage incurred by the nano- the ship’s computer is equipped with an extensive database bot ship (from collisions, bacterial toxin, and phagocytic on human immunology, thus allowing it to offer guidance cells), and the amount of time allowed to complete each during the mission and to recommend a course of action task (Figure 7). to the pilot. The pilot must use the nanobot to identify Although the “ship’s computer” character functions and explore the site of infection, emulate the function of in an advisory capacity, offering verbal and textual sup- the damaged component of the immune system, and initi- port to guide student-pilots through the various mis- ate a successful immune response. The mission’s level of sions, the ultimate course remains under the student’s dif culty, the overall sense of urgency, and the video control. Help screens, which appear in the plane of the
  6. 6. 510 PRESENCE: VOLUME 9, NUMBER 6 Figure 7. An optional display, keeping the user updated about the nanobot status (left to right: hull structural integrity, protein dye jets remaining, current viewing scale, and time remaining for current task). Figure 8. An example of the help screens providing students with more-detailed information about each cell or protein they encounter. screen upon user command, contain information that is essential to understanding the tasks to be performed, including visual simulations, as well as information about cells and proteins encountered in the simulation (Figure 8). Full-motion video animation provides outlines both of the relevant immunology and of the speci c tasks from a third-person perspective, providing crucial sup- port for students in understanding their intended roles (Figure 9). Additionally, students can pause the simulation at any time to access database information and simulation con- trols through a simple pop-up menu system (Figure 10). In this manner, mission outlines, help screens, and Figure 9. Full-motion video animation complementing audio and animated mission brie ngs can be reviewed throughout textual instructions in introducing students to assigned tasks. the simulation. Added text and spoken support serves to augment the visual cues that are provided in mission brie ngs and help screens. For those students who con- tinue to have dif culty, a “hint” functionality is also in completing the mission. Overall, this multifaceted available, which provides explicit instructions for the help system has played a key role in making the simula- task at hand and becomes increasingly speci c as the tion accessible and relevant to a broad target audience. student continues to have dif culty and requests addi- It provides students with suf cient information to make tional help. It can be reviewed as needed for assistance the Virtual Explorer accessible to inexperienced users,
  7. 7. Dean et al. 511 ever, makes it impossible for one mission to touch upon the entire range of material and issues that are presented to students in an immunology course. Eventually, we hope that others will go beyond this work and add mis- sions that detail the involvement of other components of the immune system which can be explored through the individual viewpoints of those components. Ideally, such future missions (such as “killer T cell” or “neutro- phil” missions) would expand upon the helper T cell mission’s focus and include additional facets of immu- nology, such as the innate and cell-mediated immune responses. Mission outlines were scripted to maximize user inter- action and freedom, while still providing suf cient sup- port to guide even those users with no immunology background. Missions are divided into individual tasks, thus establishing a series of mini-goals which are pre- Figure 10. The familiar pop-up menu system, providing easy access to nanobot functions for novice users. sented to the user in a scavenger-hunt fashion. Preliminary user feedback revealed that clear mission outlines must not only be presented before each task (to provide clear instructions for that task) but must also be yet without sacri cing the challenge that retains the in- continually available for review during task execution. terest of more-advanced users. Although the mission outlines and help screens have The Virtual Explorer’s immunology module currently been made clear and simple, the virtual environments contains two interactive missions (Figure 11). Following have also been carefully constructed to show as much the brief introductory movie, the user is given a training relevant detail as possible. Although much of the simu- mission in which the user can explore and observe the lation’s visual detail is not referenced in the mission out- site of a bacterial infection and must collect a bacterial lines (Figure 14), we have found that providing visual specimen for analysis (Figures 12 and 13). accuracy is essential to avoid misleading users who have This rst mission introduces the user to the look and limited immunology backgrounds and to maintain the feel of the virtual environment and also allows familiar- simulation’s relevance for more-experienced users. A ization with the controls. Students are also challenged detailed Website provides additional scienti c informa- with phagocytic components of the innate immune sys- tion about each of the models in a glossary format. tem (such as neutrophils) and must master appropriate piloting skills to complete this mission. Upon complet- 4 Educational Content ing this mission, the student can decide to emulate one of several white blood cells (currently, only the helper T We chose immunology— one of the most complex cell is available) and he or she must use the nanobot to subjects studied by students of biology and medi- ful ll this character’s role in an immune response. In the cine—as the subject for the rst module because it pre- “Helper T Cell Mission,” we present a compromised sents unique visualization challenges. Its processes occur immune system that the student can “repair” by pilot- simultaneously in diverse locations of the body and of- ing a small nanobot ship in such a way so as to ful ll the ten on time and length scales that, although too small role of a helper T cell in a humoral immune response. to be directly perceptible, still vary over several orders of The inherent complexity of the immune system, how- magnitude. Consequently, the study of basic immu-
  8. 8. 512 PRESENCE: VOLUME 9, NUMBER 6 Figure 11. The immunology module, allowing the student to select from missions that emulate the roles of key players in the immune system, as well as an introductory training mission. Figure 12. Detail from the training mission. A shard of glass creates an opportunity for bacteria to enter the body. nology presents several common conceptual pitfalls, One common misunderstanding that interactive 3-D which we feel can be clari ed with properly imple- graphics are particularly well suited to clarify is the con- mented interactive virtual environments. The compart- cept of relative scale. Textbooks and other static teach- mentalization of instructional material that is required ing materials are inherently limited in their abilities to for the ef cient organization of a textbook makes it dif- simultaneously show microscopic details and the larger cult for students to gain an overall “road map” of the macroscopic systems within which they operate. immune response while still retaining a sense of the de- Consequently, textbooks and the like are often unable tails of each microenvironment. Thus, processes and to clearly represent the vast scale differences that are key to microenvironments are usually studied individually so immunology (Figure 15). For instance, immunology texts that each can be explored in detail, but the systemic re- often utilize schematic diagrams that depict cell surface lationship among these details often remains dif cult to proteins that are oversized and underpopulated by several conceptualize. orders of magnitude. Although these diagrams are useful
  9. 9. Dean et al. 513 Figure 13. Results summaries, concluding each task with an update on the current status of the immune system and providing an overview of the next task. Additionally, students can pause the simulation at any time to access database information and simulation controls through a simple pop-up menu system (Figure 10). Figure 14. Text outlines of each task, augmented by full-motion video and available to students for review throughout each mission. for conveying cell-protein identity and for suggesting the outnumber white blood cells by a ratio of almost 700 to mediation of cell-to-cell interactions through these pro- 1. Similarly, IgM and IgD surface receptors are typically teins, students are unable to gain a sense of how much several times as abundant as MHC Class I and Class II smaller surface proteins are than typical cells. Additionally, proteins on the surfaces of mature B cells. the implications in many diagrams that cell-to-cell interac- Interactive 3-D graphics can provide students with a tions can be mediated by single surface proteins are inher- visual model that helps them gain a basic understanding ently misleading (Figure 16). of the relative frequency of occurrence of different com- The concept of relative concentration provides addi- ponents. Certain components, however, are so rare that tional conceptual challenges that are similar to those we are required to exaggerate measured concentrations encountered in the exploration of relative scale. Stu- in our VR presentation simply to include even a few dents are often required to memorize lists of average specimens. For example, the relative concentrations of concentrations, but, without a visual representation of monocytes and granulocytes in the bloodstream are so these numbers, it is very dif cult to understand the im- low that they could appear to be virtually nonexistent plications of ratios, which also can vary by several orders among the many red blood cells. The representation of of magnitude (Figure 17). important constituents with vanishingly small concen- For example, in healthy individuals, red blood cells trations requires that we include a few specimens in the
  10. 10. 514 PRESENCE: VOLUME 9, NUMBER 6 Figure 15. Differing scales. Depicting scales that differ by several orders of magnitude is a task well suited to interactive computer graphics (left to right: blood vessel at 20003 magni cation, cell surface at 1,000,0003 magni cation) lymphocytes are very dif cult to distinguish visually, although such discrimination is often critical to the un- derstanding of an immune response. “Virtual dyes”—which simulate the binding of mono- clonal antibody dyes to the surface proteins of these cells—allow the students to quickly identify subsets of B and T cells in their native environment (Figure 18). Ad- ditionally, static teaching materials such as textbooks often fail to remind students of the dynamics of the sys- tems being studied. Cell surfaces, for example, are highly uid and dynamic in nature, and surface proteins are often free to migrate and diffuse across the surface. Figure 16. Surface proteins. These proteins allow for recognition A complete immune response involves a complex se- and signaling between cells and are often misrepresented by ries of steps and interactions (Figure 19). For example, immunology textbooks in both scale and population. the immune response to a bacterial infection might in- volve immediate in ammation at the site of infection and lymphocyte activation in some subset of the lymph simulation to remind the student of their essential roles. nodes or spleen, which is then followed by an antibody Although we would have preferred to have shown exact and complement response, and so on. One common concentrations, we were limited by available computa- misconception involves the locations of the immune tional power. response: the primary adaptive immune response is actu- Another area that is particularly enhanced by interac- ally mediated in the lymph node, rather than at the site tive 3-D graphics is the description of shape and struc- of infection (Figure 20). Because the processes in an ture. The characteristic shapes of cells, proteins, and immune response occur at several different locations in receptors have critical implications for binding, func- the body and involve important processes at several dif- tion, and identi cation. Structural differences between ferent length scales, the interactive visual simulation of MHC Class I and Class II, for example, are critical in these processes is a potentially unique aid to under- determining the nature of the immune response. Also, standing. We therefore believe that immunology’s visu-
  11. 11. Dean et al. 515 Figure 17. Virtual Explorer’s depiction of the bloodstream, helping to clarify issues of relative cell size and population. Figure 18. Protein dye jets, allowing students to visually identify different types of white blood cells based on their surface protein characteristics. alization challenges make it especially well suited to six independent video signals which are split by an demonstrate the bene ts of interactive 3-D graphics for MCO board to drive three contiguous displays in ste- education. reo, while still supporting well-populated virtual envi- ronments and fast frame rates. Rapid advancement in computer hardware leads us to believe that this level 5 Hardware Con guration of computer graphics performance will be available at the educational and consumer levels in the near fu- The Virtual Explorer is currently running on a ture. In parallel, we have developed a version of our four-processor Silicon Graphics Power Onyx. This system for the Silicon Graphics O2 workstation (a level of performance allows us to render in real time
  12. 12. 516 PRESENCE: VOLUME 9, NUMBER 6 Figure 19. Full-motion video animation, supplementing the interactive real-time graphics to demonstrate tasks to be performed as well as to give students a more comprehensive look at an immune response (left to right: the nanobot facilitates an immune response by emulating a Helper T cell, shown here docking with a B cell; a complement cascade helps to carry out the nal stage of an immune response). Figure 20. Lymph nodes. Although often misunderstood or unfamiliar to students, lymph nodes take center stage as the foci of adaptive immune responses. $5,000-$10,000 platform), as well as for various closed in a small soundproof theater (approximately 4 m other Silicon Graphics workstations. The exibility of by 6 m) and employs three 52 in. rear-projection, con- the software framework has allowed us to easily adapt sumer-grade television screens arranged at 120 deg. an- the Virtual Explorer for most Silicon Graphics IRIX- gles, creating a large window into the virtual environ- based hardware systems and their supported user in- ment. (See Figure 22.) put devices. (See Figure 21.) The graphics are driven by a four-processor Silicon The Virtual Explorer installation in our lab is en- Graphics Power Onyx, with RealityEngine2 graphics
  13. 13. Dean et al. 517 Figure 21. The Virtual Explorer software in our most expansive version, running on a four-processor Silicon Graphics Power Onyx, which controls the interactive 3-D graphics and coordinates the simulation. Six-channel video output from the Power Onyx drives three large-screen displays that form a wraparound viewport into the virtual world (Figure 22). Four-channel spatialized sound is generated by a Figure 22. Three large-screen, rear-projection monitors, creating a sound server running on an SGI Indigo2 Extreme, which communicates wraparound viewport into the virtual world. with the Onyx through TCP/IP. User input from a force-feedback joystick is processed through a Windows PC which also communicates with the Onyx via TCP/IP. (See Figure 27.) Another version runs on an individual single-processor SGI computer. proximately 10 ft. to 12 ft.) and the size of the viewing room. and two RM4 raster managers. The Onyx uses an MCO 6 User Interface board to split the video signal into six independent channels, and stereoscopic multiplexers combine these Depending upon the requirements of the physical channels into the three eld-sequential stereo channels installation, the Virtual Explorer system can accommo- that are displayed on the three large TV screens. De- date multiple user input devices. To be effective, the pending upon the available graphics hardware and the interface paradigm must be easily understandable, espe- level of processor performance, the software can also cially by nontechnical users. We believe that acceptable support several other combinations of stereo and mono user input devices must provide a familiar interface that video channels. (See Figure 23.) is relatively simple and easily recognized so that students Field-sequential stereo LCD shutter glasses (Figure can focus on interacting with the simulation and not on 24), which are synchronized to the video eld frequency mastering the controls (Figure 25). with two infrared transmitters, allow multiple students We are currently using a CH Products force-feedback to experience the virtual environment simultaneously. ightstick and throttle, which—in addition to providing Although we experimented with several stereo video an interface that is already found in many computer systems, we ultimately selected the VRex Mux-1 multi- video games—also provides the level of control neces- plexer system because of its support of the NTSC video sary to successfully navigate in a dynamic three-dimen- standard and its relatively low cost. Initially, we also sional environment (Figure 26). Force-feedback capabil- considered using a head-mounted display, but preferred ities allow properties of the environment (such as the greater versatility, comfort, and ability to handle viscosity) to be tactually communicated to the user, and large numbers of users that our current large-screen sys- enhance the user’s experience of immersion in the vir- tem provides. It presently accommodates approximately tual environment by re ecting ship collisions, speed, fteen observers, and this capacity is theoretically lim- and acceleration. Although joystick control is not very ited only by the range of the infrared transmitters (ap- processor intensive, the scarcity of joystick-type input
  14. 14. 518 PRESENCE: VOLUME 9, NUMBER 6 Figure 23. The Onyx generating six-channel video (RGBS), which is processed through RGBS to composite video encoders (CV-233). Stereoscopic multiplexers (VR-MUX 1) interlace left- and right-eye images for each of three screens, which are displayed on large, rear- projection displays. Infrared transmitters, which are connected to each of the outside monitors, synchronize stereo shutter glasses to the 60Hz video eld frequency. Figure 24. Field-sequential stereo shutter glasses, providing a full three-dimensional experience. Figure 25. Stereo shutter glasses and large screen displays combine with a familiar force-feedback joystick and throttle to provide an interactive and immersive learning experience. devices for SGI computers led us to choose this system, which is driven by a Windows NT PC communicating challenging issue for users with limited computer gaming with the Onyx via TCP/IP (Figure 27). Additionally, experience. Although we’ve found that a certain degree of Virtual Explorer also supports the Nintendo 64 control- dif culty in navigation is essential in maintaining excite- ler (connected directly to an SGI serial port with an ment for experienced users, it was also clear that inexperi- adapter box) and Microsoft’s Sidewinder ForceFeedback enced users must also be able to control the most basic Pro Joystick. functions of the craft simply to complete the assigned mis- Navigating the nanobots has proven to be the most sions. Mechanisms for obtaining additional help and in-
  15. 15. Dean et al. 519 Ultimately, a much more modest solution proved most successful in providing students with the option of a simpli ed user interface while still maintaining the same level of user control. The Virtual Explorer soft- ware contains a menu-based control system (similar to familiar PC GUIs) that can be used in place of the joy- stick buttons to access online help and to control nano- bot auxiliary functions. Users who are more comfortable with this interface can use it instead of the joystick but- tons, although the joystick is still used for navigation. Audio in Virtual Explorer is carefully designed to en- hance the user’s sense of immersion, as well as to allow students to better orient themselves within the virtual envi- Figure 26. ForceFX force-feedback joystick and throttle from CH ronment. Background music (based on the ProTracker Products provide a ightstick-style navigation interface. standard) aids students in distinguishing among different scales and environments. Students can also identify spatial relationships between the “ship” and the objects in the virtual environment by 3-D sound, and thereby bene t structions had to be made easily understandable and from a heightened sense of immersion and overall en- readily identi able. Creating a simple hardware-software hanced awareness of the dynamics of the environment. interface that was easy to learn and operate—yet that still Our audio system supports multiple sound le formats and provided access to the many controls required by the user multiple independent audio channels (based on hardware during the simulation—proved to be one of the more per- capabilities), which allow for both global (mono) and lo- sistent design challenges that we encountered. Many users calized sound effects. We have created our own spatialized nd it dif cult to remember the functions of many rela- audio algorithm which allows us to successfully mimic 3-D tively nondescript buttons (such as may exist when each audio, including simple panning, localization, and Doppler button controls a separate function). shift effects. The audio system can be controlled either by In an early attempt to deal with this problem, we the same computer as the main simulation or a secondary added a speaker-independent, speech-recognition fea- IRIX-based system that is connected to the graphics hard- ture to the software. This feature was supposed to as- ware via TCP/IP. Currently, the audio server is running sume the burden of controlling many nanobot auxiliary on a Silicon Graphics Indigo2, because our Onyx lacks functions. Based upon commercially available speech- sound output. Four independent audio channels provide recognition software, the software listens for verbal quadraphonic sound and drive four high- and midrange commands such as “computer, start engines,” and relays speaker systems, two directly driven bass speaker systems, the appropriate signal to the simulation. We quickly dis- and two powered long-excursion subwoofers for visceral covered several problems, however, which convinced us effects. to pursue other solutions. The main problem was the noisy environment within which Virtual Explorer typi- Software Design cally runs; the system we tested requires that the envi- 7 ronment be virtually free of ambient background noise. The Virtual Explorer software is written in C++, Virtual Explorer, however, generates substantial back- based upon the IRIS Performer toolkit. Although we ground audio (engine hum, blood- ow pulse, and the considered other development options such as like), which made the speech recognition substantially OpenGL, Open Inventor, VRML, and proprietary pack- less accurate and essentially incompatible. ages such as World ToolKit, we ultimately chose Per-
  16. 16. 520 PRESENCE: VOLUME 9, NUMBER 6 Figure 27. User input from a Windows PC and audio output to an SGI Indigo2 Extreme, linked to the Onyx by Ethernet and communicating with the Virtual Explorer software through TCP/IP. Figure 28. Four-channel audio, generated by an audio server running on a Silicon Graphics Indigo2 Extreme that communicates with the Onyx through TCP/IP over an Ethernet connection. Front and rear audio signals are processed through separate ampli ers (AVR-10), resulting in effective spatialized sound. Four satellite speakers, two passive subwoofers, and two powered subwoofers provide a wide dynamic range. former for several reasons: it allows us to freely redistrib- Virtual Explorer software framework, which is con- ute the generated code, it provides a high-level graphics structed on top of Performer. This should facilitate eas- API while still allowing direct access to GL and lower- ier and quicker development of additional missions, level rendering details, and it supports multiprocessing. modules, and educational worlds. We constructed the immunology module within the The basic graphics-rendering pipeline for Virtual Ex-
  17. 17. Dean et al. 521 plorer is subdivided into six threads of execution, based upon Performer’s multiprocessing framework: applica- tion, cull, draw, database, intersection (object collision detection), and user I/O. The six threads can run on one to four of the available processors, depending upon machine con guration. The application thread controls the high-level simulation, including mission progress, object motions, and simple dynamics calculation (such as the translational and angular momentum of the ship and other objects). The database, user I/O, and inter- section threads run asynchronously from the application thread to maintain a constant and acceptable frame rate. Figure 29. Electric Garden at SIGGRAPH ’97. Virtual Explorer contains three basic scene types: blood vessel (which is essentially linear), cell surface (es- sentially planar), and lymph node (volume-oriented) (See Figure 2.) Variables such as clip-plane depth, fog important to the study of immunology. Cells have been effect, global lighting characteristics, database paging modeled at the scale of 1:2,000 and proteins at parameters, and motion models for the ship can be ad- 1:1,000,000, which is consistent with the two viewing justed to differentiate between individual scenes. Scenes scales available to the user. We have created these mod- are created based on a speci ed combination of xed els and de ned their interactions based upon available geometry and procedural scene generation. microscopy images, x-ray crystallography, and NMR Each scene has speci c information about xed ge- structures, as well as other structural data. Each model ometry, such as the shell of the lymph node, the nano- typically contains ve geometric levels of detail and has bot extraction needle, or the shape and position of the an associated information le with the de ning charac- blood vessel. Additional scenery is created quasi-ran- teristics that are used by the simulation. Additionally, domly and cached when the application is launched, each model is accompanied by a help screen containing based on variables such as cell population and average information of interest to the student (Figure 8). Tech- concentrations. This cached scenery can be dynamically niques such as object sequences (which allow for mor- rearranged during the simulation. Earlier versions of the phing models) and dynamic texture shifting (which al- software included actual dynamic generation of scenery lows for protein “dyeing”) show biological during the simulation, but that technique proved to be characteristics and improve the interaction between the too processor intensive to maintain a suf cient level of user and the individual objects in the simulation. graphics performance. A voxel-based paging scheme dynamically recon gures and pages cached geometry as needed during the simulation, allowing large scenes 8 Conclusions with large amounts of geometry to be simulated with- out sacri cing graphics performance and frame rate. Al- The response from the educational, scienti c, and though the overall complexity varies signi cantly be- computer graphics communities has been very positive. tween scenes, most scenes contain between 3,000 and More than 7,000 people have already participated in 8,000 textured polygons per frame. The RealityEngine2 demonstrations (Figure 29). We are distributing the allows us to maintain steady six-channel video with a complete source code and installer scripts for a variety of frame rate of approximately 20 Hz. Silicon Graphics computers, with illustrated instruction The simulation contains biologically accurate scale manuals included, as a CD-ROM. Several science and models of over thirty different cells and proteins that are technology museums have licensed Virtual Explorer for
  18. 18. 522 PRESENCE: VOLUME 9, NUMBER 6 Figure 30. Software:Theater at HeinzNixdorf Museumsforum in Paderborn, Germany. Figure 31. Life Tech Theater at the Tech Museum of Innovation in San Jose, California. permanent exhibits, and it has already been installed in interface), Institute for Research on Learning (Menlo Park, the Heinz Nixdorf MuseumsForum (Figure 30) in Pad- CA) and Stanford University (Stanford, CA); Teresa Larsen erborn, Germany (for which we wrote a German version (adviser for biology and computer animation), Scripps Re- of the text and audio track) and the Tech Museum of search Institute (La Jolla, CA); Barbara Sawrey (adviser for Innovation (Figure 31) in San Jose, California. Other multimedia education and visualization), Department of installations are in the planning stages. Future directions Chemistry and Biochemistry, UCSD (La Jolla, CA); Gabriele Wienhausen (adviser for multimedia education and visualiza- for study may include characterization of the educa- tion), Department of Biology, University of California, San tional bene ts of interactive three-dimensional virtual Diego (La Jolla, CA); and Michael Zyda (adviser for interac- environments, like Virtual Explorer, over interactive, yet tive 3-D graphics), Department of Computer Science, Naval non-immersive, two-dimensional systems. Postgraduate School (Monterey, CA). Further information on the system and how to obtain a video demonstration of Virtual Explorer (as well as the CD-ROMs of the source code and instruction manuals) References can be obtained from the Virtual Explorer Website at Bogen, J. E. (1972). A giant walk-through brain. Bulletin of the Los Angeles Neurological Society, 37(3). Dean, K.L., Asay-Davis, X. S., Finn, E, M., Friesner, J. A., Acknowledgments Naylor, B. J., Wustner, S. R., Fisher, S. S., & Wilson, K. R. (1998). Virtual Explorer: Creating interactive 3D virtual We would like to thank the following individuals for their in- environments for education. In M. T. Bolas, S. S. Fisher, valuable contributions to the Virtual Explorer project: April and J. O. Merritt (Eds.), Stereoscopic Displays and Virtual Apperson (adviser for immunology), School of Medicine, Uni- Reality Systems V, Proceedings of SPIE—the International versity of California, San Diego (La Jolla, CA); Jon Chris- Society for Optical Engineering, 3295 (p. 429), Bellingham, tensen (former project director), Painted Word, Inc. (Cam- WA. bridge, MA); Glen D. Fraser (adviser for interactive 3-D Dean, K., Asay-Davis, X., Finn, E., Friesner, J., Naylor, B., graphics), Montreal, Quebec, Canada; David Goodsell (advis- Wustner, S., Fisher, S., & Wilson, K. (1997). Electric gar- er for cellular and molecular visualization), Scripps Research den: The Virtual Explorer. Computer Graphics, 31(4), 16- Institute (La Jolla, CA); Mizuko Ito (adviser for educational 17, 81.
  19. 19. Dean et al. 523 Dean, K. L., Finn, E. M., Friesner, J. A., Naylor, B. J., Wust- Physical and Informational Technologies: Options for a New ner, S. R., Wilson, K. R., & Fisher, S. S. (1997). Electric Era in Healthcare (pp. 134-140), IOS Press. garden: Virtual Explorer. In R. Hopkins (Ed.), Visual Pro- Kuby, J. (1997). Immunology (3rd ed.). New York: W. H. ceedings: The Art and Interdisciplinary Programs of Freeman and Company. SIGGRAPH 97 (p. 110), New York: Association for Com- Rose, H., & Billinghurst, M. (1996). Zengo Sayu: An immer- puting Machinery. sive educational environment for learning Japanese (Techni- Dede, C., Salzman, M. C., & Loften, B. (1996). Science cal report). Seattle: University of Washington, Human space: Virtual realities for learning complex and abstract Interface Laboratory of the Washington Technology scienti c concepts. In Proc. IEEE Virtual Reality Annual Center. International Symposium (pp. 246-253). Salzman, M. C., Dede, C., Loftin, R. B., & Chen, J. (1999). Hoffman, H. M., & Murray, M. (1999). Anatomic Visual- A model for understanding how virtual reality aids complex izeR: Realizing the vision of a VR-based learning environ- conceptual learning. Presence: Teleoperators and Virtual En- ment. In Medicine Meets Virtual Reality, The Convergence of vironments, 8(3), 293-316.