Your SlideShare is downloading. ×
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Multiple Screens
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Multiple Screens

661

Published on

This report describes an approach of using a “Master-Slave” network communication mechanism by using Java Bindings for OpenGL API (Jogl), based on a real implementation, to achieve the effect of …

This report describes an approach of using a “Master-Slave” network communication mechanism by using Java Bindings for OpenGL API (Jogl), based on a real implementation, to achieve the effect of displaying 3D object within multiple screens.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
661
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Computer Graphics Lab Multiple Screens University of Trento Mu Qiao (133734) Supervisor: Dr. Giuseppe Conti Prof. Raffaele de Amicis -1-
  • 2. Table of Contents 1. Introduction..................................................................................................................... 3 2. Related Work .................................................................................................................. 6 2.1 Studierstube............................................................................................................... 6 2.2 VRJuggler ................................................................................................................. 7 2.3 Java3D....................................................................................................................... 9 3. Methods and Implementation of Multiple Screens....................................................... 11 3.1 Development Tools, libraries and Platform ............................................................ 11 3.2 Rotating Cube Example .......................................................................................... 13 3.2.1 Class Diagram of Original Rotating Cube ....................................................... 14 3.2.2 “Master-Slave” Rotating Cube Extension ....................................................... 14 3.2.3 Network Communication through Java Socket ............................................... 17 3.2.4 Multiple Screens Effect Realization by Adjusting Viewport .......................... 21 3.2.5 Synchronization of “Slave” Applications ........................................................ 24 3.3 World Wind Java SDK ........................................................................................... 25 3.3.1 Extension of Original World Wind Java SDK................................................. 25 3.3.2 Network Communication through Java Sockets.............................................. 28 3.3.3 Multiple Screens Effect Realization by Adjusting Viewport .......................... 30 4. Performance Evaluation and Future Work ................................................................... 31 4. Summary ....................................................................................................................... 33 References......................................................................................................................... 34 -2-
  • 3. 1. Introduction In many circumstances, wider vision is needed to fulfill special requirements like reality authentication of scenes (see Figure 1), commodity displayed in an exhibition, panorama of a soccer match, exciting and shocking effect of computer games or simulation etc (see Figure 2). A simple and costly way is to purchase a wider screen several times larger than a normal one. An alternative is to utilize and combine multiple screens to achieve the same effect. The advantage of multiple screens is the competitive cost, easy transportation, free rearrangement over their relative positions. This report describes an approach of using a “Master-Slave” network communication mechanism by using Java Bindings for OpenGL API (Jogl), based on a real implementation, to achieve the effect of displaying 3D object within multiple screens. Figure 1: Reality authentication of beautiful scenes (from the website of 9x Media) -3-
  • 4. Figure 2: Computer Simulation of Aircraft (from bp2.blogger.com) As part of the group project which aims to build a control station featuring in multiple screens, advanced control system, sensors support and collaborative features, one of the features described in this report is its multiple screens effect. By building a Java project composed of both “Master” and “Slave” applications communicating through Java sockets, the project is then planted over multiple PCs. One PC is running “Master” application which controls several “Slave” applications running on other PCs. Multiple screens effect is realized by synchronizing steps of “Slave” applications controlled by one “Master” application. The work begins with a relative simple application on a rotating 3D cube. The cube can be controlled by both mouse and keyboard. The implementation results in a wider scene composed of two screens through whose border the cube is able to “traverse”. Besides, the size of the cube is magnified on two screens compared with the original one displayed -4-
  • 5. only in one screen. The steps of the rotating and moving of the cube is simultaneous over the two “Slave” applications. After the rotating 3D cube application, further implementation based on World Wind Java SDK from NASA is conducted in order to achieve the similar result. With the much more complicated project structure of World Wind compared with the 3D cube, many classes within the original project are extended to achieve “Master-Slave” mechanism with good extensibility and transplantability. The implementation of the multiple screens effect based on World Wind Java SDK can be divided into steps below: 1. Identifying and extending classes which can be respectively inherited for different use of “Master” and “Slave” applications as well as some of their ancestors and offspring when needed. For example, classes of the drawing the earth, the controlling of the earth and the registrations of these features are realized differently from “Master” to “Slave” applications. 2. Realizing the “Master-Slave” communication through Java socket. “Master” earth controls the orientation, rotation and size of the “Slave” earths by establishing server-client communication. “Slaves”, treated as several different servers, are listening to a “Master” which is treated as one single client. The “Master” client controls several “Slave” servers by separately establishing communication to each of the “Slaves” but sending the same command to all of them in order to realize synchronization. 3. The sizes and positions of “Slave” earths need to be consistent to form a wider view of a single earth within multiple screens. The screen size, position, orientation, resolution as well as the viewport size of the scene display together decides the relative sizes and positions among “Slave” earths. This lab is focusing on the realization of multiple screens. This goal requires the understanding of the graphic animation, network communication based on Jogl implementation. The rest of the report is organized as below. Section 2 briefly introduces other VR applications like Studierstube, VR Juggler and Java3D. Section 3 is the core of the report beginning with the working environment of our application, and -5-
  • 6. mainly focusing on two multiple screens applications, one rotating cube example and one based on World Wind application. Detailed explanation of the implementation procedure is available through this part. In the end, this report ends with a summary as Section 4. 2. Related Work 2.1 Studierstube The Studierstube augmented reality project, developed in Virtual Reality Group in Technical University Vienna leaded by Prof. Dr. Dieter Schmalstieg, is one of the first collaborative augmented reality systems in the world. Its innovative aspects are represented by augmented reality, scientific visualization, multiple user support and framework for easy extension and use [CBE]. Studierstube utilizes C++ to build on top of the Open Inventor (OIV) toolkit which provides a higher layer of programming for OpenGL. OIV provides easy access to 3D graphics and the core of it is an object-oriented scene graph scoring both geometric information and active interaction objects [SFSEGP02]. OIV is extended when necessary and form a toolkit of the OIV class hierarchy. In the virtual environment, locales correspond to coordinate systems and coincide with physical places [SFSEGP02]. Environment Management for Multi-User Information Environments (EMMIE) [BHFMB99] developed by people from Columbia University utilize one physical locale by all displays. This approach is simple to understand and easy to implement. Studierstube uses a separate locale for each display in order to enable all users having convenient access to all of the data items, especially with the increasing data items [SFSEGP02]. Studierstube allows locales to be shared over displays. This means though multiple displays can have independent points of view, they show images of an identical scene graph (Figure 3). Usually, stationary displays can usually be embedded in a single locale. -6-
  • 7. Figure 3: 3D Drag & Drop of An Application Object across Display Boundaries in Studierstube Environment 2.2 VRJuggler The VR Juggler project has been developed at the Iowa Center for Emerging Manufacturing Technology (ICEMT) since 1997. VR Juggler is designed to “juggle” any combination of devices, graphics APIs, and platforms that users want to use. VR Juggler enables developers to create VR applications without knowing low-level details of VR [JBBC]. VR Juggler is an application framework and set of C++ classes for writing virtual reality applications. Developers are allowed to get direct access to various graphic APIs. The hardware that VR Juggler supports includes CAVETM and C2, projection tables, simple workstation monitor displays and traditional head-mounted displays [JBBC]. VR Juggler currently runs on the Silicon Graphics, Inc. (SGI) platform and is being ported to Hewlett Packard (HP) and Windows NT. For the multiple display effect supported hardware, CAVETM, whose full name is Cave Automatic Virtual Environment, is an immersive virtual reality environment where projectors are directed to multiple walls (see Figure 4). The lifelike visual display is created by projectors positioned outside the CAVE and controlled by physical movements from a user inside the CAVE. One of more computers, often SGI workstation, drives the projectors. Techniques for rendering the scene include OpenGL, OpenSG and OpenSceneGraph which are popular nowadays [Wiki-CAVE]. VR Juggler utilizes the power and flexibility of C++ and is thus an object-oriented system. System details are handled by a set of Managers. For example, Draw Manager deals with Graphics and Windowing API and communicates with Display Manager. All -7-
  • 8. the Managers are taken care of by a kernel on which user applications are based (see Figure 5). Figure 4: Virtual Reality - Multiple Walls of CAVE Figure 5: Structure of VR Juggler -8-
  • 9. 2.3 Java3D Java 3D is a scene graph-based 3D API for Java platform. It runs on top of either OpenGL or Direct3D. Java 3D is not only a wrapper around graphics APIs, but an interface that encapsulates the graphics programming using a real, object-oriented concept [Wiki-Java3D]. 1) Camera-Based View Model Java 3D model is different from camera-based view model in the sense of binding the scene model and view model together. The Java 3D view model incorporates head tracking directly and if not, it acts very much like the camera-based view model. Camera-based view model is controlled by three parameters: location, orientation and perspective. The lookAt method within it creates a viewing matrix whose inverse can be used to control the ViewPlatform object within the scene graph. 2) Structure of Java 3D View Model Figure 6: View Object, Its Component Objects and Their Interconnection Java 3D view model (see Figure 6) defines a Locale Object within a VirtualUniverse Object which specifies the reference point within the universe. There are two branches below Locale Object. The left branch groups define how scenes are rendered and -9-
  • 10. possibly navigated (ViewModel) and the right ones define the content of the scene (ObjectModel). For the ViewModel, the TransformGroup Node under the right BranchGroup Node specifies the position, orientation and scale of the ViewPlatform. The transformed ViewPlatform defines the view of the user in the virtual universe. In other words, ViewPlatform is the position of the viewer, it can be obtained and moved in space by a TransformGroup. Besides, ViewPlatform is referenced by a View Object which specifies all the parameters needed to render the scene from the point of view of the ViewPlatform. 3) Physical Body and Physical Environment Figure 7: Object and View Platform Transformations Physical Body and Physical Environment (see Figure 7) under the View Object encapsulates all the parameters associated with head, eyes and corresponding trackers. If no trackers are available, they are ignored and ViewPlatform becomes the position of the camera. 4) Multi-screens - 10 -
  • 11. Figure 8: A Portion of a Scene Graph Containing Three Screen3D Objects Same View Object can be displayed in several screens from different perspectives. For multi-screen environment (see Figure 8 and Figure 9), Java 3D must know how the Screen3D are placed with respect to each other, the tracking device, and the physical portion of the coexistence coordinate system. Figure 9: A Three-Screen Display Environment 3. Methods and Implementation of Multiple Screens 3.1 Development Tools, libraries and Platform Development tools: • Java programming language • Eclipse IDE(version 3.3.2) - 11 -
  • 12. Libraries: • Jogl (version JSR-231) • GlueGen Platform: • World Wind Java SDK (version 0.4.1) Jogl is a wrapper library that allows OpenGL to be used in Java programming language. Jogl [Wiki-Jogl]. Jogl is designed to provide harware-supported 3F graphics to applications written in Java and full access to the APIs in the OpenGL 2.0 specification as well as nearly all vendor extensions, and integrates with the AWT and Swing widget sets [OpenGL]. GlueGen is a Java tool which automatically generates the Java and JNI code necessary to call C libraries from Java code and it is originally developed for Jogl [Wiki-Gluegen]. World Wind Java SDK, developed by NASA, enables developers to embed World Wind technology in their own applications [NWW]. Besides, World Wind Java SDK is developed based on Jogl which implies the necessity of inclusion of Jogl library in our application (see Figure 10). Figure 10: Application structure of our application based on World Wind Java which is based on Jogl and JDK - 12 -
  • 13. 3.2 Rotating Cube Example The implementation of a rotating cube is as a preparation for our goal application based on World Wind Java SDK. The textured cube keeps rotating within a black background and the view of the observer can be controlled by keyboard with “Up”, “Down”, “Left”, “Right”, “PageUp”, “PageDown”, “A” and “Z”, respectively realizing the effect of observer walking forward, backward, left and right, as well as looking up, down and simply up, down. The desired aim of this implementation is to enable as observer through controlling a “Master” cube by keyboard controls two “Slave” cubes in two screens. Though there are two applications of “Slave” cubes existing, the final effect is like only one larger cube being controlled within two screens (see Figure 11). Figure 11: Two Screens Effect of A Rotating Cube - 13 -
  • 14. 3.2.1 Class Diagram of Original Rotating Cube Class diagram of original rotating cube is drawn to assist analyzing and further implementing “Master-Slave” mechanism upon (see Figure 12). The rotating cube is drawn within GLRenderer, which implements GLEventListener, is added as GLCanvas’ listener. Besides, the instance of glRenderer is passed as a parameter to an instance of KeyNavigator, extending KeyAdaptor, to implement the control of keyboards over the rotating cube. The control is realized by modifying the parameter in an instance of MovementStatus, an attribute of GLRenderer, while corresponding buttons are pressed. The modification of these parameters will further change the view of a virtual camera which represents the view point of the observer. 3.2.2 “Master-Slave” Rotating Cube Extension In order to achieve the goal of controlling several “Slave” applications by one “Master” application, the first step should be of extending original classes to “Master” and “Slave” separately (see Figure 13). The two classes that may need to be extended are GLRenderer and ApplicationStarter because “Master” and “Slave” have different framework, drawing methods, what is most important is their relationship as server and client (this will be specified in 3.2.3). For GLRender, we change it to BasicGLRenderer which overrides the four functions “init”, “display”, “displayChanged”, “reshape” because of its implementation of the interface - 14 -
  • 15. Figure 12: Class Diagram of Original Rotating Cube - 15 -
  • 16. Figure 13: Class Diagram of Multiple Screens Application of Rotating Cube - 16 -
  • 17. GLEventListener. Other functions of BasicGLRenderer are all abstract ones which are implemented diffrerently by its offspring. The extensions of BasicGLRenderer are MasterGLRenderer and SlaveGLRendere. They override the four functions “init”, “display”, “displayChanged”, “reshape” again from BasicGLRenderer which implements GLEventListener. The abstract functions in BasicGLRenderer are implemented differently based on the requirement of “Master” and “Slave” applications. For ApplicationStarter, extension of it is unnecessary because it only has one constructor. Creation of class MasterApplicationStarter and SlaveApplicationStarter is proceeded as needed. For example, MasterApplicationStarter includes MasterGLRenderer as an attribute while SlaveApplicationStarter has SlaveGLRenderer. Other distinctions mainly occur because of the network communication introduced in 3.2.3. 3.2.3 Network Communication through Java Socket “Master” cube controls several “Slave” cube through network communication. For example, button “Left” is pressed which moves the “Master” cube moves right (button direction indicates the view position of the observer, “Left” means the observer moves left, corresponding, object, the cube, moves right). The network communication is realized in our application by sockets. Two classes of Socket and ServerSocket are used for establishing the communication. Socket class implements client socket. A socket is an endpoint for communication between two machines. ServerSocket class implements server sockets. A server socket waits for requests to come in over the network. It performs some operation based on that request, and then possibly returns a result to the requester. We treat “Master” as client who has opened several sockets linked to different servers, performed by “Slave” applications. Each “Slave” is seen as a server which opens a serverSocket and keeps listening to specific port of the local address. Once client - “Master” connects to the corresponding server, at the server side, a socket acting as a client socket accepts what has been received from the client and then retrieves the objects - 17 -
  • 18. sent from the clients (see Figure 14). Figure 14: Socket Communication between Server - “Slave” and Client - “Master” Corresponding codes are illustrated and commented below: a) Server - “Slave” // open a new ServerSocket listening to a specified port ServerSocket arrayServer = new ServerSocket(portNumReceived); System.out.println("Server listening on port ." + portNumReceived); System.out.println("Waiting for connections."); // accept() blocks the caller until a correct connection is present // an incoming connection is acknowledged and associated with an // immediately created socket. // The original socket is returned to the listening state Socket client = arrayServer.accept(); - 18 -
  • 19. System.out.println("Accepted a connection from: " + client.getInetAddress().getHostName()); // new ObjectInputStream() recovers input stream from the client ObjectInputStream ois = new ObjectInputStream(client.getInputStream()); // object1 reads serialized objects sent from the client Object object1 = ois.readObject(); b) Client - “Master” // specify the amount of servers the client is trying to establish // communication with int arraySize = 3; Socket[] socket = new Socket[arraySize]; // try to build connection to the specified InetAddress and port try { socket[0] = new Socket("localhost", 4000); socket[1] = new Socket("192.168.253.20", 4001); socket[2] = new Socket("192.168.253.21", 4001); } catch(Exception e) { System.out.println(e.getMessage); } Thus, by using sockets, network communication is established between server - “Slave” and client - “Master”. Next step is to specify serialized objects that needed to be transferred in order to realize instant control from “Master” to “Slave”. Objects that are transferred through sockets need to implement Java Interface Serializable. Three serialized objects are specified as ObjectEvent, ObjectStatusEvent, POVEvent which includes information, respectively, regarding cube 3D positions, status (rotating or not) and point of view from the observer. For the purpose of convenience and clearness, three classes called SendObjectEvent, SendObjectStatusEvent and SendPOVEvent are built to respectively send the three serialized objects mentioned above. Simple structures can be seen as: SendObjectEvent sends ObjectEvent SendObjectStatusEvent sends ObjectStatusEvent SendPOVEvent sends POVEvent - 19 -
  • 20. SendObjectEvent and SendPOVEvent happen during the display() function within MasterGLRenderer class. This means that during each display, positions and point of views are sent from “Master” to “Slave” to let the latter keep steps as precise as possible with the former. SendPOVEvent also happens in the KeyNavigator class while keys are pressed or released to transfer to current point of view of the observer. SendObjectStatusEvent is the same case as SendPOVEvent here to command the status of the “Slave” applications whether to start or stop rotating. The positions and point of view of the rotating cube are set during the corresponding drawing period in the MasterGLRenderer. It means the current values are updated in real time. In the server - “Slave” side, after receiving corresponding objects (ObjectEvent, ObjectStatusEvent, POVEvent) from the client - “Master” side, the first thing is to distinguish which object it has received. Operator instanceof performs as the judger here. After the identity of the received object is known, the corresponding set function can be utilized in order to set the instant status of the SlaveGLRenderer. For example, if object of POVEvent is received and after being recognized, the corresponding instance of POVEvent within SlaveGLRenderer is set. Therefore, during moveCamera() function within SlaveGLRenderer, instant point of view transferred from “Master” is immediately set. // within SlaveApplicationStarter, receive and specify the received // object, pass it to the glRenderer if (object1 instanceof POVEvent) { POVEvent povEvent = (POVEvent) object1; glRenderer.setPOVPosition(povEvent); } // within SlaveGLRenderer, private member povEvent are set as what // has been received public void setPOVEvent(POVEvent povEvent) { this.povEvent= povEvent; } // command “Slave” to change its point of view according to the // parameters bound within povEvent private void moveCamera (GL gl) { gl.glRotated(povEvent.getLookupdown() , 1.0f, 0, 0); gl.glRotated(povEvent.getSceneroty() , 0, 1.0f, 0); - 20 -
  • 21. gl.glTranslatef(povEvent.getXPos(),povEvent.getYPos(),povEvent.ge tZPos()); } Overall, objects containing parameters regarding the rotating cube are transferred through sockets from the client - “Master” to the server - “Slave”. Parameters within serialized objects are set immediately after they are generated. Then the objects are transferred from “Master” to “Slave” and being recognized and passed to the renderer class within which the parameters of the objects are released to their corresponding positions and promote the movement of the “Slave”. 3.2.4 Multiple Screens Effect Realization by Adjusting Viewport After establishing network communication between “Master” and “Slave” as well as the transmission and set of the corresponding instant command issued from “Master” and implemented by several “Slaves”, simultaneous and identical movement of “Slaves” are realized from multiple screens (see Figure 15). Figure 15: Simultaneous and Identical Movement of Two “Slave” Applications - 21 -
  • 22. In order to simulate the effect of only one cube within multiple screens, adjustment over the “Slave” cubes are needed. Now, two problems regarding the adjustment are: • Different sizes of cubes from different screens • Different positions of cubes from different screens a) The first problem “different sizes of cubes from different screens” is caused by the differences of the resolution and sizes of screens. For resolution problem, if different screens have the options of the same resolution, we can just easily set their resolution to be the same. We implement our application in this way. If there is no uniform resolution that every screen conforms to, ratios among different resolutions need to be calculated and used as a factor to multiply the viewport parameters discussed below. For screen size problem, a method to equalize cube sizes from different screens is to multiply the viewport width and height with the corresponding ratios among screens. In OpenGL, glViewport specifies the affine transformation of x and y from normalized device coordinates to window coordinates. glViewport function is written as: void glViewport ( int x, int y, int width, int height ); x, y specify the lower left corner of the viewport rectangle in pixels while width, height specify the width and height of the viewport. Because the viewport size (width, height) will determine the size of the cube displayed, if we multiply one “Slave” cube’s viewport width and height with the ratio that expands it to the same size of a chosen standard “Slave” screen, then the size of all “Slave” cubes become the same. For example, we have “Slave” applications respectively displayed in screen A( width( A), height ( A)) , B ( width( B ), height ( B)) , C ( width(C ), height (C)) with the same resolution (we do not consider the case of different resolutions). If we choose screen A as the standard screen, width( A) then ratios of B to A are widthRatio( BtoA) = and width( B) height ( A) heightRatio( BtoA) = , the glViewport specified in “Slave” application B height ( B) becomes: - 22 -
  • 23. void glViewport (x, y, width( B ) × widthRatio( BtoA) , height ( B) × heightRatio( BtoA) ); (see Figure 16). Similarly, the viewport C has to be modified the same way. Viewport of B Screen border of B A B width( A) width( B) width( B ) × widthRatio( BtoA) Figure 16: Equalize cube size by adjusting glViewport b) The second problem is to match the position of the already equalized cubes to simulate the effect of only one cube is displayed and moved within multiple screens. The adjustment of the position of cubes is also implemented within the glViewport. Due to the consideration of conveniences, a class SliderPanel, extending JFrame and containing two instance JSliders (to respectively adjust width and height value) is built to manage the position of cubes, or more precisely, the viewport of the cubes. The slider.getValue() is added to the position of the lower left corner of the viewport to move it by manually adjusting the slider bar (see Figure 16). c) Another problem that has not implicitly addressed is that the cube should be independent of the change of the frame (the movement, resizing of the frame does not influence the size and the position of the cube). This function’s realization would benefit us from moving or resizing the frame at our will without damaging the multiple screens effect. We achieve this by using the instantly origin of the frame with respect to the screen and the origin of the viewport. For example, while the frame is moving down, the distance decreased is added to the y value of the origin of the viewport, than the cube is - 23 -
  • 24. moving up with the same distance. This method will result the phenomenon that as if the cube stays stable as before (see Figure 17). Cube stays stable Frame goes down Viewport origin Viewport origin moves up Figure 17: Cube Size and Position are independent of Frame Adjustment From the three steps described above: I. “Master-Slave” Rotating Cube Extension II. Network Communication Through Java Socket III. Multiple Screens Effect Realization by Adjusting Viewport The multiple screens effect based on network communication is achieved (see Figure 11). By utilizing the techniques used in this cube example and adequate extensions, we further implement the similar task based on World Wind Java SDK developed by NASA. 3.2.5 Synchronization of “Slave” Applications Because of the differences of network transmission speed, different “Slaves” may act unsynchronized after some time or under the condition of temporal network interruption. If one “Slave” undergoes normal rotating while the other is experiencing network interruption, then the phase of the former surpasses the latter. An idea to solve this unsynchronization problem is to send serialized object (that is, sendObjectEvent()) for every round of display() function. This is because the display() function keeps - 24 -
  • 25. circulating and displaying the rotating cube. If sendObjectEvent() is within the display() function, then for every round of display(), the instant positions of the rotating cube is sent through socket to different “Slaves”. While the display() goes to the next round, if at least one of the former serialized object has not been consumed in the server - “Slave” side, then the serialized object sent this time need to wait until the former serialized object is accepted by all the “Slave” applications. In this way, if the delay of network slows down one “Slave” application, all the other “Slave” applications are also dragged behind but all of them are still synchronized. Adding sendPOVEvent() within display() function also improve the synchronization effect. 3.3 World Wind Java SDK World Wind is a free open source virtual globe developed by NASA and open source community for use on personal computers [Wiki-NWW]. World Wind lets you zoom from satellite altitude into any place on Earth [NAASA]. A preview of the World Wind Java SDK was released in 2007 and the latest version 0.5.0 was released on May 2nd, 2008. The version that we are using is 0.4.1. With this SDK, we can embed World Wind technology in our own applications – multiple screens. 3.3.1 Extension of Original World Wind Java SDK Conceptually similar to what has been done for the rotating cube example, World Wind is also extended respectively to client - “Master” and server - “Slave”. We, in most of the cases, extend classes once necessary and put all the extended and newly generated classes into one folder for the consideration of clearness and software reuse. We try not to change the original code. But under special conditions, some slight changes of four classes are made in our applications. They are: • InputHandler: (functions added, for simplicity of coding) o void setSocket(Socket[] socket); o void setPOVEvent(POVEvent povEvent); • AWTInputHandler: (functions added, because of its implementation of interface InputHandler) o public void setSocket(Socket[] socket){} - 25 -
  • 26. o public void setPOVEvent(POVEvent povEvent){} • LatLon, Angle, Quaternion: (parameters need to be transferred through network, for simplicity of coding) o implements Serializable • Configuration: (codes added, for the creation of new classes by class name) o defaults.setProperty(MyAVKey.MASTER_WORLD_WINDOW_CLASS_NAME, MasterWorldWindowGLAutoDrawable.class.getName()); o defaults.setProperty(MyAVKey.SLAVE_WORLD_WINDOW_CLASS_NAME, SlaveWorldWindowGLAutoDrawable.class.getName()); o defaults.setProperty(MyAVKey.MASTER_INPUT_HANDLER_CLASS_NAM E, MasterAWTInputHandler.class.getName()); o defaults.setProperty(MyAVKey.SLAVE_INPUT_HANDLER_CLASS_NAME, SlaveAWTInputHandler.class.getName()); The above four illustrated are the original classes that has been modified within the whole World Wind source code. Other than these four classes, the other original classes remain the same. The extension of the original classes and their main differences are explained below: a) At the very top level, ApplicationStarter is replaced by MasterSimpleStarter and SlaveSimpleStarter, both of which extends Thread. Then the SlaveSimpleStarter can draw the earth at one thread, and listen and receive serialized objects at another hand. b) The original WorldWindowGLCanvas is replaced by BasicWorldWindowGLCanvas which is further extended respectively by MasterWorldWindowGLCanvas and SlaveWorldWindowGLCanvas. BasicWorldWindowGLCanvas is seens as an abstract class. The main differences between MasterWorldWindowGLCanvas and SlaveWorldWindowGLCanvas are that the former uses the function setSocket (Socket[] socket) to further pass client sockets to the associated class which has the controlling parameters, and the latter uses the function setPOVEvent(POVEvent povEvent) to pass the received povEvent from “Master” to the associated class which needs the controlling parameters to act correspondingly as commanded. Both of them has the - 26 -
  • 27. setFrame(MyFrame frame) function to realized the effect of remaining stable regardless of the moving and resizing of the frame (see Figure 17). c) WorldWindowGLAutoDrawable, originally responsible for displaying the earth, is extended to MasterWorldWindowGLAutoDrawable and SlaveWorldWindowGLAutoDrawable. They perform differently mainly in display(). The latter has the function setPOVEvent(POVEvent povEvent) to pass the transferred object to the associated class. Besides, both WorldWindowGLAutoDrawable and SlaveWorldWindowGLAutoDrawable extends MyWorldWindowGLDrawable which extends WorldWindowGLDrawable. d) MasterAWTInputHandler implements InputHandler, KeyListener, MouseListener, MouseMotionListener, MouseWheelListener, FocusListener to realize the mouse and key event through its private member viewInputBroker. SlaveAWTInputHandler does not need to implement these events since it only carries out the command from the “Master”. e) MasterOrbitViewInputBroker implements KeyListener, MouseListener, MouseMotionListener, MouseWheelListener, FocusListener. This class is associated with MasterAWTInputHandler and response to its corresponding call. The functions in MasterOrbitViewInputBroker set the parameters related to the control of either keyboard or mouse. From here, these parameters can be wrapped to the serialized object povEvent and send through sockets to “Slaves”. SlaveOrbitViewInputBroker is not necessary to implement the interface that MasterOrbitViewInputBroker does. f) Other extensions include MyAVKey extending AVKey which can be seen as an index of the necessary class names of whose classes are called by name. MyCGFrame simply extends JFrame. g) Added classes include SendPOVEvent and POVEvent which are responsible for sending controlling parameters through sockets. SliderPanel is used to manually set the position of the viewport’s lower left corner in order to move the position of the earth. DummySlave series are used to start SlaveSimpleStarter by passing the name and port number into it. - 27 -
  • 28. h) Classes that are simply moved from the original source code are KeyPollTimer and OrbitViewInputstateIterator without any change. The reason is that they are both set as private classes (in the original source code) and are used during implementation. Detailed information regarding the hierarchy of “Master-Slave” realization can be referred to the class diagram (see Figure 18). 3.3.2 Network Communication through Java Sockets The concept of network communication through sockets is similar to that of the rotating cube example (see 3.2.3). At the client - “Master” side, sockets are built at the MasterSimpleStarter class and passes through MasterWorldWindowGLCanvas, MasterAWTInputHandler till MasterOrbitViewInputBroker. MasterOrbitViewInputBroker contains parameters that response to keyboard or mouse event and are wrapped into povEvent. Through socket communication, povEvent is transferred through network and received by the run() function implemented by SlaveSimpleStarter which extends Thread class. SlaveSimpleStarter class then pass the povEvent that it has received to SlaveWorldWindowGLCanvas, SlaveAWTInputHandler till SlaveOrbitViewInputBroker. SlaveOrbitViewInputBroker unpacked the received povEvent and set the parameters to their corresponding places, achieving the synchronized drawing with the “Master”. Sketched sequence diagram shows the process respectively from the “Master” side (see Figure 19) and “Slave” side (see Figure 20). Connection between client - “Master” and server - “Slave” is assumed already built. - 28 -
  • 29. Figure 18: Class Diagram of the realization of “Master-Slave” mechanism - 29 -
  • 30. Figure 19: Socket Passing in Client - “Master” Side Figure 20: POVEvent Passing in Server - “Slave” Side 3.3.3 Multiple Screens Effect Realization by Adjusting Viewport As is the same case with the rotating cube example, different sizes and positions of “Slaves” are the main problems this step should solve. The realization method is quite - 30 -
  • 31. similar to that of the rotating cube example (see 3.2.4). Screen size ratios are calculated and used to adjust the viewport size. SliderPanel class is created to adjust the position of the lower left corner of viewports which equally means the changing of the earth inside the viewport. glViewport exists in the class SlaveWorldWindowGLDrawable along with the corresponding modification of its parameters. Besides, the problem of the size and position of the earth is independent of the frame is also solved the same way that rotating cube does. No matter how the frame is resized or moved, the earth inside it stays stable (see Figure 17). 4. Performance Evaluation and Future Work This lab generally realized the effect of multiple screens by socket communication. Experiments on two “Slaves” are made and described here. First, two screens are placed side by side as two “Slaves”. A third PC is used as the “Master” to issue commands by utilizing keyboard or mouse. By issuing commands from the “Master”, two “Slaves” conform to the order they have received and act accordingly. For the rotating cube example, the two “Slaves”, when positions are adjusted adequately, act synchronized as if they were one. This is mainly because the simplicity of in the code which enables precise synchronization under frequent commands. The rotating cube is enlarged within two screens and can traverse through the border of them (see Figure 11). For World Wind, “Slave” applications are synchronized if actions performed on “Master” are not frequent. If high frequent manipulation is operated on “Master”, then unsynchronization possibly occur. The reason might result from the complexity of the code, especially in the iterative calculation part. Because in order to reach the result of smoothly change under manipulation, a class OrbitViewInputStateIterator is used to calculate the smoothly change iteratively given an initial input parameter. The possible error during the iteration might lead to differences among “Slave” applications since the parameters they have received from the “Master” is only the initial input. A possible way to solve the unsynchronization problem under high frequency manipulation on “Master” might be to sendPOVEvent() during every round of the display() function which ensures the same step of all the “Slaves”. This possible improvement on synchronization has not been implemented yet. - 31 -
  • 32. As addressed in the body of the report, the positions of the “Slave” earths are adjusted by setting the lower left corner of the viewport. This leads to a problem of “blank” area within the screen, which means if the lower left corner is moved upper right, then the area that is lower left than the new origin will block the scene since that area does not belong to the viewport. The way to solve this problem might be moving the origin even lower left as needed to erase the “blank” area existed in the screen. The rule needs to be obey is to keep the viewport of all the “Slave” applications are the same assuming that their screen resolutions are all the same. An unfinished task is to include layers of the earth into the multiple screens effect. Because of the separation of the earth itself and the layers (added as an listener), synchronization of layers and its earth has not been achieved. The layers cannot be moved by mouse manipulation as it is in the original source code. Possible improvement includes the change of the orientation of “Slave” screens. Now, screens are placed side by side and forming a straight line. In some circumstances, certain angles between adjacent screens are necessary because it is quite possible to generate better virtual effects (see Figure 3). Physical locations and orientations of screens need to be calculated and added into the source code in order to realize such effect. - 32 -
  • 33. Figure 21: Two “Slave” Earths and One “Master” Earth Displayed 4. Summary The goal of this computer graphics lab is to realize multiple screens effects based on World Wind Java SDK. Jogl is used as our development tool on which World Wind Java SDK is based. Analysis on the source code, adequate extension of necessary classes, network communication through sockets and viewport management are the mainstreams during the lab process. In this report, we first present the related work done before in order to obtain wider view in this field which includes Studierstube, VR Juggler and Java3D. After that our work on multiple screens begins with the working environment, including development tools (Java programming language, Eclipse IDE), libraries (Jogl, GlueGen) and Platform (World Wind Java SDK), following by detailed descriptions about one rotating cube example, seen as an introduction to the self-classified three steps - “Master-Slave” Rotating Cube Extension, Network Communication through Java - 33 -
  • 34. Socket and Multiple Screens Effect Realization by Adjusting Viewport. Our final application is based on World Wind Java SDK. Extensions on the source code is wrapped for further software reusement. Corresponding Javadoc is provided as well. References [BHFMB99] Butz, A., Höllerer, T., Feiner, S., MacIntyre, B., & Beshers, C., (1999). Enveloping users and computers in a collaborative 3D augmented reality. Proceedings International Workshop on Augmented Reality (IWAR’99), 35– 44. [CCBE] Jun Cai, Seminar Report: Augmented Reality: the Studierstube Project, http://w5.cs.uni-sb.de/~butz/teaching/ie-ss03/papers/Studierstube/ [JBBC] Christopher Just, Allen Bierbaum, Albert Baker, Carolina Cruz-Neira, VR Juggler: A Framework for Virtual Reality Development, Iowa Center for Emerging Manufacturing Technology, Iowa State University. [SFSEGP02] Schmalstieg, D., Fuhrmann, A., Szalavari, G. H. Z., Encarnacao, L.M., Gervautz, M., Purgathofer, W., The Studierstube Augmented Reality Project. In Massachusetts Institute of Technology, Vol. 11, No.1, 33-54, 2002. [NAASA] http://worldwind.arc.nasa.gov/ [NWW] NASA World Wind: http://worldwind.arc.nasa.gov/java/index.html [OpenGL] http://www.opengl.org/resources/bindings/ [Wiki-CAVE] http://en.wikipedia.org/wiki/Cave_Automatic_Virtual_Environment [Wiki-Gluegen] http://en.wikipedia.org/wiki/Gluegen [Wiki-Java3D] http://en.wikipedia.org/wiki/Java_3D [Wiki-Jogl] http://en.wikipedia.org/wiki/Java_OpenGL [Wiki-NWW] http://en.wikipedia.org/wiki/NASA_World_Wind - 34 -

×