Recommended
More Related Content
Similar to Kinect Based 3D Object Manipulation on a Desktop Display
Similar to Kinect Based 3D Object Manipulation on a Desktop Display (20)
Recently uploaded
Recently uploaded (20)
Kinect Based 3D Object Manipulation on a Desktop Display
- 1. Kinect Based 3D Object Manipulation on a Desktop Display Mukund Raj, Sarah H. Creem-Regehr, Kristina M.Rand, Jeanine K. Stefanucci and William B. Thompson University of Utah
- 2. Introduction • Controlled experimental evaluation of a Kinect based user interface. • 3D object manipulation in virtual environment. • Two variations - with & without self avatars. 2
- 3. Motivation • Availability of low cost gesture recognition hardware. • 3D graphics on web platform. 3 The LeapMicrosoft Windows Kinect Nintendo Wii
- 4. Avatars • Self Avatars • Earlier studies – Inconclusive results on spatial cognition tasks. 4
- 5. Questions • Does self avatar have observable effect on interfaces for object manipulation in virtual environments on a desktop display? • Are there strong individual differences in the effect? 5
- 6. Experimental task • Orientation matching 3D object by rotating object in virtual world. 6
- 7. Interface - Display Conditions 7 • Sphere Condition • Self Avatar Condition * Between subjects design used
- 8. Interface – Interaction Methods 8 • Swipe – Rotation along hand motion. • Twist – Rotation about the wrist.
- 9. Experimental objects • Objects 9 Number of objects Trials per object Training 1 6 Test 12 1 Training Object Test Object #1
- 10. Experiment Setup 10 • Kinect • Orientation Sensor • Wireless Mouse
- 11. Result – Completion Time • No significant difference between sphere and avatar condition overall 11
- 12. Result – Completion Time • Difference as function of gender and video game experience. 12
- 13. Gaming experience / Gender ? 13
- 14. Results – Manipulation modes • Difference in manipulation mode as a function of gender/gaming experience 14
- 15. Conclusion • Effect of self avatar on performance of only a subset of users • Necessary to check for individual differences in performance data 16
- 16. Future Research • Automatic twist/swipe mode detection • Replace orientation sensor with a low cost mobile device accelerometer • 6DOF task • Stereographic displays • Symmetrical objects • Tactile Feedback • Comprehension vs. Manipulation 15
- 17. This work was supported by the National Science Foundation under Grant No. 1116636 17
Editor's Notes
- I would like to start off with a quick , overview of the paper.. 1. We have conducted a controlled… 2. The interface itself is a ,Gesture based interface, for manipulating 3D objects in virtual world using the kinect sensor, . 3. We compared two variations of the interface, one of which used self avatar and the other that did not. Why did we do this?
- Low cost tracking devices such as microsoft kinect and Nitendo Wii becoming increasingly popular. Leap is an even more cheaper device and expected to be out by the end of the year. With spread of such devices, we can expect to see a large number of gesture based interfaces built over them. We are also seeing, javascript APIs for rendering interactive 3D graphics like WebGL are becoming increasingly powerful. There are now stunningly realistic virtual worlds being rendered entirely within the web browser, using this library. Both these technologies make gesture based interfaces “&” virtual worlds not only easier to build but more accessible to a larger audience than ever before. We have made an attempt to build and evaluate a gesture based interface for object manipulation using self - avatars.
- What are avatars? Avatars are the digital representation of humans online, or as in our context, in virtual Environments. Self-avatars, the first-person representations of the users Themselves. Earlier avatar studies have shown inconclusive results on spatial cognition tasks. <break> 2. Our interface renders an animated representation of the user’s arm and hand. Previously using avatars needed expensive motion capture equipment, but now we have the technology to build interfaces using avatars at a much lower cost. That brings up the question whether there are benefits of having such a self avatar extending into the virtual world. Could they help in offloading of cognitive work or provide a frame of reference? Could they help the user perform better just by making the interface more natural?
- More concretely,… 1….. Does self avatar have observable effect on interfaces for object manipulation in virtual worlds on a desktop display? Intuitively having a self avatar does look like a more natural way of interaction, but we wanted to see if it translated into measureable effect in performance on a desktop display??? <break> 2. Also, in the real world it is also important to ask - are there strong individual differences in the effect? In other words, does performance of user groups vary significantly when using the interface?
- To evaluate the interface, we chose the orientation matching task, mainly due to the ease of measuring performance in it and also as the task is known to be non trivial for larger rotations. As seen in the picture here, the screen is split into left and right regions. Objects at different orientations , appear in both regions, in each successive trial. The object on the right is then rotated by the user, to match the orientation of the object on the left.
- The interface has two variations based upon display conditions or what the users see as feedback for their arm motion The picture on the top shows the 1st variation – also called ‘Sphere condition’ in which a sphere of the size similar to that of the hand is rendered based upon the position of the right hand while interaction occurs. Only feedback to the user in this condition is a sense of position of the hand. Also, in the Sphere condition, the frame of reference is unclear. The picture below shows the other variation – also called the self avatar condition in which motion of shoulder, elbow and wrist are accurately mapped onto the avatar. Fingers, however, are not animated. Self avatars can provide the user with an egocentric and anthropomorphic frame of reference as well as a more natural interface ----- We used a between subjects design for assigning participants to the feedback conditions. ** Also, Male and female participants were evenly distributed for the sphere and the self avatar conditions. ----
- The interface provides two modes of rotation to change the orientation of the objects. First is Rotation along hand motion also referred to as swipe mode. [VIDEO] As you can see, the user rotates the object along an axis on the plane of the display and perpendicular to the direction of hand motion. This is similar to rotating objects using the virtual sphere method, except that we use hand gestures instead of moving the mouse pointer. The second method of interaction is rotating the object about the axis of their wrist also referred to as twist mode. [VIDEO] This is closer to how we manipulate objects in the real world. Here the user can rotate the object about any axis just by aligning their wrist with that axis and performing a twist gesture. ** Both interaction methods were enabled in each of the display conditions and the user could select either mode at any time during a trial. ** For larger rotations users were able to employ ratcheting and accomplish larger rotations as sum of smaller rotations. ** Also as seen in the video, color of the object changed when the avatar’s hand came close to the object to incorporate a sense of contact. Users were only able to manipulate when the avatar hand was close to the object .
- The user had a chance to practice with a training object before the recorded trials. Each participant gets 6 trials to practice with a single practice object, which is the one shown on the picture on the left after which they were ask to perform the task on 12 distinct trial objects, one of which is shown in the right figure. ---------- ***** Completion time and number of times each mode of rotation was used were recorded for each trial for all participants. ******** ----------
- In the experimental setup, We used the Kinect for recovering joint orientations of the user for animating the avatar. An orientation sensor strapped around the hand of the user for improving the accuracy of the wrist orientation. And, a wireless mouse used by the user to indicate the mode of rotation for each gesture - whether along the hand motion/ or about the wrist axis. ----- ** Geometric and display field of views were matched to improve realism and sense of embodiment of the avatar. ///////////////////////And Gender and gaming experience were also recorded in order to check for “individual differences”.
- We looked at the time taken by each participant to complete all 12 trials. Here we have plotted the average completion time for the two display conditions. The first and second bar show the average completion time for the self avatar and sphere display conditions respectively. And, we can see that, there was no significant difference in performance between sphere and avatar conditions overall
- However.. There was difference in performance in visual display condition as function of gender and video game experience. On the graph we again have average completion time for each condition, now split by gender. It can be seen that although everyone performed similarly in the self avatar condition, female participants took noticeably longer to complete the trials in the sphere condition. ---------------split----------------- It is important to note that , Gaming experience and gender in our participant pool were highly overlapping and we cannot associate the effect to either gaming experience or gender alone from the available data. Here we see an effect of “individual differences” in our interface where the display condition had an effect on performance of a subset of participants.
- As seen in the earlier graph, and more clearly in here, gender and gaming experience gender and gaming experience significantly overlapped in our participant pool. It would be interesting to see the performance of female gamers and male non – gamers using this interface.
- We also recorded the mode of rotation indicating which mode the participants used whether along the hand motion or about the wrist axis and again we see an effect of “individual differences”. Male/Gamer participants used both modes almost equally while female/non gamers participants relied more on rotation along the hand motion. Ideally a good sense of object orientation in virtual world would afford a predisposition toward rotation about wrist axis as it would be more efficient if used correctly. Also it is closer to the natural way of manipulating objects in the real world and object can be matched to the target orientation in a single motion if the appropriate axis needed is known.
- We compared user performance in two variations of a gesture based interface for object manipulation interface in virtual world . 1. We found that there was an.. “Effect of self avatar on performance of only a subset of users” 2. More importantly when evaluating such interfaces, “Necessary to check for individual differences in performance data” Latest tracking devices such as the Kinect and leap, open up new ways of interacting with the virtual world. We did a controlled study of one such possibility and we saw one variation of the interface showing significant effect of individual differences while the other did not. The key message here is that in evaluating performance of such interfaces its important to keep a look out for individual differences, which can be significant , to really understand the strength and weaknesses of the system.
- Interesting ideas to explore further.
- 5,10,12,14,16