• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Virtualizing Real-life Lectures with vAcademia and Kinect

Virtualizing Real-life Lectures with vAcademia and Kinect



Andrey Smorkalov, Mikhail Fominykh, and Ekaterina Prasolova-Førland: "Virtualizing Real-life Lectures with vAcademia and Kinect", Workshop on Off-The-Shelf Virtual Reality ...

Andrey Smorkalov, Mikhail Fominykh, and Ekaterina Prasolova-Førland: "Virtualizing Real-life Lectures with vAcademia and Kinect", Workshop on Off-The-Shelf Virtual Reality
IEEE Virtual Reality Conference, Orlando, FL, USA, March, 16 2013.



Total Views
Views on SlideShare
Embed Views



1 Embed 5

https://twitter.com 5



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Virtualizing Real-life Lectures with vAcademia and Kinect Virtualizing Real-life Lectures with vAcademia and Kinect Presentation Transcript

    • Virtualizing Real-life Lectures with vAcademia and Kinect Andrey Smorkalov Volga State University of Technology, Russia Mikhail Fominykh and Ekaterina Prasolova-Førland Norwegian University of Science and Technology, Norway Workshop on Off-The-Shelf Virtual Reality IEEE Virtual Reality Conference March, 16 2013 | Orlando, FL, USA1
    • Goal o A low-cost technological setup for translating real- life presentations and lectures into a 3D virtual environment – Streaming real-life lectures into 3D virtual environment – Automatically creating immersive 3D recordings2
    • Motivation: learning with VR o Virtual worlds have recognized affordances for learning, but also many challenges o Cost is a limiting factor for learning with virtual worlds and other VR o Industry, military, and healthcare are the major areas where VR is currently used for educational purposes o Exploring new ways of using 3D virtual worlds for learning: capturing lectures and creating asynchronous content out of synchronous learning activities3
    • Motivation: capturing lectures o „Traditional‟ video recording of lectures and web conferences change the context of learning and do not provide immersion or sense of presence as in 3D virtual worlds o „2D‟ recordings, including Machinima, do not provide a possibility for collaborative work or a method for further developing the content o Kinect was previously used to improve video recording of presentations by designing an automatic camera control system o => Combining 3D recording in vAcademia with Kinect for advanced, immersive capturing of lectures4
    • First prototype: system implementation Kinect plugin Animation library Cal3D Script Executing Library vAcademia graphic engine Scripts vAcademia Virtualizing real-life lectures mode interface of vAcademia5
    • First prototype: system implementation o Five body parts: left arm, right arm, left leg, right leg, and head o Standing mode (all body parts) and sitting mode (only arms and head) o “Adequately recognized” status for each part o If a body part is not recognized adequately, last adequate state is used for 0.2–0.5 sec, and then the default state6
    • First prototype: system performance o Requirements of the components – vAcademia requires and actively uses one CPU core. – Kinect requires a dual-core CPU, but uses only one core, as the second is reserved for the application that uses Kinect data. o The process of animating the lecturer’s avatar based on the data from Kinect is not computationally complex. o System’s performance is satisfactory if component requirements are satisfied, which has been confirmed during the evaluation.7
    • First prototype: system evaluation o Non-systematic evaluation during iterative development process – Several evaluation sessions two-three different courses – Auditoriums of different configurations and lightning – Involving different teachers o Data – Short interviews with the lecturer while watching the 3D recording created vAcademia o Most common feedback – Too many restrictions on the lecturer’s movements – Suggestions on how to increase the educational value8
    • Applying Kinect Motion Capture in vAcademia: Challenges 1. Low accuracy in capturing gestures – We could not build a reliable avatar model that can move without unnatural poses 2. Kinect does not recognize the turn of the lecturer – Left and right arms are mixed up, unnatural pose is returned 3. Kinect cannot capture parts of the body that are covered by other body parts or foreign objects – Additional requirements to the setup – Lower recognition accuracy9
    • Applying Kinect Motion Capture in vAcademia: Solutions for 1 o Positioning Kinect device and the lecturer – < 1.8 m. for standing mode – < 1.3 m. for sitting mode – Kinect device at 0.5 m. from the floor – Software-based turn into a zero-degree position by the vertical axis o Additional filtration mechanism for sorting out unnatural positions of the body parts – Limited the acceptable values of Euler angles between the bones – Separated hands as distinct body parts10
    • Applying Kinect Motion Capture in vAcademia: Solutions for 2 o The turn is recognized relatively as a function of the position of the pelvis end points – The resultant value is valid within the range from -110 to 110 degrees against the “facing Kinect device” direction. o Colored markers – Two markers are placed on the body of a lecturer on the left and on the right side, facing the Kinect device. – The colors should be different from the lecturer’s clothing and the material should not be shimmering. – If they are recognized, the system considers that the lecturer is in the acceptable turn range. If not -> last correctly recognized state - > default state.11
    • Applying Kinect Motion Capture in vAcademia: Solutions for 2 o Testing colored markers12
    • Applying Kinect Motion Capture in vAcademia: Proposal for 3 o Multiple Kinect devices – Three Kinect devices: to the left, to the right, and in front of the lecturer o New challenges: – Increased price of the system – Data from the multiple Kinect devices should be adjusted to a single coordinate system => increased requirements for the accuracy of locating Kinect devices – Additional requirements to the auditorium (>7 m. across) – Merging the data from multiple Kinect devices13
    • Supporting Slide Presentations: Challenges 1. Matching relative positions in real and virtual worlds – The position of the lecturer against the whiteboard should match the position of the avatar against the virtual whiteboard. 2. Capturing a physical pointer – It is an important part of lecture experience, but Kinect cannot capture it. 3. The gestures switching slides in real world do not have the same meaning in 3D virtual world14
    • Supporting Slide Presentations: Solutions for 1 o Precise match between the physical whiteboard and the virtual one. – Performed once after installing the physical whiteboard and the Kinect device in the classroom. – Capturing left and right edges of the physical whiteboard in Kinect coordinate system. – Installing the Kinect device and the physical whiteboard on a specified distance from the floor. o Further improvement – Recognizing the borders of the physical whiteboard and creating the replica in the 3D virtual world keeping the proportion automatically.15
    • Supporting Slide Presentations: Solutions for 2 o Directing the virtual pointer based on the position of the lecturer’s hand. – If the half line that extends from the lecturer’s hand towards the physical whiteboard crosses it, the avatar in the 3D VW directs a virtual pointer to the same point. – In order to keep the lecturer aware of his or her hand being captured, we display a semi-transparent yellow area on the physical whiteboard on top of the slides.16
    • Supporting Slide Presentations: system in work o Capturing lecture o Streaming lecture17
    • Supporting Slide Presentations: Solutions for 3 o Switching slides functionality in PowerPoint by recognizing standard gestures Swipe Left and Swipe Right18
    • Learning Scenarios o Scenario 1: Lecturing as a synchronous mixed reality activity – Interactions between students in the physical and virtual classrooms – Recording student and lecturer activities in the same context o Scenario 2: Round-table discussion as a synchronous mixed reality activity – Participants joining through the 3D virtual world or captured from the real world – Multiple Kinect-based systems can be installed in remote locations, each of them can capturing two participants – The designed system provides a significant advantage over pure 3D virtual worlds in the non-verbal communication support19
    • Learning Scenarios (2) o Scenario 3: Motion capture for synchronous mixed reality educational role plays – Taking turns in the physical classroom or letting the users captured by Kinect play the roles of facilitators o Scenario 4: Creating immersive 3D recordings out of live lectures – Easy and low-cost creation of educational content for later (asynchronous) use, such as lectures and simulations – Any activity, including streaming Kinect-captured lectures, in the 3D virtual world can be easily saved and revisited later – The resultant 3D recordings combine the convenience of video and immersive qualities of 3D virtual worlds20
    • Questions? Feedbacks? Andrey Smorkalov Mikhail Fominykh Ekaterina Prasolova-FørlandsmorkalovAY@volgatech.net mikhail.fominykh@ntnu.no ekaterip@ntnu.no Acknowledgments Mikhail Morozov morozovMN@volgatech.net Multimedia Systems Laboratory Virtual Spaces LLC Volga State University of Technology vAcademia http://mmlab.ru http://vacademia.com21